DataX3：Mylsql8.0.27ToStarRocks3.2.8，使用MysqlWriter可以，改成StarRocksWriter报starrockswriter, mysqlreader插件加载失败

💬 StarRocks 用户问答 功能使用相关
#1

为了更快的定位您的问题，请提供以下信息，谢谢
【详述】Mylsql8.0.27ToStarRocks3.2.8，使用MysqlWriter可以，改成StarRocksWriter报starrockswriter, mysqlreader插件加载失败？
[main] WARN ConfigParser - 插件[mysqlreader,starrockswriter]加载失败，1s后重试… Exception:Code:[Framework-12], Description:[DataX插件初始化错误, 该问题通常是由于DataX安装错误引起，请联系您的运维解决 .]. - 插件加载失败，未完成指定插件加载:[starrockswriter, mysqlreader]
2024-08-20 19:41:50.707 [main] ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-12], Description:[DataX插件初始化错误, 该问题通常是由于DataX安装错误引起，请联系您的运维解决 .]. - 插件加载失败，未完成指定插件加载:[starrockswriter, mysqlreader]

【背景】Mylsql8.0.27ToStarRocks3.2.8，使用MysqlWriter可以
【业务影响】改成StarRocksWriter报starrockswriter, mysqlreader插件加载失败
【是否存算分离】否，存算一体。
【StarRocks版本】StarRocks3.2.8
【集群规模】例如：3fe（1 follower+2observer）+5be（fe与be混部）
【机器信息】CPU虚拟核/内存/网卡，8C/16G/千兆
【联系方式】601026166@qq.com
【附件】

图片
图片1691×229 18.1 KB

#2

确认一下datax3是否支持starrocks write插件。报错是提示该插件的问题

1赞
#3

感谢！
已安装StarRocksWriter驱动到 plugin/writer下，但又提示database为空，请教一下：少了哪些配置？
感谢！！
ERROR JobContainer - Exception when job run
com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-03], Description:[您缺失了必须填写的参数值.]. - 您提供配置文件有误，[database]是必填参数，不允许为空或者留白 .

#4

现在不提示找不到数据库了（Url后面少了一个/），又提示：
INFO StarRocksStreamLoadVisitor - Executing stream load to: ‘http://192.168.1.3:8030/api/regulationdb/datamonitoring_copy3/_stream_load’, size: ‘91288207’
2024-08-22 09:57:06.856 [Thread-4] WARN StarRocksWriterManager - Failed to flush batch data to StarRocks, retry times = 0
com.starrocks.shade.org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:8040 [/127.0.0.1] failed: 拒绝连接 (Connection refused)

#5

“writeMode”:“insert”,

        "preSql": [],

        "postSql": [],

        "jdbcUrl":"jdbc:mysql:loadbalance://192.168.1.3:9030/",

        "loadUrl":["192.168.1.3:8030"],

        "loadProps": {

          "format": "json",

          "strip_outer_array": true

        }
#6

图片
图片984×442 18.8 KB

#7

图片
图片1691×592 28.5 KB

图片
图片1355×576 18.9 KB

#8

datax和be的8040端口防火墙是通的吗

1赞
#9

jingdan您好，感谢！！
DataX在物理机Hyper-V的CentOS-Stream9-server运行，StarRocks在Docker中，物理机192.168.1.3是WIN10p，防火墙已全部关闭。

#10

现在是可以跑通了么？确保datax的机器能访问通fe的8030和be的8040

#11

其他驱动成功传输：
DataX数据传输测试：均成功，测试数据：120万条。

SQLServer12 To SQLServer12：OK
SQLServer12 To Mysql8.0.27：OK
SQLServer12 To StarRocks3.2.8（Writer使用Mysql驱动）：OK

Mysql8.0.27 To Mysql8.0.27：OK
Mysql8.0.27 To SQLServer12：OK
Mysql8.0.27 To StarRocks3.2.8（Writer使用Mysql驱动）：OK

StarRocks3.2.8 To Mysql8.0.27：OK
StarRocks3.2.8 To StarRocks3.2.8：OK
StarRocks3.2.8 To SQLServer12：OK

Hyper-V上CentOS-Stream9-server ping192.168.1.3:8040是通的，但一直报127.0.0.1:8040连接失败（是不是DxtaX在执行job的时候访问的是CentOS9本机的127.0.0.1:8040？）
但使用StarRocksWriter驱动修改job数据库连接方式后，就报错误了，脚本：
{

“job”: {

"setting": {

  "speed": {

       "channel": 1

  },

  "errorLimit": {

      "record": 0,

      "percentage": 0

  }

},

"content": [

  {

    "reader": {

      "name": "mysqlreader",

      "parameter": {

        "connection": [

          {

            "jdbcUrl":["jdbc:mysql://192.168.1.3:3307/regulationdb?characterEncoding=UTF-8&serverTimezone=Asia/Shanghai"],

            "table": ["datamonitoring"]

          }

        ],

        "username": "root",

        "password": "123456",

        "column": ["id","MarketID"

                   ,"MarketIName"

                   ,"TerminalId"

                   ,"TerminalName"

                   ,"ShopingCartID"

                   ,"SequenceNumberInShopingCart"

                   ,"NameID"

                   ,"NameIDSku"

                   ,"MarketDefinedProductCode"

                   ,"SellingCount"

                   ,"Price"

                   ,"SellingTime"

                   ,"Valuation"

                   ,"UploadingTime"]

      }

    },

    "writer": {

        "name":"starrockswriter",

       "parameter": {

        "username":"root",

        "password":"123456",

        "database":"regulationdb",

        "table":"datamonitoring_copy3",

        "column": ["id","MarketID"

                   ,"MarketIName"

                   ,"TerminalId"

                   ,"TerminalName"

                   ,"ShopingCartID"

                   ,"SequenceNumberInShopingCart"

                   ,"NameID"

                   ,"NameIDSku"

                   ,"MarketDefinedProductCode"

                   ,"SellingCount"

                   ,"Price"

                   ,"SellingTime"

                   ,"Valuation"

                   ,"UploadingTime"],

        "field": [

          {  

           "name": "id",  

           "type": "int"  

           },

           {  

           "name": "MarketID",  

           "type": "int"  

           },

           {  

           "name": "MarketIName",  

           "type": "varchar(255)"  

           },

           {  

           "name": "TerminalId",  

           "type": "int"  

           },

           {  

           "name": "TerminalName",  

           "type": "varchar(255)"  

           },

           {  

           "name": "ShopingCartID",  

           "type": "varchar(255)"  

           },

           {  

             "name": "SequenceNumberInShopingCart",  

             "type": "varchar(255)"  

           },

           {  

           "name": "NameID",  

           "type": "int"  

           },

           {  

           "name": "NameIDSku",  

           "type": "varchar(255)"  

           },

           {  

           "name": "MarketDefinedProductCode",  

           "type": "varchar(255)"  

           },

           {  

           "name": "SellingCount",  

           "type": "decimal(6, 3)"  

           },

           {  

           "name": "Price",  

           "type": "decimal(8, 2)"  

           },

           {  

           "name": "SellingTime",  

           "type": "datetime"  

           },

           {  

           "name": "Valuation",  

           "type": "varchar(255)"  

           },

           {  

           "name": "UploadingTime",  

           "type": "datetime "  

           }

       ],

        "writeMode":"insert",

        "jdbcUrl":"jdbc:mysql://192.168.1.3:9030/",

        "loadUrl":["192.168.1.3:8030"],

       

        "preSql": [],

        "postSql": [],

        "loadProps": {

          "format": "json",

          "strip_outer_array": true

        }

    }

  }

}

]

}

}

图片

图片
图片941×502 24.4 KB

图片
图片1691×594 28.6 KB

2024-08-28 15:10:14 ****************************************************************************************************
2024-08-28 15:10:14 * ________ _________ ________ ________ ________ ________ ________ ___ __ ________
2024-08-28 15:10:14 * |\ |_ \ __ |\ __ |\ __ |\ __ |\ ____|\ |\ \ |\ ____\
2024-08-28 15:10:14 * \ \ _|| \ _\ \ |\ \ \ |\ \ \ |\ \ \ |\ \ \ _|\ \ / /|\ \ _|_
2024-08-28 15:10:14 * \ _____ \ \ \ \ \ \ __ \ \ _ \ \ _ \ \ \\ \ \ \ \ \ ___ \ ___ \
2024-08-28 15:10:14 * ||\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ \ \ _\ \ \ \ ||\ \
2024-08-28 15:10:14 * _\ \ \ _\ \ _\ _\ _\ \ _\ \ _\ _\ _\ ___\ \
2024-08-28 15:10:14 * |_\ || |||||||||||||||||| ||_
2024-08-28 15:10:14 * || ||
2024-08-28 15:10:14 *
2024-08-28 15:10:14 * ________ ___ ___ ___ ________ _____
2024-08-28 15:10:14 * |\ __ |\ \ |\ \ |\ |\ ___ \ / __
2024-08-28 15:10:14 * \ \ |\ \ \ \ \ \ \ \ \ \ \ \ \ |/|\
2024-08-28 15:10:14 * \ \ __ \ \ \ \ \ \ \ \ \ \ \ \ |/ \ \
2024-08-28 15:10:14 * \ \ \ \ \ \ _\ \ _\ \ \ \ \ \ \ \ \
2024-08-28 15:10:14 * \ _\ _\ _\ _\ _\ _\ _\ \ _
2024-08-28 15:10:14 * |||||||||||| || ||
2024-08-28 15:10:14 *
2024-08-28 15:10:14 ****************************************************************************************************
2024-08-28 15:10:14
2024-08-28 15:10:14 StarRocks [(Blazing Fast)]> _
2024-08-28 15:10:14
2024-08-28 15:10:14
2024-08-28 15:10:15 2024-08-28 07:10:15,213 INFO Set uid to user 0 succeeded
2024-08-28 15:10:15 2024-08-28 07:10:15,516 INFO RPC interface ‘supervisor’ initialized
2024-08-28 15:10:15 2024-08-28 07:10:15,516 CRIT Server ‘unix_http_server’ running without any HTTP authentication checking
2024-08-28 15:10:15 2024-08-28 07:10:15,516 INFO supervisord started with pid 7
2024-08-28 15:10:16 2024-08-28 07:10:16,518 INFO spawned: ‘beservice’ with pid 555
2024-08-28 15:10:16 2024-08-28 07:10:16,519 INFO spawned: ‘broker’ with pid 556
2024-08-28 15:10:16 2024-08-28 07:10:16,521 INFO spawned: ‘director’ with pid 557
2024-08-28 15:10:16 2024-08-28 07:10:16,522 INFO spawned: ‘feproxy’ with pid 558
2024-08-28 15:10:16 2024-08-28 07:10:16,523 INFO spawned: ‘feservice’ with pid 560
2024-08-28 15:10:16 2024-08-28 07:10:16+00:00 INFO checking if need to perform auto registring Backend and Broker …
2024-08-28 15:10:16 2024-08-28 07:10:16,583 INFO exited: broker (exit status 1; not expected)
2024-08-28 15:10:16 2024-08-28 07:10:16+00:00 INFO checking if FE service query port:9030 alive or not …
2024-08-28 15:10:16 2024-08-28 07:10:16+00:00 WARN FE service query port:9030 is NOT alive yet!
2024-08-28 15:10:17 2024-08-28 07:10:17,790 INFO spawned: ‘broker’ with pid 3024
2024-08-28 15:10:18 2024-08-28 07:10:18+00:00 WARN FE service query port:9030 is NOT alive yet!
2024-08-28 15:10:20 2024-08-28 07:10:20+00:00 WARN FE service query port:9030 is NOT alive yet!
2024-08-28 15:10:21 2024-08-28 07:10:21,741 INFO success: beservice entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2024-08-28 15:10:21 2024-08-28 07:10:21,741 INFO success: director entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2024-08-28 15:10:21 2024-08-28 07:10:21,741 INFO success: feproxy entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2024-08-28 15:10:21 2024-08-28 07:10:21,741 INFO success: feservice entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2024-08-28 15:10:22 2024-08-28 07:10:22+00:00 INFO FE service query port:9030 is alive!
2024-08-28 15:10:22 2024-08-28 07:10:22+00:00 INFO generate my.cnf file …
2024-08-28 15:10:22 2024-08-28 07:10:22+00:00 INFO check if need to add BE into FE service …
2024-08-28 15:10:22 2024-08-28 07:10:22,929 INFO success: broker entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2024-08-28 15:10:26 2024-08-28 07:10:26+00:00 INFO BE service already added into FE service …
2024-08-28 15:10:26 2024-08-28 07:10:26+00:00 INFO check if need to add BROKER into FE service …
2024-08-28 15:10:26 2024-08-28 07:10:26+00:00 INFO broker service already added into FE service …
2024-08-28 15:10:15 Unlinking stale socket /data/deploy/starrocks/supervisor/supervisor.sock
2024-08-28 15:10:26 2024-08-28 07:10:26+00:00 INFO cluster initialization DONE!
2024-08-28 15:10:26 2024-08-28 07:10:26+00:00 INFO wait a few seconds for BE and Broker’s heartbeat …
2024-08-28 15:10:36 2024-08-28 07:10:36+00:00 INFO StarRocks Cluster information details:
2024-08-28 15:10:36 *************************** 1. row ***************************
2024-08-28 15:10:36 Name: 127.0.0.1_9010_1724058294712
2024-08-28 15:10:36 IP: 127.0.0.1
2024-08-28 15:10:36 EditLogPort: 9010
2024-08-28 15:10:36 HttpPort: 8030
2024-08-28 15:10:36 QueryPort: 9030
2024-08-28 15:10:36 RpcPort: 9020
2024-08-28 15:10:36 Role: LEADER
2024-08-28 15:10:36 ClusterId: 1869614770
2024-08-28 15:10:36 Join: true
2024-08-28 15:10:36 Alive: true
2024-08-28 15:10:36 ReplayedJournalId: 147393
2024-08-28 15:10:36 LastHeartbeat: 2024-08-28 07:10:34
2024-08-28 15:10:36 IsHelper: true
2024-08-28 15:10:36 ErrMsg:
2024-08-28 15:10:36 StartTime: 2024-08-28 07:10:21
2024-08-28 15:10:36 Version: 3.2.8-759cc78
2024-08-28 15:10:36 *************************** 1. row ***************************
2024-08-28 15:10:36 BackendId: 10004
2024-08-28 15:10:36 IP: 127.0.0.1
2024-08-28 15:10:36 HeartbeatPort: 9050
2024-08-28 15:10:36 BePort: 9060
2024-08-28 15:10:36 HttpPort: 8040
2024-08-28 15:10:36 BrpcPort: 8060
2024-08-28 15:10:36 LastStartTime: 2024-08-28 07:10:19
2024-08-28 15:10:36 LastHeartbeat: 2024-08-28 07:10:34
2024-08-28 15:10:36 Alive: true
2024-08-28 15:10:36 SystemDecommissioned: false
2024-08-28 15:10:36 ClusterDecommissioned: false
2024-08-28 15:10:36 TabletNum: 62
2024-08-28 15:10:36 DataUsedCapacity: 0.000 B
2024-08-28 15:10:36 AvailCapacity: 91.113 GB
2024-08-28 15:10:36 TotalCapacity: 165.451 GB
2024-08-28 15:10:36 UsedPct: 44.93 %
2024-08-28 15:10:36 MaxDiskUsedPct: 44.93 %
2024-08-28 15:10:36 ErrMsg:
2024-08-28 15:10:36 Version: 3.2.8-759cc78
2024-08-28 15:10:36 Status: {“lastSuccessReportTabletsTime”:“N/A”}
2024-08-28 15:10:36 DataTotalCapacity: 91.113 GB
2024-08-28 15:10:36 DataUsedPct: 0.00 %
2024-08-28 15:10:36 CpuCores: 16
2024-08-28 15:10:36 NumRunningQueries: 0
2024-08-28 15:10:36 MemUsedPct: 0.80 %
2024-08-28 15:10:36 CpuUsedPct: 0.0 %
2024-08-28 15:10:36 Location:
2024-08-28 15:10:36 *************************** 1. row ***************************
2024-08-28 15:10:36 Name: allin1broker
2024-08-28 15:10:36 IP: 127.0.0.1
2024-08-28 15:10:36 Port: 8000
2024-08-28 15:10:36 Alive: true
2024-08-28 15:10:36 LastStartTime: 2024-08-19 09:05:05
2024-08-28 15:10:36 LastUpdateTime: 2024-08-28 07:10:34
2024-08-28 15:10:36 ErrMsg:
2024-08-28 15:10:36 2024-08-28 07:10:36+00:00 INFO
2024-08-28 15:10:36 2024-08-28 07:10:36+00:00 INFO
2024-08-28 15:10:36 2024-08-28 07:10:36+00:00 INFO * IMPORTANT NOTICE!
2024-08-28 15:10:36
2024-08-28 15:10:36 If FE/BE state needs to be persisted, please be sure the following directories are mounted:
2024-08-28 15:10:36 * FE service meta: /data/deploy/starrocks/fe/meta
2024-08-28 15:10:36 * BE service storage: /data/deploy/starrocks/be/storage
2024-08-28 15:10:36
2024-08-28 15:10:36 2024-08-28 07:10:36+00:00 INFO
2024-08-28 15:10:36 2024-08-28 07:10:36+00:00 INFO FE mysql query port: 9030
2024-08-28 15:10:36 2024-08-28 07:10:36+00:00 INFO FE http service port: 8080
2024-08-28 15:10:36 2024-08-28 07:10:36+00:00 INFO
2024-08-28 15:10:36 2024-08-28 07:10:36+00:00 INFO Enjoy the journal to StarRocks blazing-fast lake-house engine!

#12

图片

#13

集群中be的ip为啥是127.0.0.1呢

#14

fe转发给be的时候取的show backends中看到的ip

#15

谢谢专家。
Docker是没有IP地址的，用的是物理机的IP，对吧，StarRocks集群中be是否需要配置?如何配置呢？
感谢！！

#16

docker run --name starrocks -p 9030:9030 -p 8030:8030 -p 8040:8040 -e MYSQL_PWD=123456 -d -v d:/starrocksdb/fe/meta:/data/deploy/starrocks/fe/meta -v d:/starrocksdb/be/storage:/data/deploy/starrocks/be/storage starrocksimg

#17

StarRocks集群中be是否需要配置IP和端口?应如何配置？感谢！！

#18

StarRocks集群中be是否需要配置IP和端口?应如何配置？感谢！！
我的QQ601026166方便交流，谢谢。

#19

容器部署，参考 https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/examples/starrocks/deploy_a_starrocks_cluster_with_fe_proxy.yaml 需要加一个proxy来写入数据

集群部署指定ip 参考 https://docs.starrocks.io/zh/docs/deployment/deploy_manually/ 官网部署文档

#21

收到，感谢！我看一下。

京ICP备2022026421号-1