【详述】扩容一台新的BE节点后运行一段时间这台节点挂掉了
【背景】扩容be节点
【业务影响】
【StarRocks版本】例如：2.5.2
【集群规模】例如：1fe+3be（fe与be分开部）
【机器信息】16C/32G/万兆
【联系方式】社区群12-TuTu
【附件】
be.WARNING.log:
W0222 17:32:02.684300 17399 tablet_sink.cpp:1269] NodeChannel[10003], tablet add chunk failed, load_id=c36b9f13-b293-11ed-aa62-00163e09a0ba, txn_id: 3698885, parallel=1, compress_type=2, node=172.16.4.128:8060, errmsg=Too many versions. tablet_id: 11588, version_count: 1010, limit: 1000, replica_state: 0
W0222 17:32:02.692546 17391 tablet_sink.cpp:1403] close channel failed. channel_name=NodeChannel[10003], load_info=load_id=c36b9f13-b293-11ed-aa62-00163e09a0ba, txn_id: 3698885, parallel=1, compress_type=2, error_msg=Too many versions. tablet_id: 11588, version_count: 1010, limit: 1000, replica_state: 0
W0222 17:32:02.859742 17397 tablet_sink.cpp:1046] NodeChannel[10003], tablet open failed, load_id=c3865307-b293-11ed-aa62-00163e09a0ba, txn_id: 3698889, parallel=1, compress_type=2, node=172.16.4.128:8060, errmsg=Too many versions. tablet_id: 11588, version_count: 1010, limit: 1000, replica_state: 0
W0222 17:32:02.859810 17397 tablet_sink.cpp:1269] NodeChannel[10003], tablet add chunk failed, load_id=c3865307-b293-11ed-aa62-00163e09a0ba, txn_id: 3698889, parallel=1, compress_type=2, node=172.16.4.128:8060, errmsg=Too many versions. tablet_id: 11588, version_count: 1010, limit: 1000, replica_state: 0
W0222 17:32:02.868075 17391 tablet_sink.cpp:1403] close channel failed. channel_name=NodeChannel[10003], load_info=load_id=c3865307-b293-11ed-aa62-00163e09a0ba, txn_id: 3698889, parallel=1, compress_type=2, error_msg=Too many versions. tablet_id: 11588, version_count: 1010, limit: 1000, replica_state: 0
W0222 17:32:10.381326 17503 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:12.250614 17501 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:12.256338 17500 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:13.339745 17507 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=write() send(): Broken pipe
W0222 17:32:13.641145 17502 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=write() send(): Broken pipe
W0222 17:32:22.923779 17507 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:24.019738 17499 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=write() send(): Broken pipe
W0222 17:32:24.373365 17502 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=write() send(): Broken pipe
W0222 17:32:33.048362 17505 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:37.594656 17503 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:39.867841 17501 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:50.318459 17380 exec_state_reporter.cpp:129] Retrying ReportExecStatus: No more data to read.
W0222 17:32:51.078778 17500 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:53.736864 17507 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:56.646121 17499 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:56.697708 17504 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:32:58.258889 17502 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:00.538504 17498 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:04.336248 17380 exec_state_reporter.cpp:129] Retrying ReportExecStatus: No more data to read.
W0222 17:33:12.601855 17506 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:14.323025 17505 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:14.422642 17503 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:19.100251 17501 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:21.247141 17500 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:33.253919 17507 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:34.988373 17499 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:35.164398 17504 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:39.779054 17502 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:41.948328 17498 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:53.916692 17506 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:55.661013 17505 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:33:55.839257 17503 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:00.441751 17501 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:02.635696 17507 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:02.636129 17500 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:12.316316 17499 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:14.629745 17504 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:16.316238 17502 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:16.491552 17498 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:21.120805 17506 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:23.313896 17505 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:23.715238 17503 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:25.777930 17501 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:28.428308 17507 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=write() send(): Broken pipe
W0222 17:34:36.960930 17500 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:37.173467 17499 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:41.780236 17504 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:43.989976 17502 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:55.327129 17498 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:57.573346 17505 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:57.573774 17506 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:34:57.818733 17503 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:02.436578 17501 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:04.674706 17507 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:06.769333 17500 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:08.879587 17499 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:09.182361 17504 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=write() send(): Broken pipe
W0222 17:35:13.661175 17502 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:14.424145 17498 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:16.004420 17505 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:16.527225 17506 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:17.649144 17503 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:23.140355 17499 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:25.411084 17504 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:36.661032 17502 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:39.937505 17498 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:44.815876 17505 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:45.087819 17506 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:46.127209 17503 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:57.444772 17501 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:58.306644 17507 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:35:59.030462 17500 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:36:03.802048 17499 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:36:07.440924 17504 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:36:19.682948 29116 thrift_rpc_helper.cpp:56] Connect frontend failed, address=TNetworkAddress(hostname=, port=0), status=Couldn’t open transport for :0 (Could not resolve host for client socket.)
W0222 17:36:19.683043 29116 routine_load_task_executor.cpp:445] commit failed
W0222 17:36:19.683127 29116 thrift_rpc_helper.cpp:56] Connect frontend failed, address=TNetworkAddress(hostname=, port=0), status=Couldn’t open transport for :0 (Could not resolve host for client socket.)
W0222 17:36:19.683136 29116 stream_load_executor.cpp:289] transaction rollback failed. errmsg=Couldn’t open transport for :0 (Could not resolve host for client socket.)id=6ae70b23be72454b-92b0e1ba7a9b6911, job_id=3814248, txn_id: 3700057, label=loadobc_order_order_warehouse_out_detail-3814248-6ae70b23-be72-454b-92b0-e1ba7a9b6911-3700057, db=dwd_flink
W0222 17:36:36.355657 28998 exec_state_reporter.cpp:129] Retrying ReportExecStatus: No more data to read.
W0222 17:36:45.020153 29121 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:36:56.271037 28998 exec_state_reporter.cpp:129] Retrying ReportExecStatus: No more data to read.
W0222 17:37:11.056728 29119 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:37:19.536922 29120 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
W0222 17:37:31.707090 29121 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
be.log:
I0222 17:41:53.632166 29242 data_consumer.cpp:60] init kafka consumer with group id: loadljkc_ar_gatherbill_8e80b3b1-e4b4-433a-844e-c4e56eeec189
I0222 17:41:53.632560 29246 data_consumer.cpp:60] init kafka consumer with group id: loadljkc_ar_gatheritem_8bef37f6-9011-434c-a93d-bbbb516f17bb
I0222 17:41:53.787814 29432 heartbeat_server.cpp:80] Updating master info: TMasterInfo(network_address=TNetworkAddress(hostname=172.16.4.127, port=9020), cluster_id=1531098948, epoch=4, token=85a39bb5-d5cd-479a-9e5f-7a6676a88eae, backend_ip=172.16.4.124, http_port=8030, heartbeat_flags=0, backend_id=7256462, min_active_txn_id=3701482)
I0222 17:41:53.944128 29247 data_consumer.cpp:60] init kafka consumer with group id: loadljkc_er_bxcontrast_b8e2d807-38e5-43f5-acc8-966efb6f550b
I0222 17:41:54.016021 9312 data_consumer.cpp:266] kafka consume done: a540fbcbb4bb9691-1ecea8150856218c, grp: 094151f5426a64cc-f4dab1db80d17bac. cancelled: 0, left time(ms): 14499, total cost(ms): 501, consume cost(ms): 501, received rows: 3, put rows: 3
I0222 17:41:54.016034 9312 data_consumer_group.cpp:80] all consumers are finished. shutdown queue. group id: 094151f5426a64cc-f4dab1db80d17bac
I0222 17:41:54.016038 29122 data_consumer_group.cpp:130] consumer group done: 094151f5426a64cc-f4dab1db80d17bac. consume time(ms)=501, received rows=3, received bytes=1918, eos: 1, left_time: 14499, left_bytes: 4294965378, blocking get time(us): 501797, blocking put time(us): 0
I0222 17:41:54.016049 29122 data_consumer.cpp:365] kafka consumer cancelled. a540fbcbb4bb9691-1ecea8150856218c
I0222 17:41:54.026497 28981 tablet_sink.cpp:1441] Olap table sink statistics. load_id: 70834e7a-e616-4da6-ac9f-e9b2480453b2, txn_id: 3701482, add chunk time(ms)/wait lock time(ms)/num: {42565:(3)(0)(1)} {10008:(8)(0)(1)} {10003:(3)(0)(1)}
I0222 17:41:54.026532 28981 profile_report_worker.cpp:24] unregister_non_pipeline_load fragment_instance_id=70834e7a-e616-4da6-ac9f-e9b2480453b3
I0222 17:41:54.026592 28981 plan_fragment_executor.cpp:480] Fragment 70834e7a-e616-4da6-ac9f-e9b2480453b3:(Active: 511.631ms, non-child: 0.00%)
OlapTableSink:(Active: 11.013ms, non-child: 2.15%)
- TxnID: 3701482
- IndexNum: 1
- ReplicatedStorage: false
FILE_SCAN_NODE (id=0):(Active: 501.307ms, non-child: 97.98%)
FileScanner:
FilePRead:
W0222 17:41:54.026695 29122 thrift_rpc_helper.cpp:63] retrying call frontend service after 100 ms, address=TNetworkAddress(hostname=172.16.4.127, port=9020), reason=No more data to read.
I0222 17:41:54.198609 29122 data_consumer.cpp:60] init kafka consumer with group id: loadobc_order_order_warehouse_out_resume_f2eb954f-bd80-4401-a709-5f4b6ea1ebb1
I0222 17:41:54.201306 29122 routine_load_task_executor.cpp:276] finished routine load task id=70834e7ae6164da6-ac9fe9b2480453b2, job_id=3814249, txn_id: 3701482, label=loadobc_order_order_warehouse_out_resume-3814249-70834e7a-e616-4da6-ac9f-e9b2480453b2-3701482, db=dwd_flink, status: , current tasks num: 0
I0222 17:41:54.486425 29298 agent_server.cpp:383] Submit task success. type=CLONE, signature=5276875
I0222 17:41:54.486441 29298 agent_server.cpp:383] Submit task success. type=CLONE, signature=4844364
I0222 17:41:54.486449 29298 agent_server.cpp:383] Submit task success. type=CLONE, signature=4857900
I0222 17:41:54.486452 29298 agent_server.cpp:383] Submit task success. type=CLONE, signature=5618253
I0222 17:41:54.486461 29182 engine_clone_task.cpp:204] Creating a new replica of tablet 5276875 by clone. version:1
I0222 17:41:54.486464 29183 engine_clone_task.cpp:204] Creating a new replica of tablet 4844364 by clone. version:1
I0222 17:41:54.486486 29182 storage_engine.cpp:1051] Obtained shard path=/opt/module/storage/data/10
I0222 17:41:54.486491 29183 storage_engine.cpp:1051] Obtained shard path=/opt/module/storage/data/11
I0222 17:41:54.490121 29182 engine_clone_task.cpp:586] Copied tablet 5276875 files=1. bytes=2656 cost=0 ms rate=0 MB/s
I0222 17:41:54.490419 29183 engine_clone_task.cpp:586] Copied tablet 4844364 files=1. bytes=2936 cost=0 ms rate=0 MB/s
I0222 17:41:54.490586 29182 snapshot_manager.cpp:176] Assigning new rowset id for cloned rowsets
I0222 17:41:54.490825 29183 snapshot_manager.cpp:176] Assigning new rowset id for cloned rowsets
I0222 17:41:54.492039 29182 engine_clone_task.cpp:376] Cloned snapshot from http://172.16.4.129:8040/api/_tablet/_download?token=85a39bb5-d5cd-479a-9e5f-7a6676a88eae&type=V2&file=/data/starrocks/snapshot/20230222174154.6080.180//5276875/42944869/ to /opt/module/storage/data/10/5276875/42944869
I0222 17:41:54.492048 29182 tablet_manager.cpp:735] Loading tablet 5276875 from /opt/module/storage/data/10/5276875/42944869. force=0 restore=0
I0222 17:41:54.492058 29183 engine_clone_task.cpp:376] Cloned snapshot from http://172.16.4.129:8040/api/_tablet/_download?token=85a39bb5-d5cd-479a-9e5f-7a6676a88eae&type=V2&file=/data/starrocks/snapshot/20230222174154.6081.180//4844364/1179653723/ to /opt/module/storage/data/11/4844364/1179653723
I0222 17:41:54.492069 29183 tablet_manager.cpp:735] Loading tablet 4844364 from /opt/module/storage/data/11/4844364/1179653723. force=0 restore=0
I0222 17:41:54.493536 29182 tablet_manager.cpp:779] Reporting tablet info. tablet_id=5276875
I0222 17:41:54.493549 29182 engine_clone_task.cpp:315] clone get tablet info success. tablet_id:5276875, schema_hash:42944869, signature:5276875, version:1, min_readable_version:1
I0222 17:41:54.493556 29182 agent_task.cpp:324] clone success, set tablet infos. status:0, signature:5276875
I0222 17:41:54.493571 29183 tablet_manager.cpp:779] Reporting tablet info. tablet_id=4844364
I0222 17:41:54.493578 29183 engine_clone_task.cpp:315] clone get tablet info success. tablet_id:4844364, schema_hash:1179653723, signature:4844364, version:1, min_readable_version:1
I0222 17:41:54.493582 29183 agent_task.cpp:324] clone success, set tablet infos. status:0, signature:4844364
I0222 17:41:54.498945 29182 engine_clone_task.cpp:204] Creating a new replica of tablet 4857900 by clone. version:1
I0222 17:41:54.498951 29183 engine_clone_task.cpp:204] Creating a new replica of tablet 5618253 by clone. version:1
I0222 17:41:54.498962 29182 storage_engine.cpp:1051] Obtained shard path=/opt/module/storage/data/12
I0222 17:41:54.498971 29183 storage_engine.cpp:1051] Obtained shard path=/opt/module/storage/data/13
I0222 17:41:54.502525 29183 engine_clone_task.cpp:586] Copied tablet 5618253 files=1. bytes=1899 cost=0 ms rate=0 MB/s
I0222 17:41:54.502604 29182 engine_clone_task.cpp:586] Copied tablet 4857900 files=1. bytes=2933 cost=0 ms rate=0 MB/s
I0222 17:41:54.503042 29183 snapshot_manager.cpp:176] Assigning new rowset id for cloned rowsets
I0222 17:41:54.503067 29182 snapshot_manager.cpp:176] Assigning new rowset id for cloned rowsets
I0222 17:41:54.504585 29183 engine_clone_task.cpp:376] Cloned snapshot from http://172.16.4.130:8040/api/_tablet/_download?token=85a39bb5-d5cd-479a-9e5f-7a6676a88eae&type=V2&file=/data/starrocks/snapshot/20230222174154.3807.180//5618253/1481912711/ to /opt/module/storage/data/13/5618253/1481912711
I0222 17:41:54.504592 29183 tablet_manager.cpp:735] Loading tablet 5618253 from /opt/module/storage/data/13/5618253/1481912711. force=0 restore=0
I0222 17:41:54.504606 29182 engine_clone_task.cpp:376] Cloned snapshot from http://172.16.4.129:8040/api/_tablet/_download?token=85a39bb5-d5cd-479a-9e5f-7a6676a88eae&type=V2&file=/data/starrocks/snapshot/20230222174154.6082.180//4857900/1179653723/ to /opt/module/storage/data/12/4857900/1179653723
I0222 17:41:54.504616 29182 tablet_manager.cpp:735] Loading tablet 4857900 from /opt/module/storage/data/12/4857900/1179653723. force=0 restore=0
I0222 17:41:54.506011 29183 tablet_manager.cpp:779] Reporting tablet info. tablet_id=5618253
I0222 17:41:54.506017 29183 engine_clone_task.cpp:315] clone get tablet info success. tablet_id:5618253, schema_hash:1481912711, signature:5618253, version:1, min_readable_version:1
I0222 17:41:54.506031 29183 agent_task.cpp:324] clone success, set tablet infos. status:0, signature:5618253
I0222 17:41:54.506081 29182 tablet_manager.cpp:779] Reporting tablet info. tablet_id=4857900
I0222 17:41:54.506089 29182 engine_clone_task.cpp:315] clone get tablet info success. tablet_id:4857900, schema_hash:1179653723, signature:4857900, version:1, min_readable_version:1
I0222 17:41:54.506105 29182 agent_task.cpp:324] clone success, set tablet infos. status:0, signature:4857900
I0222 17:41:54.631047 29241 data_consumer.cpp:60] init kafka consumer with group id: loadljkc_gl_docfree1_ac002b4d-2137-43e2-9973-5e22262ba5b7
I0222 17:41:54.633329 29241 data_consumer.cpp:60] init kafka consumer with group id: loadljkc_po_purchaseinfi_b_8f4194ce-b367-4086-84b2-ac5ca6c2e2df
I0222 17:41:55.020308 29240 data_consumer.cpp:60] init kafka consumer with group id: loadljkc_to_presettleout_b_1739358c-bd72-449f-8b50-4fabf22ab034
I0222 17:41:55.286976 29248 data_consumer.cpp:60] init kafka consumer with group id: loadljkc_so_squareinv_d_d1b4da26-cc4a-4b94-a141-32cc3fe4f360
I0222 17:41:55.350914 29249 data_consumer.cpp:60] init kafka consumer with group id: loadljkc_so_saleorder_exe_00270e96-c9d6-45e3-869a-9c4150bac87a
I0222 17:41:55.486912 29298 agent_server.cpp:383] Submit task success. type=CLONE, signature=5620870
I0222 17:41:55.486928 29298 agent_server.cpp:383] Submit task success. type=CLONE, signature=4594691
I0222 17:41:55.486937 29298 agent_server.cpp:383] Submit task success. type=CLONE, signature=4770856
I0222 17:41:55.486940 29298 agent_server.cpp:383] Submit task success. type=CLONE, signature=4857896
I0222 17:41:55.486948 29183 engine_clone_task.cpp:204] Creating a new replica of tablet 5620870 by clone. version:1
I0222 17:41:55.486948 29182 engine_clone_task.cpp:204] Creating a new replica of tablet 4594691 by clone. version:1
I0222 17:41:55.486972 29183 storage_engine.cpp:1051] Obtained shard path=/opt/module/storage/data/14
I0222 17:41:55.486975 29182 storage_engine.cpp:1051] Obtained shard path=/opt/module/storage/data/15
I0222 17:41:55.490303 29182 engine_clone_task.cpp:586] Copied tablet 4594691 files=1. bytes=3408 cost=0 ms rate=0 MB/s
I0222 17:41:55.490720 29183 engine_clone_task.cpp:586] Copied tablet 5620870 files=1. bytes=1898 cost=0 ms rate=0 MB/s
I0222 17:41:55.490794 29182 snapshot_manager.cpp:176] Assigning new rowset id for cloned rowsets
I0222 17:41:55.491205 29183 snapshot_manager.cpp:176] Assigning new rowset id for cloned rowsets
I0222 17:41:55.492274 29182 engine_clone_task.cpp:376] Cloned snapshot from http://172.16.4.128:8040/api/_tablet/_download?token=85a39bb5-d5cd-479a-9e5f-7a6676a88eae&type=V2&file=/data/starrocks/snapshot/20230222174155.4429.180//4594691/1346481182/ to /opt/module/storage/data/15/4594691/1346481182
I0222 17:41:55.492285 29182 tablet_manager.cpp:735] Loading tablet 4594691 from /opt/module/storage/data/15/4594691/1346481182. force=0 restore=0
I0222 17:41:55.493049 29183 engine_clone_task.cpp:376] Cloned snapshot from http://172.16.4.129:8040/api/_tablet/_download?token=85a39bb5-d5cd-479a-9e5f-7a6676a88eae&type=V2&file=/data/starrocks/snapshot/20230222174155.6083.180//5620870/1481912711/ to /opt/module/storage/data/14/5620870/1481912711
I0222 17:41:55.493059 29183 tablet_manager.cpp:735] Loading tablet 5620870 from /opt/module/storage/data/14/5620870/1481912711. force=0 restore=0
I0222 17:41:55.494482 29182 tablet_manager.cpp:779] Reporting tablet info. tablet_id=4594691
I0222 17:41:55.494489 29182 engine_clone_task.cpp:315] clone get tablet info success. tablet_id:4594691, schema_hash:1346481182, signature:4594691, version:1, min_readable_version:1
I0222 17:41:55.494494 29182 agent_task.cpp:324] clone success, set tablet infos. status:0, signature:4594691
I0222 17:41:55.494827 29183 tablet_manager.cpp:779] Reporting tablet info. tablet_id=5620870
I0222 17:41:55.494833 29183 engine_clone_task.cpp:315] clone get tablet info success. tablet_id:5620870, schema_hash:1481912711, signature:5620870, version:1, min_readable