【详述】导完数据后进行查询，报Fail to scan tablet. error: fail to init reader 的错误
错误如图所示：
查询报错的tablet 1409187
查询该tablet的明细
查询集群平衡状态
【背景】导入对应表的数据
【业务影响】 无法查询，每次查询都报错
10.177.81.193对应时间段的be.WARNING 信息
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/runtime/plan_fragment_executor.cpp:217 _get_next_internal_vectorized(&chunk)
W1116 02:42:01.389093 21298 tablet_updates.cpp:945] wait_for_version timeout(60002ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
W1116 02:43:01.400303 21063 tablet_updates.cpp:945] wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
W1116 02:43:01.400427 21063 tablet_reader.cpp:61] fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000)
W1116 02:43:01.400444 21063 tablet_scanner.cpp:64] Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
E1116 02:43:01.400463 21063 olap_scan_node.cpp:97] Failed to start scan node: Internal error: Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:519 scanner->init(state, scanner_params)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:369 _start_scan_thread(state)
W1116 02:43:01.400688 21063 plan_fragment_executor.cpp:188] Fail to open fragment, instance_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea29, status=Internal error: Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:519 scanner->init(state, scanner_params)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:369 _start_scan_thread(state)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:562 child(0)->get_next(state, &_cur_left_input_chunk, &_probe_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:288 _probe(state, probe_timer, chunk, tmp_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:562 child(0)->get_next(state, &_cur_left_input_chunk, &_probe_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:288 _probe(state, probe_timer, chunk, tmp_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/topn_node.cpp:170 child->get_next(state, &chunk, &eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/runtime/plan_fragment_executor.cpp:203 _plan->open(_runtime_state)
W1116 02:43:01.417160 21063 fragment_mgr.cpp:195] Fail to open fragment 0d7e1a56-6515-11ed-aa4a-843e92e0ea29: Internal error: Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:519 scanner->init(state, scanner_params)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:369 _start_scan_thread(state)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:562 child(0)->get_next(state, &_cur_left_input_chunk, &_probe_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:288 _probe(state, probe_timer, chunk, tmp_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:562 child(0)->get_next(state, &_cur_left_input_chunk, &_probe_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:288 _probe(state, probe_timer, chunk, tmp_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/topn_node.cpp:170 child->get_next(state, &chunk, &eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/runtime/plan_fragment_executor.cpp:203 _plan->open(_runtime_state)
10.177.81.193 对应的时间点附近的be.INFO 信息
I1116 02:29:59.529531 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:30:01.910670 21179 update_manager.cpp:156] index cache expire: before:(3 0) after:(3 0) expire: (0 0)
I1116 02:30:17.142248 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3121
I1116 02:30:59.529703 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:31:17.227878 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3133
I1116 02:31:59.529878 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:32:17.322232 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3145
I1116 02:32:59.530037 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:33:17.419095 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3157
I1116 02:33:59.530195 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:34:17.514519 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3169
I1116 02:34:59.530347 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:35:17.617909 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3181
I1116 02:35:59.530516 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:36:01.932932 21179 update_manager.cpp:156] index cache expire: before:(3 0) after:(3 0) expire: (0 0)
I1116 02:36:17.713502 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3193
I1116 02:36:59.530683 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:37:17.800619 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3205
I1116 02:37:59.530844 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:38:17.912693 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3217
I1116 02:38:59.531008 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:39:18.009964 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3229
I1116 02:39:59.531157 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:40:18.101238 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3241
I1116 02:40:59.531307 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:41:01.386487 21298 internal_service.cpp:189] exec plan fragment, fragment_instance_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea29, coord=TNetworkAddress(hostname=10.177.81.152, port=9020), backend=40, is_pipeline=0, chunk_size=4096
I1116 02:41:01.386549 21298 plan_fragment_executor.cpp:69] Prepare(): query_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea01 fragment_instance_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea29 backend_num=40
I1116 02:41:18.189472 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3253
I1116 02:41:59.531455 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
W1116 02:42:01.389093 21298 tablet_updates.cpp:945] wait_for_version timeout(60002ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
I1116 02:42:01.389520 21063 plan_fragment_executor.cpp:183] Open(): fragment_instance_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea29
I1116 02:42:01.955129 21179 update_manager.cpp:156] index cache expire: before:(3 0) after:(3 0) expire: (0 0)
I1116 02:42:18.303162 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3265
I1116 02:42:59.531610 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
W1116 02:43:01.400303 21063 tablet_updates.cpp:945] wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
W1116 02:43:01.400427 21063 tablet_reader.cpp:61] fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000)
W1116 02:43:01.400444 21063 tablet_scanner.cpp:64] Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
E1116 02:43:01.400463 21063 olap_scan_node.cpp:97] Failed to start scan node: Internal error: Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:519 scanner->init(state, scanner_params)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:369 _start_scan_thread(state)
W1116 02:43:01.400688 21063 plan_fragment_executor.cpp:188] Fail to open fragment, instance_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea29, status=Internal error: Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:519 scanner->init(state, scanner_params)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:369 _start_scan_thread(state)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:562 child(0)->get_next(state, &_cur_left_input_chunk, &_probe_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:288 _probe(state, probe_timer, chunk, tmp_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:562 child(0)->get_next(state, &_cur_left_input_chunk, &_probe_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:288 _probe(state, probe_timer, chunk, tmp_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/topn_node.cpp:170 child->get_next(state, &chunk, &eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/runtime/plan_fragment_executor.cpp:203 _plan->open(_runtime_state)
W1116 02:43:01.417160 21063 fragment_mgr.cpp:195] Fail to open fragment 0d7e1a56-6515-11ed-aa4a-843e92e0ea29: Internal error: Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:519 scanner->init(state, scanner_params)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/olap_scan_node.cpp:369 _start_scan_thread(state)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:562 child(0)->get_next(state, &_cur_left_input_chunk, &_probe_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:288 _probe(state, probe_timer, chunk, tmp_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:562 child(0)->get_next(state, &_cur_left_input_chunk, &_probe_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/hash_join_node.cpp:288 _probe(state, probe_timer, chunk, tmp_eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/project_node.cpp:122 _children[0]->get_next(state, chunk, eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/exec/vectorized/topn_node.cpp:170 child->get_next(state, &chunk, &eos)
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/runtime/plan_fragment_executor.cpp:203 _plan->open(_runtime_state)
I1116 02:43:01.419195 21063 plan_fragment_executor.cpp:450] Fragment 0d7e1a56-6515-11ed-aa4a-843e92e0ea29:(Active: 1m, non-child: 0.00%)
-
AverageThreadTokens: 1.00
-
MemoryLimit: 500.00 GB
-
PeakMemoryUsage: 160.29 KB
-
RowsProduced: 0
DataStreamSender (dst_id=10, dst_fragments=[0d7e1a56651511ed-aa4a843e92e0ea2b]):(Active: 1.731ms, non-child: 0.00%)
- PartType: UNPARTITIONED
- BytesSent: 0
- CompressTime: 0.000ns
- IgnoreRows: 0
- OverallThroughput: 0.00 /sec
- SendRequestTime: 508.515us
- SerializeBatchTime: 0.000ns
- ShuffleDispatchTime: 0.000ns
- ShuffleHashTime: 0.000ns
- UncompressedBytes: 0
- WaitResponseTime: 963.689us
SORT_NODE (id=9):(Active: 1m, non-child: 0.00%)
- SortKeys: 22: replace DESC
- SortType: All
- ChunksSorter: 14.120us
- 1-BuildingTime: 0.000ns
- 2-SortingTime: 0.000ns
- 3-MergingTime: 0.000ns
- 4-OutputTime: 0.000ns
- PeakMemoryUsage: 16.00 KB
- RowsReturned: 0
- RowsReturnedRate: 0
PROJECT_NODE (id=8):(Active: 1m, non-child: 0.00%)
- CommonSubExprComputeTime: 0.000ns
- ExprComputeTime: 0.000ns
- PeakMemoryUsage: 0
- RowsReturned: 0
- RowsReturnedRate: 0
HASH_JOIN_NODE (id=7):(Active: 1m, non-child: 0.00%)
- JoinPredicates: 6: app_type = 19: app_type
- JoinType: LeftOuterJoin
- AvgInputProbeChunkSize: 0
- AvgOutputChunkSize: 0
- BuildBuckets: 64
- BuildRows: 33
- BuildTime: 42.590us
- 1-CopyRightTableChunkTime: 7.610us
- 2-BuildHashTableTime: 23.990us
- 3-BuildPushDownExprTime: 660.000ns
- 4-BuildConjunctEvaluateTime: 2.180us
- PeakMemoryUsage: 1.91 KB
- ProbeRows: 0
- ProbeTime: 1.860us
- 1-MergeInputChunkTimer: 0.000ns
- 2-SearchHashTableTimer: 0.000ns
- 3-OutputBuildColumnTimer: 0.000ns
- 4-OutputProbeColumnTimer: 0.000ns
- 5-OutputTupleColumnTimer: 0.000ns
- 6-ProbeConjunctEvaluateTime: 0.000ns
- 7-OtherJoinConjunctEvaluateTime: 0.000ns
- 8-WhereConjunctEvaluateTime: 0.000ns
- PushDownExprNum: 0
- RowsReturned: 0
- RowsReturnedRate: 0
EXCHANGE_NODE (id=6):(Active: 7.266ms, non-child: 0.01%)
- BytesPassThrough: 0
- BytesReceived: 801.00 B
- DecompressRowBatchTimer: 0.000ns
- DeserializeRowBatchTimer: 15.380us
- PeakMemoryUsage: 0
- RequestReceived: 1.00 B
- RowsReturned: 33
- RowsReturnedRate: 4.54 K/sec
- SenderTotalTime: 27.680us
- SenderWaitLockTime: 170.000ns
PROJECT_NODE (id=4):(Active: 1m, non-child: 0.00%)
- CommonSubExprComputeTime: 0.000ns
- ExprComputeTime: 0.000ns
- PeakMemoryUsage: 0
- RowsReturned: 0
- RowsReturnedRate: 0
HASH_JOIN_NODE (id=3):(Active: 1m, non-child: 0.00%)
- JoinPredicates: 5: app_code = 14: app_code
- JoinType: LeftOuterJoin
- AvgInputProbeChunkSize: 0
- AvgOutputChunkSize: 0
- BuildBuckets: 4.10K
- BuildRows: 2.94K
- BuildTime: 274.282us
- 1-CopyRightTableChunkTime: 126.151us
- 2-BuildHashTableTime: 124.641us
- 3-BuildPushDownExprTime: 1.110us
- 4-BuildConjunctEvaluateTime: 1.050us
- PeakMemoryUsage: 235.16 KB
- ProbeRows: 0
- ProbeTime: 3.500us
- 1-MergeInputChunkTimer: 0.000ns
- 2-SearchHashTableTimer: 0.000ns
- 3-OutputBuildColumnTimer: 0.000ns
- 4-OutputProbeColumnTimer: 0.000ns
- 5-OutputTupleColumnTimer: 0.000ns
- 6-ProbeConjunctEvaluateTime: 0.000ns
- 7-OtherJoinConjunctEvaluateTime: 0.000ns
- 8-WhereConjunctEvaluateTime: 0.000ns
- PushDownExprNum: 0
- RowsReturned: 0
- RowsReturnedRate: 0
OLAP_SCAN_NODE (id=2):(Active: 463.025us, non-child: 0.00%)
- Table: DIM_APP_RANK_CODE_COLOCATE_JOIN_app_code_index
- Rollup: DIM_APP_RANK_CODE_COLOCATE_JOIN_app_code_index
- BytesRead: 121.33 KB
- IOTaskCount : 3
- NumDiskAccess: 0
- PeakMemoryUsage: 0
- RowsRead: 2.94K
- RowsReturned: 2.94K
- RowsReturnedRate: 6.34 M/sec
- ScanConcurrency : 3
- ScanTime: 707.567us
- ScannerThreadsInvoluntaryContextSwitches: 0
- ScannerThreadsTotalWallClockTime: 0.000ns
- MaterializeTupleTime(*): 0.000ns
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 0.000ns
- ScannerThreadsVoluntaryContextSwitches: 0
- TabletCount : 3
- TotalRawReadTime(*): 0.000ns
- TotalReadThroughput: 1.01 KB/sec
SCAN:(Active: 692.027us, non-child: 0.00%)
- CachedPagesNum: 12
- CompressedBytesRead: 0
- CreateSegmentIter: 63.592us
- IOTime: 0.000ns
- PushdownPredicates: 0
- RawRowsRead: 2.94K
- ReadPagesNum: 12
- SegmentInit: 372.214us
- BitmapIndexFilter: 0.000ns
- BitmapIndexFilterRows: 0
- BloomFilterFilterRows: 0
- SegmentZoneMapFilterRows: 0
- ShortKeyFilterRows: 0
- ZoneMapIndexFilterRows: 0
- SegmentRead: 312.623us
- BlockFetch: 218.883us
- BlockFetchCount: 3
- BlockSeek: 83.530us
- BlockSeekCount: 3
- ChunkCopy: 0.000ns
- DecompressT: 0.000ns
- DelVecFilterRows: 0
- IndexLoad: 0.000ns
- PredFilter: 0.000ns
- PredFilterRows: 0
- RowsetsReadCount: 3
- SegmentsReadCount: 3
- TotalColumnsDataPageCount: 9
- UncompressedBytesRead: 0
PROJECT_NODE (id=1):(Active: 1m, non-child: 0.00%)
- CommonSubExprComputeTime: 0.000ns
- ExprComputeTime: 0.000ns
- PeakMemoryUsage: 0
- RowsReturned: 0
- RowsReturnedRate: 0
OLAP_SCAN_NODE (id=0):(Active: 1m, non-child: 99.99%)
- Table: TARGET_D_APP_RANK_COLOCATE_JOIN_app_code_index
- Rollup: TARGET_D_APP_RANK_COLOCATE_JOIN_app_code_index
- Predicates: 1: month_id = '202206', 2: day_id = '20220624', 3: prov_id = '111', 4: area_code = '-1', 8: user_type = '0', 9: stat_type = '2', 12: flow_pr IS NOT NULL
- BytesRead: 0
- IOTaskCount : 2
- NumDiskAccess: 0
- PeakMemoryUsage: 0
- RowsRead: 0
- RowsReturned: 0
- RowsReturnedRate: 0
- ScanConcurrency : 0
- ScanTime: 0.000ns
- ScannerThreadsInvoluntaryContextSwitches: 0
- ScannerThreadsTotalWallClockTime: 0.000ns
- MaterializeTupleTime(*): 0.000ns
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 0.000ns
- ScannerThreadsVoluntaryContextSwitches: 0
- TabletCount : 3
- TotalRawReadTime(*): 0.000ns
- TotalReadThroughput: 0.00 /sec
SCAN:
- CachedPagesNum: 0
- CompressedBytesRead: 0
- CreateSegmentIter: 0.000ns
- IOTime: 0.000ns
- PushdownPredicates: 7
- RawRowsRead: 0
- ReadPagesNum: 0
- SegmentInit: 0.000ns
- BitmapIndexFilter: 0.000ns
- BitmapIndexFilterRows: 0
- BloomFilterFilterRows: 0
- SegmentZoneMapFilterRows: 0
- ShortKeyFilterRows: 0
- ZoneMapIndexFilterRows: 0
- SegmentRead: 0.000ns
- BlockFetch: 0.000ns
- BlockFetchCount: 0
- BlockSeek: 0.000ns
- BlockSeekCount: 0
- ChunkCopy: 0.000ns
- DecompressT: 0.000ns
- DelVecFilterRows: 0
- IndexLoad: 0.000ns
- PredFilter: 0.000ns
- PredFilterRows: 0
- RowsetsReadCount: 2
- SegmentsReadCount: 0
- TotalColumnsDataPageCount: 0
- UncompressedBytesRead: 0
I1116 02:43:18.389427 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3277
I1116 02:43:59.531762 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:44:18.480756 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3289
I1116 02:44:59.531913 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:45:18.569357 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3301
I1116 02:45:59.532068 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:46:18.661458 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3313
I1116 02:46:59.532220 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:47:18.745100 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3325
I1116 02:47:59.532361 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:48:01.977308 21179 update_manager.cpp:156] index cache expire: before:(3 0) after:(3 0) expire: (0 0)
I1116 02:48:18.833091 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3337
I1116 02:48:59.532513 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:49:18.920627 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3349
I1116 02:49:59.532665 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
I1116 02:50:19.011677 21459 heartbeat_server.cpp:57] get heartbeat from FE.host:10.177.81.154, port:9020, cluster id:1623223024, counter:3361
I1116 02:50:59.532814 21178 load_channel_mgr.cpp:175] Memory consumption(bytes) limit=107374182400 current=0 peak=3691944
对应时间段的 fe.log 的信息
2022-11-16 02:40:08,094 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601918, replay to journal id is 601919
2022-11-16 02:40:11,633 WARN (tablet stat mgr|41) [TabletStatMgr.runAfterCatalogReady():71] task exec error. backend[11007]
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.thrift.BackendService$Client.recv_get_tablet_stat(BackendService.java:454) ~[starrocks-fe.jar:?]
at com.starrocks.thrift.BackendService$Client.get_tablet_stat(BackendService.java:442) ~[starrocks-fe.jar:?]
at com.starrocks.catalog.TabletStatMgr.runAfterCatalogReady(TabletStatMgr.java:64) [starrocks-fe.jar:?]
at com.starrocks.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:61) [starrocks-fe.jar:?]
at com.starrocks.common.util.Daemon.run(Daemon.java:119) [starrocks-fe.jar:?]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_333]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_333]
at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_333]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_333]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_333]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_333]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_333]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 10 more
2022-11-16 02:40:11,636 INFO (tablet stat mgr|41) [TabletStatMgr.runAfterCatalogReady():80] finished to get tablet stat of all backends. cost: 2402266 ms
2022-11-16 02:40:11,638 INFO (tablet stat mgr|41) [TabletStatMgr.runAfterCatalogReady():123] finished to update index row num of all databases. cost: 2 ms
2022-11-16 02:40:12,297 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601919, replay to journal id is 601920
2022-11-16 02:40:13,098 INFO (nioEventLoopGroup-3-166|355) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:13,100 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601920, replay to journal id is 601921
2022-11-16 02:40:18,106 INFO (nioEventLoopGroup-3-167|356) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:18,108 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601921, replay to journal id is 601922
2022-11-16 02:40:22,298 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601922, replay to journal id is 601923
2022-11-16 02:40:23,114 INFO (nioEventLoopGroup-3-168|357) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:23,115 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601923, replay to journal id is 601924
2022-11-16 02:40:28,121 INFO (nioEventLoopGroup-3-169|358) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:28,122 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601924, replay to journal id is 601925
2022-11-16 02:40:32,299 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601925, replay to journal id is 601926
2022-11-16 02:40:33,130 INFO (nioEventLoopGroup-3-170|359) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:33,132 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601926, replay to journal id is 601927
2022-11-16 02:40:38,136 INFO (nioEventLoopGroup-3-171|360) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:38,139 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601927, replay to journal id is 601928
2022-11-16 02:40:42,300 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601928, replay to journal id is 601929
2022-11-16 02:40:43,144 INFO (nioEventLoopGroup-3-172|361) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:43,145 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601929, replay to journal id is 601930
2022-11-16 02:40:48,151 INFO (nioEventLoopGroup-3-173|362) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:48,152 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601930, replay to journal id is 601931
2022-11-16 02:40:52,301 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601931, replay to journal id is 601932
2022-11-16 02:40:53,157 INFO (nioEventLoopGroup-3-174|363) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:53,159 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601932, replay to journal id is 601933
2022-11-16 02:40:58,165 INFO (nioEventLoopGroup-3-175|364) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:40:58,167 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601933, replay to journal id is 601934
2022-11-16 02:41:01,358 INFO (starrocks-mysql-nio-pool-20|403) [QeProcessorImpl.registerQuery():69] register query id = 0d7e1a56-6515-11ed-aa4a-843e92e0ea01, job: -1
2022-11-16 02:41:02,302 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601934, replay to journal id is 601935
2022-11-16 02:41:03,173 INFO (nioEventLoopGroup-3-176|365) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:03,175 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601935, replay to journal id is 601936
2022-11-16 02:41:08,180 INFO (nioEventLoopGroup-3-177|366) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:08,181 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601936, replay to journal id is 601937
2022-11-16 02:41:12,304 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601937, replay to journal id is 601938
2022-11-16 02:41:13,187 INFO (nioEventLoopGroup-3-178|367) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:13,189 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601938, replay to journal id is 601939
2022-11-16 02:41:18,194 INFO (nioEventLoopGroup-3-179|368) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:18,214 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601939, replay to journal id is 601940
2022-11-16 02:41:22,305 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601940, replay to journal id is 601941
2022-11-16 02:41:23,219 INFO (nioEventLoopGroup-3-180|369) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:23,221 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601941, replay to journal id is 601942
2022-11-16 02:41:28,227 INFO (nioEventLoopGroup-3-181|370) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:28,228 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601942, replay to journal id is 601943
2022-11-16 02:41:32,306 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601943, replay to journal id is 601944
2022-11-16 02:41:33,234 INFO (nioEventLoopGroup-3-182|371) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:33,236 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601944, replay to journal id is 601945
2022-11-16 02:41:38,241 INFO (nioEventLoopGroup-3-183|372) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:38,242 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601945, replay to journal id is 601946
2022-11-16 02:41:42,307 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601946, replay to journal id is 601947
2022-11-16 02:41:43,248 INFO (nioEventLoopGroup-3-184|373) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:43,249 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601947, replay to journal id is 601948
2022-11-16 02:41:48,261 INFO (nioEventLoopGroup-3-185|374) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:48,264 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601948, replay to journal id is 601949
2022-11-16 02:41:52,308 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601949, replay to journal id is 601950
2022-11-16 02:41:53,268 INFO (nioEventLoopGroup-3-186|375) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:53,271 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601950, replay to journal id is 601951
2022-11-16 02:41:58,275 INFO (nioEventLoopGroup-3-187|376) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:41:58,280 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601951, replay to journal id is 601952
2022-11-16 02:42:02,310 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601952, replay to journal id is 601953
2022-11-16 02:42:03,286 INFO (nioEventLoopGroup-3-188|377) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:03,288 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601953, replay to journal id is 601954
2022-11-16 02:42:08,293 INFO (nioEventLoopGroup-3-189|378) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:08,296 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601954, replay to journal id is 601955
2022-11-16 02:42:12,310 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601955, replay to journal id is 601956
2022-11-16 02:42:13,300 INFO (nioEventLoopGroup-3-190|379) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:13,303 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601956, replay to journal id is 601957
2022-11-16 02:42:18,307 INFO (nioEventLoopGroup-3-191|380) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:18,309 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601957, replay to journal id is 601958
2022-11-16 02:42:22,312 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601958, replay to journal id is 601959
2022-11-16 02:42:23,315 INFO (nioEventLoopGroup-3-192|381) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:23,317 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601959, replay to journal id is 601960
2022-11-16 02:42:28,323 INFO (nioEventLoopGroup-3-1|96) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:28,325 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601960, replay to journal id is 601961
2022-11-16 02:42:32,313 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601961, replay to journal id is 601962
2022-11-16 02:42:33,330 INFO (nioEventLoopGroup-3-2|98) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:33,331 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601962, replay to journal id is 601963
2022-11-16 02:42:38,337 INFO (nioEventLoopGroup-3-3|102) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:38,338 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601963, replay to journal id is 601964
2022-11-16 02:42:42,314 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601964, replay to journal id is 601965
2022-11-16 02:42:43,343 INFO (nioEventLoopGroup-3-4|104) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:43,344 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601965, replay to journal id is 601966
2022-11-16 02:42:48,350 INFO (nioEventLoopGroup-3-5|105) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:48,352 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601966, replay to journal id is 601967
2022-11-16 02:42:52,315 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601967, replay to journal id is 601968
2022-11-16 02:42:53,357 INFO (nioEventLoopGroup-3-6|106) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:53,358 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601968, replay to journal id is 601969
2022-11-16 02:42:58,364 INFO (nioEventLoopGroup-3-7|107) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:42:58,367 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601969, replay to journal id is 601970
2022-11-16 02:43:01,402 WARN (thrift-server-pool-33|256) [Coordinator.updateFragmentExecStatus():1560] one instance report fail errorCode INTERNAL_ERROR Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193, query_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea01 instance_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea29
2022-11-16 02:43:01,402 WARN (thrift-server-pool-33|256) [Coordinator.updateStatus():712] one instance report fail throw updateStatus(), need cancel. job id: -1, query id: 0d7e1a56-6515-11ed-aa4a-843e92e0ea01, instance id: 0d7e1a56-6515-11ed-aa4a-843e92e0ea29
2022-11-16 02:43:01,408 WARN (starrocks-mysql-nio-pool-20|403) [Coordinator.getNext():731] get next fail, need cancel. status errorCode CANCELLED Cancelled BufferControlBlock::cancel, query id: 0d7e1a56-6515-11ed-aa4a-843e92e0ea01
2022-11-16 02:43:01,416 INFO (thrift-server-pool-33|256) [Coordinator.cancelInternal():807] unfinished instance: 0d7e1a56-6515-11ed-aa4a-843e92e0ea0e
2022-11-16 02:43:01,416 WARN (starrocks-mysql-nio-pool-20|403) [Coordinator.getNext():752] query failed: Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
2022-11-16 02:43:01,416 WARN (thrift-server-pool-29|252) [Coordinator.updateFragmentExecStatus():1560] one instance report fail errorCode CANCELLED Cancelled because of runtime state is cancelled, query_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea01 instance_id=0d7e1a56-6515-11ed-aa4a-843e92e0ea2b
2022-11-16 02:43:01,417 INFO (starrocks-mysql-nio-pool-20|403) [QeProcessorImpl.unregisterQuery():79] deregister query id 0d7e1a56-6515-11ed-aa4a-843e92e0ea01
2022-11-16 02:43:01,417 INFO (starrocks-mysql-nio-pool-20|403) [StmtExecutor.execute():455] execute Exception. Fail to scan tablet. error: fail to init reader. tablet=1409187.1272809200.3e45f2d617d78f7e-f79128ba26321fb7res=Timeout: wait_for_version timeout(60003ms) version:2 tablet:1409187 #version:2 [1 1@0 2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [0/1/14209172/0/500.85 MB/-468.85 MB]
/data/starrocks/src/starrocks-2.1.11/starrocks-2.1.11/be/src/storage/tablet_updates.cpp:1926 _wait_for_version(EditVersion(version, 0), 60000), backend: 10.177.81.193
2022-11-16 02:43:02,317 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601970, replay to journal id is 601971
2022-11-16 02:43:03,371 INFO (nioEventLoopGroup-3-8|108) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:43:03,374 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601971, replay to journal id is 601972
2022-11-16 02:43:08,380 INFO (nioEventLoopGroup-3-9|109) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
2022-11-16 02:43:08,381 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601972, replay to journal id is 601973
2022-11-16 02:43:12,318 INFO (replayer|82) [Catalog.replayJournal():2484] replayed journal id is 601973, replay to journal id is 601974
2022-11-16 02:43:13,387 INFO (nioEventLoopGroup-3-10|110) [RestBaseAction.handleRequest():55] receive http request. url=/api/bootstrap?cluster_id=1623223024&token=effc8dcb-251d-41dd-91f8-7468afda5fd5
什么版本？导入的目标表是什么模型的表？导入频率和批次都是多大？报错时看看后台磁盘io高不高
2.1.1的arm版本，主键模型，导入数据量116GB左右， 频率不是很高，但集群间的带宽只有千兆。
磁盘IO昨晚没看，现在进行相同的查询已经没问题了。
ok，集群建议万兆网卡，不然就很有可能哪个环节timeout了