starrocks 3.3.20 3台 be服务不可用

starrocks 3.3.20 3台 be服务不可用

【详述】
starrocks 3.3.20
starrocks 3fe 3be 混部

帮我分析一下,starrocks报错是因为什么原因。
文件分别为be_info,be_out,be_warning,还有/var/log/messages系统日志
文件的命名方式为xx_ip_日期(0317).txt。
我之前用promethues配置过报警信息,报警的时间如下:

2026-03-17 14:39:08
To平台與運維[1.主题]:10.77.46.136-8040 異常[2.告警时间]:2026.03.17 14:38:38[3.问题详情]:Port 8040:0[4.告警集群]:starrocks1-ty[5.告警內容]:StarRocks-be,have some problem.
2026-03-17 14:39:08
To平台與運維[1.主题]:10.77.46.137-8040 異常[2.告警时间]:2026.03.17 14:38:38[3.问题详情]:Port 8040:0[4.告警集群]:starrocks1-ty[5.告警內容]:StarRocks-be,have some problem.
2026-03-17 14:59:08
To平台與運維[1.主题]:10.77.46.137-8040 恢復[2.恢复时间]:2026.03.17 14:57:53[3.恢复详情]:Port 8040:1[4.恢复集群]:starrocks1-ty[5.恢复內容]:StarRocks-be,have some problem.[6.告警開始時間]:2026.03.17 14:47:38
2026-03-17 14:59:08
To平台與運維[1.主题]:10.77.46.136-8040 恢復[2.恢复时间]:2026.03.17 14:57:53[3.恢复详情]:Port 8040:1[4.恢复集群]:starrocks1-ty[5.恢复內容]:StarRocks-be,have some problem.[6.告警開始時間]:2026.03.17 14:54:08
2026-03-17 15:00:17
To平台與運維[1.主题]:10.77.46.135-9100 異常[2.告警时间]:2026.03.17 14:59:47[3.问题详情]:Port 9100:0[4.告警集群]:node_exporter_ty_sr1_fe/be[5.告警內容]:TCP_ESTABLISHED大于1500%(目前使用-1548%)
2026-03-17 15:00:53
To平台與運維[1.主题]:10.77.46.136-8040 異常[2.告警时间]:2026.03.17 15:00:23[3.问题详情]:Port 8040:0[4.告警集群]:starrocks1-ty[5.告警內容]:StarRocks-be,have some problem.
2026-03-17 15:01:08
To平台與運維[1.主题]:10.77.46.137-8040 異常[2.告警时间]:2026.03.17 15:00:38[3.问题详情]:Port 8040:0[4.告警集群]:starrocks1-ty[5.告警內容]:StarRocks-be,have some problem.
2026-03-17 15:05:53
To平台與運維[1.主题]:10.77.46.136-8040 恢復[2.恢复时间]:2026.03.17 15:04:23[3.恢复详情]:Port 8040:1[4.恢复集群]:starrocks1-ty[5.恢复內容]:StarRocks-be,have some problem.[6.告警開始時間]:2026.03.17 15:00:23
2026-03-17 15:06:08
To平台與運維[1.主题]:10.77.46.137-8040 恢復[2.恢复时间]:2026.03.17 15:04:23[3.恢复详情]:Port 8040:1[4.恢复集群]:starrocks1-ty[5.恢复內容]:StarRocks-be,have some problem.[6.告警開始時間]:2026.03.17 15:00:38

To平台與運維[1.主题]:10.77.46.135-8040 異常[2.告警时间]:2026.03.17 15:06:23[3.问题详情]:Port 8040:0[4.告警集群]:starrocks1-ty[5.告警內容]:StarRocks-be,have some problem.
2026-03-17 15:10:17
To平台與運維[1.主题]:10.77.46.135-9100 恢復[2.恢复时间]:2026.03.17 15:06:32[3.恢复详情]:Port 9100:1[4.恢复集群]:node_exporter_ty_sr1_fe/be[5.恢复內容]:TCP_ESTABLISHED大于1500%(目前使用-1668%)[6.告警開始時間]:2026.03.17 14:59:47
2026-03-17 15:56:21
To平台与运维[1.主题]: 10.77.46.135发生system.uptime异常![2.告警时间]: 2026.03.17 15:56:11 [3.问题详情]:system.uptime:00:01:39
2026-03-17 15:56:53
To平台與運維[1.主题]:10.77.46.135-8040 恢復[2.恢复时间]:2026.03.17 15:56:23[3.恢复详情]:Port 8040:1[4.恢复集群]:starrocks1-ty[5.恢复內容]:StarRocks-be,have some problem.[6.告警開始時間]:2026.03.17 15:56:08
2026-03-17 16:05:17
To平台与运维[1.主题]: 10.77.46.135已修复system.uptime问题![2.恢复时间]: 2026.03.17 16:05:11 [3.恢复详情]:system.uptime: 00:10:39

表项是,136、137先报警重启,但是重启后集群只读不写,没有解决。最终发现报错日志,好多和135 8060端口通信timeout有关,又想重启135,发现系统判定为僵尸进程,无法通过systemctl重启,也kill不掉。
最终由重启135的机器,才解决了这个问题

这是所有 相关日志,帮我分析出事故原因是为何?

be_info_135_0317.txt (6.5 MB) be_info_136_0317.txt (4.0 MB) 上传中:be_info_137_0317.txt… be_out_135_0317.txt (769.6 KB) be_out_136_0317.txt (384.7 KB) be_out_137_0317.txt (777.7 KB) be_warning_135_0317.txt (647.4 KB) 上传中:be_warning_136_0317.txt… messages_135_0317.txt (232.2 KB) messages_136_0317.txt (2.5 KB) messages_137_0317.txt (2.5 KB)

@trueeyu @Loong 大神帮忙看看 @夏天12


没能看出来多少门道,建议你把leader的日志也发一下,不过是否有routine load任务配置了防火墙不通的kafka?里面貌似提示有个kafka的一直在请求连接,有没有可能它在申请连接把连接数打上来

这个kafka一直都有,不是新的。几个routine load任务一直在跑,近一个月没有新增。
所有的日志,都放在附件里了。不知道当时的leader是谁了,目前是136了

今天又dump了。三台机器日志差不多如下:
be.out

3.3.20 RELEASE (build bc0523a distro centos arch x86_64)
query_id:929a3ce2-2330-11f1-aa1b-005056abf325, fragment_instance:929a3ce2-2330-11f1-aa1b-005056abf326
tracker:process consumption: 31719327280
tracker:jemalloc_metadata consumption: 1396377504
tracker:query_pool consumption: 71754128
tracker:query_pool/connector_scan consumption: 0
tracker:load consumption: 33484897
tracker:metadata consumption: 4801307637
tracker:tablet_metadata consumption: 590551755
tracker:rowset_metadata consumption: 206820887
tracker:segment_metadata consumption: 410236870
tracker:column_metadata consumption: 3593698125
tracker:tablet_schema consumption: 1770907
tracker:segment_zonemap consumption: 93436850
tracker:short_key_index consumption: 281001653
tracker:column_zonemap_index consumption: 1323731317
tracker:ordinal_index consumption: 1513791304
tracker:bitmap_index consumption: 0
tracker:bloom_filter_index consumption: 0
tracker:compaction consumption: 1341360
tracker:schema_change consumption: 0
tracker:column_pool consumption: 0
tracker:page_cache consumption: 17488893728
tracker:jit_cache consumption: 0
tracker:update consumption: 4709220241
tracker:chunk_allocator consumption: 0
tracker:passthrough consumption: 0
tracker:clone consumption: 0
tracker:consistency consumption: 0
tracker:datacache consumption: 0
tracker:replication consumption: 0
*** Aborted at 1773882695 (unix time) try “date -d @1773882695” if you are using GNU date ***
PC: @ 0x7f7ff9f6b27a __memset_evex_unaligned_erms
*** SIGSEGV (@0x7f7c24ff9000) received by PID 926519 (TID 0x7f7ed5494640) from PID 620728320; stack trace: ***
@ 0x7f7ff9e8ef38 __pthread_once_slow
@ 0x7d29900 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
@ 0x7f7ffae04bcb os::Linux::chained_handler(int, siginfo*, void*)
@ 0x7f7ffae098de JVM_handle_linux_signal
@ 0x7f7ffadfd538 signalHandler(int, siginfo*, void*)
@ 0x7f7ff9e3e730 (/usr/lib64/libc.so.6+0x3e72f)
@ 0x7f7ff9f6b27a __memset_evex_unaligned_erms
@ 0x406f39d std::__cxx11::basic_string<char, std::char_traits, std::allocator > starrocks::DecimalV3Cast::to_string(std::enable_if<is_underlying_type_of_decimal, long>::type const&, int, int)
@ 0x406f537 starrocks::DecimalV3Column::put_mysql_row_buffer(starrocks::MysqlRowBuffer*, unsigned long, bool) const
@ 0x6be730c starrocks::MysqlResultWriter::process_chunk(starrocks::Chunk*)
@ 0x44253ed starrocks::pipeline::ResultSinkOperator::push_chunk(starrocks::RuntimeState*, std::shared_ptrstarrocks::Chunk const&)
@ 0x44b71e4 starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int)
@ 0x4779cf3 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x39667c3 starrocks::ThreadPool::dispatch_thread()
@ 0x395de46 starrocks::thread::supervise_thread(void*)
@ 0x7f7ff9e89d22 start_thread
@ 0x7f7ff9f0ed40 __clone3
[1773882695.692][thread:140182720955968] je_mallctl execute purge successstart time: Thu Mar 19 09:11:47 AM CST 2026, server uptime: 09:11:47 up 341 days, 16:31, 0 users, load average: 91.01, 38.43, 21.33
[0.001s][warning][gc] -XX:+PrintGCDetails is deprecated. Will use -Xlog:gc* instead.

/var/log/messages:

Mar 19 09:11:36 node2 systemd[1]: Created slice Slice /system/systemd-coredump.
Mar 19 09:11:36 node2 systemd[1]: Started Process Core Dump (PID 3714202/UID 0).
Mar 19 09:11:38 node2 systemd-coredump[3714203]: Process 926519 (starrocks_be) of user 1001 dumped core.#012#012Module /home/starrocks/StarRocks/be/lib/starrocks_be without build-id.#012Stack trace of thread 1200602:#012#0 0x00007f7ff9e8d004 n/a (n/a + 0x0)#012ELF object binary architecture: AMD x86-64
Mar 19 09:11:38 node2 systemd[1]: systemd-coredump@0-3714202-0.service: Deactivated successfully.
Mar 19 09:11:38 node2 systemd[1]: systemd-coredump@0-3714202-0.service: Consumed 1.504s CPU time.
Mar 19 09:11:41 node2 systemd[1]: starrocks_be.service: Main process exited, code=dumped, status=11/SEGV
Mar 19 09:11:41 node2 systemd[1]: starrocks_be.service: Failed with result ‘core-dump’.
Mar 19 09:11:41 node2 systemd[1]: starrocks_be.service: Consumed 3w 5d 12h 13min 39.677s CPU time.
Mar 19 09:11:46 node2 systemd[1]: starrocks_be.service: Scheduled restart job, restart counter is at 1.
Mar 19 09:11:46 node2 systemd[1]: Stopped starrocks_be Service.
Mar 19 09:11:46 node2 systemd[1]: starrocks_be.service: Consumed 3w 5d 12h 13min 39.677s CPU time.
Mar 19 09:11:46 node2 systemd[1]: Starting starrocks_be Service…
Mar 19 09:11:47 node2 systemd[1]: Started starrocks_be Service.
Mar 19 09:13:55 node2 sshd-session[3716411]: Accepted password for root from 10.251.172.190 port 52594 ssh2
Mar 19 09:13:55 node2 systemd[1]: Starting Hostname Service…
Mar 19 09:13:55 node2 systemd[1]: Started Hostname Service.
Mar 19 09:13:55 node2 sshd-session[3716422]: Accepted password for root from 10.251.172.190 port 52595 ssh2
Mar 19 09:14:25 node2 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Mar 19 09:21:57 node2 sshd-session[3722452]: Accepted password for root from 10.251.172.190 port 53077 ssh2
Mar 19 09:21:58 node2 sshd-session[3722454]: Connection closed by 10.251.172.190 port 53077
Mar 19 09:22:16 node2 cupsd[1419]: REQUEST localhost - - “POST / HTTP/1.1” 200 182 Renew-Subscription successful-ok
Mar 19 09:30:10 node2 sshd-session[3728599]: Accepted password for root from 10.251.172.244 port 65475 ssh2
Mar 19 09:30:10 node2 systemd[1]: Starting Hostname Service…
Mar 19 09:30:10 node2 systemd[1]: Started Hostname Service.
Mar 19 09:30:10 node2 sshd-session[3728620]: Accepted password for root from 10.251.172.244 port 65478 ssh2
Mar 19 09:30:40 node2 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Mar 19 09:32:17 node2 sshd-session[3730442]: Accepted password for root from 10.251.172.244 port 65506 ssh2
Mar 19 09:32:18 node2 systemd[1]: Starting Hostname Service…
Mar 19 09:32:18 node2 systemd[1]: Started Hostname Service.
Mar 19 09:32:18 node2 sshd-session[3730460]: Accepted password for root from 10.251.172.244 port 65507 ssh2
Mar 19 09:32:48 node2 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Mar 19 09:34:04 node2 systemd[1]: Started Process Core Dump (PID 3732022/UID 0).
Mar 19 09:34:05 node2 systemd-coredump[3732023]: Process 3714706 (starrocks_be) of user 1001 dumped core.#012#012Module /home/starrocks/StarRocks/be/lib/starrocks_be without build-id.#012Stack trace of thread 3719509:#012#0 0x00007f717508d004 n/a (n/a + 0x0)#012ELF object binary architecture: AMD x86-64
Mar 19 09:34:05 node2 systemd[1]: systemd-coredump@1-3732022-0.service: Deactivated successfully.
Mar 19 09:34:05 node2 systemd[1]: systemd-coredump@1-3732022-0.service: Consumed 1.406s CPU time.
Mar 19 09:34:08 node2 systemd[1]: starrocks_be.service: Main process exited, code=dumped, status=11/SEGV
Mar 19 09:34:08 node2 systemd[1]: starrocks_be.service: Failed with result ‘core-dump’.
Mar 19 09:34:08 node2 systemd[1]: starrocks_be.service: Consumed 5h 53min 5.195s CPU time.
Mar 19 09:34:13 node2 systemd[1]: starrocks_be.service: Scheduled restart job, restart counter is at 2.
Mar 19 09:34:13 node2 systemd[1]: Stopped starrocks_be Service.
Mar 19 09:34:13 node2 systemd[1]: starrocks_be.service: Consumed 5h 53min 5.195s CPU time.
Mar 19 09:34:13 node2 systemd[1]: Starting starrocks_be Service…
Mar 19 09:34:13 node2 systemd[1]: Started starrocks_be Service.

be.warning,09:11:36 down的:
W20260319 09:06:29.005096 140182491207232 pipeline_driver_executor.cpp:184] [Driver] Process error, query_id=dc2e689d-232f-11f1-a184-005056abe7c2, instance_id=dc2e689d-232f-11f1-a184-005056abe7c5, status=Global dictionary error: no global dict: BE:10006
W20260319 09:06:29.005223 140182625490496 pipeline_driver_executor.cpp:184] [Driver] Process error, query_id=dc30d9a7-232f-11f1-a184-005056abe7c2, instance_id=dc30d9a7-232f-11f1-a184-005056abe7c5, status=Global dictionary error: no global dict: BE:10006
W20260319 09:06:29.005624 140183023093312 fragment_context.cpp:191] [Driver] Canceled, query_id=dc306475-232f-11f1-a184-005056abe7c2, instance_id=dc306475-232f-11f1-a184-005056abe7c3, reason=InternalError
E20260319 09:06:29.012754 140182314960448 scan_operator.cpp:465] scan fragment dc31eb1d-232f-11f1-a184-005056abe7c5 driver 0 Scan tasks error: Global dictionary error: no global dict
be/src/storage/meta_reader.cpp:221 _collect(_params->fields[i], _params->cids[i], (*dsts)[i], _params->field_type[i])
be/src/storage/meta_reader.cpp:86 _collect_context.seg_collecters[_collect_context.cursor_idx]->collect(&columns)
be/src/storage/olap_meta_reader.cpp:151 _read((*result).get(), chunk_capacity - chunk_start)
E20260319 09:06:29.012826 140182314960448 scan_operator.cpp:465] scan fragment dc31eb1d-232f-11f1-a184-005056abe7c5 driver 0 Scan tasks error: Global dictionary error: no global dict
be/src/storage/meta_reader.cpp:221 _collect(_params->fields[i], _params->cids[i], (*dsts)[i], _params->field_type[i])
be/src/storage/meta_reader.cpp:86 _collect_context.seg_collecters[_collect_context.cursor_idx]->collect(&columns)
be/src/storage/olap_meta_reader.cpp:151 _read((*result).get(), chunk_capacity - chunk_start)
E20260319 09:06:29.012851 140182314960448 scan_operator.cpp:465] scan fragment dc31eb1d-232f-11f1-a184-005056abe7c5 driver 0 Scan tasks error: Global dictionary error: no global dict
be/src/storage/meta_reader.cpp:221 _collect(_params->fields[i], _params->cids[i], (*dsts)[i], _params->field_type[i])
be/src/storage/meta_reader.cpp:86 _collect_context.seg_collecters[_collect_context.cursor_idx]->collect(&columns)
be/src/storage/olap_meta_reader.cpp:151 _read((*result).get(), chunk_capacity - chunk_start)
W20260319 09:06:29.012863 140182709417536 pipeline_driver.cpp:322] pull_chunk returns not ok status Global dictionary error: no global dict
be/src/storage/meta_reader.cpp:221 _collect(_params->fields[i], _params->cids[i], (*dsts)[i], _params->field_type[i])
be/src/storage/meta_reader.cpp:86 _collect_context.seg_collecters[_collect_context.cursor_idx]->collect(&columns)
be/src/storage/olap_meta_reader.cpp:151 _read((*result).get(), chunk_capacity - chunk_start)
be/src/exec/pipeline/scan/scan_operator.cpp:267 _get_scan_status()
W20260319 09:06:29.012895 140182709417536 pipeline_driver_executor.cpp:184] [Driver] Process error, query_id=dc31eb1d-232f-11f1-a184-005056abe7c2, instance_id=dc31eb1d-232f-11f1-a184-005056abe7c5, status=Global dictionary error: no global dict: BE:10006
E20260319 09:06:29.033930 140182214248000 scan_operator.cpp:465] scan fragment dc314edb-232f-11f1-a184-005056abe7c5 driver 0 Scan tasks error: Global dictionary error: no global dict
be/src/storage/meta_reader.cpp:221 _collect(_params->fields[i], _params->cids[i], (*dsts)[i], _params->field_type[i])
be/src/storage/meta_reader.cpp:86 _collect_context.seg_collecters[_collect_context.cursor_idx]->collect(&columns)
be/src/storage/olap_meta_reader.cpp:151 _read((*result).get(), chunk_capacity - chunk_start)
E20260319 09:06:29.033951 140182231033408 scan_operator.cpp:465] scan fragment dc314edb-232f-11f1-a184-005056abe7c5 driver 0 Scan tasks error: Global dictionary error: no global dict
be/src/storage/meta_reader.cpp:221 _collect(_params->fields[i], _params->cids[i], (*dsts)[i], _params->field_type[i])
be/src/storage/meta_reader.cpp:86 _collect_context.seg_collecters[_collect_context.cursor_idx]->collect(&columns)
be/src/storage/olap_meta_reader.cpp:151 _read((*result).get(), chunk_capacity - chunk_start)
E20260319 09:06:29.033957 140182298175040 scan_operator.cpp:465] scan fragment dc314edb-232f-11f1-a184-005056abe7c5 driver 0 Scan tasks error: Global dictionary error: no global dict
be/src/storage/meta_reader.cpp:221 _collect(_params->fields[i], _params->cids[i], (*dsts)[i], _params->field_type[i])
be/src/storage/meta_reader.cpp:86 _collect_context.seg_collecters[_collect_context.cursor_idx]->collect(&columns)
be/src/storage/olap_meta_reader.cpp:151 _read((*result).get(), chunk_capacity - chunk_start)
W20260319 09:06:29.034065 140182633883200 pipeline_driver.cpp:322] pull_chunk returns not ok status Global dictionary error: no global dict
be/src/storage/meta_reader.cpp:221 _collect(_params->fields[i], _params->cids[i], (*dsts)[i], _params->field_type[i])
be/src/storage/meta_reader.cpp:86 _collect_context.seg_collecters[_collect_context.cursor_idx]->collect(&columns)
be/src/storage/olap_meta_reader.cpp:151 _read((*result).get(), chunk_capacity - chunk_start)
be/src/exec/pipeline/scan/scan_operator.cpp:267 _get_scan_status()
W20260319 09:06:29.034092 140182633883200 pipeline_driver_executor.cpp:184] [Driver] Process error, query_id=dc314edb-232f-11f1-a184-005056abe7c2, instance_id=dc314edb-232f-11f1-a184-005056abe7c5, status=Global dictionary error: no global dict: BE:10006
E20260319 09:06:29.034011 140182432458304 scan_operator.cpp:465] scan fragment dc314edb-232f-11f1-a184-005056abe7c5 driver 0 Scan tasks error: Global dictionary error: no global dict
be/src/storage/meta_reader.cpp:221 _collect(_params->fields[i], _params->cids[i], (*dsts)[i], _params->field_type[i])
be/src/storage/meta_reader.cpp:86 _collect_context.seg_collecters[_collect_context.cursor_idx]->collect(&columns)
be/src/storage/olap_meta_reader.cpp:151 _read((*result).get(), chunk_capacity - chunk_start)
W20260319 09:06:48.653521 140183014700608 fragment_context.cpp:191] [Driver] Canceled, query_id=e7e7f72d-232f-11f1-a184-005056abe7c2, instance_id=e7e7f72d-232f-11f1-a184-005056abe7c5, reason=InternalError
W20260319 09:06:48.654197 140182913988160 fragment_context.cpp:191] [Driver] Canceled, query_id=e7e7f72f-232f-11f1-a184-005056abe7c2, instance_id=e7e7f72f-232f-11f1-a184-005056abe7c5, reason=InternalError
W20260319 09:06:48.654732 140183132198464 fragment_context.cpp:191] [Driver] Canceled, query_id=e7e7f72c-232f-11f1-a184-005056abe7c2, instance_id=e7e7f72c-232f-11f1-a184-005056abe7c5, reason=InternalError
W20260319 09:06:48.654800 140183132198464 fragment_context.cpp:191] [Driver] Canceled, query_id=e7e7f72c-232f-11f1-a184-005056abe7c2, instance_id=e7e7f72c-232f-11f1-a184-005056abe7c3, reason=InternalError
W20260319 09:06:48.655406 140182997915200 fragment_context.cpp:191] [Driver] Canceled, query_id=e7e7f731-232f-11f1-a184-005056abe7c2, instance_id=e7e7f731-232f-11f1-a184-005056abe7c5, reason=InternalError
W20260319 09:06:48.655459 140182955951680 fragment_context.cpp:191] [Driver] Canceled, query_id=e7e7f731-232f-11f1-a184-005056abe7c2, instance_id=e7e7f731-232f-11f1-a184-005056abe7c3, reason=InternalError
W20260319 09:06:48.656828 140182964344384 fragment_context.cpp:191] [Driver] Canceled, query_id=e7e7f729-232f-11f1-a184-005056abe7c2, instance_id=e7e7f729-232f-11f1-a184-005056abe7c5, reason=InternalError
W20260319 09:06:48.660167 140182947558976 fragment_context.cpp:191] [Driver] Canceled, query_id=e7e89373-232f-11f1-a184-005056abe7c2, instance_id=e7e89373-232f-11f1-a184-005056abe7c5, reason=InternalError
W20260319 09:06:53.135914 140175094031936 engine_clone_task.cpp:410] Fail to make snapshot from 10.210.3.45: Not found: get_rowsets_for_snapshot: no version to clone tablet:19178044 #version:5 [3720.3 3722.2@4 3722.2] #pending:0 request_version:3723, tablet:19178044
W20260319 09:06:53.135919 140175076120128 engine_clone_task.cpp:410] Fail to make snapshot from 10.210.3.47: Not found: get_rowsets_for_snapshot: no version to clone tablet:19178052 #version:5 [3720.3 3722.2@4 3722.2] #pending:0 request_version:3723, tablet:19178052
W20260319 09:06:53.138730 140174933612096 engine_clone_task.cpp:410] Fail to make snapshot from 10.210.3.45: Not found: get_rowsets_for_snapshot: no version to clone tablet:19178048 #version:5 [3720.3 3722.2@4 3722.2] #pending:0 request_version:3723, tablet:19178048
E20260319 09:10:00.209351 140187266967104 persistent_index.cpp:480] find a empty shard with kvs, key size: 27, kv_num: 1
E20260319 09:10:14.552767 140187266967104 persistent_index.cpp:480] find a empty shard with kvs, key size: 44, kv_num: 1
W20260319 09:11:35.467781 140179146290752 data_dir.cpp:779] reach capacity limit. used pct: 647030, left bytes: -3472327136091410480, path: /data/StarRocks/be/storage
W20260319 09:11:35.467836 140179146290752 compaction_task.cpp:116] compaction task:2312315 failed, skip tablet:105047168 because data dir reaches capacity limit. input rowsets size:3472328296227680304
W20260319 09:11:35.467920 140179146290752 data_dir.cpp:779] reach capacity limit. used pct: 647030, left bytes: -3472327136091794353, path: /data/StarRocks/be/storage
W20260319 09:11:35.467926 140179146290752 compaction_task.cpp:116] compaction task:2312316 failed, skip tablet:105048222 because data dir reaches capacity limit. input rowsets size:3472328296228064177
W20260319 09:11:48.093403 140125297354560 timezone_utils.cpp:93] not found timezone:Africa/Casablanca
W20260319 09:11:48.094759 140125297354560 timezone_utils.cpp:93] not found timezone:Africa/El_Aaiun

be.info:
I20260319 09:11:34.915043 140179288966720 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538356 load_id: b2411708012328b7-6a64eb5b242b6bbf index_id: 302576 tablet_ids:118002363,114789296,113170673,93701915,106667331,92162716,90608657,85902861,82772103,81316503,75821333,69815201,68280771,66766250,109909952,63635818,79879057,60539202,54076331,52356926,50821287,49216790,47598446,44352206,39559391,38057886,36540145,41095766,35057261,96964789,33512245,30407051,27387777,25863728,22878205,21073362,20623511,20219464,18938575,16996656,57356485,15475485,14972956,12927856,11913547,12420356,11068291,84348789,10856410,78451372,111538350,74297584,9376937,9237369,9102199,10217849,8973308,8865661,7803043,2014463,1911375,1843101,1318868,709220,1634825,1456080,1388046,9965048,2048777,1945554,1559280,763321,100247540,2867359,2152064,24373083,21927460,4763467,72898483,1161922,1490453,6892879,65210580,2760114,7675278,31977989,871648,302765,3196142,14461360,790384,736214,62115322,75165636,1705257,101832921,483629,682140,1350350,10431974,3922726,3788099,627355,3018571,8741215,302705,302717,6315391,1287630,13954831,1256350,305376,10642547,77030383,1192527,4395676,565190,1807309,311055,17979163,654365,4702889,302669,2795609,444690,302633,397300,302753,302609,11399339,1773375,95318788,537588,302657,119601371,1524937,302645,108290129,2901929,6076205,2256384,103442686,16484957,5447018,817323,58951008,302597,3300278,21522260,1037921,1068549,516219,967584,89050450,302729,302777,3510863,1131171,385065,504393,4064112,1877319,9656167,424496,9516466,1099127,1979706,329416,334826,473688,1599623,403016,302741,116377357,1224342,7008890,340528,414645,408852,1421718,317154,4111313,4506117,8234153,302585,493391,3334894,434802,4253180,302621,898661,2830029,600372,9796559,18474700,390660,4601533,2725440,2083333,4830741,323731,2187960,2291400,13444710,3652240,2326253,2360563,2395970,2980499,2440048,3440918,2474519,4966906,17485384,8488071,2539395,15970535,2584905,6546422,2652241,87481246,2690105,3581344,2940940,19373409,3053512,8090822,3088123,302693,8361060,5741233,3126584,19802863,3161392,3230796,3265560,4553739,121193522,3370039,3405812,3475842,1007347,3698817,3743466,4205743,1739314,3616855,3545879,3969521,4016851,4158555,45977459,4301243,7930511,28883916,1671274,5238927,7419266,4348449,98602902,71376952,4458783,4649553,7548081,3877291,4899789,5034412,5101674,5169326,3832707,2507156,5310815,5514772,844306,5379007,5583269,302681,5651388,2222168,5845275,5960824,6193421,6431162,2117541,6662231,55760916,6777495,463564,7262070,7134629,8615046, reason:
I20260319 09:11:34.919274 140178715121216 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538357 load_id: 8a4b123c7a4856f7-7449225f92352cac index_id: 112505426 tablet_ids:112508851,112508839,112508827,112508815,112508791,112508779,112508767,112508755,112508743,112508731,112508719,112508707,112508695,112508683,112508671,112508659,112508647,112508635,112508623,112508611,112508599,112508587,112508575,112508563,112508551,112508539,112508527,112508515,112508503,112508491,112508479,112508467,112508455,112508443,112508431,112508419,112508407,112508395,112508383,112508371,112508359,112508347,112508335,112508323,114789524,112508311,112508299,112508287,112508275,112508263,112508251,112508239,112508227,112508215,112508203,112508191,112508179,112508167,112508151,112508139,112508127,112508115,112508103,112508091,112508079,112508067,112508055,112508043,112508031,112508019,112508007,112507995,112507983,112507971,112507959,112507947,112507935,112507923,112507911,112507899,112507887,112507875,112507863,112507851,112507839,112507827,112507815,112507803,112507791,112507779,112507767,112507755,112507743,112507731,112507719,112507707,112507695,112507683,112507671,112507659,112507647,112507635,112507623,112507611,112507599,112506447,112506435,112506423,112506411,112506399,118002591,112506387,112506375,112506363,112506351,112506339,112506327,112506315,112506303,112506291,112506279,112506267,112506255,112506243,112506231,119601609,112506219,112506207,112506195,112506183,112506171,112506159,112506147,112506135,112506123,112506111,112506099,112506087,112506075,116377586,112506063,112506051,112506039,112506027,112506015,112506003,112505991,112505979,112505967,112505955,112505943,112505655,112505643,112505631,112505619,112505607,112505595,112505583,112505571,112505559,112505547,112505475,112505463,112505451,112505439,112505427,112505487,112505499,113170903,112505511,112505523,112505535,112505667,112505679,112505691,112505703,112505715,112505727,112505739,112505751,112505763,112505775,112505787,112505799,112505811,112505823,112505835,121193753,112505847,112505859,112505871,112505883,112505895,112505907,112505919,112505931,112506459,112506471,112506483,112506495,112506507,112506519,112506531,112506543,112506555,112506567,112506579,112506591,112506603,112506615,112506627,112506639,112506651,112506663,112506675,112506687,112506699,112506711,112506723,112506735,112506747,112506759,112506771,112506783,112506795,112506807,112506819,112506831,112506843,112506855,112506867,112506879,112506891,112506903,112506915,112506927,112506939,112506951,112506963,112506975,112506987,112506999,112507011,112507023,112507035,112507047,112507059,112507071,112507083,112507095,112507107,112507119,112507131,112507143,112507155,112507167,112507179,112507191,112507203,112507215,112507227,112507239,112507251,112507263,112507275,112507287,112507299,112507311,112507323,112507335,112507347,112507359,112507371,112507383,112507395,112507407,112507419,112507431,112507443,112507455,112507467,112507479,112507491,112507503,112507515,112507527,112507539,112507551,112507563,112507575,112507587, reason:
I20260319 09:11:34.920399 140178815833664 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538357 load_id: 8a4b123c7a4856f7-7449225f92352cac index_id: 112505426 tablet_ids:112508847,112508835,112508823,112508811,112508787,112508775,112508763,112508751,112508739,112508727,112508715,112508703,112508691,112508679,112508667,112508655,112508643,112508631,112508619,112508607,112508595,112508583,112508571,112508559,112508547,112508535,112508523,112508511,112508499,112508487,112508475,112508463,112508451,112508439,112508427,112508415,112508403,112508391,112508379,112508367,112508355,112508343,112508331,112508319,114789520,112508307,112508295,112508283,112508271,112508259,112508247,112508235,112508223,112508211,112508199,112508187,112508175,112508163,112508159,112508147,112508135,112508123,112508111,112508099,112508087,112508075,112508063,112508051,112508039,112508027,112508015,112508003,112507991,112507979,112507967,112507955,112507943,112507931,112507919,112507907,112507895,112507883,112507871,112507859,112507847,112507835,112507823,112507811,112507799,112507787,112507775,112507763,112507751,112507739,112507727,112507715,112507703,112507691,112507679,112507667,112507655,112507643,112507631,112507619,112507607,112507595,112506443,112506431,112506419,112506407,112506395,118002587,112506383,112506371,112506359,112506347,112506335,112506323,112506311,112506299,112506287,112506275,112506263,112506251,112506239,112506227,119601605,112506215,112506203,112506191,112506179,112506167,112506155,112506143,112506131,112506119,112506107,112506095,112506083,112506071,116377582,112506059,112506047,112506035,112506023,112506011,112505999,112505987,112505975,112505963,112505951,112505939,112505651,112505639,112505627,112505615,112505603,112505591,112505579,112505567,112505555,112505543,112505471,112505459,112505447,112505435,112505483,112505495,113170899,112505507,112505519,112505531,112505663,112505675,112505687,112505699,112505711,112505723,112505735,112505747,112505759,112505771,112505783,112505795,112505807,112505819,112505831,121193749,112505843,112505855,112505867,112505879,112505891,112505903,112505915,112505927,112506455,112506467,112506479,112506491,112506503,112506515,112506527,112506539,112506551,112506563,112506575,112506587,112506599,112506611,112506623,112506635,112506647,112506659,112506671,112506683,112506695,112506707,112506719,112506731,112506743,112506755,112506767,112506779,112506791,112506803,112506815,112506827,112506839,112506851,112506863,112506875,112506887,112506899,112506911,112506923,112506935,112506947,112506959,112506971,112506983,112506995,112507007,112507019,112507031,112507043,112507055,112507067,112507079,112507091,112507103,112507115,112507127,112507139,112507151,112507163,112507175,112507187,112507199,112507211,112507223,112507235,112507247,112507259,112507271,112507283,112507295,112507307,112507319,112507331,112507343,112507355,112507367,112507379,112507391,112507403,112507415,112507427,112507439,112507451,112507463,112507475,112507487,112507499,112507511,112507523,112507535,112507547,112507559,112507571,112507583, reason:
I20260319 09:11:34.920764 140178731906624 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538355 load_id: ae45d502ff3e7c83-697261a229aad1a4 index_id: 2172859 tablet_ids:121194024,116377856,114789790,111538842,109910445,106667828,103443197,96965281,90609146,89050973,87481738,85903355,82772602,75821822,75166125,74298077,100248035,72898986,98603393,71377443,69815722,65211070,60539694,58951523,57356981,55761411,54076850,52357444,49217254,47598915,44352704,41096233,33512684,31978432,30407500,24373521,21927950,21522676,19373951,118002858,18938991,18475149,17979566,16997059,16485360,15475904,14461763,12420759,11913966,11399745,11068694,10642950,10432377,2585282,2474896,2173996,2725817,5034805,2256761,38058352,6777898,13445116,2174092,2174536,2174524,2174512,2174500,2174488,2174476,9237772,2396347,2174464,12928433,5845678,2174416,2174404,2222545,2174380,7009293,2174368,19803457,2174356,2174332,2174440,3969911,93702424,2174308,3546256,9796962,2690482,2174260,3126961,113171170,4601923,2174248,4064502,2174008,2174224,2173936,7675681,2173924,2188337,2173912,84349315,4459173,2980876,2173900,2173888,3511240,108290665,68281272,8741620,2173876,2173864,17485819,2173852,50822022,2360940,2174236,4554150,20623944,20219880,2795986,2173756,3300655,2173744,78451861,2173732,3441295,2507533,25864158,2173720,2173768,2173780,2173792,2539772,2174056,2173804,3370419,2173816,2173948,2173960,2174320,3652617,2326630,2174068,2173972,36540630,8361463,4158945,2760494,2174020,2174032,2174044,101833437,3476219,2174080,2440425,3877689,4900179,2652618,2174272,28884349,2174104,3161769,4703279,2174116,2902306,2174128,79879548,2174140,66766744,3335271,6431565,2174152,15971187,7419688,2174164,2174176,2174188,21073778,2174200,2174212,2830406,2867736,2941317,4396066,6315794,3018948,2173984,10856813,2174452,3053889,3088508,3196519,2174296,3231173,3265937,119601881,5102067,5169716,2174284,3406383,2174392,2291946,3581721,14973394,3617234,3699194,5583666,3743843,4206133,3788489,3923116,4017241,13955298,4506507,4111703,63636326,7930914,4253570,2174428,9516871,8973711,4301639,5239317,10218252,9102602,4348839,45977929,4649943,81317018,4763857,2173840,5311205,5515162,4967309,39559864,7803653,5379403,35057705,5447409,5651778,8234556,27388207,5741636,5961227,3833097,6076608,6193824,8866064,6546825,2173828,8488474,6662634,77030874,6893282,7135032,7262491,7548625,62115811,92163433,8091256,2174344,95319287,8615449,9377340,4831338,9656585,22878639,9965451, reason:
I20260319 09:11:34.920951 140178790655552 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538355 load_id: ae45d502ff3e7c83-697261a229aad1a4 index_id: 2172859 tablet_ids:121194016,118002850,116377848,111538834,109910437,106667820,114789782,90609138,89050965,87481730,85903347,82772594,75821814,75166117,74298069,92163425,100248027,72898978,69815714,108290657,68281264,63636318,60539686,58951515,57356973,55761403,54076842,52357436,49217246,47598907,44352696,41096225,36540622,35057697,30407492,24373513,22878631,21927942,21522668,20623936,19803449,19373943,18938983,16997051,16485352,33512676,15475896,14461755,13955290,13445108,12420751,11913958,11399737,11068686,10642942,2725809,5034797,4967301,39559856,2173832,2396339,62115803,2174096,2474888,2174528,2174516,2174504,2174492,2174480,2174468,9237764,2174456,4601915,5845670,10432369,4703271,3161761,6431557,2174420,2174408,2174372,31978424,7009285,2174360,2690474,2174336,2174324,2174300,5583658,3546248,25864150,2174264,9796954,2256753,38058344,6777890,2174252,3126953,4158937,2174240,4064494,2174384,2291938,2174000,20219872,2173940,2173928,7675673,2173712,2173916,2188329,2173904,2173892,2173880,3511232,2173868,2173856,17485811,50822014,2360932,2173760,2173748,3300647,2173736,78451853,2173724,3441287,2507525,2173772,2173784,2539764,2174048,2173796,3370411,2174312,3652609,2173808,2173952,2326622,2174060,2173964,2173976,2760486,2174348,2174012,2795978,4017233,2585274,2174024,3476211,2174036,2174072,4900171,2652610,28884341,101833429,2440417,3877681,45977921,4506499,4111695,2174084,2174108,2902298,2174120,79879540,2174132,66766736,3335263,2174144,15971179,7419680,8741612,2174156,2174168,2174180,21073770,2174192,2174204,2174216,2830398,2867728,96965273,2941309,4396058,3018940,10856805,2174444,3053881,5961219,12928425,2173988,3088500,3196511,2174288,3231165,3923108,3265929,84349307,2980868,4459165,3581713,2174396,14973386,3617226,3833089,3699186,10218244,3743835,3788481,7135024,2174276,3406375,7262483,4206125,4253562,4301631,93702416,2174432,3969903,8866056,4348831,113171162,17979558,2174228,4554142,4649935,7548617,81317010,4763849,4831330,5311197,5515154,2222537,2173844,103443189,119601873,5169708,5102059,95319279,8615441,9102594,65211062,5447401,5651770,18475141,27388199,5741628,6076600,5239309,6193816,6315786,6546817,2173820,8488466,6662626,6893274,77030866,8091248,7803645,7930906,8234548,8361455,8973703,9377332,5379395,9516863,9656577,9965443,98603385,71377435, reason:
I20260319 09:11:34.936449 140178248808000 stream_load.cpp:253] new income streaming load request.id=7b4281f2e9a86fbb-dc65cef37d6f76a5, job_id=-1, txn_id: -1, label=f91dac82-b2c7-4918-b8aa-9ee6e1a95ef6, db=pstage_band, db=pstage_band, tbl=ods_log_equip_state
I20260319 09:11:34.940937 140178248808000 stream_load_executor.cpp:77] begin to execute job. label=f91dac82-b2c7-4918-b8aa-9ee6e1a95ef6, txn_id: 390538358, query_id=7b4281f2-e9a8-6fbb-dc65-cef37d6f76a5
I20260319 09:11:34.951256 140179297359424 local_tablets_channel.cpp:782] LocalTabletsChannel txn_id: 390538358 load_id: 7b4281f2-e9a8-6fbb-dc65-cef37d6f76a5 sink_id: 0 open 390 delta writer: [119599982:1][119599978:2][118001004:1][118001000:2][114787925:2][113169320:1][113169316:2][108288715:1][108288711:2][108288707:2][103441305:2][101831334:1][101831330:2][101831326:2][105046476:2][98601543:2][96963423:1][96963419:2][116375961:1][93700487:1][116375957:2][93700483:2][116375953:2][93700479:2][92160433:2][111536984:2][90607317:1][111536980:2][90607313:2][90607309:2][85901498:1][85901494:2][85901490:2][84347385:1][84347381:2][84347377:2][82770671:1][89048972:1][82770667:2][89048968:2][82770663:2][79877718:1][79877714:2][79877710:2][114787921:2][78450033:1][78450029:2][78450025:2][77029029:2][77029025:2][75819990:2][75819986:2][75164297:1][74296176:1][69813751:1][69813747:2][69813743:2][68279408:1][68279404:2][68279400:2][66764897:1][66764893:2][66764889:2][65209228:1][65209224:2][63634342:2][63634338:2][62113977:1][62113973:2][62113969:2][55759547:1][55759543:2][55759539:2][35055919:2][10855196:2][72897080:2][10855200:1][10641332:2][95317333:2][14460014:1][47597080:2][95317329:2][14460010:2][9515282:1][10641336:1][9654927:2][9515286:2][10216639:1][6939632:2][9236189:2][9795366:2][58949588:1][8613866:2][58949584:2][8613862:1][8486891:2][8359872:2][81315140:2][12926248:1][8232909:2][8232905:1][8232901:2][41094463:1][106665943:2][9375749:2][8359876:1][81315144:1][12926252:2][13953589:2][75819994:1][57355123:2][17977897:2][16995441:2][7929323:2][75164293:2][12419113:2][7801065:2][35055915:2][100246130:1][19801594:1][7929331:2][71375594:1][7801061:1][7929327:1][21521029:2][7674098:2][118000996:2][11398028:2][6939688:1]0 failed_tablets: _num_remaining_senders: 1
I20260319 09:11:34.959182 140178782262848 local_tablets_channel.cpp:673] LocalTabletsChannel txn_id: 390538358 load_id: 7b4281f2-e9a8-6fbb-dc65-cef37d6f76a5 sink_id: 0 commit 1 tablets: 105046484
I20260319 09:11:34.963467 140178773870144 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538358 load_id: 7b4281f2e9a86fbb-dc65cef37d6f76a5 index_id: 6939569 tablet_ids:108288707,103441305,101831326,98601543,116375953,93700479,92160433,111536980,90607309,85901490,84347377,82770663,79877710,114787921,78450025,77029025,75819986,69813743,68279400,66764889,63634338,62113969,55759539,72897080,47597080,95317329,9515286,9236189,9795366,8613866,8486891,8232909,106665943,12926252,13953589,57355123,12419113,7801065,35055915,7929331,21521029,7674098,118000996,11398028,14460018,100246122,7418051,109908589,44350750,9654935,10641340,89048964,7133449,30405665,6939666,10216643,10855204,52355224,6939590,10430768,60537845,27386447,6939692,6939640,6939627,87479891,15968359,6939653,9963843,31976664,6939614,8739982,33510931,6939705,71375586,7546787,6939731,45976083,8359880,6939757,113169312,9375757,8089628,7260738,6939744,11067033,6939718,11912053,28882534,75164289,81315136,14971707,25862428,20622217,58949580,15474279,13443417,119599974,16483751,21071972,16995449,17977905,8864465,8972128,18473170,18937226,121192162,19372175,19801598,96963415,20218237,21926234,6939602,22876921,6939679,24371781,65209220,50818935,6939578,9101019,36538808,38056569,39558013,74296168,41094455,7007710,17484175,49215448,54074889, reason:
I20260319 09:11:34.963624 140178689943104 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538358 load_id: 7b4281f2e9a86fbb-dc65cef37d6f76a5 index_id: 6939569 tablet_ids:119599978,118001000,114787925,113169316,108288711,101831330,96963419,116375957,93700483,111536984,90607313,85901494,84347381,82770667,89048968,79877714,78450029,77029029,75819990,69813747,68279404,66764893,65209224,63634342,62113973,55759543,35055919,10855196,10641332,95317333,14460010,9654927,6939632,58949584,8359872,81315140,8232901,9375749,17977897,16995441,7929323,75164293,100246126,19801590,6939684,9515278,47597084,10216635,6939671,7418043,6939658,72897084,19372167,52355228,6939594,60537849,27386451,92160437,7674090,11398020,8864457,103441309,38056573,9963835,6939645,45976087,9795358,9236181,8486883,22876925,6939606,6939619,7546779,6939570,6939697,9101011,33510935,6939710,7007702,17484167,6939749,49215452,71375590,7801057,11067025,6939736,10430760,106665947,121192166,57355127,7133441,30405669,20218229,98601547,7260730,11912045,12419105,36538812,12926244,13953581,14971699,15474271,8613858,16483743,21071964,8972120,18473162,18937218,8089620,20622209,21926226,21521021,87479895,15968351,31976668,24371785,25862432,13443409,28882538,6939582,39558017,74296172,41094459,109908593,44350754,8739974,50818939,6939723,54074893, reason:
I20260319 09:11:35.138556 140179255395904 local_tablets_channel.cpp:782] LocalTabletsChannel txn_id: 390538359 load_id: 384251dd-c5d0-6efa-7a3c-d3a18113028e sink_id: 0 open 540 delta writer: [119600459:2][119600455:1][119600451:2][118001466:2][118001462:1][118001458:2][116376443:2][116376439:1][116376435:2][114788394:2][111537450:1][109909050:2][106666422:2][103441775:2][103441771:1][101831954:2][101831950:1][101831946:2][100246644:2][100246640:1][100246636:2][96963873:2][93700989:2][92161265:2][90607765:2][87480356:2][87480352:1][87480348:2][85901949:2][81315594:2][78450484:1][78450480:2][92161261:1][77029491:2][92161257:2][77029487:1][77029483:2][75820449:2][74296659:1][121192627:2][72897557:2][69814251:2][66765351:2][65209687:2][65209683:1][65209679:2][62114433:2][62114429:1][74296663:2][62114425:2][57355587:1][57355583:2][54075390:2][54075386:1][54075382:2][50819767:2][79878169:1][49215912:2][81315602:2][79878165:2][49215908:1][81315598:1][49215904:2][47597564:2][47597560:1][45976560:2][45976556:1][41094902:1][41094898:2][39558495:2][36539264:2][44351289:2][35056368:2][44351285:1][35056364:1][44351281:2][35056360:2][30406134:2][27386906:1][27386902:2][75820445:1][25862882:2][75820441:2][69814259:2][25862878:1][69814255:1][25862874:2][24372236:2][22877370:2][21926663:2][21926659:1][21926655:2][21521463:2][21521459:1][21521455:2][39558491:1][21072521:2][39558487:2][21072517:1][21072513:2][20218666:2][60538310:2][20218662:1][60538306:1][20218658:2][89049505:2][19802046:2][19802042:1][19802038:2][58950050:2][18937690:2][58950046:1][18937686:1][58950042:2][18937682:2][18473760:2][18473756:1][18473752:2][75164752:2][47597556:2][38057021:2][17978347:2][89049513:2][75164748:1][38057017:1][17978343:1][89049509:1][75164744:2][38057013:2]0 failed_tablets: _num_remaining_senders: 1
I20260319 09:11:35.160106 140178782262848 local_tablets_channel.cpp:673] LocalTabletsChannel txn_id: 390538359 load_id: 384251dd-c5d0-6efa-7a3c-d3a18113028e sink_id: 0 commit 1 tablets: 105046970
I20260319 09:11:35.164082 140178670536256 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538359 load_id: 384251ddc5d06efa-7a3cd3a18113028e index_id: 17507 tablet_ids:119600459,118001466,116376443,114788394,106666422,103441775,101831954,100246644,93700989,92161265,87480356,77029491,75820449,121192627,72897557,65209687,62114433,74296663,54075390,50819767,49215912,81315602,47597564,45976560,39558495,36539264,44351289,35056368,25862882,69814259,24372236,22877370,21926663,21521463,21072521,20218666,60538310,19802046,58950050,18937690,18473760,75164752,38057021,17978347,89049513,16995866,16484167,15474695,14972142,14460461,79878173,13954005,52355833,13443867,95317802,12926748,11398460,11067475,55760022,10855620,96963881,10431184,10217059,9795769,9655364,4648841,109909058,4600820,5238176,4394964,9515689,11912574,4016138,63634845,4347737,28883020,9964250,6776716,31977118,4458071,68279868,4157843,19372591,111537454,4829556,41094906,9236592,4252468,3922014,4110601,8614269,7929734,3831995,71376056,3160719,3195469,7261209,30406142,3264887,27386910,7674501,3510190,3087414,8360283,20622686,3580645,4702138,12419549,4205005,3230123,3299605,8972531,108289182,8090031,85901957,6892076,4762716,3052839,66765359,3125911,4552918,3404711,3787387,5100914,3369362,10641756,3876551,4300529,3968809,3545206,84347875,3440243,17484591,3334195,82771162,3475169,5033617,113169781,3742793,4899038,4966093,57355591,5168575,3616178,5378249,3698144,5446241,5582451,5514021,8233335,98602011,15969130,5650637,5740443,5960034,4063400,4505405,6075387,3651567,6192644,6430369,5844485,6314614,8487294,6545645,6661454,5310057,7008113,33511381,7133852,7418456,7547190,90607773,8740411,7801780,78450488,8864884,9101422,9376160, reason:
I20260319 09:11:35.169134 140179314144832 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538359 load_id: 384251ddc5d06efa-7a3cd3a18113028e index_id: 17507 tablet_ids:119600451,118001458,116376435,109909050,101831946,100246636,96963873,90607765,87480348,85901949,81315594,78450480,92161257,77029483,69814251,66765351,65209679,62114425,57355583,54075382,79878165,49215904,41094898,44351281,35056360,30406134,27386902,75820441,25862874,21926655,21521455,39558487,21072513,20218658,89049505,19802038,58950042,18937682,18473752,47597556,75164744,38057013,17978339,16995858,16484159,121192619,72897549,14972134,14460453,13953997,111537446,11398452,11067467,19372583,10855612,10217051,28883012,9964242,9795761,4762708,6661446,5378241,4600812,6430361,82771154,3475161,71376048,9515681,63634837,4347729,4300521,31977110,4458063,4110593,8614261,36539256,3545198,3968801,3125903,45976552,5446233,22877362,4829548,9236584,4252460,7418448,3922006,10641748,3876543,3160711,114788386,3831987,50819759,8972523,3299597,3787379,3087406,8360275,15474687,7674493,20622678,4702130,3580637,3404703,3369354,33511373,17484583,3334187,52355825,13443859,9101414,3264879,4899030,3052831,11912566,4016130,7929726,84347867,3440235,103441767,3698136,113169773,3742785,3195461,3616170,7261201,4552910,12419541,3230115,4204997,6192636,3651559,4648833,93700981,3510182,24372228,10431176,4063392,5582443,4966085,68279860,4157835,5514013,5033609,5168567,60538302,5238168,5310049,4394956,106666414,98602003,15969122,5650629,5740435,5960026,4505397,6075379,5100906,74296655,7008105,7547182,95317794,12926740,6776708,6892068,7133844,55760014,7801772,108289174,8090023,8233327,6545637,8487286,8740403,8864876,5844477,6314606,9376152,9655356, reason:
I20260319 09:11:35.228967 140178282378816 stream_load.cpp:253] new income streaming load request.id=354a60cb4d943954-f299c1dfcf3b22b9, job_id=-1, txn_id: -1, label=3203a85c-5e1a-474e-a475-a454e7f2af3b, db=pstage_band, db=pstage_band, tbl=ods_sfc_req_scrap_bad_d
I20260319 09:11:35.231124 140178282378816 stream_load_executor.cpp:77] begin to execute job. label=3203a85c-5e1a-474e-a475-a454e7f2af3b, txn_id: 390538360, query_id=354a60cb-4d94-3954-f299-c1dfcf3b22b9
I20260319 09:11:35.233478 140179305752128 local_tablets_channel.cpp:782] LocalTabletsChannel txn_id: 390538360 load_id: 354a60cb-4d94-3954-f299-c1dfcf3b22b9 sink_id: 0 open 42 delta writer: [92162764:1][92162760:2][50821326:2][15970574:2][15970570:1][7803082:2][4830780:2][202545:2][1224360:2][92162756:2][24226:2][65926:2][24213:2][24200:2][15970566:2][7803078:1][4830776:1][50821334:1][24222:1][7803074:2][4830772:2][24209:1][65922:1][50821330:2][24218:2][65918:2][24205:2][24192:2][202549:1][1224364:1][24196:1][202553:2][1224368:2][2291431:2][424514:2][3405843:2][2291435:1][424518:1][3405847:1][2291439:2][424522:2][3405851:2]0 failed_tablets: _num_remaining_senders: 1
I20260319 09:11:35.235704 140178765477440 local_tablets_channel.cpp:673] LocalTabletsChannel txn_id: 390538360 load_id: 354a60cb-4d94-3954-f299-c1dfcf3b22b9 sink_id: 0 commit 1 tablets: 15970570
I20260319 09:11:35.237031 140178815833664 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538360 load_id: 354a60cb4d943954-f299c1dfcf3b22b9 index_id: 24191 tablet_ids:50821326,7803082,4830780,92162756,24226,65926,24213,24200,202553,1224368,2291439,424522,3405851, reason:
I20260319 09:11:35.237075 140178799048256 local_tablets_channel.cpp:826] cancel LocalTabletsChannel txn_id: 390538360 load_id: 354a60cb4d943954-f299c1dfcf3b22b9 index_id: 24191 tablet_ids:92162760,202545,1224360,7803074,4830772,50821330,24218,65918,24205,24192,2291431,424514,3405843, reason:
I20260319 09:11:35.354043 140178232022592 stream_load.cpp:253] new income streaming load request.id=ea46f870c56699f9-82e3d0bb80520987, job_id=-1, txn_id: -1, label=5fe66ce5-4256-4988-84e6-9d75ac29fcd6, db=pstage_band, db=pstage_band, tbl=ods_sfc_scan_bind
I20260319 09:11:35.363549 140178232022592 stream_load_executor.cpp:77] begin to execute job. label=5fe66ce5-4256-4988-84e6-9d75ac29fcd6, txn_id: 390538361, query_id=ea46f870-c566-99f9-82e3-d0bb80520987
I20260319 09:11:35.467409 140175001609792 size_tiered_compaction_policy.cpp:353] pick tablet 105047168 for size-tiered compaction rowset version=3472328296227680304-3472328296227680304 score=8.08464e+08 level_size=1310720000000 total_size=3472328296227680304 segment_num=808464432 force_base_compaction=0 reached_max_versions=0
I20260319 09:11:35.467491 140174950229568 size_tiered_compaction_policy.cpp:353] pick tablet 105047172 for size-tiered compaction rowset version=695-706 score=16.4619 level_size=19432 total_size=53068 segment_num=5 force_base_compaction=0 reached_max_versions=0
I20260319 09:11:35.467655 140179129505344 compaction_manager.cpp:93] submit task to compaction pool, task_id:2312315, tablet_id:105047168, compaction_type:cumulative, compaction_score:8.08464e+08 for round:4335818, candidates_size:174
I20260319 09:11:35.467664 140175168411200 size_tiered_compaction_policy.cpp:353] pick tablet 105047381 for size-tiered compaction rowset version=824-872 score=16.431 level_size=70070 total_size=120204 segment_num=7 force_base_compaction=0 reached_max_versions=0
I20260319 09:11:35.467744 140179129505344 compaction_manager.cpp:93] submit task to compaction pool, task_id:2312316, tablet_id:105048222, compaction_type:cumulative, compaction_score:41.7934 for round:4335819, candidates_size:173
W20260319 09:11:35.467781 140179146290752 data_dir.cpp:779] reach capacity limit. used pct: 647030, left bytes: -3472327136091410480, path: /data/StarRocks/be/storage
W20260319 09:11:35.467836 140179146290752 compaction_task.cpp:116] compaction task:2312315 failed, skip tablet:105047168 because data dir reaches capacity limit. input rowsets size:3472328296227680304
W20260319 09:11:35.467920 140179146290752 data_dir.cpp:779] reach capacity limit. used pct: 647030, left bytes: -3472327136091794353, path: /data/StarRocks/be/storage
W20260319 09:11:35.467926 140179146290752 compaction_task.cpp:116] compaction task:2312316 failed, skip tablet:105048222 because data dir reaches capacity limit. input rowsets size:3472328296228064177
I20260319 09:11:35.468342 140179129505344 compaction_manager.cpp:93] submit task to compaction pool, task_id:2312317, tablet_id:105046692, compaction_type:cumulative, compaction_score:36.8253 for round:4335820, candidates_size:172
I20260319 09:11:35.468386 140179129505344 compaction_manager.cpp:93] submit task to compaction pool, task_id:2312318, tablet_id:105047535, compaction_type:cumulative, compaction_score:36.7132 for round:4335821, candidates_size:171
I20260319 09:11:35.468457 140179129505344 compaction_manager.cpp:93] submit task to compaction pool, task_id:2312319, tablet_id:105048034, compaction_type:base, compaction_score:15.0497 for round:4335822, candidates_size:170
I20260319 09:11:47.935403 140125297354560 daemon.cpp:335] version 3.3.20-bc0523a

你在fe audit里面根据"929a3ce2-2330-11f1-aa1b-005056abf325"这个queryid把语句拿下来,看下这是什么语句,有点像类型转换触发了bug