为了更快的定位您的问题,请提供以下信息,谢谢
【详述】测试insert性能时Be崩溃
flink导入数据到starrocks,测试insert性能,使用web访问/api/compaction/show?tablet_id=tablet_id页面,随着导入线程并行度的增加和sink.buffer-flush.interval-ms参数的减少,Be会崩溃。该问题为必现。
【背景】做过哪些操作?
- flink导入数据到starrocks,测试insert性能。Be崩溃时,并行度为40,每个线程数据量:10000/s,sink.buffer-flush.interval-ms设置为5000(ms).
- 使用web访问/api/compaction/show?tablet_id=tablet_id页面, 隔几秒手动刷新查看
- 几分钟后Be崩溃
Be be.out:
start time: Mon Nov 11 09:57:38 CST 2024, server uptime: 09:57:38 up 17 days, 17:24, 6 users, load average: 0.52, 0.97, 0.70
3.3.5 RELEASE (build 6d81f75)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000
tracker:process consumption: 859107200
tracker:jemalloc_metadata consumption: 210060720
tracker:jemalloc_fragmentation consumption: 0
tracker:query_pool consumption: 243405480
tracker:query_pool/connector_scan consumption: 0
tracker:load consumption: 54016
tracker:metadata consumption: 4108202
tracker:tablet_metadata consumption: 215169
tracker:rowset_metadata consumption: 2830038
tracker:segment_metadata consumption: 128947
tracker:column_metadata consumption: 934048
tracker:tablet_schema consumption: 26385
tracker:segment_zonemap consumption: 103596
tracker:short_key_index consumption: 0
tracker:column_zonemap_index consumption: 173152
tracker:ordinal_index consumption: 445376
tracker:bitmap_index consumption: 0
tracker:bloom_filter_index consumption: 0
tracker:compaction consumption: 139173400
tracker:schema_change consumption: 0
tracker:column_pool consumption: 0
tracker:page_cache consumption: 20288544
tracker:jit_cache consumption: 0
tracker:update consumption: 66495
tracker:chunk_allocator consumption: 0
tracker:clone consumption: 0
tracker:consistency consumption: 0
tracker:datacache consumption: 0
tracker:replication consumption: 0
*** Aborted at 1731290430 (unix time) try "date -d @1731290430" if you are using GNU date ***
PC: @ 0x3e5ce27 bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::PrettyWriter<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::x▒▒
*** SIGSEGV (@0x312d3933343f) received by PID 46399 (TID 0x7f3ece9f1700) from PID 959657023; stack trace: ***
@ 0x7f413ac361cb __pthread_once_slow
@ 0x7aff560 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
@ 0x7f413ac3f5f0 (/usr/lib64/libpthread-2.17.so+0xf5ef)
@ 0x3e5ce27 bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::PrettyWriter<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::x▒▒
@ 0x3e5cee2 bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::PrettyWriter<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::x▒▒
@ 0x3e5cee2 bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::PrettyWriter<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::x▒▒
@ 0x3e5d156 bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::PrettyWriter<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::x▒▒
@ 0x3e5d156 bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::PrettyWriter<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::x▒▒
@ 0x3e5d156 bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::PrettyWriter<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::x▒▒
@ 0x600f7d1 starrocks::Tablet::get_compaction_status(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
@ 0x3e560bd starrocks::CompactionAction::_handle_show_compaction(starrocks::HttpRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
@ 0x3e5b1c8 starrocks::CompactionAction::handle(starrocks::HttpRequest*)
@ 0x7b91547 evhttp_handle_request
@ 0x7b921f3 bufferevent_readcb
@ 0x7b7e932 event_process_active_single_queue
@ 0x7b7f06f event_base_loop
@ 0x3e25ce4 starrocks::EvHttpServer::start()::{lambda()#1}::operator()() const
@ 0xc257790 execute_native_thread_routine
@ 0x7f413ac37e65 start_thread
@ 0x7f413a03988d __clone
Be崩溃前,Be使用内存并不多(2g),机器还有很多空闲内存:
【业务影响】
【是否存算分离】否
【StarRocks版本】例如:3.3.5
【集群规模】例如:1fe+1be(fe与be混部,单机环境)
【机器信息】CPU虚拟核/内存/网卡,例如:56C/160G/万兆
【联系方式】
18638489474@163.com
【附件】
be_log.zip (521.9 KB) fe_log.zip (1.2 MB)