256g内存的时候，查询报错内容也发一下
256gb 和这个差不多SQLSyntaxErrorException: Memory of process exceed limit. Start execute plan fragment. Used: 227634391912, Limit: 227634303466. Mem usage has exceed the limit of BE backen 但是实际使用才130gb 每次这个时候就报错 太浪费了吧 报错的时候只有小查询能查询
是哇 但是 128gb的系统总内存才70gb使用 就报错 256gb 130就报错 这能避免不 主要报错时候会影响导入和稍大的查询 觉得这个不太合理 是因为分离架构 很多表开启了cache么
存算分离的cache 是缓存在磁盘上的
在报错的这个 be.INFO 中搜一下报错前后的 内存统计信息，Current memory statistics
I1218 16:34:50.746234 9848 daemon.cpp:211] Current memory statistics: process(105685322816), query_pool(0), load(0), metadata(437552748), compaction(1010120016), schema_change(0), column_pool(65496358273), page_cache(339230848), update(27580040573), chunk_allocator(157780136), clone(0), consistency(0)
I1218 16:35:05.750115 9848 daemon.cpp:211] Current memory statistics: process(111992735560), query_pool(0), load(0), metadata(440315868), compaction(538701352), schema_change(0), column_pool(71239417690), page_cache(344498672), update(27915786173), chunk_allocator(157780136), clone(0), consistency(0)
I1218 16:35:20.754274 9848 daemon.cpp:211] Current memory statistics: process(113704838904), query_pool(-47408704), load(0), metadata(442195867), compaction(1079213504), schema_change(0), column_pool(72221713582), page_cache(340346928), update(27915786173), chunk_allocator(157694120), clone(0), consistency(0)
I1218 16:35:35.761488 9848 daemon.cpp:211] Current memory statistics: process(112521752008), query_pool(1307016), load(0), metadata(444732188), compaction(1859840144), schema_change(0), column_pool(70517661507), page_cache(328100336), update(27932594901), chunk_allocator(157771944), clone(0), consistency(0)
W1218 16:35:50.757510 10122 pipeline_driver_executor.cpp:161] [Driver] Process error, query_id=730c3afd-9d80-11ee-b55a-1a5e7986dccb, instance_id=730c3afd-9d80-11ee-b55a-1a5e7986dccc, status=Memory limit exceeded: Memory of process exceed limit. Pipeline Backend: 172.31.143.160, fragment: 730c3afd-9d80-11ee-b55a-1a5e7986dccc Used: 117493358375, Limit: 117363112279. Mem usage has exceed the limit of BE
I1218 16:35:50.765421 9848 daemon.cpp:211] Current memory statistics: process(117392726264), query_pool(1420032), load(0), metadata(446900077), compaction(3322393272), schema_change(0), column_pool(70596151497), page_cache(344030720), update(27647350717), chunk_allocator(157780136), clone(0), consistency(0)
I1218 16:36:05.769121 9848 daemon.cpp:211] Current memory statistics: process(109173265984), query_pool(2126512), load(11984), metadata(447996526), compaction(2524388992), schema_change(0), column_pool(65461688539), page_cache(334846832), update(27412107613), chunk_allocator(157042856), clone(0), consistency(0)
是的 表全是这样建的 填的oss的桶
“datacache.enable” = “true”,
“datacache.partition_duration” = “1 MONTH”,
“enable_async_write_back” = “false”,
“replication_num” = “1”
在这个时间点附近报 查询超限了么，现在机器已经是 256G 内存了么
主要应该是导入数据估计消耗点么
top看 每个节点的内存统计都是79G左右的样子么
curl -XGET -s http://be_ip:8040/metrics | grep “^starrocks_be_.*_mem_bytes|^starrocks_be_tcmalloc_bytes_in_use”
发一下两个cn的这个返回结果截图
这样就对了
curl -X GET -s http://172.31.143.161:8041/metrics | grep -E “starrocks_be_.*_mem_bytes|starrocks_be_tcmalloc_bytes_in_use”
TYPE starrocks_be_bitmap_index_mem_bytes gauge
starrocks_be_bitmap_index_mem_bytes 0
TYPE starrocks_be_bloom_filter_index_mem_bytes gauge
starrocks_be_bloom_filter_index_mem_bytes 0
TYPE starrocks_be_chunk_allocator_mem_bytes gauge
starrocks_be_chunk_allocator_mem_bytes 204283016
TYPE starrocks_be_clone_mem_bytes gauge
starrocks_be_clone_mem_bytes 0
TYPE starrocks_be_column_metadata_mem_bytes gauge
starrocks_be_column_metadata_mem_bytes 1045454667
TYPE starrocks_be_column_pool_mem_bytes gauge
starrocks_be_column_pool_mem_bytes 77461658913
TYPE starrocks_be_column_zonemap_index_mem_bytes gauge
starrocks_be_column_zonemap_index_mem_bytes 186247915
TYPE starrocks_be_compaction_mem_bytes gauge
starrocks_be_compaction_mem_bytes 990414504
TYPE starrocks_be_consistency_mem_bytes gauge
starrocks_be_consistency_mem_bytes 0
TYPE starrocks_be_load_mem_bytes gauge
starrocks_be_load_mem_bytes 0
TYPE starrocks_be_metadata_mem_bytes gauge
starrocks_be_metadata_mem_bytes 1163738761
TYPE starrocks_be_ordinal_index_mem_bytes gauge
starrocks_be_ordinal_index_mem_bytes 663445952
TYPE starrocks_be_process_mem_bytes gauge
starrocks_be_process_mem_bytes 110788966320
TYPE starrocks_be_query_mem_bytes gauge
starrocks_be_query_mem_bytes 243429477
TYPE starrocks_be_rowset_metadata_mem_bytes gauge
starrocks_be_rowset_metadata_mem_bytes 0
TYPE starrocks_be_schema_change_mem_bytes gauge
starrocks_be_schema_change_mem_bytes 0
TYPE starrocks_be_segment_metadata_mem_bytes gauge
starrocks_be_segment_metadata_mem_bytes 117788093
TYPE starrocks_be_segment_zonemap_mem_bytes gauge
starrocks_be_segment_zonemap_mem_bytes 96053244
TYPE starrocks_be_short_key_index_mem_bytes gauge
starrocks_be_short_key_index_mem_bytes 1179267
TYPE starrocks_be_storage_page_cache_mem_bytes gauge
starrocks_be_storage_page_cache_mem_bytes 337246752
TYPE starrocks_be_tablet_metadata_mem_bytes gauge
starrocks_be_tablet_metadata_mem_bytes 496001
TYPE starrocks_be_tablet_schema_mem_bytes gauge
starrocks_be_tablet_schema_mem_bytes 496001
TYPE starrocks_be_update_mem_bytes gauge
starrocks_be_update_mem_bytes 19996417541
改过http_port默认值么，截图看写的 8041，http_port 默认8040端口
curl -X GET -s http://172.31.143.160:8041/metrics | grep -E “starrocks_be_.*_mem_bytes|starrocks_be_tcmal
loc_bytes_in_use”
TYPE starrocks_be_bitmap_index_mem_bytes gauge
starrocks_be_bitmap_index_mem_bytes 0
TYPE starrocks_be_bloom_filter_index_mem_bytes gauge
starrocks_be_bloom_filter_index_mem_bytes 0
TYPE starrocks_be_chunk_allocator_mem_bytes gauge
starrocks_be_chunk_allocator_mem_bytes 59394936
TYPE starrocks_be_clone_mem_bytes gauge
starrocks_be_clone_mem_bytes 0
TYPE starrocks_be_column_metadata_mem_bytes gauge
starrocks_be_column_metadata_mem_bytes 300621129
TYPE starrocks_be_column_pool_mem_bytes gauge
starrocks_be_column_pool_mem_bytes 65286531755
TYPE starrocks_be_column_zonemap_index_mem_bytes gauge
starrocks_be_column_zonemap_index_mem_bytes 67046985
TYPE starrocks_be_compaction_mem_bytes gauge
starrocks_be_compaction_mem_bytes 2090386920
TYPE starrocks_be_consistency_mem_bytes gauge
starrocks_be_consistency_mem_bytes 0
TYPE starrocks_be_load_mem_bytes gauge
starrocks_be_load_mem_bytes 0
TYPE starrocks_be_metadata_mem_bytes gauge
starrocks_be_metadata_mem_bytes 343648327
TYPE starrocks_be_ordinal_index_mem_bytes gauge
starrocks_be_ordinal_index_mem_bytes 164355168
TYPE starrocks_be_process_mem_bytes gauge
starrocks_be_process_mem_bytes 110089923672
TYPE starrocks_be_query_mem_bytes gauge
starrocks_be_query_mem_bytes 0
TYPE starrocks_be_rowset_metadata_mem_bytes gauge
starrocks_be_rowset_metadata_mem_bytes 0
TYPE starrocks_be_schema_change_mem_bytes gauge
starrocks_be_schema_change_mem_bytes 0
TYPE starrocks_be_segment_metadata_mem_bytes gauge
starrocks_be_segment_metadata_mem_bytes 42636475
TYPE starrocks_be_segment_zonemap_mem_bytes gauge
starrocks_be_segment_zonemap_mem_bytes 34998184
TYPE starrocks_be_short_key_index_mem_bytes gauge
starrocks_be_short_key_index_mem_bytes 322927
TYPE starrocks_be_storage_page_cache_mem_bytes gauge
starrocks_be_storage_page_cache_mem_bytes 339547888
TYPE starrocks_be_tablet_metadata_mem_bytes gauge
starrocks_be_tablet_metadata_mem_bytes 390723
TYPE starrocks_be_tablet_schema_mem_bytes gauge
starrocks_be_tablet_schema_mem_bytes 390723
TYPE starrocks_be_update_mem_bytes gauge
starrocks_be_update_mem_bytes 31635176848
be.conf 中修改一下 disable_column_pool=1 ，修改后需要重启。column pool 这个没啥用还占用了较多的内存，后面会默认禁掉
感谢大佬的回复 现在是1比1的内存了 我再观察观察看看待会会慢了还有可用内存不 