[问题排查] 3.3.2版本be节点出现了内存泄露, 怎么排查?

💬 StarRocks 用户问答 功能使用相关
#1

为了更快的定位您的问题，请提供以下信息，谢谢
【详述】问题详细描述
process显示消耗29G内存, 但是query_pool显示-2G内存消耗.
很多sql出现了内存溢出的情况, 怎么判断哪里产生了内存泄漏呢

img_v3_02e5_a0f75a6a-f3f2-4694-84ab-1d88a49a193g
img_v3_02e5_a0f75a6a-f3f2-4694-84ab-1d88a49a193g2986×1482 336 KB

【背景】做过哪些操作？
【业务影响】
任务大量失败
【是否存算分离】否
【StarRocks版本】例如：3.3.2
【集群规模】例如：3fe（3 follower）+3be（fe与be混部）
【机器信息】CPU虚拟核/内存/网卡，例如：24C/96G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息，请补充下您的联系方式，例如：社区群16-可乐鸡或者邮箱，谢谢
【附件】

  • fe.log/beINFO/相应截图
  • 慢查询：
    • Profile信息
    • 并行度：show variables like ‘%parallel_fragment_exec_instance_num%’;
    • pipeline是否开启：show variables like ‘%pipeline%’;
    • be节点cpu和内存使用率截图
  • 查询报错：
  • be crash
    • be.out
  • 外表查询报错
    • be.out和fe.warn.log
#2

参考 https://github.com/StarRocks/starrocks/pull/35322 拿下heapprof

#3

大佬 帮忙看下


digraph "/opt/starrocks/be/lib/starrocks_be; 11.6 MB" {                                                                                                                                          
 node [width=0.375,height=0.25];                                                                                                                                                                  
 Legend [shape=box,fontsize=24,shape=plaintext,label="/opt/starrocks/be/lib/starrocks_be\lTotal MB: 11.6\lFocusing on: 11.6\lDropped nodes with <= 0.1 abs(MB)\lDropped edges with <= 0.0 MB\l"]; 
 N1 [label="__clone@@GLIBC_2.2.5\n0.0 (0.0%)\rof 11.1 (95.7%)\r",shape=box,fontsize=8.0];                                                                                                         
 N2 [label="pthread_condattr_setpshared@GLIBC_2.2.5\n0.0 (0.0%)\rof 11.1 (95.7%)\r",shape=box,fontsize=8.0];                                                                                      
 N3 [label="execute_native_thread_routine\n0.0 (0.0%)\rof 9.0 (77.3%)\r",shape=box,fontsize=8.0];                                                                                                 
 N4 [label="starrocks\nReportOlapTableTaskWorkerPool\n_worker_thread_callback\n0.0 (0.0%)\rof 9.0 (77.3%)\r",shape=box,fontsize=8.0];                                                             
 N5 [label="starrocks\nTabletManager\nreport_all_tablets_info\n0.0 (0.0%)\rof 9.0 (77.3%)\r",shape=box,fontsize=8.0];                                                                             
 N6 [label="std\n_Rb_tree\n_M_emplace_unique\n4.0 (34.4%)\rof 9.0 (77.3%)\r",shape=box,fontsize=37.3];                                                                                            
 N7 [label="starrocks\nTTablet\nTTablet\n0.0 (0.0%)\rof 5.0 (43.0%)\r",shape=box,fontsize=8.0];                                                                                                   
 N8 [label="std\nvector\noperator=\n[clone\n.isra.0]\n5.0 (43.0%)\r",shape=box,fontsize=40.8];                                                                                                    
 N9 [label="LZ4F_compressBegin_usingCDict\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                                 
 N10 [label="my_calloc\n2.1 (18.4%)\r",shape=box,fontsize=29.4];                                                                                                                                  
 N11 [label="starrocks\nFlushToken\n_flush_memtable\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                       
 N12 [label="starrocks\nHorizontalRowsetWriter\n_flush_chunk\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                              
 N13 [label="starrocks\nHorizontalRowsetWriter\n_flush_segment_writer\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                     
 N14 [label="starrocks\nHorizontalRowsetWriter\nflush_chunk\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                               
 N15 [label="starrocks\nLz4fBlockCompression\n_compress\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                   
 N16 [label="starrocks\nLz4fBlockCompression\ncompress\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                    
 N17 [label="starrocks\nMemTable\nflush\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                                   
 N18 [label="starrocks\nMemTableRowsetWriterSink\nflush_chunk\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                             
 N19 [label="starrocks\nMemtableFlushTask\nrun\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                            
 N20 [label="starrocks\nPageIO\ncompress_page_body\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                        
 N21 [label="starrocks\nSegmentWriter\nfinalize\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                           
 N22 [label="starrocks\nSegmentWriter\nfinalize_columns\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                   
 N23 [label="starrocks\nThread\nsupervise_thread\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                          
 N24 [label="starrocks\nThreadPool\ndispatch_thread\n0.0 (0.0%)\rof 2.1 (18.4%)\r",shape=box,fontsize=8.0];                                                                                       
 N25 [label="starrocks\nScalarColumnWriter\nfinish\n0.0 (0.0%)\rof 1.4 (12.2%)\r",shape=box,fontsize=8.0];                                                                                        
 N26 [label="starrocks\nScalarColumnWriter\nfinish_current_page\n0.0 (0.0%)\rof 1.4 (12.2%)\r",shape=box,fontsize=8.0];                                                                           
 N27 [label="starrocks\nScalarColumnWriter\nwrite_data\n0.0 (0.0%)\rof 0.7 (6.1%)\r",shape=box,fontsize=8.0];                                                                                     
 N28 [label="starrocks\nStringColumnWriter\nwrite_data\n0.0 (0.0%)\rof 0.7 (6.1%)\r",shape=box,fontsize=8.0];                                                                                     
 N29 [label="addConstant\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                                   
 N30 [label="brpc\nInputMessenger\nOnNewMessages\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                           
 N31 [label="brpc\nProcessInputMessage\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                     
 N32 [label="brpc\nSocket\nProcessEvent\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                    
 N33 [label="brpc\npolicy\nProcessRpcRequest\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                               
 N34 [label="bthread\nTaskGroup\ntask_runner\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                               
 N35 [label="bthread_make_fcontext\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                         
 N36 [label="classDefinition\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                               
 N37 [label="compileInModule\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                               
 N38 [label="method\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                                        
 N39 [label="realloc\n0.5 (4.3%)\r",shape=box,fontsize=18.4];                                                                                                                                     
 N40 [label="resizeMap\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                                     
 N41 [label="runInterpreter\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                                
 N42 [label="starrocks\nPInternalServiceImplBase\nexecute_command\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                          
 N43 [label="starrocks\nexecute_command\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                    
 N44 [label="starrocks\nexecute_script\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                     
 N45 [label="wrenCompile\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                                   
 N46 [label="wrenMapSet\n0.0 (0.0%)\rof 0.5 (4.3%)\r",shape=box,fontsize=8.0];                                                                                                                    
 N1 -> N2 [label=11.1, weight=88551, style="setlinewidth(2.000000)"];                                                                                                                             
 N5 -> N6 [label=9.0, weight=76283, style="setlinewidth(2.000000)"];                                                                                                                              
 N2 -> N3 [label=9.0, weight=76283, style="setlinewidth(2.000000)"];                                                                                                                              
 N4 -> N5 [label=9.0, weight=76283, style="setlinewidth(2.000000)"];                                                                                                                              
 N3 -> N4 [label=9.0, weight=76283, style="setlinewidth(2.000000)"];                                                                                                                              
 N7 -> N8 [label=5.0, weight=50553, style="setlinewidth(2.000000)"];                                                                                                                              
 N6 -> N7 [label=5.0, weight=50553, style="setlinewidth(2.000000)"];                                                                                                                              
 N11 -> N17 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N15 -> N9 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                             
 N17 -> N18 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N23 -> N24 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N16 -> N15 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N18 -> N14 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N14 -> N12 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N12 -> N13 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N9 -> N10 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                             
 N21 -> N22 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N2 -> N23 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                             
 N19 -> N11 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N20 -> N16 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N13 -> N21 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N24 -> N19 [label=2.1, weight=27881, style="setlinewidth(1.101440)"];                                                                                                                            
 N26 -> N20 [label=1.4, weight=20991, style="setlinewidth(0.734293)"];                                                                                                                            
 N22 -> N25 [label=1.4, weight=20991, style="setlinewidth(0.734293)"];                                                                                                                            
 N25 -> N26 [label=1.4, weight=20991, style="setlinewidth(0.734293)"];                                                                                                                            
 N27 -> N20 [label=0.7, weight=12922, style="setlinewidth(0.367147)"];                                                                                                                            
 N22 -> N28 [label=0.7, weight=12922, style="setlinewidth(0.367147)"];                                                                                                                            
 N28 -> N27 [label=0.7, weight=12922, style="setlinewidth(0.367147)"];                                                                                                                            
 N46 -> N40 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N40 -> N39 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N30 -> N31 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N45 -> N36 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N36 -> N38 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N29 -> N46 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N32 -> N30 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N34 -> N32 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N37 -> N45 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N38 -> N29 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N42 -> N43 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N35 -> N34 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N33 -> N42 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N44 -> N41 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N43 -> N44 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N41 -> N37 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 N31 -> N33 [label=0.5, weight=10140, style="setlinewidth(0.259698)"];                                                                                                                            
 }
#4

当前内存占用不高？

#5

是在发生了问题之后 我才dump的 可能没有包含那段时间的内存 我重启一下 先开启 然后等出问题我再dump

京ICP备2022026421号-1