为了更快的定位您的问题,请提供以下信息,谢谢
【详述】问题详细描述
【背景】做过哪些操作?
【业务影响】
【是否存算分离】
【StarRocks版本】3.2.13
【集群规模】例如:3fe(1 follower+2observer)+2be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:32C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息,请补充下您的联系方式,例如:社区群4-小李或者邮箱,谢谢
【附件】
通过系统命令查看当前starRocks_be进程RSS很高。
cat /proc/533515/status | grep -E ‘VmRSS|VmSize|VmPeak|VmHWM’
VmPeak: 48250624 kB
VmSize: 48250504 kB
VmHWM: 38418624 kB
VmRSS: 38273496 kB
fe.log日志看到大量的打印日志:
I0105 07:29:56.648064 533792 daemon.cpp:186] Current memory statistics: process(29826196532), query_pool(0), load(0), metadata(1316019048), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), passthrough(0), clone(0), consistency(0), datacache(0)
I0105 07:30:11.650743 533792 daemon.cpp:186] Current memory statistics: process(29826051118), query_pool(0), load(0), metadata(1316019048), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), passthrough(0), clone(0), consistency(0), datacache(0)
I0105 07:30:26.653534 533792 daemon.cpp:186] Current memory statistics: process(29832543432), query_pool(0), load(0), metadata(1316019048), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), passthrough(0), clone(0), consistency(0), datacache(0)
I0105 07:30:33.905939 534246 load_channel_mgr.cpp:257] Memory consumption(bytes) limit=15255646064 current=0 peak=24352
I0105 07:30:41.658394 533792 daemon.cpp:186] Current memory statistics: process(29825964772), query_pool(0), load(0), metadata(1316019048), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), passthrough(0), clone(0), consistency(0), datacache(0)
I0105 07:30:56.661129 533792 daemon.cpp:186] Current memory statistics: process(29834570844), query_pool(0), load(0), metadata(1316019048), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), passthrough(0), clone(0), consistency(0), datacache(0)
I0105 07:31:11.667626 533792 daemon.cpp:186] Current memory statistics: process(29829494320), query_pool(0), load(0), metadata(1316019048), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), passthrough(0), clone(0), consistency(0), datacache(0)
I0105 07:31:26.681152 533792 daemon.cpp:186] Current memory statistics: process(29830254408), query_pool(0), load(0), metadata(1316019048), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), passthrough(0), clone(0), consistency(0), datacache(0)
I0105 07:31:33.906956 534247 load_channel_mgr.cpp:257] Memory consumption(bytes) limit=15255646064 current=0 peak=24352
I0105 07:31:41.689064 533792 daemon.cpp:186] Current memory statistics: process(29828025936), query_pool(0), load(0), metadata(1316061663), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), passthrough(0), clone(0), consistency(0), datacache(0)
process使用很高,但是其余具体的子项目几乎都因为0,通过页面查看 mem_tracker,也是发现:
levelcc Labelar Parent Limits Current Consumption Peak Consumption
← process 47G 32G 32G
jemalloc metadata process nones 414M 414M
N jemalloc fragmentation process none 50M 212M
N query pool process 42G 。 427K
N load process 14G 。 23K
~ compaction process 47G 。 。
N schema change process nones 。 。
N column pool process none 。 。
N page cache process none 。 。
No chunk allocator process none 。 。
尝试通过系统命令分析进程的smaps内存信息发现如下:
发现如下内存段比较大:
7fe63f400000-7fcb9ec00000 rw-p 00000000 00:00 0
Size: 22536192 kB
KernelPagesize: 4 kB
MMUPagesize: 4 kB
RSS: 22105448 kB
PSS: 22105448 kB
shared_Clean:a 0 kB
shared_Dirty:s 0 kB
Private_Clean: 124 ke
Private_Dirty: 22105324 kB
Referenced: 19287468 kB
Anonymous: 22105448 kB
LazyFree: 124 kB
AnonHugePages: 19054592 kB
shmemPmdMapped: 0 kB
Private_Hugetlb:
swap: 0 kB
Shared_Huget1b: 0 kB
Swappss: 0kB
Locked: 0 Kb
并且这段smaps多次观察发现这4个值是在变动的
Private_Dirty: 22105324 kB
Referenced: 19287468 kB
Anonymous: 22105448 kB
AnonHugePages: 19054592 kB
想请问下,为啥会有这么大段的RSS占用,或者是我理解有问题,还有就是这类问题是否有通用的分析思路?
感谢