【详述】
存算分离版本使用datax持续导入主键表，表规模在百亿级，发现CN节点内存一直上升，直到报错Memory of process exceed limit
查看指标，只有starrocks_be_process_mem_bytes在异常上升，其他几个内存的指标正常
【背景】
3.2.6 没有出现该问题
【是否存算分离】
是
【StarRocks版本】
3.3.0
【集群规模】
3fe+3cn
【机器信息】
36C/96G
【附件】
fe.warn.log
W0701 08:19:11.009248 393 tablet_sink_sender.cpp:120] NodeChannel[1286019], tablet add chunk failed, load_id=e8452b7d-7fae-1105-6dd9-5d54b2380683, txn_id: 1116491, parallel=1, compress_type=2, node=192.168.97.155:8060, errmsg=Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77385966456, Limit: 77760000000. Mem usage has exceed the limit of BE
W0701 08:19:11.009287 393 plan_fragment_executor.cpp:207] Fail to open fragment, instance_id=e8452b7d-7fae-1105-6dd9-5d54b2380684, status=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77385966456, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:19:11.011020 393 fragment_mgr.cpp:202] Fail to open fragment e8452b7d-7fae-1105-6dd9-5d54b2380684: Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77385966456, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:19:11.012881 393 stream_load_executor.cpp:112] fragment execute failed, query_id=e8452b7d7fae1105-6dd95d54b2380683, err_msg=Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77385966456, Limit: 77760000000. Mem usage has exceed the limit of BE, id=e8452b7d7fae1105-6dd95d54b2380683, job_id=-1, txn_id: 1116491, label=f5db719f-61c0-4f62-a6d7-d31ff43353dd, db=opentelemetry
W0701 08:19:11.012931 952 stream_load.cpp:376] append body content failed. errmsg=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77385966456, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get()) context=id=e8452b7d7fae1105-6dd95d54b2380683, job_id=-1, txn_id: 1116491, label=f5db719f-61c0-4f62-a6d7-d31ff43353dd, db=opentelemetry
W0701 08:19:11.013746 395 tablet_sink_sender.cpp:120] NodeChannel[1286019], tablet add chunk failed, load_id=a847495b-af29-4454-6f14-1793caa60887, txn_id: 1116505, parallel=1, compress_type=2, node=192.168.97.155:8060, errmsg=Memory limit exceeded: Memory of process exceed limit. try consume:114688 Used: 77317005440, Limit: 77760000000. Mem usage has exceed the limit of BE
W0701 08:19:11.013769 395 plan_fragment_executor.cpp:207] Fail to open fragment, instance_id=a847495b-af29-4454-6f14-1793caa60888, status=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:114688 Used: 77317005440, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:19:11.013777 395 fragment_mgr.cpp:202] Fail to open fragment a847495b-af29-4454-6f14-1793caa60888: Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:114688 Used: 77317005440, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:19:11.014081 395 stream_load_executor.cpp:112] fragment execute failed, query_id=a847495baf294454-6f141793caa60887, err_msg=Memory limit exceeded: Memory of process exceed limit. try consume:114688 Used: 77317005440, Limit: 77760000000. Mem usage has exceed the limit of BE, id=a847495baf294454-6f141793caa60887, job_id=-1, txn_id: 1116505, label=d3000140-6a38-4ecc-8bd7-8f8bd5396a4e, db=test_ods
W0701 08:19:11.014132 955 stream_load.cpp:376] append body content failed. errmsg=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:114688 Used: 77317005440, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get()) context=id=a847495baf294454-6f141793caa60887, job_id=-1, txn_id: 1116505, label=d3000140-6a38-4ecc-8bd7-8f8bd5396a4e, db=test_ods
W0701 08:20:49.891906 475 fragment_context.cpp:154] [Driver] Canceled, query_id=d03c8da7-3782-11ef-a84a-961a03213c62, instance_id=d03c8da7-3782-11ef-a84a-961a03213c65, reason=InternalError
W0701 08:20:49.891932 459 fragment_context.cpp:154] [Driver] Canceled, query_id=cd8ef9aa-3782-11ef-a84a-961a03213c62, instance_id=cd8ef9aa-3782-11ef-a84a-961a03213c63, reason=InternalError
W0701 08:22:18.452643 395 fragment_mgr.cpp:331] Retrying ReportExecStatus: write() send(): Broken pipe
W0701 08:22:40.168756 463 fragment_context.cpp:154] [Driver] Canceled, query_id=10efd0ed-3783-11ef-a84a-961a03213c62, instance_id=10efd0ed-3783-11ef-a84a-961a03213c63, reason=InternalError
W0701 08:22:40.168807 458 fragment_context.cpp:154] [Driver] Canceled, query_id=10efd0ed-3783-11ef-a84a-961a03213c62, instance_id=10efd0ed-3783-11ef-a84a-961a03213c65, reason=InternalError
W0701 08:22:40.460222 393 tablet_sink_sender.cpp:120] NodeChannel[1286031], tablet add chunk failed, load_id=864b4422-3f92-60b2-5f38-c5b718d0c787, txn_id: 1116579, parallel=1, compress_type=2, node=192.168.96.92:8060, errmsg=Memory limit exceeded: Memory of process exceed limit. try consume:160 Used: 76173735336, Limit: 77760000000. Mem usage has exceed the limit of BE
W0701 08:22:40.460253 393 plan_fragment_executor.cpp:207] Fail to open fragment, instance_id=864b4422-3f92-60b2-5f38-c5b718d0c788, status=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:160 Used: 76173735336, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:22:40.460861 393 fragment_mgr.cpp:202] Fail to open fragment 864b4422-3f92-60b2-5f38-c5b718d0c788: Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:160 Used: 76173735336, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:22:40.461598 393 stream_load_executor.cpp:112] fragment execute failed, query_id=864b44223f9260b2-5f38c5b718d0c787, err_msg=Memory limit exceeded: Memory of process exceed limit. try consume:160 Used: 76173735336, Limit: 77760000000. Mem usage has exceed the limit of BE, id=864b44223f9260b2-5f38c5b718d0c787, job_id=-1, txn_id: 1116579, label=a8ca04b5-d38b-405e-bd89-598f63d4d152, db=opentelemetry
W0701 08:22:40.461652 951 stream_load.cpp:376] append body content failed. errmsg=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:160 Used: 76173735336, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get()) context=id=864b44223f9260b2-5f38c5b718d0c787, job_id=-1, txn_id: 1116579, label=a8ca04b5-d38b-405e-bd89-598f63d4d152, db=opentelemetry
W0701 08:22:40.523216 387 tablet_sink_sender.cpp:120] NodeChannel[1286031], tablet add chunk failed, load_id=934fe408-7486-35e3-8560-915daa5575bb, txn_id: 1116609, parallel=1, compress_type=2, node=192.168.96.92:8060, errmsg=Memory limit exceeded: Memory of process exceed limit. try consume:1310720 Used: 75799011472, Limit: 77760000000. Mem usage has exceed the limit of BE
W0701 08:22:40.523257 387 plan_fragment_executor.cpp:207] Fail to open fragment, instance_id=934fe408-7486-35e3-8560-915daa5575bc, status=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:1310720 Used: 75799011472, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:22:40.523265 387 fragment_mgr.cpp:202] Fail to open fragment 934fe408-7486-35e3-8560-915daa5575bc: Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:1310720 Used: 75799011472, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:22:40.523633 387 stream_load_executor.cpp:112] fragment execute failed, query_id=934fe408748635e3-8560915daa5575bb, err_msg=Memory limit exceeded: Memory of process exceed limit. try consume:1310720 Used: 75799011472, Limit: 77760000000. Mem usage has exceed the limit of BE, id=934fe408748635e3-8560915daa5575bb, job_id=-1, txn_id: 1116609, label=ba1566d4-6b02-472e-926a-017ae6d10f08, db=test_ods
W0701 08:22:40.523696 952 stream_load.cpp:376] append body content failed. errmsg=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:1310720 Used: 75799011472, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:853 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:427 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get()) context=id=934fe408748635e3-8560915daa5575bb, job_id=-1, txn_id: 1116609, label=ba1566d4-6b02-472e-926a-017ae6d10f08, db=test_ods
W0701 08:24:45.454680 461 fragment_context.cpp:154] [Driver] Canceled, query_id=5f7bffd4-3783-11ef-a84a-961a03213c62, instance_id=5f7bffd4-3783-11ef-a84a-961a03213c63, reason=InternalError
W0701 08:29:36.271827 953 thrift_rpc_helper.cpp:130] Rpc error: FE RPC failure, address=TNetworkAddress(hostname=starrocks-1740675210545377282-fe-2.starrocks-1740675210545377282-fe-search.starrocks.svc.cluster.local, port=9020), reason=No more data to read.
W0701 08:31:46.989780 464 fragment_context.cpp:154] [Driver] Canceled, query_id=550e0ea1-3784-11ef-a84a-961a03213c62, instance_id=550e0ea1-3784-11ef-a84a-961a03213c63, reason=InternalError
W0701 08:31:55.373148 468 fragment_context.cpp:154] [Driver] Canceled, query_id=5aced5f3-3784-11ef-a84a-961a03213c62, instance_id=5aced5f3-3784-11ef-a84a-961a03213c63, reason=InternalError
W0701 08:31:58.970990 388 tablet_sink_sender.cpp:120] NodeChannel[1286019], tablet add chunk failed, load_id=2442521b-2783-88f0-1ef2-9b99a860c886, txn_id: 1116945, parallel=1, compress_type=2, node=192.168.97.155:8060, errmsg=Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77747015832, Limit: 77760000000. Mem usage has exceed the limit of BE
W0701 08:31:58.971040 388 plan_fragment_executor.cpp:207] Fail to open fragment, instance_id=2442521b-2783-88f0-1ef2-9b99a860c887, status=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77747015832, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:837 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:581 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:31:58.971453 388 fragment_mgr.cpp:202] Fail to open fragment 2442521b-2783-88f0-1ef2-9b99a860c887: Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77747015832, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:837 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:581 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get())
W0701 08:31:58.972447 388 stream_load_executor.cpp:112] fragment execute failed, query_id=2442521b278388f0-1ef29b99a860c886, err_msg=Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77747015832, Limit: 77760000000. Mem usage has exceed the limit of BE, id=2442521b278388f0-1ef29b99a860c886, job_id=-1, txn_id: 1116945, label=040126eb-6e22-49e2-a18a-17ebe2c95944, db=opentelemetry
W0701 08:31:58.972496 949 stream_load.cpp:376] append body content failed. errmsg=Internal error: Memory limit exceeded: Memory of process exceed limit. try consume:1280 Used: 77747015832, Limit: 77760000000. Mem usage has exceed the limit of BE
be/src/exec/tablet_sink_index_channel.cpp:837 _wait_request(_add_batch_closures[_current_request_index])
be/src/exec/tablet_sink_index_channel.cpp:581 _wait_one_prev_request()
be/src/exec/tablet_sink_sender.cpp:72 _send_chunk_by_node(chunk, _channels[i], validate_select_idx)
be/src/runtime/plan_fragment_executor.cpp:252 _sink->send_chunk(runtime_state(), chunk.get()) context=id=2442521b278388f0-1ef29b99a860c886, job_id=-1, txn_id: 1116945, label=040126eb-6e22-49e2-a18a-17ebe2c95944, db=opentelemetry
W0701 08:34:35.857787 471 fragment_context.cpp:154] [Driver] Canceled, query_id=be232650-3784-11ef-a84a-961a03213c62, instance_id=be232650-3784-11ef-a84a-961a03213c63, reason=InternalError
W0701 08:35:19.128970 952 thrift_rpc_helper.cpp:130] Rpc error: FE RPC failure, address=TNetworkAddress(hostname=starrocks-1740675210545377282-fe-2.starrocks-1740675210545377282-fe-search.starrocks.svc.cluster.local, port=9020), reason=No more data to read.
W0701 08:35:48.028038 954 thrift_rpc_helper.cpp:130] Rpc error: FE RPC failure, address=TNetworkAddress(hostname=starrocks-1740675210545377282-fe-2.starrocks-1740675210545377282-fe-search.starrocks.svc.cluster.local, port=9020), reason=No more data to read.
W0701 08:35:56.804461 955 thrift_rpc_helper.cpp:130] Rpc error: FE RPC failure, address=TNetworkAddress(hostname=starrocks-1740675210545377282-fe-2.starrocks-1740675210545377282-fe-search.starrocks.svc.cluster.local, port=9020), reason=No more data to read.
W0701 08:37:06.919268 389 fragment_mgr.cpp:331] Retrying ReportExecStatus: No more data to read.
