【详述】问题详细描述:broker load 大概5亿数据量
【导入/导出方式】broker load
【背景】做过哪些操作?
fe.conf
max_broker_concurrency = 5
max_bytes_per_broker_scanner=32212254720
be.conf
#1.be节点base compaction线程数量
base_compaction_num_threads_per_disk=4
#2.base compaction时写磁盘的限速,单位为M
base_compaction_write_mbytes_per_sec=20
#3.be节点cumulative compaction的线程数量
cumulative_compaction_num_threads_per_disk=8
#4.be节点cumulative compactiond写磁盘的限速,单位为M
cumulative_compaction_write_mbytes_per_sec=300
#5.be节点cumulative compactiond线程轮询的间隔
cumulative_compaction_check_interval_seconds=2
【业务影响】
【StarRocks版本】例如:1.19.5
【集群规模】例如:3fe(1 follower+2observer)+5be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/256G/万兆
【附件】be.WARNING
W0424 09:00:43.232565 49415 fragment_mgr.cpp:300] Retrying ReportExecStatus: No more data to read.
W0424 09:01:51.845860 49288 memtable.cpp:207] Too many segment files in one load. tablet=858996, segment_count=1001, limit=1000
W0424 09:01:52.400563 49653 internal_service.cpp:205] tablet writer add chunk failed, message=tablet_id = 858996 flush_status error , id=a4240e3c-3996-48be-92cf-d1d7c55fd52b, index_id=632630, sender_id=2
W0424 09:01:52.400704 49677 tablet_sink.cpp:199] NodeChannel[632630-10008] add batch req success but status isn’t ok, load_id=a4240e3c-3996-48be-92cf-d1d7c55fd52b, txn_id=1067004, node=10.20.52.196:8060, errmsg=tablet_id = 858996 flush_status error
W0424 09:16:17.161778 49410 tablet_sink.cpp:991] close channel failed. channel_name=NodeChannel[632630-10008], load_info=load_id=a4240e3c-3996-48be-92cf-d1d7c55fd52b, txn_id=1067004, errror_msg=already stopped, skip waiting for close. cancelled/!eos: : 1/1
- fe.warn.log/be.warn.log/相应截图