【详述】starrocks be在2023-03-15 15:44:00开始,compaction score持续性变的很高,并且很有规律性
【背景】新增加了几张表,对这几张表有定时的ETL操作(读写数据)
【业务影响】
【StarRocks版本】例如:2.2.10
【集群规模】例如:3fe(3 follower)+ 5be(fe与be分开部署)
【机器信息】CPU虚拟核/内存/网卡,例如:8C/32G
【联系方式】社区群15-老张,谢谢
be compaction截图如下:
fe master节点内存截图如下:
另外,观察到fe master的内存和be compaction score升高的时间点是吻合的。
其中一台be节点的be.WARNING部分日志如下:
W0316 22:17:19.585459 2093 storage_engine.cpp:630] Fail to vectorized compact table=8309629.125424368.d4477c3f6d419029-5781725a92f145ab, err=Internal error: cumulative compaction miss version error.
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:70 check_version_continuity_with_cumulative_point(candidate_rowsets)
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:35 pick_rowsets_to_compact()
W0316 22:18:57.625792 2239 stream_load_executor.cpp:202] commit transaction failed, errmsg=Publish timeout. The data will be visible after a whileid=114274a62c5bf776-ab43348e580197ad, job_id=-1, txn_id: 17746753, label=a65e9a27-faff-482b-ab86-911c5e25bdc6, db=ods
W0316 22:18:57.693936 2240 stream_load_executor.cpp:202] commit transaction failed, errmsg=Publish timeout. The data will be visible after a whileid=584c9081c5ff309a-b9251115d93e0c83, job_id=-1, txn_id: 17746756, label=f5a420cb-847f-49e8-bb8f-08d904a9f49d, db=ods
W0316 22:19:10.997452 2093 cumulative_compaction.cpp:178] candidate rowsets misses version behind cumulative point, which may be fixed by clone later. cumulative point: 327 rowset start version: 343 rowset end version: 608
W0316 22:19:10.997483 2093 storage_engine.cpp:630] Fail to vectorized compact table=8308717.125424368.c24d694d0a3807f1-a1bfe83382602b83, err=Internal error: cumulative compaction miss version error.
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:70 check_version_continuity_with_cumulative_point(candidate_rowsets)
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:35 pick_rowsets_to_compact()
W0316 22:19:12.094529 2093 cumulative_compaction.cpp:178] candidate rowsets misses version behind cumulative point, which may be fixed by clone later. cumulative point: 327 rowset start version: 343 rowset end version: 608
W0316 22:19:12.094559 2093 storage_engine.cpp:630] Fail to vectorized compact table=8309845.125424368.e24758616cff20f7-6ff14ddd37efe7a7, err=Internal error: cumulative compaction miss version error.
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:70 check_version_continuity_with_cumulative_point(candidate_rowsets)
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:35 pick_rowsets_to_compact()
W0316 22:19:13.210690 2093 cumulative_compaction.cpp:178] candidate rowsets misses version behind cumulative point, which may be fixed by clone later. cumulative point: 274 rowset start version: 285 rowset end version: 504
W0316 22:19:13.210721 2093 storage_engine.cpp:630] Fail to vectorized compact table=8310022.125424368.c843e2ed73ddefed-57879a15c3b0cd99, err=Internal error: cumulative compaction miss version error.
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:70 check_version_continuity_with_cumulative_point(candidate_rowsets)
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:35 pick_rowsets_to_compact()
W0316 22:19:14.329663 2093 cumulative_compaction.cpp:178] candidate rowsets misses version behind cumulative point, which may be fixed by clone later. cumulative point: 274 rowset start version: 285 rowset end version: 504
W0316 22:19:14.329701 2093 storage_engine.cpp:630] Fail to vectorized compact table=8310025.125424368.7b485c4fe908294c-d6c9d680cb057885, err=Internal error: cumulative compaction miss version error.
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:70 check_version_continuity_with_cumulative_point(candidate_rowsets)
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:35 pick_rowsets_to_compact()
W0316 22:19:16.779247 2093 cumulative_compaction.cpp:178] candidate rowsets misses version behind cumulative point, which may be fixed by clone later. cumulative point: 275 rowset start version: 285 rowset end version: 502
W0316 22:19:16.779286 2093 storage_engine.cpp:630] Fail to vectorized compact table=8309575.125424368.b747b8e6adf5031f-49a0365f17b784a6, err=Internal error: cumulative compaction miss version error.
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:70 check_version_continuity_with_cumulative_point(candidate_rowsets)
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:35 pick_rowsets_to_compact()
W0316 22:19:17.878597 2093 cumulative_compaction.cpp:178] candidate rowsets misses version behind cumulative point, which may be fixed by clone later. cumulative point: 273 rowset start version: 285 rowset end version: 502
W0316 22:19:17.878619 2093 storage_engine.cpp:630] Fail to vectorized compact table=8309098.125424368.9349b407cad8c3e0-32abdc8060ca8ea4, err=Internal error: cumulative compaction miss version error.
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:70 check_version_continuity_with_cumulative_point(candidate_rowsets)
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:35 pick_rowsets_to_compact()
W0316 22:19:19.013361 2093 cumulative_compaction.cpp:178] candidate rowsets misses version behind cumulative point, which may be fixed by clone later. cumulative point: 273 rowset start version: 285 rowset end version: 502
W0316 22:19:19.013404 2093 storage_engine.cpp:630] Fail to vectorized compact table=8309626.125424368.c340eb2f0d7ae31d-b2c2efa8b01df794, err=Internal error: cumulative compaction miss version error.
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:70 check_version_continuity_with_cumulative_point(candidate_rowsets)
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:35 pick_rowsets_to_compact()
W0316 22:19:20.140684 2093 cumulative_compaction.cpp:178] candidate rowsets misses version behind cumulative point, which may be fixed by clone later. cumulative point: 273 rowset start version: 285 rowset end version: 502
W0316 22:19:20.140756 2093 storage_engine.cpp:630] Fail to vectorized compact table=8309629.125424368.d4477c3f6d419029-5781725a92f145ab, err=Internal error: cumulative compaction miss version error.
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:70 check_version_continuity_with_cumulative_point(candidate_rowsets)
/root/starrocks/be/src/storage/vectorized/cumulative_compaction.cpp:35 pick_rowsets_to_compact()
table=8309629.125424368.d4477c3f6d419029-5781725a92f145ab
这个table对应的tablet如下:
不知道这个waring日志是否和be compaction score升高有没有关系,麻烦帮忙分析,提供下排查思路。