【详述】停电后好多副本无法修复,即使修复后同一tabletid的rowcount也不一致,导致查出来的数据每次都不同,统一重启be后副本大部分修复了,但是数据不统一也没有报错,导致每次查询结果不一致
【背景】机房断电,跳闸了,一周之内连续断了两次,非主动关闭服务器
【业务影响】
【StarRocks版本】例如:2.4.5
【集群规模】例如:3fe(1 LEADER+2FOLLOWER)+3be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】社区群13-妈妈说名字起太长点赞时候能占一行,邮箱363027825@qq.com谢谢
【附件】
- beINFO
W0630 09:18:23.916455 35317 tablet_manager.cpp:910] Fail to remove or move /opt/StarRocks-2.5.4/be/storage/data/105/1613450 :Internal error: get_applied_rowsets failed, tablet updates is in error state: tablet:1613450 actual row size changed after compaction 1055 -> 0tablet:1613450 #version:310 [423159 423454.1@309 423454.1] #pending:0 - 查询错误:
主键表竟然出现相同id
执行 SHOW PROC ‘/cluster_balance/history_tablets’;
ErrMsg列有错误,Internal error: load snapshot failed, tablet updates is in error state: tablet:1615492 _apply_rowset_commit error: load primary index failed: Internal error: FixedMutableIndex<8> insert found duplicate key 0FBB180000000000
/root/starrocks/be/src/storage/persistent_index.cpp:1536 _shards[shard_offset + i]->insert(keys, values, idxes_by_shard[i])
/root/starrocks/be/src/storage/persistent_index.cpp:2913 _l0->insert(n, keys, values, check_l1_key_sizes)
/root/starrocks/be/src/storage/persistent_index.cpp:2694 _insert_rowsets(tablet, rowsets, pkey_schema, apply_version, std::move(pk_column)) tablet:1615492 #version:19 [43002 43002@0 43020] pending: rowsets:22
56407 [seg:1 row:17 del:0 bytes:2688 compaction:33551744]
56406 [seg:1 row:11 del:0 bytes:2312 compaction:33552120]
56405 [seg:1 row:9 del:0 bytes:2373 compaction:33554432]
56404 [seg:1 row:816 del:0 bytes:26960 compaction:33527472]
56408 [seg:1 row:43 del:0 bytes:3966 compaction:33550466]
56409 [seg:1 row:6 del:0 bytes:2086 compaction:33554432]
56410 [seg:1 row:39 del:0 bytes:3707 compaction:33550725]
56411 [seg:1 row:14 del:0 bytes:2494 compaction:33551938]
56412 [seg:1 row:18 del:0 bytes:2730 compaction:33551702]
56413 [seg:1 row:16 del:0 bytes:2491 compaction:33551941]
56414 [seg:1 row:33 del:0 bytes:3501 compaction:33550931]
56415 [seg:1 row:24 del:0 bytes:2937 compaction:33551495]
56416 [seg:1 row:19 del:0 bytes:2730 compaction:33551702]
56417 [seg:1 row:6 del:0 bytes:2080 compaction:33554432]
56418 [seg:1 row:8 del:0 bytes:2099 compaction:33554432]
56419 [seg:1 row:14 del:0 bytes:2526 compaction:33551906]
56420 [seg:1 row:6 del:0 bytes:2041 compaction:33554432]
56421 [seg:1 row:35 del:0 bytes:3475 compaction:33550957]
56422 [seg:1 row:23 del:0 bytes:2974 compaction:33551458]
56423 [seg:1 row:10 del:0 bytes:2480 compaction:33551952]
56424 [seg:1 row:10 del:0 bytes:2361 compaction:33552071]
56425 [seg:1 row:26 del:0 bytes:3127 compaction:33551305]
/root/starrocks/be/src/storage/task/engine_clone_task.cpp:867 tablet->updates()->load_snapshot(snapshot_meta)