同一张表多次查询结果不一致

【业务影响】数据查询结果不一致,该表无删除操作,查询期间也无导入任务
【StarRocks版本】例如:2.5.10
【集群规模】例如:3fe + 12be
【联系方式】微信ID: zzDuke1688
【做过什么操作】之前其他表有过类似情况,原因没有排查到,解决方案是重建表 数据重新导入之后再多次查询结果一致,show proc ‘/statistic’; 中也没有不健康tablet;尝试使用FE主节点和从节点查询均有该问题。
【背景】:表结构:

 CREATE TABLE `xxxxx` (
  `token_address` varchar(42) NOT NULL COMMENT "",
  `account_address` varchar(42) NOT NULL COMMENT "",
  `start_at` date NOT NULL COMMENT "",
  `end_at` date NULL COMMENT "",
  `balance` double NULL DEFAULT "0" COMMENT "",
  `symbol` varchar(200) NULL COMMENT "",
  `start_block` bigint(20) NULL COMMENT "",
  `end_block` bigint(20) NULL COMMENT "",
  `etl_time` datetime NOT NULL COMMENT ""
) ENGINE=OLAP 
PRIMARY KEY(`token_address`, `account_address`, `start_at`)
COMMENT "token balance "
PARTITION BY RANGE(`start_at`)
(PARTITION p0 VALUES [("2015-01-01"), ("2018-01-01")),
PARTITION p201801 VALUES [("2018-01-01"), ("2018-06-01")),
..........
PARTITION p202310 VALUES [("2023-10-01"), ("2023-11-01")))
DISTRIBUTED BY HASH(`token_address`, `account_address`, `start_at`) BUCKETS 20 
PROPERTIES (
"replication_num" = "3",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "MONTH",
"dynamic_partition.time_zone" = "Etc/UTC",
"dynamic_partition.start" = "-2147483648",
"dynamic_partition.end" = "2",
"dynamic_partition.prefix" = "p",
"dynamic_partition.buckets" = "32",
"dynamic_partition.history_partition_num" = "0",
"dynamic_partition.replication_num" = "3",
"dynamic_partition.start_day_of_month" = "1",
"in_memory" = "false",
"storage_format" = "V2",
"enable_persistent_index" = "true",
"compression" = "LZ4"
);

【现象】:

mysql> select now();
+---------------------+
| now()               |
+---------------------+
| 2023-08-15 10:09:09 |
+---------------------+
1 row in set (0.00 sec)

mysql>  select count(*) from dws_token_balance_history_eth_dil;
+------------+
| count(*)   |
+------------+
| 1545819724 |
+------------+
1 row in set (0.16 sec)

mysql> select now();
+---------------------+
| now()               |
+---------------------+
| 2023-08-15 10:09:14 |
+---------------------+
1 row in set (0.01 sec)

mysql>  select count(*) from dws_token_balance_history_eth_dil;
+------------+
| count(*)   |
+------------+
| 1545819097 |
+------------+
1 row in set (0.16 sec)

mysql> show frontends;
+------------------------------------+-----------------+-------------+----------+-----------+---------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+----------------+
| Name                               | IP              | EditLogPort | HttpPort | QueryPort | RpcPort | Role     | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg | StartTime           | Version        |
+------------------------------------+-----------------+-------------+----------+-----------+---------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+----------------+
| 192.168.100.102_9010_1639122375786 | 192.168.100.102 | 9010        | 8030     | 9030      | 9020    | LEADER   | 127394295 | true | true  | 831415193         | 2023-08-15 10:09:20 | true     |        | 2023-08-08 05:28:26 | 2.5.10-9feb716 |
| 192.168.100.103_9010_1659417153866 | 192.168.100.103 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | 127394295 | true | true  | 831415186         | 2023-08-15 10:09:20 | true     |        | 2023-08-08 05:29:01 | 2.5.10-9feb716 |
| 192.168.100.101_9010_1639115964538 | 192.168.100.101 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | 127394295 | true | true  | 831415186         | 2023-08-15 10:09:20 | true     |        | 2023-08-08 05:27:46 | 2.5.10-9feb716 |
+------------------------------------+-----------------+-------------+----------+-----------+---------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+----------------+
3 rows in set (0.00 sec)

发现有一台BE在周期性的重启,BE.out和部分Warning如下
warning中大量的 Fail to delete,目前将该BE节点服务停掉,然后观察上面表还有多次查询不一致的问题

tracker:bloom_filter_index consumption: 16497712
tracker:compaction consumption: 91124104
tracker:schema_change consumption: 0
tracker:column_pool consumption: 1825318879
tracker:page_cache consumption: 10729706432
tracker:update consumption: 3389026932
tracker:chunk_allocator consumption: 2123851968
tracker:clone consumption: 0
tracker:consistency consumption: 0
*** Aborted at 1692094991 (unix time) try "date -d @1692094991" if you are using GNU date ***
PC: @     0x7f0af78a8acf __GI_raise
*** SIGABRT (@0x1f8b91) received by PID 2067345 (TID 0x7f0ae5077700) from PID 2067345; stack trace: ***
    @          0x5aed0a2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f0af83bfcf0 (unknown)
    @     0x7f0af78a8acf __GI_raise
    @     0x7f0af787bea5 __GI_abort
    @          0x2b3b906 _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold
    @          0x7f80416 __cxxabiv1::__terminate()
    @          0x7f80481 std::terminate()
    @          0x7f805d4 __cxa_throw
    @          0x2b3b81e _Znwm.cold
    @          0x41d860d std::__cxx11::basic_string<>::_M_mutate()
    @          0x443ca70 std::__cxx11::basic_string<>::_M_replace_aux()
    @          0x44d4337 starrocks::SliceMutableIndex::load_snapshot()
    @          0x442ec86 starrocks::ShardByLengthMutableIndex::load_snapshot()
    @          0x443d765 starrocks::ShardByLengthMutableIndex::load()
    @          0x4448dac starrocks::PersistentIndex::_load()
    @          0x4449bfe starrocks::PersistentIndex::load()
    @          0x444ff3b starrocks::PersistentIndex::load_from_tablet()
    @          0x416d462 starrocks::PrimaryIndex::_do_load()
    @          0x416e97f starrocks::PrimaryIndex::load()
    @          0x4285488 starrocks::UpdateManager::on_rowset_finished()
    @          0x48a4af7 starrocks::vectorized::DeltaWriter::commit()
    @          0x5358fcb starrocks::vectorized::AsyncDeltaWriter::_execute()
    @          0x5c33b8c bthread::ExecutionQueueBase::_execute()
    @          0x5c34908 bthread::ExecutionQueueBase::_execute_tasks()
    @          0x4af5bc2 starrocks::ThreadPool::dispatch_thread()
    @          0x4af06ba starrocks::Thread::supervise_thread()
    @     0x7f0af83b51ca start_thread
    @     0x7f0af7893e73 __GI___clone
    @                0x0 (unknown)
start time: Tue Aug 15 10:23:12 UTC 2023
W0815 10:24:17.775736 2084866 rowset.cpp:246] Fail to delete /data_ssd1/sr_storage/data/887/139886181/178797581/02000000000616112b465f05a5294f35ce4544378ed4b3b9_0.dat: Not found: /data_ssd1/sr_storage/data/887/139886181/178797581/02000000000616112b465f05a5294f35ce4544378ed4b3b9_0.dat: No such file or directory
W0815 10:24:17.775743 2084866 rowset.cpp:246] Fail to delete /data_ssd1/sr_storage/data/91/139663516/1692485426/020000000004b8042b465f05a5294f35ce4544378ed4b3b9_0.dat: Not found: /data_ssd1/sr_storage/data/91/139663516/1692485426/020000000004b8042b465f05a5294f35ce4544378ed4b3b9_0.dat: No such file or directory
W0815 10:24:17.775745 2084866 rowset.cpp:246] Fail to delete /data_ssd1/sr_storage/data/91/139663516/1692485426/020000000007eecd0645112b0f17e15627563cc76b5de492_0.dat: Not found: /data_ssd1/sr_storage/data/91/139663516/1692485426/020000000007eecd0645112b0f17e15627563cc76b5de492_0.dat: No such file or directory
W0815 10:24:17.775750 2084866 rowset.cpp:246] Fail to delete /data_ssd/sr_storage/data/904/139886245/178797581/02000000000300d32b465f05a5294f35ce4544378ed4b3b9_0.dat: Not found: /data_ssd/sr_storage/data/904/139886245/178797581/02000000000300d32b465f05a5294f35ce4544378ed4b3b9_0.dat: No such file or directory
W0815 10:24:17.775755 2084866 rowset.cpp:246] Fail to delete /data_ssd1/sr_storage/data/790/108207147/1153140909/02000000000621fc0645112b0f17e15627563cc76b5de492_0.dat: Not found: /data_ssd1/sr_storage/data/790/108207147/1153140909/02000000000621fc0645112b0f17e15627563cc76b5de492_0.dat: No such file or directory
W0815 10:24:17.775759 2084866 rowset.cpp:246] Fail to delete /data_ssd1/sr_storage/data/91/139663516/1692485426/020000000003c5120645112b0f17e15627563cc76b5de492_0.dat: Not found: /data_ssd1/sr_storage/data/91/139663516/1692485426/020000000003c5120645112b0f17e15627563cc76b5de492_0.dat: No such file or directory
W0815 10:24:17.775776 2084866 rowset.cpp:246] Fail to delete /data_ssd1/sr_storage/data/91/139663516/1692485426/020000000006307d0645112b0f17e15627563cc76b5de492_0.dat: Not found: /data_ssd1/sr_storage/data/91/139663516/1692485426/020000000006307d0645112b0f17e15627563cc76b5de492_0.dat: No such file or directory
W0815 10:24:17.775794 2084866 rowset.cpp:246] Fail to delete /data_ssd1/sr_storage/data/91/139663516/1692485426/0200000000052f6c0645112b0f17e15627563cc76b5de492_0.dat: Not found: /data_ssd1/sr_storage/data/91/139663516/1692485426/0200000000052f6c0645112b0f17e15627563cc76b5de492_0.dat: No such file or directory
W0815 10:24:17.775810 2084866 rowset.cpp:246] Fail to delete /data_ssd1/sr_storage/data/91/139663516/1692485426/02000000000207110645112b0f17e15627563cc76b5de492_0.dat: Not found: /data_ssd1/sr_storage/data/91/139663516/1692485426/02000000000207110645112b0f17e15627563cc76b5de492_0.dat: No such file or directory
W0815 10:24:17.775815 2084866 rowset.cpp:246] Fail to delete /data_ssd1/sr_storage/data/91/139663516/1692485426/02000000000175a10645112b0f17e15627563cc76b5de492_0.dat: Not found: /data_ssd1/sr_storage/data/91/139663516/1692485426/02000000000175a10645112b0f17e15627563cc76b5de492_0.dat: No such file or directory

发下show tablet from dws_token_balance_history_eth_dil的结果

另外找这个表有数据的tablet,show tablet xxx,然后再执行上一步的detailcmd对应的命令,拿到metaurl的结果发下

昨晚将问题BE节点数据清空后重新加入集群,然后早晨测试看该表依旧有该现象。
该表数据量较大三副本共约300G数据
tablet信息:tablet.log (1.1 MB)