【详述】
大数据量写入的使用报表状态异常,无法继续写入并且无法查询
【背景】
测试数据写入情况,前次测试大约10万条每秒的写入,后续接近每秒20万条数据写入
【业务影响】
表无法继续写入,并且无法查询,不知道如何恢复正常
【StarRocks版本】2.4.0
【集群规模】1fe+1be（fe与be混部）
【机器信息】
硬件：
CPU：2Intel® Xeon® Silver 4216 CPU @ 2.10GHz
内存：832G 2666Mhz
磁盘：4T*8 Raid0磁盘阵列+256GB SSD系统盘
系统
CentOS Linux release 7.4.1708 (Core)
【附件】
使用
com.starrocks.connector.flink.StarRocksSink
执行下show tablet 24422;
执行下最后一列 DetailCmd 看下结果呢。
今天看的时候,已经启动起来了,看最近还能不能复现这个问题
应该是系统把这个有问题的tablet自动修复了。
方便的话在be 日志里面搜索下这个 24422 以文件形式拿下日志。
E1117 22:14:34.039036 12392 delta_writer.cpp:110] Too many versions. tablet_id: 16029, version_count: 1001, limit: 1000
E1117 22:14:34.256508 12481 delta_writer.cpp:110] Too many versions. tablet_id: 16029, version_count: 1001, limit: 1000
E1117 22:14:34.490525 12465 delta_writer.cpp:110] Too many versions. tablet_id: 16029, version_count: 1001, limit: 1000
E1117 22:14:34.702881 12395 delta_writer.cpp:110] Too many versions. tablet_id: 16029, version_count: 1001, limit: 1000
E1117 22:14:34.935237 12449 delta_writer.cpp:110] Too many versions. tablet_id: 16029, version_count: 1001, limit: 1000
E1117 22:14:35.162009 12479 delta_writer.cpp:110] Too many versions. tablet_id: 16029, version_count: 1001, limit: 1000
E1117 22:14:35.383827 12469 delta_writer.cpp:110] Too many versions. tablet_id: 16029, version_count: 1001, limit: 1000
E1117 22:14:35.589105 12457 delta_writer.cpp:110] Too many versions. tablet_id: 16029, version_count: 1001, limit: 1000
E1117 22:14:35.824373 12481 delta_writer.cpp:110] Too many versions. tablet_id: 16029, version_count: 1001, limit: 1000
E1121 12:46:37.413417 30547 tablet_updates.cpp:1456] actual row size changed after compaction 50000000 -> 48696294tablet:24422 #version:182 [1162.1 1322.1@161 1342] pending: rowsets:51
304 [seg:1 row:3748415 del:0 bytes:38881199 compaction:-5326767]
391 [seg:1 row:3748416 del:0 bytes:38662301 compaction:-5107869]
478 [seg:1 row:3836612 del:0 bytes:39889765 compaction:-6335333]
568 [seg:1 row:3924812 del:0 bytes:40545380 compaction:-6990948]
657 [seg:1 row:3779419 del:0 bytes:39304031 compaction:-5749599]
750 [seg:1 row:4018600 del:0 bytes:41771844 compaction:-8217412]
844 [seg:1 row:4018600 del:0 bytes:41680115 compaction:-8125683]
939 [seg:1 row:4061810 del:0 bytes:42202934 compaction:-8648502]
1036 [seg:1 row:4148232 del:0 bytes:43118115 compaction:-9563683]
1133 [seg:1 row:4148232 del:0 bytes:43036596 compaction:-9482164]
1316 [seg:1 row:44103 del:0 bytes:958468 compaction:32595964]
1317 [seg:1 row:44048 del:0 bytes:946058 compaction:32608374]
1318 [seg:1 row:44075 del:0 bytes:934698 compaction:32619734]
1319 [seg:1 row:44100 del:0 bytes:957367 compaction:32597065]
1320 [seg:1 row:44083 del:0 bytes:943089 compaction:32611343]
1321 [seg:1 row:44062 del:0 bytes:956359 compaction:32598073]
1322 [seg:1 row:44155 del:0 bytes:961087 compaction:32593345]
1323 [seg:1 row:44125 del:0 bytes:935212 compaction:32619220]
1324 [seg:1 row:44154 del:0 bytes:949381 compaction:32605051]
1325 [seg:1 row:44085 del:0 bytes:952047 compaction:32602385]
1326 [seg:1 row:44075 del:0 bytes:955461 compaction:32598971]
1327 [seg:1 row:44057 del:0 bytes:935248 compaction:32619184]
1328 [seg:1 row:44094 del:0 bytes:949668 compaction:32604764]
1329 [seg:1 row:44144 del:0 bytes:952962 compaction:32601470]
1330 [seg:1 row:44023 del:0 bytes:962728 compaction:32591704]
1331 [seg:1 row:44151 del:0 bytes:948750 compaction:32605682]
1332 [seg:1 row:44152 del:0 bytes:965287 compaction:32589145]
1333 [seg:1 row:44096 del:0 bytes:946599 compaction:32607833]
1334 [seg:1 row:44080 del:0 bytes:923295 compaction:32631137]
1335 [seg:1 row:44118 del:0 bytes:936893 compaction:32617539]
1336 [seg:1 row:10566852 del:2185686 bytes:108254108 compaction:37258684]
1337 [seg:1 row:44096 del:0 bytes:951090 compaction:32603342]
1338 [seg:1 row:44084 del:0 bytes:953512 compaction:32600920]
1339 [seg:1 row:44060 del:0 bytes:950811 compaction:32603621]
1340 [seg:1 row:44133 del:0 bytes:951304 compaction:32603128]
1341 [seg:1 row:44109 del:0 bytes:965632 compaction:32588800]
1342 [seg:1 row:44150 del:0 bytes:979183 compaction:32575249]
1343 [seg:1 row:44150 del:0 bytes:947993 compaction:32606439]
1344 [seg:1 row:44089 del:0 bytes:951416 compaction:32603016]
1345 [seg:1 row:44099 del:0 bytes:951897 compaction:32602535]
1346 [seg:1 row:44020 del:0 bytes:931294 compaction:32623138]
1347 [seg:1 row:44104 del:0 bytes:951190 compaction:32603242]
1348 [seg:1 row:44138 del:0 bytes:953178 compaction:32601254]
1349 [seg:1 row:44052 del:0 bytes:946281 compaction:32608151]
1350 [seg:1 row:44099 del:0 bytes:943823 compaction:32610609]
1351 [seg:1 row:44134 del:0 bytes:965682 compaction:32588750]
1352 [seg:1 row:44063 del:0 bytes:949636 compaction:32604796]
1353 [seg:1 row:44103 del:0 bytes:953137 compaction:32601295]
1354 [seg:1 row:44131 del:0 bytes:936947 compaction:32617485]
1355 [seg:1 row:44057 del:0 bytes:945089 compaction:32609343]
1356 [seg:1 row:44109 del:0 bytes:954587 compaction:32599845]
E1121 12:48:06.314029 30741 tablet_updates.cpp:1456] actual row size changed after compaction 50000000 -> 48393818tablet:24426 #version:181 [1163 1342.1@180 1342.1] pending: rowsets:11
300 [seg:1 row:3704316 del:0 bytes:38312274 compaction:-4757842]
387 [seg:1 row:3748415 del:0 bytes:38892306 compaction:-5337874]
474 [seg:1 row:3836613 del:0 bytes:39792086 compaction:-6237654]
558 [seg:1 row:3660217 del:0 bytes:37827887 compaction:-4273455]
652 [seg:1 row:4047567 del:0 bytes:42107173 compaction:-8552741]
745 [seg:1 row:3975389 del:0 bytes:41301190 compaction:-7746758]
838 [seg:1 row:3975389 del:0 bytes:41257050 compaction:-7702618]
933 [seg:1 row:4061810 del:0 bytes:42220826 compaction:-8666394]
1030 [seg:1 row:4148232 del:0 bytes:43127269 compaction:-9572837]
1126 [seg:1 row:4105022 del:0 bytes:42463536 compaction:-8909104]
1356 [seg:1 row:10737030 del:1606182 bytes:109966924 compaction:5838788]
E1121 12:48:36.150180 30768 tablet_updates.cpp:1456] actual row size changed after compaction 50000000 -> 48091344tablet:24428 #version:181 [1163 1342.1@180 1342.1] pending: rowsets:11
296 [seg:1 row:3704317 del:0 bytes:38305020 compaction:-4750588]
383 [seg:1 row:3748415 del:0 bytes:38881664 compaction:-5327232]
470 [seg:1 row:3792513 del:0 bytes:39269991 compaction:-5715559]
563 [seg:1 row:4101208 del:0 bytes:42629958 compaction:-9075526]
647 [seg:1 row:3611017 del:0 bytes:37387765 compaction:-3833333]
739 [seg:1 row:3932178 del:0 bytes:40760131 compaction:-7205699]
832 [seg:1 row:3975390 del:0 bytes:41319493 compaction:-7765061]
927 [seg:1 row:4061810 del:0 bytes:42183203 compaction:-8628771]
1023 [seg:1 row:4061810 del:0 bytes:42021798 compaction:-8467366]
1119 [seg:1 row:4148233 del:0 bytes:43099597 compaction:-9545165]
1356 [seg:1 row:10863109 del:1908656 bytes:111374654 compaction:20022808]
E1121 18:14:27.892555 64149 tablet_updates.cpp:1456] actual row size changed after compaction 50000000 -> 47875290tablet:24678 #version:392 [1147 1526.1@381 1536] pending: rowsets:31
568 [seg:1 row:8952077 del:1455247 bytes:93455545 compaction:16059412]
780 [seg:1 row:9088124 del:0 bytes:94562444 compaction:-61008012]
940 [seg:1 row:7129774 del:0 bytes:74137965 compaction:-40583533]
1091 [seg:1 row:6481612 del:0 bytes:67387151 compaction:-33832719]
1398 [seg:1 row:9979390 del:0 bytes:101987384 compaction:-68432952]
1519 [seg:1 row:44056 del:0 bytes:956805 compaction:32597627]
1520 [seg:1 row:44138 del:0 bytes:942233 compaction:32612199]
1521 [seg:1 row:44135 del:0 bytes:957515 compaction:32596917]
1522 [seg:1 row:44102 del:0 bytes:959649 compaction:32594783]
1523 [seg:1 row:44064 del:0 bytes:965227 compaction:32589205]
1524 [seg:1 row:44131 del:0 bytes:948354 compaction:32606078]
1525 [seg:1 row:44068 del:0 bytes:946489 compaction:32607943]
1526 [seg:1 row:44053 del:0 bytes:939170 compaction:32615262]
1527 [seg:1 row:44085 del:0 bytes:945062 compaction:32609370]
1528 [seg:1 row:44158 del:0 bytes:943357 compaction:32611075]
1529 [seg:1 row:44058 del:0 bytes:946990 compaction:32607442]
1530 [seg:1 row:44069 del:0 bytes:950742 compaction:32603690]
1531 [seg:1 row:44192 del:0 bytes:954258 compaction:32600174]
1532 [seg:1 row:44089 del:0 bytes:930454 compaction:32623978]
1533 [seg:1 row:44087 del:0 bytes:960168 compaction:32594264]
1534 [seg:1 row:9162785 del:2124710 bytes:94540492 compaction:48626425]
1535 [seg:1 row:44075 del:0 bytes:954643 compaction:32599789]
1536 [seg:1 row:44154 del:0 bytes:967505 compaction:32586927]
1537 [seg:1 row:44070 del:0 bytes:946383 compaction:32608049]
1538 [seg:1 row:44038 del:0 bytes:943729 compaction:32610703]
1539 [seg:1 row:44158 del:0 bytes:958251 compaction:32596181]
1540 [seg:1 row:44143 del:0 bytes:975098 compaction:32579334]
1541 [seg:1 row:44131 del:0 bytes:932014 compaction:32622418]
1542 [seg:1 row:44069 del:0 bytes:942635 compaction:32611797]
1543 [seg:1 row:44095 del:0 bytes:965815 compaction:32588617]
1544 [seg:1 row:44057 del:0 bytes:944148 compaction:32610284]
这张表是不是存在导入呢。可以 告知下导入频率么？
是的,用的SR的flink sink,看了以下应该是 stream load的方式,具体频率还没细看,数据量超过了每秒10万条
目前高频导入还是建议设置为10s以上。否则会频繁出现compaction的问题的。
好的，这个导入频率是那个参数控制的呢？
好的,感谢! 
问题如果解决了，麻烦标注下该问题已经解决了。
请问 如何临时解决这个问题呢 表数据可以恢复吗