在某大表上添加索引出现了报错

【详述】在某大表上添加索引出现了报错
【背景】在某大表上添加索引出现了报错
【业务影响】
【StarRocks版本】例如:2.5.2
【集群规模】例如:1fe+8be
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息,请补充下您的联系方式,例如:社区群4-小李或者邮箱,谢谢
【附件】
报错信息:
schema change task failed after try three times: task type: ALTER, status_code: RUNTIME_ERROR, backendId: Backend [id=7256462, host=172.16.4.124, heartbeatPort=9050, alive=true], signature: 26933309

查[root@doris-5 log]# tail -2000000 be.INFO | grep 26933309
I0526 13:48:03.888797 31870 agent_server.cpp:365] Submit task success. type=CREATE, signature=26933309
I0526 13:48:03.888842 9385 tablet_manager.cpp:139] Creating tablet 26933309
I0526 13:48:04.007004 9385 tablet_manager.cpp:208] Created tablet 26933309
I0526 13:48:14.626623 31870 agent_server.cpp:422] Submit task success. type=ALTER, signature=26933309
I0526 13:48:14.626650 9388 agent_task.cpp:217] get alter table task, signature: 26933309
I0526 13:48:14.626693 9388 schema_change.cpp:675] begin to do request alter tablet: base_tablet_id=25890489, base_schema_hash=683863723, new_tablet_id=26933309, new_schema_hash=1183981242, alter_version=1069
I0526 13:48:14.626700 9388 schema_change.cpp:725] finish to validate alter tablet request. begin to convert data from base tablet to new tablet base_tablet=25890489.683863723.9c4411ad3cdb6e8a-16034138d1e930a3 new_tablet=26933309.1183981242.e24b0e3ae3a52fbe-047f7574168b5695
I0526 13:48:14.626881 9388 tablet_updates.cpp:2483] convert_from start tablet:26933309 #pending:0 base_tablet:25890489 request_version:1069
I0526 13:48:19.567503 9430 local_tablets_channel.cpp:483] LocalTabletsChannel txn_id: 30226982 load_id: eb07477b-fb88-11ed-9347-00163e09a0ba open delta writer: [26933324:0][26933315:0][26933309:0][26933297:0][26933294:0] failed_tablets:
I0526 13:48:29.699595 9431 local_tablets_channel.cpp:343] LocalTabletsChannel txn_id: 30226982 load_id: eb07477b-fb88-11ed-9347-00163e09a0ba commit 5 tablets: 26933324,26933315,26933309,26933297,26933294
I0526 13:48:29.767590 8193 txn_manager.cpp:213] Commit txn successfully. tablet: 26933309, txn_id: 30226982, rowsetid: 02000000023b55489747c320ef7064f4d9ee7c72b388b686 #segment:1 #delfile:0
I0526 13:48:30.495616 9374 tablet_updates.cpp:505] add rowset to pending commits tablet:26933309 version:1070 txn_id: 30226982 #pending:1
I0526 13:48:30.495632 9374 publish_version.cpp:110] Publish txn success tablet:26933309 version:1070 tablet_max_version:1 partition:25890462 txn_id: 30226982 rowset:02000000023b55489747c320ef7064f4d9ee7c72b388b686
/root/starrocks/be/src/storage/rowset/segment_iterator.cpp:868 _read(chunk, rowid, std::max(chunk_capacity - chunk_start, chunk_capacity / 4)) base_tablet_id=25890489, base_schema_hash=683863723, new_tablet_id=26933309, new_schema_hash=1183981242
W0526 13:48:49.363077 9388 agent_task.cpp:122] alter failed. signature: 26933309
I0526 13:48:57.353583 31870 agent_server.cpp:422] Submit task success. type=ALTER, signature=26933309
I0526 13:48:57.353613 9389 agent_task.cpp:217] get alter table task, signature: 26933309
I0526 13:48:57.353653 9389 schema_change.cpp:675] begin to do request alter tablet: base_tablet_id=25890489, base_schema_hash=683863723, new_tablet_id=26933309, new_schema_hash=1183981242, alter_version=1069
I0526 13:48:57.353662 9389 schema_change.cpp:725] finish to validate alter tablet request. begin to convert data from base tablet to new tablet base_tablet=25890489.683863723.9c4411ad3cdb6e8a-16034138d1e930a3 new_tablet=26933309.1183981242.e24b0e3ae3a52fbe-047f7574168b5695
I0526 13:48:57.353714 9389 tablet_updates.cpp:2483] convert_from start tablet:26933309 #pending:1 base_tablet:25890489 request_version:1069
/root/starrocks/be/src/storage/rowset/segment_iterator.cpp:868 _read(chunk, rowid, std::max(chunk_capacity - chunk_start, chunk_capacity / 4)) base_tablet_id=25890489, base_schema_hash=683863723, new_tablet_id=26933309, new_schema_hash=1183981242
W0526 13:49:28.834744 9389 agent_task.cpp:122] alter failed. signature: 26933309
I0526 13:49:37.354722 31870 agent_server.cpp:422] Submit task success. type=ALTER, signature=26933309
I0526 13:49:37.355566 9388 agent_task.cpp:217] get alter table task, signature: 26933309
I0526 13:49:37.355583 9388 schema_change.cpp:675] begin to do request alter tablet: base_tablet_id=25890489, base_schema_hash=683863723, new_tablet_id=26933309, new_schema_hash=1183981242, alter_version=1069
I0526 13:49:37.355595 9388 schema_change.cpp:725] finish to validate alter tablet request. begin to convert data from base tablet to new tablet base_tablet=25890489.683863723.9c4411ad3cdb6e8a-16034138d1e930a3 new_tablet=26933309.1183981242.e24b0e3ae3a52fbe-047f7574168b5695
I0526 13:49:37.355655 9388 tablet_updates.cpp:2483] convert_from start tablet:26933309 #pending:1 base_tablet:25890489 request_version:1069
/root/starrocks/be/src/storage/rowset/segment_iterator.cpp:868 _read(chunk, rowid, std::max(chunk_capacity - chunk_start, chunk_capacity / 4)) base_tablet_id=25890489, base_schema_hash=683863723, new_tablet_id=26933309, new_schema_hash=1183981242
W0526 13:50:08.625442 9388 agent_task.cpp:122] alter failed. signature: 26933309
I0526 13:51:02.947757 31870 agent_server.cpp:369] Submit task success. type=DROP, signature=26933309
I0526 13:51:02.952147 9383 tablet_manager.cpp:1255] Start to drop tablet 26933309
I0526 13:51:02.952275 9383 tablet_manager.cpp:1299] Succeed to drop tablet 26933309
I0526 13:53:15.345469 9437 tablet_manager.cpp:906] Moved /opt/module/storage/data/991/26933309
寻了172.16.4.124上关于这个id的be.INFO

麻烦您留下微信/企业微信/飞书等联系方式,方便我们联系您快速解决这个问题,谢谢!

微信号:ljw1209113096

您好,微信找不到这个号。

12群 名称TuTu

加了 麻烦通过一下

日志里发现类似信息,说明schema change的时候内存使用超过限制了默认2GB,如果表分区比较大的情况下可能会遇到。

I0526 13:50:08.625403  9388 schema_change.cpp:693] finished alter tablet process, status=Internal error: segment iterator failed to get next chunk, status is:Memory limit exceeded: Memory of schema change task exceed limit. read and decompress page Used: 2148062352, Limit: 2147483648. You can change the limit by modify BE config [memory_limitation_per_thread_for_schema_change]

可以通过调整大be配置memory_limitation_per_thread_for_schema_change(默认值2)来绕过。