be crash, 多列动态过滤导致三个be全挂了, 报错显示vector::_M_default_append large memory alloc

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】问题详细描述

join键是两个字段拼接的, 可能是多列动态过滤导致的问题.

be.WARN 报错日志

W1024 18:07:09.459998   785 pipeline_driver_executor.cpp:170] [Driver] Process error, query_id=bb02d196-91ef-11ef-b560-0242634969bc, instance_id=bb02d196-91ef-11ef-b560-0242634969c9, status=Internal error: Internal error: vector::_M_default_append
W1024 18:07:09.460419   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10005], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460455   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10006], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460464   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10004], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460538   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10005], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460549   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10006], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460558   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10004], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460628   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10005], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460637   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10006], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460649   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10004], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460767   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10005], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460779   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10006], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.460801   782 tablet_sink_sender.cpp:248] close channel failed. channel_name=NodeChannel[10004], load_info=load_id=bb02d196-91ef-11ef-b560-0242634969bc, txn_id: 6249261, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W1024 18:07:09.312206   798 mem_hook.cpp:249] large memory alloc, query_id:bb02d196-91ef-11ef-b560-0242634969bc instance: bb02d196-91ef-11ef-b560-0242634969c9 acquire:139771120713728 bytes, stack:
    @          0x69eef12  malloc
    @          0xa59596c  operator new()
    @          0x424d2c1  starrocks::raw::make_room<>()
    @          0x737af10  starrocks::concat_const_not_null()
    @          0x73b71df  starrocks::string_func_const<>()
    @          0x739b894  starrocks::StringFunctions::concat()
    @          0x40e62f0  starrocks::VectorizedFunctionCallExpr::evaluate_checked()
    @          0x40ebfd9  starrocks::DictMappingExpr::evaluate_checked()
    @          0x40c6053  starrocks::ExprContext::evaluate()
    @          0x40e5e8e  starrocks::VectorizedFunctionCallExpr::evaluate_checked()
    @          0x40c6053  starrocks::ExprContext::evaluate()
    @          0x5f671aa  starrocks::RuntimeFilterProbeCollector::update_selectivity()
    @          0x5f682f8  starrocks::RuntimeFilterProbeCollector::evaluate()
    @          0x5f5f7d3  starrocks::pipeline::Operator::eval_runtime_bloom_filters()
    @          0x5f53124  starrocks::pipeline::ScanOperator::pull_chunk()
    @          0x5f4abcf  starrocks::pipeline::PipelineDriver::process()
    @          0x67eb3be  starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x6b7b90c  starrocks::ThreadPool::dispatch_thread()
    @          0x6b74e1a  starrocks::Thread::supervise_thread()
    @     0x7f1fb7b26ac3  (unknown)
    @     0x7f1fb7bb7a04  clone
    @              (nil)  (unknown)
SELECT COUNT(*) AS `_count`
                                                                       FROM (SELECT COUNT(*) AS `DUMMY`
                                                                             FROM (SELECT `$f15`, CAST(`xsje` AS DECIMAL(38, 8)) AS `$f0`
                                                                                   FROM (SELECT `t`.`pay_time`,
                                                                                                `t`.`xsje`,
                                                                                                `dim_shop_area`.`qdmc`,
                                                                                                CONCAT(CONCAT(CASE
                                                                                                                  WHEN `t`.`product_code` IS NOT NULL
                                                                                                                      THEN `t`.`product_code`
                                                                                                                  ELSE '' END,
                                                                                                              '--'),
                                                                                                       CASE
                                                                                                           WHEN `t`.`product_name` IS NOT NULL
                                                                                                               THEN `t`.`product_name`
                                                                                                           ELSE '' END) AS `$f15`
                                                                                         FROM ((SELECT `pay_time`     AS `pay_time`,
                                                                                                       `product_code` AS `product_code`,
                                                                                                       `product_name` AS `product_name`,
                                                                                                       `xsje`         AS `xsje`,
                                                                                                       `$f9`
                                                                                                FROM (SELECT `pay_time`,
                                                                                                             `product_code`,
                                                                                                             `product_name`,
                                                                                                             `xsje`,
                                                                                                             CONCAT(
                                                                                                                     CONCAT(
                                                                                                                             CASE
                                                                                                                                 WHEN `shop_name` IS NOT NULL
                                                                                                                                     THEN `shop_name`
                                                                                                                                 ELSE '' END,
                                                                                                                             '--'),
                                                                                                                     CASE
                                                                                                                         WHEN `shop_code` IS NOT NULL
                                                                                                                             THEN `shop_code`
                                                                                                                         ELSE '' END) AS `$f9`
                                                                                                      FROM (SELECT `_persist_TABLE_wNOseQh9gr_WORKING`.`pay_time`,
                                                                                                                   `_persist_TABLE_wNOseQh9gr_WORKING`.`shop_code`,
                                                                                                                   `_persist_TABLE_wNOseQh9gr_WORKING`.`shop_name`,
                                                                                                                   `_persist_TABLE_wNOseQh9gr_WORKING`.`product_code`,
                                                                                                                   `_persist_TABLE_wNOseQh9gr_WORKING`.`product_name`,
                                                                                                                   `_persist_TABLE_wNOseQh9gr_WORKING`.`xsje`
                                                                                                            FROM `cubeappdata`.`_persist_TABLE_wNOseQh9gr_WORKING`) AS `_persist_TABLE_wNOseQh9gr_WORKING`
                                                                                                      WHERE DATE_FORMAT(`pay_time`, '%Y') = '2024') AS `_persist_TABLE_wNOseQh9gr_WORKING`)
                                                                                               UNION ALL
                                                                                               (SELECT `pay_time`     AS `pay_time`,
                                                                                                       `product_code` AS `product_code`,
                                                                                                       `product_name` AS `product_name`,
                                                                                                       `xsje`,
                                                                                                       `$f9`
                                                                                                FROM (SELECT `pay_time`,
                                                                                                             `product_code`,
                                                                                                             `product_name`,
                                                                                                             `xspe`                   AS `xsje`,
                                                                                                             CONCAT(
                                                                                                                     CONCAT(
                                                                                                                             CASE
                                                                                                                                 WHEN `shop_name` IS NOT NULL
                                                                                                                                     THEN `shop_name`
                                                                                                                                 ELSE '' END,
                                                                                                                             '--'),
                                                                                                                     CASE
                                                                                                                         WHEN `shop_code` IS NOT NULL
                                                                                                                             THEN `shop_code`
                                                                                                                         ELSE '' END) AS `$f9`
                                                                                                      FROM (SELECT `_persist_TABLE_3zeOfGldR8_WORKING`.`pay_time`,
                                                                                                                   `_persist_TABLE_3zeOfGldR8_WORKING`.`shop_code`,
                                                                                                                   `_persist_TABLE_3zeOfGldR8_WORKING`.`shop_name`,
                                                                                                                   `_persist_TABLE_3zeOfGldR8_WORKING`.`product_code`,
                                                                                                                   `_persist_TABLE_3zeOfGldR8_WORKING`.`product_name`,
                                                                                                                   `_persist_TABLE_3zeOfGldR8_WORKING`.`xspe`
                                                                                                            FROM `cubeappdata`.`_persist_TABLE_3zeOfGldR8_WORKING`) AS `_persist_TABLE_3zeOfGldR8_WORKING`
                                                                                                      WHERE DATE_FORMAT(`pay_time`, '%Y') = '2024') AS `_persist_TABLE_3zeOfGldR8_WORKING`)) AS `t`
                                                                                                  INNER JOIN (SELECT `qdmc`, `$f8`
                                                                                                              FROM (SELECT `level_name3`            AS `qdmc`,
                                                                                                                           CONCAT(
                                                                                                                                   CONCAT(
                                                                                                                                           CASE
                                                                                                                                               WHEN `shop_name` IS NOT NULL
                                                                                                                                                   THEN `shop_name`
                                                                                                                                               ELSE '' END,
                                                                                                                                           '--'),
                                                                                                                                   CASE
                                                                                                                                       WHEN `shop_id` IS NOT NULL
                                                                                                                                           THEN `shop_id`
                                                                                                                                       ELSE '' END) AS `$f8`
                                                                                                                    FROM (SELECT `dim_shop_area`.`level_name3`,
                                                                                                                                 `dim_shop_area`.`shop_id`,
                                                                                                                                 `dim_shop_area`.`shop_name`
                                                                                                                          FROM `cubeappdata`.`dim_shop_area`) AS `dim_shop_area`
                                                                                                                    WHERE `level_name3` = '品牌部'
                                                                                                                       OR `level_name3` = '渠道及大客户部'
                                                                                                                       OR `level_name3` = '电商部'
                                                                                                                       OR `level_name3` = '国际(FUGA)'
                                                                                                                       OR `level_name3` = '线下(FUGA)'
                                                                                                                       OR `level_name3` = '直营部'
                                                                                                                       OR `level_name3` = '专业市场部'
                                                                                                                       OR `level_name3` = '加盟部'
                                                                                                                       OR `level_name3` = '电商(FUGA)') AS `dim_shop_area`) AS `dim_shop_area`
                                                                                                             ON `t`.`$f9` = `dim_shop_area`.`$f8`) AS `t1`) AS `t3`
                                                                                      INNER JOIN (SELECT `$f16`
                                                                                                  FROM (SELECT `big_category_id`,
                                                                                                               `brand_id`,
                                                                                                               `type2_id`,
                                                                                                               CONCAT(
                                                                                                                       CONCAT(
                                                                                                                               CASE
                                                                                                                                   WHEN `product_code` IS NOT NULL
                                                                                                                                       THEN `product_code`
                                                                                                                                   ELSE '' END,
                                                                                                                               '--'),
                                                                                                                       CASE
                                                                                                                           WHEN `product_name` IS NOT NULL
                                                                                                                               THEN `product_name`
                                                                                                                           ELSE '' END) AS `$f16`
                                                                                                        FROM (SELECT `dim_sys_product`.`product_code`,
                                                                                                                     `dim_sys_product`.`product_name`,
                                                                                                                     `dim_sys_product`.`big_category_id`,
                                                                                                                     `dim_sys_product`.`brand_id`,
                                                                                                                     `dim_sys_product`.`type2_id`
                                                                                                              FROM `cubeappdata`.`dim_sys_product`) AS `dim_sys_product`
                                                                                                        WHERE ((
                                                                                                            ((`type2_id` = '2' OR `type2_id` = '4') OR `type2_id` = '5') OR
                                                                                                            `type2_id` =
                                                                                                            '6'))
                                                                                                          AND ((
                                                                                                            ((((((((((((((`big_category_id` = '33' OR `big_category_id` = '22') OR
                                                                                                                         `big_category_id` =
                                                                                                                         '23') OR
                                                                                                                        `big_category_id` =
                                                                                                                        '01') OR
                                                                                                                       `big_category_id` =
                                                                                                                       '24') OR
                                                                                                                      `big_category_id` =
                                                                                                                      '02') OR
                                                                                                                     `big_category_id` =
                                                                                                                     '14') OR
                                                                                                                    `big_category_id` =
                                                                                                                    '03') OR
                                                                                                                   `big_category_id` =
                                                                                                                   '04') OR
                                                                                                                  `big_category_id` =
                                                                                                                  '05') OR
                                                                                                                 `big_category_id` =
                                                                                                                 '17') OR
                                                                                                                `big_category_id` =
                                                                                                                '06') OR
                                                                                                               `big_category_id` =
                                                                                                               '19') OR
                                                                                                              `big_category_id` =
                                                                                                              '09') OR
                                                                                                             `big_category_id` =
                                                                                                             '000') OR
                                                                                                            `big_category_id` =
                                                                                                            '10'))
                                                                                                          AND ((`brand_id` = '01' OR `brand_id` = '09'))) AS `dim_sys_product`) AS `dim_sys_product`
                                                                                                 ON `t3`.`$f15` = `dim_sys_product`.`$f16`) AS `t6`

【背景】做过哪些操作?
【业务影响】
【是否存算分离】
【StarRocks版本】例如:3.2.6
【集群规模】例如:3fe(3 follower)+3be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息,请补充下您的联系方式,例如:社区群16-可乐鸡或者邮箱,谢谢
【附件】

  1. 看be.out 中的堆栈,发一下挂掉时候的 be.out 堆栈日志,从queryid 到start time 完整的堆栈信息
  2. dmesg -T 查看是否有oom

大佬 相关的日志 1秒内 集群就挂了
be_1024_174458.info.log (30.3 KB)

发的是 be.INFO 的,需要 be.out 的日志

start time: Thu Oct 24 17:33:17 CST 2024
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/starrocks/be/lib/jni-packages/starrocks-jdbc-bridge-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/starrocks/be/lib/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
3.2.6 RELEASE (build 2585333)
query_id:a189da62-91ec-11ef-b42b-0242af73969c, fragment_instance:a189da62-91ec-11ef-b42b-0242af7396a9
tracker:process consumption: 3806938472
tracker:query_pool consumption: 815093848
tracker:query_pool/connector_scan consumption: 0
tracker:load consumption: 79440
tracker:metadata consumption: 130993388
tracker:tablet_metadata consumption: 57620976
tracker:rowset_metadata consumption: 33663200
tracker:segment_metadata consumption: 7109370
tracker:column_metadata consumption: 32599842
tracker:tablet_schema consumption: 3920136
tracker:segment_zonemap consumption: 5830660
tracker:short_key_index consumption: 185182
tracker:column_zonemap_index consumption: 13884690
tracker:ordinal_index consumption: 7252832
tracker:bitmap_index consumption: 0
tracker:bloom_filter_index consumption: 0
tracker:compaction consumption: 0
tracker:schema_change consumption: 0
tracker:column_pool consumption: 0
tracker:page_cache consumption: 1948384
tracker:update consumption: 168990151
tracker:chunk_allocator consumption: 2143588080
tracker:clone consumption: 0
tracker:consistency consumption: 0
tracker:datacache consumption: 0
tracker:replication consumption: 0
*** Aborted at 1729763098 (unix time) try "date -d @1729763098" if you are using GNU date ***
PC: @          0x737af71 starrocks::concat_const_not_null()
*** SIGSEGV (@0x4) received by PID 25 (TID 0x7f2c07969640) from PID 4; stack trace: ***
    @          0x7fb5eda google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f2cc8d92256 os::Linux::chained_handler()
    @     0x7f2cc8d97f4b JVM_handle_linux_signal
    @     0x7f2cc8d8aa8c signalHandler()
    @     0x7f2cc7d3b520 (unknown)
    @          0x737af71 starrocks::concat_const_not_null()
    @          0x73b71df starrocks::string_func_const<>()
    @          0x739b894 starrocks::StringFunctions::concat()
    @          0x40e62f0 starrocks::VectorizedFunctionCallExpr::evaluate_checked()
    @          0x40ebfd9 starrocks::DictMappingExpr::evaluate_checked()
    @          0x40c6053 starrocks::ExprContext::evaluate()
    @          0x40e5e8e starrocks::VectorizedFunctionCallExpr::evaluate_checked()
    @          0x40c6053 starrocks::ExprContext::evaluate()
    @          0x5f671aa starrocks::RuntimeFilterProbeCollector::update_selectivity()
    @          0x5f682f8 starrocks::RuntimeFilterProbeCollector::evaluate()
    @          0x5f5f7d3 starrocks::pipeline::Operator::eval_runtime_bloom_filters()
    @          0x5f53124 starrocks::pipeline::ScanOperator::pull_chunk()
    @          0x5f4abcf starrocks::pipeline::PipelineDriver::process()
    @          0x67eb3be starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x6b7b90c starrocks::ThreadPool::dispatch_thread()
    @          0x6b74e1a starrocks::Thread::supervise_thread()
    @     0x7f2cc7d8dac3 (unknown)
    @     0x7f2cc7e1ea04 clone
    @                0x0 (unknown)

该sql导致 be crash 是必现的么,提供下querydump,获取方式参考 怎么获取query_dump文件

3.2.6是必现的 升级到3.2.11解决了这个问题