原始json array数据字符串字段过长导致insert into select 内存溢出

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】问题详细描述

审计日志中显示sql使用了7.115668744G数据
审计日志显示同时运行的 只有这一个sql

2024-08-02 04:40:58.725+08:00 [query] |Timestamp=1722544770130|Client=10.0.0.80:22786|User=fabric|AuthorizedUser=‘fabric’@’%’|ResourceGroup=load_resource_group|Catalog=default_catalog|Db=|State=ERR|ErrorCode=|Time=88595|ScanBytes=80436233333|ScanRows=7424052|ReturnRows=0|CpuCostNs=168158909447|MemCostBytes=7115668744|StmtId=1543510|QueryId=26d68358-5046-11ef-bdc2-02423c37a781|IsQuery=false|feIp=mdw|Stmt=INSERT INTO cubeappdata.ods_api_wdtqjqm_sale_order_info_f (dt, trade_id, trade_no, platform_id, warehouse_type, src_tids, trade_status, trade_type, delivery_term, receiver_ring, freeze_reason, refund_status, fenxiao_type, fenxiao_nick, trade_time, pay_time, consign_time, receiver_province, receiver_city, receiver_district, receiver_zip, receiver_area, receiver_dtb, bad_reason, logistics_no, buyer_message, cs_remark, remark_flag, print_remark, goods_type_count, goods_count, goods_amount, post_amount, other_amount, discount, receivable, cod_amount, ext_cod_fee, goods_cost, post_cost, weight, profit, tax, tax_rate, commission, invoice_type, invoice_title, invoice_content, salesman_name, checker_name, fchecker_name, checkouter_name, stockout_no, flag_name, trade_from, single_spec_no, raw_goods_count, raw_goods_type_count, currency, invoice_id, version_id, modified, created, check_time, id_card_type, shop_no, shop_name, shop_remark, warehouse_no, customer_no, logistics_name, logistics_code, logistics_type_name, to_deliver_time, delay_to_time, estimate_consign_time, shop_id, warehouse_id, trade_label, trade_mask, shop_platform_id, sub_platform_id, package_name, package_id, paid, large_type, gift_mask, customer_id, other_cost, is_sealed, customer_type, logistics_id, cancel_reason, revert_reason, detail_list, origin_data, yuce_cube_shop_id) SELECT tmp_ods_api_wdtqjqm_sale_order_info_f.dt, tmp_ods_api_wdtqjqm_sale_order_info_f.trade_id, tmp_ods_api_wdtqjqm_sale_order_info_f.trade_no, tmp_ods_api_wdtqjqm_sale_order_info_f.platform_id, tmp_ods_api_wdtqjqm_sale_order_info_f.warehouse_type, tmp_ods_api_wdtqjqm_sale_order_info_f.src_tids, tmp_ods_api_wdtqjqm_sale_order_info_f.trade_status, tmp_ods_api_wdtqjqm_sale_order_info_f.trade_type, tmp_ods_api_wdtqjqm_sale_order_info_f.delivery_term, tmp_ods_api_wdtqjqm_sale_order_info_f.receiver_ring, tmp_ods_api_wdtqjqm_sale_order_info_f.freeze_reason, tmp_ods_api_wdtqjqm_sale_order_info_f.refund_status, tmp_ods_api_wdtqjqm_sale_order_info_f.fenxiao_type, tmp_ods_api_wdtqjqm_sale_order_info_f.fenxiao_nick, tmp_ods_api_wdtqjqm_sale_order_info_f.trade_time, tmp_ods_api_wdtqjqm_sale_order_info_f.pay_time, tmp_ods_api_wdtqjqm_sale_order_info_f.consign_time, tmp_ods_api_wdtqjqm_sale_order_info_f.receiver_province, tmp_ods_api_wdtqjqm_sale_order_info_f.receiver_city, tmp_ods_api_wdtqjqm_sale_order_info_f.receiver_district, tmp_ods_api_wdtqjqm_sale_order_info_f.receiver_zip, tmp_ods_api_wdtqjqm_sale_order_info_f.receiver_area, tmp_ods_api_wdtqjqm_sale_order_info_f.receiver_dtb, tmp_ods_api_wdtqjqm_sale_order_info_f.bad_reason, tmp_ods_api_wdtqjqm_sale_order_info_f.logistics_no, tmp_ods_api_wdtqjqm_sale_order_info_f.buyer_message, tmp_ods_api_wdtqjqm_sale_order_info_f.cs_remark, tmp_ods_api_wdtqjqm_sale_order_info_f.remark_flag, tmp_ods_api_wdtqjqm_sale_order_info_f.print_remark, tmp_ods_api_wdtqjqm_sale_order_info_f.goods_type_count, tmp_ods_api_wdtqjqm_sale_order_info_f.goods_count, tmp_ods_api_wdtqjqm_sale_order_info_f.goods_amount, tmp_ods_api_wdtqjqm_sale_order_info_f.post_amount, tmp_ods_api_wdtqjqm_sale_order_info_f.other_amount, tmp_ods_api_wdtqjqm_sale_order_info_f.discount, tmp_ods_api_wdtqjqm_sale_order_info_f.receivable, tmp_ods_api_wdtqjqm_sale_order_info_f.cod_amount, tmp_ods_api_wdtqjqm_sale_order_info_f.ext_cod_fee, tmp_ods_api_wdtqjqm_sale_order_info_f.goods_cost, tmp_ods_api_wdtqjqm_sale_order_info_f.post_cost, tmp_ods_api_wdtqjqm_sale_order_info_f.weight, tmp_ods_api_wdtqjqm_sale_order_info_f.profit, tmp_ods_api_wdtqjqm_sale_order_info_f.tax, tmp_ods_api_wdtqjqm_sale_order_info_f.tax_rate, tmp_ods_api_wdtqjqm_sale_order_info_f.commission, tmp_ods_api_wdtqjqm_sale_order_info_f.invoice_type, tmp_ods_api_wdtqjqm_sale_order_info_f.invoice_title, tmp_ods_api_wdtqjqm_sale_order_info_f.invoice_content, tmp_ods_api_wdtqjqm_sale_order_info_f.salesman_name, tmp_ods_api_wdtqjqm_sale_order_info_f.checker_name, tmp_ods_api_wdtqjqm_sale_order_info_f.fchecker_name, tmp_ods_api_wdtqjqm_sale_order_info_f.checkouter_name, tmp_ods_api_wdtqjqm_sale_order_info_f.stockout_no, tmp_ods_api_wdtqjqm_sale_order_info_f.flag_name, tmp_ods_api_wdtqjqm_sale_order_info_f.trade_from, tmp_ods_api_wdtqjqm_sale_order_info_f.single_spec_no, tmp_ods_api_wdtqjqm_sale_order_info_f.raw_goods_count, tmp_ods_api_wdtqjqm_sale_order_info_f.raw_goods_type_count, tmp_ods_api_wdtqjqm_sale_order_info_f.currency, tmp_ods_api_wdtqjqm_sale_order_info_f.invoice_id, tmp_ods_api_wdtqjqm_sale_order_info_f.version_id, tmp_ods_api_wdtqjqm_sale_order_info_f.modified, tmp_ods_api_wdtqjqm_sale_order_info_f.created, tmp_ods_api_wdtqjqm_sale_order_info_f.check_time, tmp_ods_api_wdtqjqm_sale_order_info_f.id_card_type, tmp_ods_api_wdtqjqm_sale_order_info_f.shop_no, tmp_ods_api_wdtqjqm_sale_order_info_f.shop_name, tmp_ods_api_wdtqjqm_sale_order_info_f.shop_remark, tmp_ods_api_wdtqjqm_sale_order_info_f.warehouse_no, tmp_ods_api_wdtqjqm_sale_order_info_f.customer_no, tmp_ods_api_wdtqjqm_sale_order_info_f.logistics_name, tmp_ods_api_wdtqjqm_sale_order_info_f.logistics_code, tmp_ods_api_wdtqjqm_sale_order_info_f.logistics_type_name, tmp_ods_api_wdtqjqm_sale_order_info_f.to_deliver_time, tmp_ods_api_wdtqjqm_sale_order_info_f.delay_to_time, tmp_ods_api_wdtqjqm_sale_order_info_f.estimate_consign_time, tmp_ods_api_wdtqjqm_sale_order_info_f.shop_id, tmp_ods_api_wdtqjqm_sale_order_info_f.warehouse_id, tmp_ods_api_wdtqjqm_sale_order_info_f.trade_label, tmp_ods_api_wdtqjqm_sale_order_info_f.trade_mask, tmp_ods_api_wdtqjqm_sale_order_info_f.shop_platform_id, tmp_ods_api_wdtqjqm_sale_order_info_f.sub_platform_id, tmp_ods_api_wdtqjqm_sale_order_info_f.package_name, tmp_ods_api_wdtqjqm_sale_order_info_f.package_id, tmp_ods_api_wdtqjqm_sale_order_info_f.paid, tmp_ods_api_wdtqjqm_sale_order_info_f.large_type, tmp_ods_api_wdtqjqm_sale_order_info_f.gift_mask, tmp_ods_api_wdtqjqm_sale_order_info_f.customer_id, tmp_ods_api_wdtqjqm_sale_order_info_f.other_cost, tmp_ods_api_wdtqjqm_sale_order_info_f.is_sealed, tmp_ods_api_wdtqjqm_sale_order_info_f.customer_type, tmp_ods_api_wdtqjqm_sale_order_info_f.logistics_id, tmp_ods_api_wdtqjqm_sale_order_info_f.cancel_reason, tmp_ods_api_wdtqjqm_sale_order_info_f.revert_reason, tmp_ods_api_wdtqjqm_sale_order_info_f.detail_list, tmp_ods_api_wdtqjqm_sale_order_info_f.origin_data, tmp_ods_api_wdtqjqm_sale_order_info_f.yuce_cube_shop_id FROM cubeappdata.tmp_ods_api_wdtqjqm_sale_order_info_f|Digest=|IsForwardToLeader=true

【背景】做过哪些操作?
【业务影响】
【是否存算分离】
【StarRocks版本】例如:3.2.7
【集群规模】例如:3fe(3 follower)+3be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:16C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息,请补充下您的联系方式,例如:社区群15-可乐鸡或者邮箱,谢谢
【附件】

  • fe.log/beINFO/相应截图

  • 慢查询:

    • Profile信息

    • 并行度:show variables like ‘%parallel_fragment_exec_instance_num%’;
      parallel_fragment_exec_instance_num = 1

    • pipeline是否开启:show variables like ‘%pipeline%’;
      ±-----------------------±------+
      | Variable_name | Value |
      ±-----------------------±------+
      | enable_pipeline_engine | true |
      | max_pipeline_dop | 64 |
      | pipeline_dop | 2 |
      | pipeline_profile_level | 1 |
      | pipeline_sink_dop | 2 |
      ±-----------------------±------+

    • be节点cpu和内存使用率截图

  • 查询报错:

  • be crash

    • be.out
  • 外表查询报错

    • be.out和fe.warn.log

可以参考这个获取一个heap profile https://github.com/StarRocks/starrocks/pull/35322