iceberg表Parquet-ZSTD格式v2读取问题

💬 StarRocks 用户问答 功能使用相关
#1

为了更快的定位您的问题，请提供以下信息，谢谢
【详述】查询iceberg外表Parquet-ZSTD格式v2问题
【背景】
【业务影响】
【是否存算分离】是
【StarRocks版本】例如：3.1.4
【集群规模】例如：3fe（1 follower+2observer）+3be（fe与be混部）
【机器信息】CPU虚拟核/内存/网卡，例如：48C/64G/万兆
【联系方式】
【附件】
查询iceberg Parquet-ZSTD格式v2
报错信息：ERROR 1064 (HY000): Unsupported iceberg file content: 2

image
image1288×376 67.4 KB

be日志
W1214 07:19:19.821959 3144 iceberg_delete_builder.h:91] Unsupported iceberg file content: 2
W1214 07:19:19.822075 3144 iceberg_delete_builder.h:91] Unsupported iceberg file content: 2
W1214 07:19:19.822147 3144 iceberg_delete_builder.h:91] Unsupported iceberg file content: 2
W1214 07:19:19.822222 3132 pipeline_driver.cpp:308] pull_chunk returns not ok status Internal error: Unsupported iceberg file content: 2
/build/starrocks/be/src/exec/hdfs_scanner_parquet.cpp:33 iceberg_delete_builder->build_parquet(runtime_state->timezone(), *tdelete_file)
/build/starrocks/be/src/connector/hive_connector.cpp:653 scanner->init(state, scanner_params)
/build/starrocks/be/src/connector/hive_connector.cpp:110 _init_scanner(state)
/build/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:504 _data_source->open(state)
/build/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:533 _open_data_source(state)
/build/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:227 _get_scan_status()
W1214 07:19:19.822255 3144 iceberg_delete_builder.h:91] Unsupported iceberg file content: 2
W1214 07:19:19.822384 3144 iceberg_delete_builder.h:91] Unsupported iceberg file content: 2
W1214 07:19:19.822541 3144 iceberg_delete_builder.h:91] Unsupported iceberg file content: 2
W1214 07:19:19.822258 3132 pipeline_driver_executor.cpp:161] [Driver] Process error, query_id=18c3882c-9a51-11ee-898d-021f8add70b0, instance_id=18c3882c-9a51-11ee-898d-021f8add70b2, status=Internal error: Unsupported iceberg file content: 2
/build/starrocks/be/src/exec/hdfs_scanner_parquet.cpp:33 iceberg_delete_builder->build_parquet(runtime_state->timezone(), *tdelete_file)
/build/starrocks/be/src/connector/hive_connector.cpp:653 scanner->init(state, scanner_params)
/build/starrocks/be/src/connector/hive_connector.cpp:110 _init_scanner(state)
/build/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:504 _data_source->open(state)
/build/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:533 _open_data_source(state)
/build/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:227 _get_scan_status()

#2

CREATE TABLE iceberg_db.iceberg_xxl_job_log (
id STRING,
job_group STRING COMMENT ‘’,
job_id STRING COMMENT ‘’,
executor_address STRING COMMENT ‘’,
executor_handler STRING COMMENT ‘’,
executor_param STRING COMMENT ‘’,
executor_sharding_param STRING COMMENT ‘’,
executor_fail_retry_count STRING COMMENT ‘’,
trigger_time STRING COMMENT ‘’,
trigger_code STRING COMMENT ‘’,
trigger_msg STRING COMMENT ‘’,
handle_time STRING COMMENT ‘’,
handle_code STRING COMMENT ‘’,
handle_msg STRING COMMENT ‘’,
alarm_status STRING COMMENT ‘’,
execute_date STRING COMMENT ‘’,
handle_type STRING COMMENT ‘’,
call_back_msg STRING COMMENT ‘’,
wait_time STRING COMMENT ‘’,
wait_status STRING COMMENT ‘’,
release_code STRING COMMENT ‘’,
time_zone STRING COMMENT ‘’)
USING iceberg
PARTITIONED BY (truncate(trigger_time, 10))
LOCATION ‘’
TBLPROPERTIES(
‘current-snapshot-id’ = ‘7573267705329840892’,
‘format’ = ‘iceberg/parquet’,
‘write.format.default’ = ‘parquet’,
‘write.metadata.delete-after-commit.enabled’ = ‘true’,
‘write.metadata.previous-versions-max’ = ‘1’,
‘write.parquet.compression-codec’ = ‘zstd’,
‘format-version’=‘2’)

#3

使用spark 建表

#4

这个报错有点问题，在main分支优化了，意思是当前不支持iceberg v2 equality delete，现在支持的是position

#5

老师您好！我想问一下，大概什么时候支持equality delete呢？如果是v2的表更新，应该都会有这个文件吧

#6

你们当前还在用iceberg吗？

#7

在用喔

京ICP备2022026421号-1