3.2.4存算分离版本审计日志AuditLoader插件写入失败 all partitions have no load data

💬 StarRocks 用户问答
#1

为了更快的定位您的问题，请提供以下信息，谢谢
【是否存算分离】是
【StarRocks版本】3.2.4
【集群规模】K8S+S3

审计日志auditLoader插件stream_load写入失败，all partitions have no load data
插件从官方文档下载https://docs.starrocks.io/zh/docs/administration/management/audit_loader/

审计日志内容

2024-05-08 15:33:02,797 [query] |Timestamp=1715182382675|Client=172.16.108.64:59186|User=restore_read|AuthorizedUser=‘restore_read’@’%’|ResourceGroup=default_wg|Catalog=default_catalog|Db=link_restore|State=EOF|ErrorCode=OTHER_ERR|Time=122|ScanBytes=0|ScanRows=0|ReturnRows=0|CpuCostNs=60799|MemCostBytes=2322408|StmtId=8199|QueryId=41f2cb1a-0d50-11ef-8100-820dd7d1d0a7|IsQuery=true|feIp=starrocks-share-data-clr-fe-8.starrocks-share-data-clr-fe-search.starrocks.svc.cluster.local|Stmt=select event_id as eventId,trace_id as traceId,rpc_id as rpcId,app as app from temp_hit_risk where 1=1 and ds>=‘2024-05-07 16:23’ and ds<=‘2024-05-07 17:23’ and from_user_id=‘1’ limit 0,50;|Digest=|PlanCpuCost=839.6203693606594|PlanMemCost=0.0|IsForwardToLeader=false

建表语句

CREATE TABLE starrocks_audit_db__.starrocks_audit_tbl__ (
queryId VARCHAR(64) COMMENT “查询的唯一ID”,
timestamp DATETIME NOT NULL COMMENT “查询开始时间”,
queryType VARCHAR(12) COMMENT “查询类型（query, slow_query, connection）”,
clientIp VARCHAR(32) COMMENT “客户端IP”,
user VARCHAR(64) COMMENT “查询用户名”,
authorizedUser VARCHAR(64) COMMENT “用户唯一标识，既user_identity”,
resourceGroup VARCHAR(64) COMMENT “资源组名”,
catalog VARCHAR(32) COMMENT “Catalog名”,
db VARCHAR(96) COMMENT “查询所在数据库”,
state VARCHAR(8) COMMENT “查询状态（EOF，ERR，OK）”,
errorCode VARCHAR(512) COMMENT “错误码”,
queryTime BIGINT COMMENT “查询执行时间（毫秒）”,
scanBytes BIGINT COMMENT “查询扫描的字节数”,
scanRows BIGINT COMMENT “查询扫描的记录行数”,
returnRows BIGINT COMMENT “查询返回的结果行数”,
cpuCostNs BIGINT COMMENT “查询CPU耗时（纳秒）”,
memCostBytes BIGINT COMMENT “查询消耗内存（字节）”,
stmtId INT COMMENT “SQL语句增量ID”,
isQuery TINYINT COMMENT “SQL是否为查询（1或0）”,
feIp VARCHAR(128) COMMENT “执行该语句的FE IP”,
stmt VARCHAR(1048576) COMMENT “原始SQL语句”,
digest VARCHAR(32) COMMENT “慢SQL指纹”,
planCpuCosts DOUBLE COMMENT “查询规划阶段CPU占用（纳秒）”,
planMemCosts DOUBLE COMMENT “查询规划阶段内存占用（字节）”
) ENGINE = OLAP
DUPLICATE KEY (queryId, timestamp, queryType)
COMMENT “审计日志表”
PARTITION BY RANGE (timestamp) ()
DISTRIBUTED BY HASH (queryId) BUCKETS 3
PROPERTIES (
“dynamic_partition.time_unit” = “DAY”,
“dynamic_partition.start” = “-30”, --表示只保留最近30天的审计信息，可视需求调整。
“dynamic_partition.end” = “3”,
“dynamic_partition.prefix” = “p”,
“dynamic_partition.buckets” = “3”,
“dynamic_partition.enable” = “true”,
“replication_num” = “3” --若集群中BE个数不大于3，可调整副本数为1，生产集群不推荐调整。
);

#2

是审计日志插件还不支持3.2版本的吗？ 日志内容多了个IsForwardToLeader，schema映射不上吗，NumberTotalRows一直是0

failed to load audit via AuditLoader plugin with label: audit_20240509_034114_starrocks-share-data-clr-fe-8_starrocks-share-data-clr-fe-search_starrocks_svc_cluster_local_9010
java.lang.Exception: status is not TEMPORARY_REDIRECT 307, status: 200, response: { “TxnId”: 326622, “Label”: “audit_20240509_034114_starrocks-share-data-clr-fe-8_starrocks-share-data-clr-fe-search_starrocks_svc_cluster_local_9010”, “Status”: “Fail”, “Message”: “fail to execute commit task: all partitions have no load data”, “NumberTotalRows”: 0, “NumberLoadedRows”: 0, “NumberFilteredRows”: 0, “NumberUnselectedRows”: 0, “LoadBytes”: 0, “LoadTimeMs”: 12, “BeginTxnTimeMs”: 1, “StreamLoadPlanTimeMs”: 2, “ReadDataTimeMs”: 0, “WriteDataTimeMs”: 6, “CommitAndPublishTimeMs”: 0}, request is: curl -v -X PUT
-H “Authorization”:“Basic cmVzdG9yZV93cml0ZTp0b3JlX3dyaXRlXzc4OQ==”
-H “Expect”:“100-continue”
-H “Content-Type”:“text/plain; charset=UTF-8”
-H “max_filter_ratio”:“1.0”
-H “column_separator”:"\x01"
-H “row_delimiter”:"\x02"
-H “columns”:“queryId, timestamp, queryType, clientIp, user, authorizedUser, resourceGroup, catalog, db, state, errorCode,queryTime, scanBytes, scanRows, returnRows, cpuCostNs, memCostBytes, stmtId, isQuery, feIp, stmt, digest, planCpuCosts, planMemCosts”
"http://sr-proxy-prod:31102/api/starrocks_audit_db__/starrocks_audit_tbl__/_stream_load?"

#3

插件版本 4.0.0
Available for versions 2.3+. Load audit log to starrocks, and user can view the statistic of queries.
version：4.0.0
javaVersion：1.8.0
ClassName：com.starrocks.plugin.audit.AuditLoaderPlugin

#4

如社区群内沟通的，出现该问题是由于plugin.conf中frontend_host_port配置了K8s中starrocks-fe-proxy-service的ip及端口，这种方式下stream load的重定向动作在Nginx内完成，http返回码不再是307而是200，进而代码抛出异常。目前可通过修改plugin.conf中frontend_host_port为默认的127.0.0.1:8030解决，下个小版本会对该问题做一下优化，支持配置proxy。

京ICP备2022026421号-1