为了更快的定位您的问题,请提供以下信息,谢谢
【详述】使用EXPORT导出数据导出到hdfs,kerberos认证怎么配置
【背景】
【业务影响】
【是否存算分离】
【StarRocks版本】例如:2.4.2
【集群规模】例如:3fe(1 follower+2observer)+3be(fe与be混部)
【机器信息】48C/64G/万兆
【表模型】主键模型
【导入或者导出方式】EXPORT导出数据导出到hdfs
配置:
EXPORT TABLE yldw.dwd_ncdos_renewal_patient_info
TO “hdfs://hdfs:8020/user/hive/warehouse/ncdos.db/dwd_ncdos_renewal_patient_info_tmp/”
PROPERTIES
(
“column_separator”=",",
“line_delimiter”="\n",
“load_mem_limit”=“2147483648”,
“include_query_id”=“false”
)
WITH BROKER “broker_104”
(
“hadoop.security.authentication” = “kerberos”,
“kerberos_principal” = “hive/presto@hadoop.com”,
“kerberos_keytab” = “/home/admin/presto.keytab”
);
报错信息:
报错日志
2023-12-21 13:35:35 [ TThreadPoolServer WorkerProcess-%d:10462600 ] - [ INFO ] could not find file system for path hdfs://hdfs:8020/user/hive/warehouse/ncdos.db/dwd_ncdos_renewal_patient_info_tmp/__starrocks_export_tmp_c122f724-9fc2-11ee-bd21-52540081aa80 create a new one
2023-12-21 13:35:35 [ TThreadPoolServer WorkerProcess-%d:10462610 ] - [ ERROR ] errors while connect to hdfs://hdfs:8020/user/hive/warehouse/ncdos.db/dwd_ncdos_renewal_patient_info_tmp/__starrocks_export_tmp_c122f724-9fc2-11ee-bd21-52540081aa80java.lang.IllegalArgumentException: Can’t get Kerberos realm
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:71)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315)
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:366)
at com.starrocks.broker.hdfs.FileSystemManager.getDistributedFileSystem(FileSystemManager.java:337)
at com.starrocks.broker.hdfs.FileSystemManager.getFileSystem(FileSystemManager.java:199)
at com.starrocks.broker.hdfs.FileSystemManager.deletePath(FileSystemManager.java:764)
at com.starrocks.broker.hdfs.HDFSBrokerServiceImpl.deletePath(HDFSBrokerServiceImpl.java:92)
at com.starrocks.thrift.TFileBrokerService$Processor$deletePath.getResult(TFileBrokerService.java:840)
at com.starrocks.thrift.TFileBrokerService$Processor$deletePath.getResult(TFileBrokerService.java:820)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: KrbException: Cannot locate default realm
at javax.security.auth.kerberos.KerberosPrincipal.(KerberosPrincipal.java:154)
at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:120)
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:69)
… 14 more
2023-12-21 13:35:35 [ TThreadPoolServer WorkerProcess-%d:10462611 ] - [ WARN ] failed to delete path: hdfs://hdfs:8020/user/hive/warehouse/ncdos.db/dwd_ncdos_renewal_patient_info_tmp/__starrocks_export_tmp_c122f724-9fc2-11ee-bd21-52540081aa80
com.starrocks.broker.hdfs.BrokerException: java.lang.IllegalArgumentException: Can’t get Kerberos realm
at com.starrocks.broker.hdfs.FileSystemManager.getDistributedFileSystem(FileSystemManager.java:425)
at com.starrocks.broker.hdfs.FileSystemManager.getFileSystem(FileSystemManager.java:199)
at com.starrocks.broker.hdfs.FileSystemManager.deletePath(FileSystemManager.java:764)
at com.starrocks.broker.hdfs.HDFSBrokerServiceImpl.deletePath(HDFSBrokerServiceImpl.java:92)
at com.starrocks.thrift.TFileBrokerService$Processor$deletePath.getResult(TFileBrokerService.java:840)
at com.starrocks.thrift.TFileBrokerService$Processor$deletePath.getResult(TFileBrokerService.java:820)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Can’t get Kerberos realm
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:71)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315)
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:366)
at com.starrocks.broker.hdfs.FileSystemManager.getDistributedFileSystem(FileSystemManager.java:337)
… 11 more
Caused by: java.lang.IllegalArgumentException: KrbException: Cannot locate default realm
at javax.security.auth.kerberos.KerberosPrincipal.(KerberosPrincipal.java:154)
at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:120)
at org.apache.