为了更快的定位您的问题,请提供以下信息,谢谢
【详述】通过StarRocks中的Catalog访问Hudi中的数据时,StarRocks不能自动的增量的更新hive的元数据信息,表像如下:我通过Flink任务,已经往Hudi的表中写入了一条新的数据,通过Hive的客户端都能查到这条新的数据了,但是通过Hudi 的Catalog去查,第一次查不到,再一次查询时,就可查到这条数据。
【背景】我们在排查时,发现以下一些异常信息:Hive中日志报出来的:
Caused by: com.starrocks.connector.hive.events.MetastoreNotificationFetchException: Failed to get next notification based on last event id: 2652902, msg: null
2023-07-21 17:01:39,142 INFO (com.starrocks.connector.hive.events.MetastoreEventsProcessor|30) [MetastoreEventsProcessor.runAfterCatalogReady():204] Start to pull [[catalog_juslink_hive]] events. resource mapping catalog size [0], normal catalog log size [1]
2023-07-21 17:01:39,142 INFO (com.starrocks.connector.hive.events.MetastoreEventsProcessor|30) [MetastoreEventsProcessor.getNextHMSEvents():111] Start to pull events on catalog [catalog_juslink_hive]
2023-07-21 17:01:39,145 INFO (com.starrocks.connector.hive.events.MetastoreEventsProcessor|30) [HiveMetaStoreThriftClient.open():415] Trying to connect to metastore with URI thrift://cdh2:9083
2023-07-21 17:01:39,145 INFO (com.starrocks.connector.hive.events.MetastoreEventsProcessor|30) [HiveMetaStoreThriftClient.open():495] Opened a connection to metastore, current connections: 2
2023-07-21 17:01:39,146 INFO (com.starrocks.connector.hive.events.MetastoreEventsProcessor|30) [HiveMetaStoreThriftClient.open():550] Connected to metastore.
2023-07-21 17:01:39,358 ERROR (com.starrocks.connector.hive.events.MetastoreEventsProcessor|30) [HiveMetaClient.callRPC():144] Failed to get next notification based on last event id: 2652902
at com.starrocks.connector.hive.events.MetastoreEventsProcessor.getNextHMSEvents(MetastoreEventsProcessor.java:118) [starrocks-fe.jar:?]
at com.starrocks.connector.hive.events.MetastoreEventsProcessor.getNextHMSEvents(MetastoreEventsProcessor.java:143) [starrocks-fe.jar:?]
at com.starrocks.connector.hive.events.MetastoreEventsProcessor.runAfterCatalogReady(MetastoreEventsProcessor.java:210) [starrocks-fe.jar:?]
2023-07-21 17:01:39,358 ERROR (com.starrocks.connector.hive.events.MetastoreEventsProcessor|30) [HiveMetaClient.callRPC():151] An exception occurred when using the current long link to access metastore. msg: Failed to get next notification based on last event id: 2652902
2023-07-21 17:01:39,359 INFO (com.starrocks.connector.hive.events.MetastoreEventsProcessor|30) [HiveMetaStoreThriftClient.close():575] Closed a connection to metastore, current connections: 1
2023-07-21 17:01:39,359 ERROR (com.starrocks.connector.hive.events.MetastoreEventsProcessor|30) [HiveMetastore.getNextEventResponse():177] Unable to fetch notifications from metastore. Last synced event id is 2652902
【业务影响】
【StarRocks版本】2.5.3
【集群规模】4fe,2 follower+1ob+3be be和fe混部
谢谢
