starrocks-cluster-sync同步出现Failed to get source backend info for tablet 12058154 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install

迁移工具 ‌:使用 starrocks-cluster-sync 工具
迁移架构 ‌:从‌ 华为云存算一体集群同步到‌ 阿里云存算一体集群 ‌,走公网传输20M带宽
源集群配置:华为云单台Backend节点(原3节点集群缩容下线10003、10004节点,仅保留10002节点),节点规格24核96G

  • 目标集群配置 ‌:阿里云Backend节点规格为16核64G
  • StarRocks版本 ‌:StarRocks-3.3.4
  • 集群架构 ‌:非存算分离,源集群目前为单节点混部部署
    在目标搭建的同步工具配置如下 看是否有问题
    root@iZt4n6r5j89xjndxj8zk28Z:/data# cat starrocks-cluster-sync/conf/sync.properties

If true, all tables will be synchronized only once, and the program will exit automatically after completion.

one_time_run_mode=false

source_fe_host=
source_fe_query_port=9030
source_cluster_user=
source_cluster_password=
source_cluster_password_secret_key=
source_cluster_token=

target_fe_host=
target_fe_query_port=9030
target_cluster_user=root
target_cluster_password=
target_cluster_password_secret_key=

jdbc_connect_timeout_ms=30000
jdbc_socket_timeout_ms=60000
source_cluster_jdbc_url_suffix=?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC

Comma-separated list of database/table/partition names:

<db_name> or <db_name.table_name> or <db_name.table_name.partition_name>

example: db1,db2.tbl2,db3.tbl3.p20240101

Effective order: 1. include 2. exclude

include_data_list=ad_network
exclude_data_list=

If there are no special requirements, please maintain the default values for the following configurations.

target_cluster_storage_volume=
target_cluster_replication_num=1
target_cluster_max_disk_used_percent=80

Modified as true to prevent non-persistent indexes from filling up memory during replication.

To maintain consistency with the source cluster, use null.

target_cluster_enable_persistent_index=true

Whether to use builtin_storage_volume on the target cluster

When set to true, tables created on the target cluster will use builtin_storage_volume uniformly,

instead of using the source cluster’s storage_volume configuration.

This is useful for scenarios where the target cluster does not support custom storage volumes yet.

target_cluster_use_builtin_storage_volume_only=false

max_replication_data_size_per_job_in_gb=50
meta_job_interval_seconds=600
meta_job_threads=1
ddl_job_interval_seconds=10
ddl_job_batch_size=1

table config

ddl_job_allow_drop_target_only=false
ddl_job_allow_drop_schema_change_table=true
ddl_job_allow_drop_inconsistent_partition=true
ddl_job_allow_drop_inconsistent_time_partition = true
ddl_job_allow_drop_partition_target_only=true

index config

enable_bitmap_index_sync=false
ddl_job_allow_drop_inconsistent_bitmap_index=true
ddl_job_allow_drop_bitmap_index_target_only=true

MV config

enable_materialized_view_sync=false
ddl_job_allow_drop_inconsistent_materialized_view=true
ddl_job_allow_drop_materialized_view_target_only=false

View config

enable_view_sync=false
ddl_job_allow_drop_inconsistent_view=true
ddl_job_allow_drop_view_target_only=false

replication_job_interval_seconds=10
replication_job_batch_size=1
report_interval_seconds=300
enable_table_property_sync=false

  • 完整的报错异常

26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11443226 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250621, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11443226 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250621, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11443230 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250621, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11443230 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250621, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291945 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291945 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291949 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291949 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291953 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291953 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291957 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291957 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291961 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291961 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291965 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11291965 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250617, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837264 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837264 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837268 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837268 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837272 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837272 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837276 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837276 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837280 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837280 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837284 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11837284 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250629, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659588 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659588 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659592 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659592 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659596 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659596 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659600 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659600 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659604 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659604 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659608 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11659608 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250626, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556648 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556648 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556652 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556652 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556656 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556656 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556660 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556660 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556664 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556664 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556668 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11556668 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250624, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778050 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778050 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778054 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778054 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778058 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778058 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778062 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778062 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778066 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778066 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778070 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11778070 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250628, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008023 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008023 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008027 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008027 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008031 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008031 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008035 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008035 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008039 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008039 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008043 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12008043 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250702, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718148 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718148 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718152 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718152 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718156 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718156 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718160 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718160 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718164 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718164 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718168 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11718168 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250627, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954230 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954230 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954234 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954234 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954238 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954238 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954242 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954242 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954246 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954246 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954250 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 11954250 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250701, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058134 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058134 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058138 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058138 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058142 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058142 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058146 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058146 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058150 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058150 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058154 and backend ID 10004. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 ERROR [meta-handler] produceReplicationJob(ClusterMetaKeeper.java:251): Failed to get source backend info for tablet 12058154 and backend ID 10003. Detail: db->ad_network, table->postback_install, partition->p20250703, index->postback_install
26/05/30 20:20:23 INFO [meta-handler] splitReplicationJobIfNeeded(ClusterMetaKeeper.java:328): Split replication job for ad_network.postback_install, batch partition count: 100
26/05/30 20:20:23 INFO [meta-handler] splitReplicationJobIfNeeded(ClusterMetaKeeper.java:328): Split replication job for ad_network.postback_install, batch partition count: 100
26/05/30 20:20:23 INFO [meta-handler] splitReplicationJobIfNeeded(ClusterMetaKeeper.java:328): Split replication job for ad_network.postback_install, batch partition count: 100
26/05/30 20:21:50 INFO [replication-job-handler] sendReplicationJob(Utils.java:313): Sent replication request, job token: 11001-40255_1780143623351-7786, table: ad_network.click_report
26/05/30 20:24:28 INFO [sync-reporter] report(SyncJob.java:310): DDL job queue size: 0
26/05/30 20:24:28 INFO [sync-reporter] report(SyncJob.java:312): Replication job queue size: 0
26/05/30 20:24:28 INFO [sync-reporter] report(SyncJob.java:317): Total estimated size of data pending synchronization: 0
26/05/30 20:24:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2346): Databases pending synchronization: []
26/05/30 20:24:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2347): Databases that only exist in the target cluster: []
26/05/30 20:24:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2366): Database ad_network, source size: 2.556 TB, target size: 2.093 GB, diff: 2.554 TB
26/05/30 20:24:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2378): Database ad_network , source running txn: 0, source finished txn: 0, target running txn: 1, target finished txn: 280
26/05/30 20:24:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2382): Total source size: 2.556 TB, total target size: 2.093 GB, total diff: 2.554 TB
26/05/30 20:24:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2385): Total source running txn: 0, total source finished txn: 0, total target running txn: 1, total target finished txn: 280
26/05/30 20:24:28 INFO [sync-reporter] report(SyncJob.java:403): Sync job progress: 33.33%, total: 3, ddlPending: 0, ddlRunning: 0, jobPending: 0, sent: 1, jobRunning: 1, finished: 1, failed: 10, sent_failed: 177, unknown: 0
26/05/30 20:24:28 INFO [sync-reporter] report(SyncJob.java:436): Running table detail: [ad_network.track_impression_log]
26/05/30 20:24:28 INFO [sync-reporter] report(SyncJob.java:462): Sync table progress, finishedTableRatio: 22.22%, expiredTableRatio: 77.78%, total table: 9, finished table: 2, unfinished table: 7, expired table: 7.
26/05/30 20:24:28 INFO [sync-reporter] report(SyncJob.java:467): Unfinished table detail: [ad_network.postback_event, ad_network.track_impression_log, ad_network.dsp_data, ad_network.postback_issued_log, ad_network.click_report, ad_network.track_click_log, ad_network.postback_install]
26/05/30 20:24:28 INFO [sync-reporter] report(SyncJob.java:468): Expired table detail: [ad_network.postback_event, ad_network.track_impression_log, ad_network.dsp_data, ad_network.postback_issued_log, ad_network.click_report, ad_network.track_click_log, ad_network.postback_install]
26/05/30 20:29:28 INFO [sync-reporter] report(SyncJob.java:310): DDL job queue size: 0
26/05/30 20:29:28 INFO [sync-reporter] report(SyncJob.java:312): Replication job queue size: 0
26/05/30 20:29:28 INFO [sync-reporter] report(SyncJob.java:317): Total estimated size of data pending synchronization: 0
26/05/30 20:29:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2346): Databases pending synchronization: []
26/05/30 20:29:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2347): Databases that only exist in the target cluster: []
26/05/30 20:29:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2366): Database ad_network, source size: 2.556 TB, target size: 2.093 GB, diff: 2.554 TB
26/05/30 20:29:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2378): Database ad_network , source running txn: 0, source finished txn: 0, target running txn: 1, target finished txn: 280
26/05/30 20:29:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2382): Total source size: 2.556 TB, total target size: 2.093 GB, total diff: 2.554 TB
26/05/30 20:29:28 INFO [sync-reporter] reportDbProgress(ClusterMetaKeeper.java:2385): Total source running txn: 0, total source finished txn: 0, total target running txn: 1, total target finished txn: 280
26/05/30 20:29:28 INFO [sync-reporter] report(SyncJob.java:403): Sync job progress: 33.33%, total: 3, ddlPending: 0, ddlRunning: 0, jobPending: 0, sent: 1, jobRunning: 1, finished: 1, failed: 10, sent_failed: 177, unknown: 0
26/05/30 20:29:28 INFO [sync-reporter] report(SyncJob.java:436): Running table detail: [ad_network.track_impression_log]
26/05/30 20:29:28 INFO [sync-reporter] report(SyncJob.java:462): Sync table progress, finishedTableRatio: 22.22%, expiredTableRatio: 77.78%, total table: 9, finished table: 2, unfinished table: 7, expired table: 7.
26/05/30 20:29:28 INFO [sync-reporter] report(SyncJob.java:467): Unfinished table detail: [ad_network.postback_event, ad_network.track_impression_log, ad_network.dsp_data, ad_network.postback_issued_log, ad_network.click_report, ad_network.track_click_log, ad_network.postback_install]
26/05/30 20:29:28 INFO [sync-reporter] report(SyncJob.java:468): Expired table detail: [ad_network.postback_event, ad_network.track_impression_log, ad_network.dsp_data, ad_network.postback_issued_log, ad_network.click_report, ad_network.track_click_log, ad_network.postback_install]