starrocks_be_max_disk_io_util_percent 监控信息一直是0

【详述】
在进行小批量写入数据时,观察BE节点监控项starrocks_be_max_disk_io_util_percent 的值一直是0,而通过iostat -dxk 1命令查看磁盘负载信息确是有值的,详见下面截图,请问下 starrocks_be_max_disk_io_util_percent 这个监控项的监控原理是什么?要如何排查?

BE节点服务器执行命令iostat -dxk 1 的部分截图

通过Prometheus+Grafana配置的监控图形截图

【StarRocks版本】2.4.1
【集群规模】2fe(1 follower+1observer)+3be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,48C/48G/千兆
【附件】
http://BE_IP:BE_HTTP_PORT/metrics 的详情如下:

# TYPE starrocks_be_active_scan_context_count gauge
starrocks_be_active_scan_context_count 0
# TYPE starrocks_be_binary_column_pool_bytes gauge
starrocks_be_binary_column_pool_bytes 0
# TYPE starrocks_be_bitmap_index_mem_bytes gauge
starrocks_be_bitmap_index_mem_bytes 0
# TYPE starrocks_be_blocks_created_total counter
starrocks_be_blocks_created_total 0
# TYPE starrocks_be_blocks_deleted_total counter
starrocks_be_blocks_deleted_total 0
# TYPE starrocks_be_blocks_open_reading gauge
starrocks_be_blocks_open_reading 0
# TYPE starrocks_be_blocks_open_writing gauge
starrocks_be_blocks_open_writing 0
# TYPE starrocks_be_bloom_filter_index_mem_bytes gauge
starrocks_be_bloom_filter_index_mem_bytes 1092960
# TYPE starrocks_be_broker_count gauge
starrocks_be_broker_count 0
# TYPE starrocks_be_brpc_endpoint_stub_count gauge
starrocks_be_brpc_endpoint_stub_count 3
# TYPE starrocks_be_bytes_read_total counter
starrocks_be_bytes_read_total 0
# TYPE starrocks_be_bytes_written_total counter
starrocks_be_bytes_written_total 0
# TYPE starrocks_be_central_column_pool_bytes gauge
starrocks_be_central_column_pool_bytes 0
# TYPE starrocks_be_chunk_allocator_mem_bytes gauge
starrocks_be_chunk_allocator_mem_bytes 65596400
# TYPE starrocks_be_chunk_pool_local_core_alloc_count counter
starrocks_be_chunk_pool_local_core_alloc_count 2001369
# TYPE starrocks_be_chunk_pool_other_core_alloc_count counter
starrocks_be_chunk_pool_other_core_alloc_count 802803
# TYPE starrocks_be_chunk_pool_system_alloc_cost_ns counter
starrocks_be_chunk_pool_system_alloc_cost_ns 5912747
# TYPE starrocks_be_chunk_pool_system_alloc_count counter
starrocks_be_chunk_pool_system_alloc_count 1091
# TYPE starrocks_be_chunk_pool_system_free_cost_ns counter
starrocks_be_chunk_pool_system_free_cost_ns 0
# TYPE starrocks_be_chunk_pool_system_free_count counter
starrocks_be_chunk_pool_system_free_count 0
# TYPE starrocks_be_clone_mem_bytes gauge
starrocks_be_clone_mem_bytes 0
# TYPE starrocks_be_column_metadata_mem_bytes gauge
starrocks_be_column_metadata_mem_bytes 68340664
# TYPE starrocks_be_column_pool_mem_bytes gauge
starrocks_be_column_pool_mem_bytes 3237072299
# TYPE starrocks_be_column_zonemap_index_mem_bytes gauge
starrocks_be_column_zonemap_index_mem_bytes 5840296
# TYPE starrocks_be_compaction_bytes_total counter
starrocks_be_compaction_bytes_total{type="base"} 0
starrocks_be_compaction_bytes_total{type="cumulative"} 0
starrocks_be_compaction_bytes_total{type="update"} 1517443730
# TYPE starrocks_be_compaction_deltas_total counter
starrocks_be_compaction_deltas_total{type="base"} 29
starrocks_be_compaction_deltas_total{type="cumulative"} 0
starrocks_be_compaction_deltas_total{type="update"} 2874
# TYPE starrocks_be_compaction_mem_bytes gauge
starrocks_be_compaction_mem_bytes 0
# TYPE starrocks_be_consistency_mem_bytes gauge
starrocks_be_consistency_mem_bytes 0
# TYPE starrocks_be_cpu counter
starrocks_be_cpu{mode="guest"} 0
starrocks_be_cpu{mode="guest_nice"} 0
starrocks_be_cpu{mode="idle"} 6851560120
starrocks_be_cpu{mode="iowait"} 3181445
starrocks_be_cpu{mode="irq"} 0
starrocks_be_cpu{mode="nice"} 71
starrocks_be_cpu{mode="soft_irq"} 267921
starrocks_be_cpu{mode="steal"} 0
starrocks_be_cpu{mode="system"} 5679279
starrocks_be_cpu{mode="user"} 74324707
# TYPE starrocks_be_data_stream_receiver_count gauge
starrocks_be_data_stream_receiver_count 0
# TYPE starrocks_be_date_column_pool_bytes gauge
starrocks_be_date_column_pool_bytes 0
# TYPE starrocks_be_datetime_column_pool_bytes gauge
starrocks_be_datetime_column_pool_bytes 0
# TYPE starrocks_be_decimal_column_pool_bytes gauge
starrocks_be_decimal_column_pool_bytes 0
# TYPE starrocks_be_disk_sync_total counter
starrocks_be_disk_sync_total 0
# TYPE starrocks_be_disks_avail_capacity gauge
starrocks_be_disks_avail_capacity{path="/opt/star24/storage"} 597405855744
# TYPE starrocks_be_disks_data_used_capacity gauge
starrocks_be_disks_data_used_capacity{path="/opt/star24/storage"} 15328947220
# TYPE starrocks_be_disks_state gauge
starrocks_be_disks_state{path="/opt/star24/storage"} 1
# TYPE starrocks_be_disks_total_capacity gauge
starrocks_be_disks_total_capacity{path="/opt/star24/storage"} 842234789888
# TYPE starrocks_be_double_column_pool_bytes gauge
starrocks_be_double_column_pool_bytes 0
# TYPE starrocks_be_engine_requests_total counter
starrocks_be_engine_requests_total{status="failed",type="base_compaction"} 0
starrocks_be_engine_requests_total{status="failed",type="clone"} 0
starrocks_be_engine_requests_total{status="failed",type="create_rollup"} 0
starrocks_be_engine_requests_total{status="failed",type="create_tablet"} 0
starrocks_be_engine_requests_total{status="failed",type="cumulative_compaction"} 0
starrocks_be_engine_requests_total{status="failed",type="delete"} 0
starrocks_be_engine_requests_total{status="failed",type="finish_task"} 0
starrocks_be_engine_requests_total{status="failed",type="publish"} 0
starrocks_be_engine_requests_total{status="failed",type="report_all_tablets"} 2
starrocks_be_engine_requests_total{status="failed",type="report_disk"} 2
starrocks_be_engine_requests_total{status="failed",type="report_tablet"} 0
starrocks_be_engine_requests_total{status="failed",type="report_task"} 1
starrocks_be_engine_requests_total{status="failed",type="schema_change"} 0
starrocks_be_engine_requests_total{status="total",type="base_compaction"} 8778
starrocks_be_engine_requests_total{status="total",type="clone"} 0
starrocks_be_engine_requests_total{status="total",type="create_rollup"} 8
starrocks_be_engine_requests_total{status="total",type="create_tablet"} 8
starrocks_be_engine_requests_total{status="total",type="cumulative_compaction"} 63
starrocks_be_engine_requests_total{status="total",type="delete"} 0
starrocks_be_engine_requests_total{status="total",type="drop_tablet"} 8
starrocks_be_engine_requests_total{status="total",type="finish_task"} 699
starrocks_be_engine_requests_total{status="total",type="publish"} 656
starrocks_be_engine_requests_total{status="total",type="report_all_tablets"} 4385
starrocks_be_engine_requests_total{status="total",type="report_disk"} 4385
starrocks_be_engine_requests_total{status="total",type="report_tablet"} 8
starrocks_be_engine_requests_total{status="total",type="report_task"} 26298
starrocks_be_engine_requests_total{status="total",type="schema_change"} 0
starrocks_be_engine_requests_total{status="total",type="storage_migrate"} 0
# TYPE starrocks_be_fd_num_limit gauge
starrocks_be_fd_num_limit 4878217
# TYPE starrocks_be_fd_num_used gauge
starrocks_be_fd_num_used 8352
# TYPE starrocks_be_float_column_pool_bytes gauge
starrocks_be_float_column_pool_bytes 0
# TYPE starrocks_be_fragment_endpoint_count gauge
starrocks_be_fragment_endpoint_count 0
# TYPE starrocks_be_fragment_request_duration_us counter
starrocks_be_fragment_request_duration_us 2583313815
# TYPE starrocks_be_fragment_requests_total counter
starrocks_be_fragment_requests_total 22033
# TYPE starrocks_be_http_request_send_bytes counter
starrocks_be_http_request_send_bytes 0
# TYPE starrocks_be_http_requests_total counter
starrocks_be_http_requests_total 0
# TYPE starrocks_be_int128_column_pool_bytes gauge
starrocks_be_int128_column_pool_bytes 0
# TYPE starrocks_be_int16_column_pool_bytes gauge
starrocks_be_int16_column_pool_bytes 0
# TYPE starrocks_be_int32_column_pool_bytes gauge
starrocks_be_int32_column_pool_bytes 0
# TYPE starrocks_be_int64_column_pool_bytes gauge
starrocks_be_int64_column_pool_bytes 0
# TYPE starrocks_be_int8_column_pool_bytes gauge
starrocks_be_int8_column_pool_bytes 0
# TYPE starrocks_be_jemalloc_active_bytes gauge
starrocks_be_jemalloc_active_bytes 15808241664
# TYPE starrocks_be_jemalloc_allocated_bytes gauge
starrocks_be_jemalloc_allocated_bytes 14707149472
# TYPE starrocks_be_jemalloc_mapped_bytes gauge
starrocks_be_jemalloc_mapped_bytes 16072241152
# TYPE starrocks_be_jemalloc_metadata_bytes gauge
starrocks_be_jemalloc_metadata_bytes 167450688
# TYPE starrocks_be_jemalloc_metadata_thp gauge
starrocks_be_jemalloc_metadata_thp 0
# TYPE starrocks_be_jemalloc_resident_bytes gauge
starrocks_be_jemalloc_resident_bytes 15997411328
# TYPE starrocks_be_jemalloc_retained_bytes gauge
starrocks_be_jemalloc_retained_bytes 13217632256
# TYPE starrocks_be_load_bytes counter
starrocks_be_load_bytes 238919048
# TYPE starrocks_be_load_channel_count gauge
starrocks_be_load_channel_count 0
# TYPE starrocks_be_load_mem_bytes gauge
starrocks_be_load_mem_bytes 0
# TYPE starrocks_be_load_rows counter
starrocks_be_load_rows 137453
# TYPE starrocks_be_local_column_pool_bytes gauge
starrocks_be_local_column_pool_bytes 0
# TYPE starrocks_be_lz4_compress_context_pool_create_count gauge
starrocks_be_lz4_compress_context_pool_create_count 4
# TYPE starrocks_be_lz4f_compress_context_pool_create_count gauge
starrocks_be_lz4f_compress_context_pool_create_count 3
# TYPE starrocks_be_lz4f_decompress_context_pool_create_count gauge
starrocks_be_lz4f_decompress_context_pool_create_count 23
# TYPE starrocks_be_max_disk_io_util_percent gauge
starrocks_be_max_disk_io_util_percent 0
# TYPE starrocks_be_max_network_receive_bytes_rate gauge
starrocks_be_max_network_receive_bytes_rate 1052597
# TYPE starrocks_be_max_network_send_bytes_rate gauge
starrocks_be_max_network_send_bytes_rate 347630
# TYPE starrocks_be_memory_pool_bytes_total gauge
starrocks_be_memory_pool_bytes_total 0
# TYPE starrocks_be_memtable_flush_duration_us counter
starrocks_be_memtable_flush_duration_us 167666463
# TYPE starrocks_be_memtable_flush_total counter
starrocks_be_memtable_flush_total 2302
# TYPE starrocks_be_meta_request_duration counter
starrocks_be_meta_request_duration{type="read"} 5858
starrocks_be_meta_request_duration{type="write"} 1440673
# TYPE starrocks_be_meta_request_total counter
starrocks_be_meta_request_total{type="read"} 18
starrocks_be_meta_request_total{type="write"} 10902
# TYPE starrocks_be_metadata_mem_bytes gauge
starrocks_be_metadata_mem_bytes 71768373
# TYPE starrocks_be_network_receive_bytes counter
starrocks_be_network_receive_bytes{device="ens160"} 703666467299
starrocks_be_network_receive_bytes{device="lo"} 240510649066
starrocks_be_network_receive_bytes{device="virbr0"} 0
# TYPE starrocks_be_network_receive_packets counter
starrocks_be_network_receive_packets{device="ens160"} 791553738
starrocks_be_network_receive_packets{device="lo"} 31371307
starrocks_be_network_receive_packets{device="virbr0"} 0
# TYPE starrocks_be_network_send_bytes counter
starrocks_be_network_send_bytes{device="ens160"} 528111000108
starrocks_be_network_send_bytes{device="lo"} 240510649066
starrocks_be_network_send_bytes{device="virbr0"} 0
# TYPE starrocks_be_network_send_packets counter
starrocks_be_network_send_packets{device="ens160"} 136949322
starrocks_be_network_send_packets{device="lo"} 31371307
starrocks_be_network_send_packets{device="virbr0"} 0
# TYPE starrocks_be_ordinal_index_mem_bytes gauge
starrocks_be_ordinal_index_mem_bytes 26826816
# TYPE starrocks_be_page_cache_capacity gauge
starrocks_be_page_cache_capacity 8176363388
# TYPE starrocks_be_page_cache_hit_count gauge
starrocks_be_page_cache_hit_count 2062734248
# TYPE starrocks_be_page_cache_lookup_count gauge
starrocks_be_page_cache_lookup_count 2063036432
# TYPE starrocks_be_pip_exe_driver_queue_len gauge
starrocks_be_pip_exe_driver_queue_len 0
# TYPE starrocks_be_pip_exe_poller_block_queue_len gauge
starrocks_be_pip_exe_poller_block_queue_len 0
# TYPE starrocks_be_pip_query_ctx_cnt gauge
starrocks_be_pip_query_ctx_cnt 0
# TYPE starrocks_be_plan_fragment_count gauge
starrocks_be_plan_fragment_count 0
# TYPE starrocks_be_process_fd_num_limit_hard gauge
starrocks_be_process_fd_num_limit_hard 65535
# TYPE starrocks_be_process_fd_num_limit_soft gauge
starrocks_be_process_fd_num_limit_soft 65535
# TYPE starrocks_be_process_fd_num_used gauge
starrocks_be_process_fd_num_used 1090
# TYPE starrocks_be_process_mem_bytes gauge
starrocks_be_process_mem_bytes 14476446928
# TYPE starrocks_be_process_thread_num gauge
starrocks_be_process_thread_num 736
# TYPE starrocks_be_push_request_duration_us counter
starrocks_be_push_request_duration_us 0
# TYPE starrocks_be_push_request_write_bytes counter
starrocks_be_push_request_write_bytes 0
# TYPE starrocks_be_push_request_write_bytes_per_second gauge
starrocks_be_push_request_write_bytes_per_second 0
# TYPE starrocks_be_push_request_write_rows counter
starrocks_be_push_request_write_rows 0
# TYPE starrocks_be_push_requests_total counter
starrocks_be_push_requests_total{status="FAIL"} 0
starrocks_be_push_requests_total{status="SUCCESS"} 0
# TYPE starrocks_be_query_mem_bytes gauge
starrocks_be_query_mem_bytes 0
# TYPE starrocks_be_query_scan_bytes counter
starrocks_be_query_scan_bytes 34187913980452
# TYPE starrocks_be_query_scan_bytes_per_second gauge
starrocks_be_query_scan_bytes_per_second 0
# TYPE starrocks_be_query_scan_rows counter
starrocks_be_query_scan_rows 1136187174539
# TYPE starrocks_be_readable_blocks_total counter
starrocks_be_readable_blocks_total 0
# TYPE starrocks_be_resource_group_bigquery_count gauge
starrocks_be_resource_group_bigquery_count{name="default_wg"} 0
# TYPE starrocks_be_resource_group_concurrency_overflow_count gauge
starrocks_be_resource_group_concurrency_overflow_count{name="default_wg"} 0
# TYPE starrocks_be_resource_group_connector_scan_use_ratio gauge
starrocks_be_resource_group_connector_scan_use_ratio{name="default_wg"} 0.000000
# TYPE starrocks_be_resource_group_cpu_limit_ratio gauge
starrocks_be_resource_group_cpu_limit_ratio{name="default_wg"} 1.000000
# TYPE starrocks_be_resource_group_cpu_use_ratio gauge
starrocks_be_resource_group_cpu_use_ratio{name="default_wg"} 0.000000
# TYPE starrocks_be_resource_group_mem_inuse_bytes gauge
starrocks_be_resource_group_mem_inuse_bytes{name="default_wg"} 0
# TYPE starrocks_be_resource_group_mem_limit_bytes gauge
starrocks_be_resource_group_mem_limit_bytes{name="default_wg"} 36793635249
# TYPE starrocks_be_resource_group_running_queries gauge
starrocks_be_resource_group_running_queries{name="default_wg"} 0
# TYPE starrocks_be_resource_group_scan_use_ratio gauge
starrocks_be_resource_group_scan_use_ratio{name="default_wg"} 0.000000
# TYPE starrocks_be_resource_group_total_queries gauge
starrocks_be_resource_group_total_queries{name="default_wg"} 0
# TYPE starrocks_be_result_block_queue_count gauge
starrocks_be_result_block_queue_count 0
# TYPE starrocks_be_result_buffer_block_count gauge
starrocks_be_result_buffer_block_count 0
# TYPE starrocks_be_routine_load_task_count gauge
starrocks_be_routine_load_task_count 0
# TYPE starrocks_be_rowset_count_generated_and_in_use gauge
starrocks_be_rowset_count_generated_and_in_use 2334
# TYPE starrocks_be_rowset_metadata_mem_bytes gauge
starrocks_be_rowset_metadata_mem_bytes 1653205
# TYPE starrocks_be_schema_change_mem_bytes gauge
starrocks_be_schema_change_mem_bytes 0
# TYPE starrocks_be_segment_metadata_mem_bytes gauge
starrocks_be_segment_metadata_mem_bytes 1618517
# TYPE starrocks_be_segment_read counter
starrocks_be_segment_read{type="segment_rows_by_short_key"} 2543367204480
starrocks_be_segment_read{type="segment_rows_read_by_zone_map"} 2539638381402
starrocks_be_segment_read{type="segment_total_read_times"} 27695707
starrocks_be_segment_read{type="segment_total_row_num"} 2553551898202
# TYPE starrocks_be_segment_zonemap_mem_bytes gauge
starrocks_be_segment_zonemap_mem_bytes 723992
# TYPE starrocks_be_short_key_index_mem_bytes gauge
starrocks_be_short_key_index_mem_bytes 158625
# TYPE starrocks_be_small_file_cache_count gauge
starrocks_be_small_file_cache_count 0
# TYPE starrocks_be_snmp counter
starrocks_be_snmp{name="tcp_in_errs"} 0
starrocks_be_snmp{name="tcp_in_segs"} 180020823
starrocks_be_snmp{name="tcp_out_segs"} 508474389
starrocks_be_snmp{name="tcp_retrans_segs"} 63085
# TYPE starrocks_be_storage_page_cache_mem_bytes gauge
starrocks_be_storage_page_cache_mem_bytes 9999981808
# TYPE starrocks_be_stream_load counter
starrocks_be_stream_load{type="load_rows"} 137421
starrocks_be_stream_load{type="receive_bytes"} 388816985
# TYPE starrocks_be_stream_load_pipe_count gauge
starrocks_be_stream_load_pipe_count 0
# TYPE starrocks_be_streaming_load_bytes counter
starrocks_be_streaming_load_bytes 388816985
# TYPE starrocks_be_streaming_load_current_processing gauge
starrocks_be_streaming_load_current_processing 0
# TYPE starrocks_be_streaming_load_duration_ms counter
starrocks_be_streaming_load_duration_ms 84825
# TYPE starrocks_be_streaming_load_requests_total counter
starrocks_be_streaming_load_requests_total 186
# TYPE starrocks_be_tablet_base_max_compaction_score gauge
starrocks_be_tablet_base_max_compaction_score 2
# TYPE starrocks_be_tablet_cumulative_max_compaction_score gauge
starrocks_be_tablet_cumulative_max_compaction_score 2
# TYPE starrocks_be_tablet_metadata_mem_bytes gauge
starrocks_be_tablet_metadata_mem_bytes 155987
# TYPE starrocks_be_tablet_schema_mem_bytes gauge
starrocks_be_tablet_schema_mem_bytes 40427
# TYPE starrocks_be_tablet_update_max_compaction_score gauge
starrocks_be_tablet_update_max_compaction_score 4284201310
# TYPE starrocks_be_tablet_writer_count gauge
starrocks_be_tablet_writer_count 0
# TYPE starrocks_be_thrift_connections_total counter
starrocks_be_thrift_connections_total{name="BackendService"} 3
starrocks_be_thrift_connections_total{name="heartbeat"} 1
# TYPE starrocks_be_thrift_current_connections gauge
starrocks_be_thrift_current_connections{name="BackendService"} 2
starrocks_be_thrift_current_connections{name="heartbeat"} 1
# TYPE starrocks_be_thrift_opened_clients gauge
starrocks_be_thrift_opened_clients{name="backend"} 0
starrocks_be_thrift_opened_clients{name="broker"} 0
starrocks_be_thrift_opened_clients{name="frontend"} 4
# TYPE starrocks_be_thrift_used_clients gauge
starrocks_be_thrift_used_clients{name="backend"} 0
starrocks_be_thrift_used_clients{name="broker"} 0
starrocks_be_thrift_used_clients{name="frontend"} 0
# TYPE starrocks_be_total_column_pool_bytes gauge
starrocks_be_total_column_pool_bytes 0
# TYPE starrocks_be_transaction_streaming_load_bytes counter
starrocks_be_transaction_streaming_load_bytes 137634
# TYPE starrocks_be_transaction_streaming_load_current_processing gauge
starrocks_be_transaction_streaming_load_current_processing 0
# TYPE starrocks_be_transaction_streaming_load_duration_ms counter
starrocks_be_transaction_streaming_load_duration_ms 0
# TYPE starrocks_be_transaction_streaming_load_requests_total counter
starrocks_be_transaction_streaming_load_requests_total 19055824
# TYPE starrocks_be_txn_request counter
starrocks_be_txn_request{type="begin"} 374
starrocks_be_txn_request{type="commit"} 186
starrocks_be_txn_request{type="exec"} 186
starrocks_be_txn_request{type="rollback"} 4
# TYPE starrocks_be_uint8_column_pool_bytes gauge
starrocks_be_uint8_column_pool_bytes 0
# TYPE starrocks_be_unused_rowsets_count gauge
starrocks_be_unused_rowsets_count 0
# TYPE starrocks_be_update_compaction_duration_us counter
starrocks_be_update_compaction_duration_us{type="update"} 52265257
# TYPE starrocks_be_update_compaction_outputs_bytes_total counter
starrocks_be_update_compaction_outputs_bytes_total{type="update"} 1418109246
# TYPE starrocks_be_update_compaction_outputs_total counter
starrocks_be_update_compaction_outputs_total{type="update"} 62
# TYPE starrocks_be_update_del_vector_bytes_total gauge
starrocks_be_update_del_vector_bytes_total 95556
# TYPE starrocks_be_update_del_vector_deletes_new counter
starrocks_be_update_del_vector_deletes_new 137634
# TYPE starrocks_be_update_del_vector_deletes_total counter
starrocks_be_update_del_vector_deletes_total 19055824
# TYPE starrocks_be_update_del_vector_dels_num gauge
starrocks_be_update_del_vector_dels_num 0
# TYPE starrocks_be_update_del_vector_num gauge
starrocks_be_update_del_vector_num 2487
# TYPE starrocks_be_update_mem_bytes gauge
starrocks_be_update_mem_bytes 539584627
# TYPE starrocks_be_update_primary_index_bytes_total gauge
starrocks_be_update_primary_index_bytes_total 539489423
# TYPE starrocks_be_update_primary_index_num gauge
starrocks_be_update_primary_index_num 4
# TYPE starrocks_be_update_rowset_commit_apply_duration_us counter
starrocks_be_update_rowset_commit_apply_duration_us 1983017
# TYPE starrocks_be_update_rowset_commit_apply_total counter
starrocks_be_update_rowset_commit_apply_total 3194
# TYPE starrocks_be_update_rowset_commit_request_failed counter
starrocks_be_update_rowset_commit_request_failed 0
# TYPE starrocks_be_update_rowset_commit_request_total counter
starrocks_be_update_rowset_commit_request_total 3194
# TYPE starrocks_be_wg_pip_exe_driver_queue_len gauge
starrocks_be_wg_pip_exe_driver_queue_len 0
# TYPE starrocks_be_wg_pip_exe_poller_block_queue_len gauge
starrocks_be_wg_pip_exe_poller_block_queue_len 0
# TYPE starrocks_be_writable_blocks_total counter
starrocks_be_writable_blocks_total 0

请问是否有设置 enable_metric_calculator = false呢?

image
我看BE的节点参数配置是1呢。。

收到 研发看下有消息给您答复啊

starrocks_be_max_disk_io_util_percent这个监控项是每隔15s采集操作系统的/proc/diskstats数据,然后根据差值/15s,用来计算ioutil。

你好,请问可以详细说下是根据哪个值做的差值/15 得来的呢?这是我在服务器上查到的当前的磁盘信息

是根据倒数第二列的数据(也就是正数第13列),两次采样做差值然后除以采样周期计算得到ioutil. 参见:https://www.kernel.org/doc/Documentation/ABI/testing/procfs-diskstats

我搭了一个环境,给磁盘一定的写入压力, starrocks_be_max_disk_io_util_percent这个指标是可以和iostat里的ioutil对的上的

我也测试了,但是这个监控值一直是0 呀,查看/proc/diskstats数据也是有变化的,请问这个要如何排查呢?

假设你的数据目录是在SDX这个设备上。先这样做下验证,每5s curl一次metric接口,然后给SDX这个设备一个写入压力,比如用dd持续写入一个文件。观察下metric接口吐出的starrocks_be_max_disk_io_util_percent这个数据是否和iostat的数据保持一致

get_iostat.log (87.5 KB) get_metrics.log (10.7 KB)

这是每个5秒 分别查看的 iostat 的信息和 监控 starrocks_be_max_disk_io_util_percent 的值,大概是在 11:26 的时候进行的数据写入。麻烦看一下 starrocks_be_max_disk_io_util_percent 确实一直是 0

enable_system_metrics 这个配置开启了吗

image
参数配置显示是 1

你的storage_root_path是怎么配置的,配置到哪个设备了?

image
配置在根目录下了,磁盘设备应该是 sda2 和 sdb1

@U_1668579146121_1345 hi 这个问题还存在吗

目前还没解决呢。。

@U_1668579146121_1345 你微信多少?微信沟通吧。

BE日志有打印类似下面的日志吗:
Current memory statistics: process(226956400), query_pool(0), load(0), metadata(41267), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), clone(0), consistency(0)


这是BE日志中的一小段,好像有这部分内容