【tpc-h性能】StarRocks3.0 on k8s集群的TPC-H测试性能问题

💬 StarRocks 用户问答
#1

【详述】
在k8s上部署了一个StarRocks3.0集群，配置如下：
apiVersion: v1
kind: ConfigMap
metadata:
name: starrocks-configmap
namespace: starrocks
data:
be.conf: |
sys_log_level = INFO
be_port = 9060
webserver_port = 8040
heartbeat_service_port = 9050
brpc_port = 8060
storage_root_path = /data1,medium:ssd;/data2,medium:ssd
brpc_max_body_size = 8589934592
fe.conf: |
# the output dir of stderr/stdout/gc
LOG_DIR = ${STARROCKS_HOME}/log
DATE = “$(date +%Y%m%d-%H%M%S)”
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:${LOG_DIR}/fe.gc.log.$DATE -XX:+PrintConcurrentLocks"
# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
sys_log_level = INFO
http_port = 8030
rpc_port = 9020
query_port = 9030
edit_log_port = 9010
mysql_service_nio_enabled = true
meta_dir = /data1
—
apiVersion: starrocks.com/v1alpha1
kind: StarRocksCluster
metadata:
name: starrockscluster-sample
namespace: starrocks
spec:
starRocksFeSpec:
image: fe-ubuntu:3.0.0
replicas: 3
requests:
cpu: 4
memory: 16Gi
storageVolumes:
- name: starrocks-fe-data1
storageClassName: local-path
storageSize: 256Gi
mountPath: /data1
nodeSelector:
starrocks-schedulable: “true”
configMapInfo:
configMapName: starrocks-configmap
resolveKey: fe.conf
service:
type: NodePort
starRocksBeSpec:
image: be-ubuntu:3.0.0
replicas: 10
requests:
cpu: 4
memory: 16Gi
limits:
cpu: 16
memory: 64Gi
storageVolumes:
- name: starrocks-be-data1
storageClassName: local-path
storageSize: 256Gi
mountPath: /data1
- name: starrocks-be-data2
storageClassName: local-path
storageSize: 256Gi
mountPath: /data2
nodeSelector:
starrocks-schedulable: “true”
configMapInfo:
configMapName: starrocks-configmap
resolveKey: be.conf

参考https://docs.starrocks.io/en-us/latest/benchmarking/TPC-H_Benchmarking进行了benchmark测试，结果与文档差距较大，结果如下所示：
root@mysql-client:/data1/tpch-poc-0.1.2/benchmark# bash ./bin/benchmark.sh -p -d tpch
[INFO] 2023-05-15 07:37:23 benchmark.py[217] benchmark args:Namespace(performance=True, check_result=False, scale=100, dataset=‘tpch’, sql_file=’’, log_quiet=False, log_verbose=False)
[INFO] 2023-05-15 07:37:23 benchmark.py[97] test sql in dirs:[tpch]
[INFO] 2023-05-15 07:37:23 starrocks_lib.py[266] get sql info from sql_dir:/data1/tpch-poc-0.1.2/benchmark/sql/tpch/query/tpch
------ dataset: tpch, concurrency: 1 ------
sql\time(ms)\parallel_num 8
q1 6034.0
q2 358.0
q3 1842.0
q4 1310.0
q5 3672.0
q6 120.0
q7 1837.0
q8 1649.0
q9 4746.0
q10 1452.0
q11 521.0
q12 627.0
q13 5002.0
q14 523.0
q15 739.0
q16 572.0
q17 1245.0
q18 9418.0
q19 1049.0
q20 622.0
q21 4226.0
q22 792.0

【StarRocks版本】3.0
【集群规模】3fe + 3be
【机器信息】k8s集群物理机配置为96核CPU(Intel®Xeon®Gold5220R@2.20GHz)，512G内存，SSD数据盘

1赞
#2

物理机器有几个节点, 看到BE的replica配置为10, 这10个POD写的SSD盘对应物理机的盘是同一个SSD盘吗?

#3

BE实际为3个副本，贴的配置忘记更新了

物理机节点60个，卷用的local-pv，不是同一个盘

#4

是否有条件在BE的pod里跑一跑fio测试，压测一下挂上去的/data1,/data2的裸跑性能是本地盘SSD的性能？

#5

root@ubuntu-fio:/data1# fio --name=disk-test --directory=/data1 --ioengine=libaio --rw=write --bs=1m --size=1g --numjobs=32 --direct=1 --group_reporting
fio --name=disk-test --directory=/data1 --ioengine=libaio --rw=read --bs=1m --size=1g --numjobs=32 --direct=1 --group_reporting
fio --name=disk-test --directory=/data1 --ioengine=libaio --rw=randwrite --bs=1m --size=1g --numjobs=32 --direct=1 --group_reporting
fio --name=disk-test --directory=/data1 --ioengine=libaio --rw=randread --bs=1m --size=1g --numjobs=32 --direct=1 --group_reporting
disk-test: (g=0): rw=write, bs=® 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
…
fio-3.28
Starting 32 processes
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
disk-test: Laying out IO file (1 file / 1024MiB)
Jobs: 21 (f=21): [(1),W(1),(4),W(3),(1),W(1),(1),W(3),(1),W(3),(1),W(2),(1),W(6),(1),W(2)][96.9%][w=475MiB/s][w=475 IOPS][eta 00m:02s]
disk-test: (groupid=0, jobs=32): err= 0: pid=80: Tue May 16 06:27:51 2023
write: IOPS=513, BW=514MiB/s (539MB/s)(32.0GiB/63758msec); 0 zone resets
slat (usec): min=36, max=92751, avg=138.67, stdev=1554.63
clat (usec): min=208, max=182130, avg=61313.26, stdev=34237.18
lat (usec): min=257, max=209130, avg=61452.29, stdev=34256.99
clat percentiles (usec):
| 1.00th=[ 1205], 5.00th=[ 4621], 10.00th=[ 5080], 20.00th=[ 5342],
| 30.00th=[ 65274], 40.00th=[ 67634], 50.00th=[ 67634], 60.00th=[ 67634],
| 70.00th=[ 91751], 80.00th=[ 91751], 90.00th=[ 92799], 95.00th=[ 93848],
| 99.00th=[120062], 99.50th=[128451], 99.90th=[166724], 99.95th=[173016],
| 99.99th=[179307]
bw ( KiB/s): min=260096, max=5601280, per=100.00%, avg=531616.62, stdev=19236.15, samples=4008
iops : min= 254, max= 5470, avg=519.12, stdev=18.78, samples=4008
lat (usec) : 250=0.23%, 500=0.21%, 750=0.16%, 1000=0.18%
lat (msec) : 2=0.88%, 4=2.57%, 10=18.42%, 20=0.34%, 50=2.64%
lat (msec) : 100=71.29%, 250=3.07%
cpu : usr=0.11%, sys=0.09%, ctx=33374, majf=0, minf=431
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=514MiB/s (539MB/s), 514MiB/s-514MiB/s (539MB/s-539MB/s), io=32.0GiB (34.4GB), run=63758-63758msec

Disk stats (read/write):
sde: ios=0/32895, merge=0/2271, ticks=0/1978649, in_queue=1978649, util=99.74%
disk-test: (g=0): rw=read, bs=® 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
…
fio-3.28
Starting 32 processes
Jobs: 8 (f=4): [f(1),(1),R(1),f(1),R(1),f(1),(3),f(1),(1),R(1),(10),R(1),_(9)][95.7%][r=1600MiB/s][r=1599 IOPS][eta 00m:03s]
disk-test: (groupid=0, jobs=32): err= 0: pid=170: Tue May 16 06:28:57 2023
read: IOPS=496, BW=497MiB/s (521MB/s)(32.0GiB/65968msec)
slat (usec): min=25, max=994, avg=42.28, stdev=23.94
clat (usec): min=202, max=102735, avg=63629.21, stdev=21133.70
lat (usec): min=230, max=102779, avg=63671.74, stdev=21136.72
clat percentiles (usec):
| 1.00th=[ 217], 5.00th=[ 225], 10.00th=[ 57410], 20.00th=[ 65274],
| 30.00th=[ 65274], 40.00th=[ 65274], 50.00th=[ 65274], 60.00th=[ 65274],
| 70.00th=[ 65274], 80.00th=[ 66323], 90.00th=[ 91751], 95.00th=[ 93848],
| 99.00th=[ 95945], 99.50th=[ 96994], 99.90th=[100140], 99.95th=[100140],
| 99.99th=[101188]
bw ( KiB/s): min=327680, max=2293760, per=95.40%, avg=485241.39, stdev=6932.38, samples=4152
iops : min= 320, max= 2240, avg=473.87, stdev= 6.77, samples=4152
lat (usec) : 250=6.35%, 500=1.17%
lat (msec) : 2=0.01%, 4=0.05%, 10=0.08%, 20=0.22%, 50=1.47%
lat (msec) : 100=90.63%, 250=0.02%
cpu : usr=0.01%, sys=0.09%, ctx=33100, majf=0, minf=8621
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=497MiB/s (521MB/s), 497MiB/s-497MiB/s (521MB/s-521MB/s), io=32.0GiB (34.4GB), run=65968-65968msec

Disk stats (read/write):
sde: ios=32594/88, merge=0/221, ticks=2084767/186, in_queue=2084953, util=99.85%
disk-test: (g=0): rw=randwrite, bs=® 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
…
fio-3.28
Starting 32 processes
Jobs: 17 (f=16): [(2),w(2),(1),f(1),(1),w(1),(6),w(2),(1),w(3),(1),w(2),(1),w(2),(1),w(3),_(1),w(1)][98.5%][w=476MiB/s][w=476 IOPS][eta 00m:01s]
disk-test: (groupid=0, jobs=32): err= 0: pid=260: Tue May 16 06:30:03 2023
write: IOPS=502, BW=503MiB/s (527MB/s)(32.0GiB/65202msec); 0 zone resets
slat (usec): min=35, max=91109, avg=509.38, stdev=4680.56
clat (usec): min=201, max=125798, avg=62274.99, stdev=32058.63
lat (usec): min=253, max=207880, avg=62784.69, stdev=32288.82
clat percentiles (usec):
| 1.00th=[ 832], 5.00th=[ 4555], 10.00th=[ 5080], 20.00th=[ 5538],
| 30.00th=[ 66847], 40.00th=[ 67634], 50.00th=[ 67634], 60.00th=[ 67634],
| 70.00th=[ 91751], 80.00th=[ 91751], 90.00th=[ 92799], 95.00th=[ 92799],
| 99.00th=[100140], 99.50th=[100140], 99.90th=[102237], 99.95th=[110625],
| 99.99th=[121111]
bw ( KiB/s): min=301056, max=4689920, per=100.00%, avg=520735.19, stdev=16976.55, samples=4096
iops : min= 294, max= 4580, avg=508.53, stdev=16.58, samples=4096
lat (usec) : 250=0.48%, 500=0.24%, 750=0.23%, 1000=0.63%
lat (msec) : 2=0.61%, 4=2.01%, 10=15.99%, 20=0.28%, 50=2.87%
lat (msec) : 100=75.54%, 250=1.13%
cpu : usr=0.10%, sys=0.08%, ctx=33555, majf=0, minf=428
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=503MiB/s (527MB/s), 503MiB/s-503MiB/s (527MB/s-527MB/s), io=32.0GiB (34.4GB), run=65202-65202msec

Disk stats (read/write):
sde: ios=0/32756, merge=0/39, ticks=0/2041587, in_queue=2041587, util=99.86%
disk-test: (g=0): rw=randread, bs=® 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
…
fio-3.28
Starting 32 processes
Jobs: 15 (f=15): [r(2),(2),r(3),(1),r(6),(6),r(1),(2),r(2),(5),r(1),(1)][94.4%][r=1108MiB/s][r=1108 IOPS][eta 00m:04s]
disk-test: (groupid=0, jobs=32): err= 0: pid=348: Tue May 16 06:31:11 2023
read: IOPS=485, BW=486MiB/s (509MB/s)(32.0GiB/67439msec)
slat (usec): min=25, max=3006, avg=46.04, stdev=36.73
clat (usec): min=196, max=111485, avg=65200.04, stdev=20454.47
lat (usec): min=231, max=111513, avg=65246.36, stdev=20456.95
clat percentiles (usec):
| 1.00th=[ 219], 5.00th=[ 237], 10.00th=[ 63701], 20.00th=[ 65274],
| 30.00th=[ 65274], 40.00th=[ 65274], 50.00th=[ 65274], 60.00th=[ 65274],
| 70.00th=[ 65274], 80.00th=[ 66323], 90.00th=[ 92799], 95.00th=[ 94897],
| 99.00th=[ 96994], 99.50th=[ 98042], 99.90th=[101188], 99.95th=[101188],
| 99.99th=[107480]
bw ( KiB/s): min=327680, max=2897962, per=98.13%, avg=488234.24, stdev=7366.67, samples=4257
iops : min= 320, max= 2827, avg=476.53, stdev= 7.19, samples=4257
lat (usec) : 250=5.27%, 500=1.05%, 750=0.05%, 1000=0.01%
lat (msec) : 4=0.06%, 10=0.02%, 20=0.26%, 50=1.15%, 100=91.86%
lat (msec) : 250=0.26%
cpu : usr=0.02%, sys=0.10%, ctx=33151, majf=0, minf=8570
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=486MiB/s (509MB/s), 486MiB/s-486MiB/s (509MB/s-509MB/s), io=32.0GiB (34.4GB), run=67439-67439msec

Disk stats (read/write):
sde: ios=32682/116, merge=0/235, ticks=2136215/4671, in_queue=2140886, util=99.85%

#6

测试了下，基本是符合预期性能的

#7

大佬有空帮忙看下这个问题吗？

#8

可以手动跑其中一个单独的query, 提供一下query的profile, 我们这边工程师可以根据profile进行分析对比.

#9

以TPC-H q1为例，我这边测试是需要跑6秒，profile如下，大佬帮忙分析下

Query:
Summary:
- Query ID: 911072f7-f944-11ed-82d9-a69c7b9835dc
- Start Time: 2023-05-23 08:34:00
- End Time: 2023-05-23 08:34:06
- Total: 6s451ms
- Query Type: Query
- Query State: EOF
- StarRocks Version: 3.0.0-48f4d81d7b
- User: root
- Default Db: tpch
- Sql Statement: select l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extendedprice * (1 - l_discount)) as sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discount) as avg_disc, count(*) as count_order from lineitem where l_shipdate <= date ‘1998-12-01’ group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus;
- Variables: parallel_fragment_exec_instance_num=8,max_parallel_scan_instance_num=-1,pipeline_dop=0,enable_adaptive_sink_dop=true,enable_runtime_adaptive_dop=false
- Collect Profile Time: 93ms
Planner:
- Analyzer: 1ms / 1
- CoordDeliverExec: 64ms / 1
- CoordPrepareExec: 2ms / 1
- ExecPlanBuild: 0ms / 1
- Optimizer: 2ms / 1
- Total: 5ms / 1
- Transformer: 0ms / 1
Optimizer:
- CostBaseOptimize: 1ms / 1
- PhysicalRewrite: 0ms / 1
- RuleBaseOptimize: 0ms / 1
- preprocessMvs: 0ms / 1
Execution Profile 911072f7-f944-11ed-82d9-a69c7b9835dc:
- QueryAllocatedMemoryUsage: 172.92 GB
- QueryCumulativeCpuTime: 2m58s
- QueryCumulativeOperatorTime: 15s285ms
- QueryDeallocatedMemoryUsage: 172.84 GB
- QueryExecutionWallTime: 6s411ms
- QueryPeakMemoryUsage: 963.31 MB
Fragment 0:
- BackendAddresses: starrockscluster-sample-be-1.starrockscluster-sample-be-search.starrocks.svc.cluster.local:9060
- BackendNum: 1
- FragmentInstancePrepareTime: 328.147us
- prepare-fragment-ctx: 81.612us
- prepare-pipeline-driver: 117.132us
- prepare-query-ctx: 9.6us
- prepare-runtime-state: 117.132us
- InstanceAllocatedMemoryUsage: 78.59 KB
- InstanceDeallocatedMemoryUsage: 72.94 KB
- InstanceNum: 1
- InstancePeakMemoryUsage: 16.82 KB
- QueryMemoryLimit: 46.66 GB
Pipeline (id=0):
- ActiveTime: 451.803us
- BlockByInputEmpty: 0
- BlockByOutputFull: 0
- BlockByPrecondition: 0
- DegreeOfParallelism: 1
- DriverPrepareTime: 217.889us
- DriverTotalTime: 6s411ms
- OverheadTime: 34.495us
- PendingTime: 6s411ms
- InputEmptyTime: 6s411ms
- FirstInputEmptyTime: 6s411ms
- FollowupInputEmptyTime: 0ns
- OutputFullTime: 0ns
- PendingFinishTime: 0ns
- PreconditionBlockTime: 0ns
- ScheduleCount: 1
- ScheduleTime: 32.364us
- TotalDegreeOfParallelism: 1
- YieldByPreempt: 0
- YieldByTimeLimit: 0
RESULT_SINK:
CommonMetrics:
- CloseTime: 5.235us
- OperatorAllocatedMemoryUsage: 9.97 KB
- OperatorDeallocatedMemoryUsage: 9.38 KB
- OperatorPeakMemoryUsage: 600.00 B
- OperatorTotalTime: 122.18us
- PrepareTime: 25.586us
- PullChunkNum: 0
- PullRowNum: 0
- PullTotalTime: 0ns
- PushChunkNum: 4
- PushRowNum: 4
- PushTotalTime: 116.560us
- SetFinishedTime: 29ns
- SetFinishingTime: 194ns
UniqueMetrics:
GLOBAL_MERGE_SOURCE (plan_node_id=6):
CommonMetrics:
- CloseTime: 381ns
- JoinRuntimeFilterEvaluate: 0
- JoinRuntimeFilterHashTime: 0ns
- JoinRuntimeFilterInputRows: 0
- JoinRuntimeFilterOutputRows: 0
- JoinRuntimeFilterTime: 0ns
- OperatorAllocatedMemoryUsage: 62.08 KB
- OperatorDeallocatedMemoryUsage: 58.99 KB
- OperatorPeakMemoryUsage: 4.09 KB
- OperatorTotalTime: 295.290us
- PrepareTime: 161.110us
- PullChunkNum: 4
- PullRowNum: 4
- PullTotalTime: 286.368us
- PushChunkNum: 0
- PushRowNum: 0
- PushTotalTime: 0ns
- RuntimeBloomFilterNum: 0
- RuntimeInFilterNum: 0
- SetFinishedTime: 125ns
- SetFinishingTime: 8.416us
UniqueMetrics:
- BufferUnplugCount: 0
- BytesPassThrough: 0.00
- BytesReceived: 980.00 B
- ClosureBlockCount: 0
- ClosureBlockTime: 0ns
- DecompressChunkTime: 0ns
- DeserializeChunkTime: 151.957us
- ReceiverProcessTotalTime: 204.150us
- RequestReceived: 3
- SenderTotalTime: 203.445us
- SenderWaitLockTime: 1.144us
Fragment 1:
- BackendAddresses: starrockscluster-sample-be-1.starrockscluster-sample-be-search.starrocks.svc.cluster.local:9060,starrockscluster-sample-be-2.starrockscluster-sample-be-search.starrocks.svc.cluster.local:9060,starrockscluster-sample-be-0.starrockscluster-sample-be-search.starrocks.svc.cluster.local:9060
- BackendNum: 3
- FragmentInstancePrepareTime: 818.951us
- __MAX_OF_FragmentInstancePrepareTime: 1.17ms
- __MIN_OF_FragmentInstancePrepareTime: 699.151us
- prepare-fragment-ctx: 65.826us
- __MAX_OF_prepare-fragment-ctx: 122.702us
- __MIN_OF_prepare-fragment-ctx: 23.644us
- prepare-pipeline-driver: 137.702us
- __MAX_OF_prepare-pipeline-driver: 175.9us
- __MIN_OF_prepare-pipeline-driver: 79.535us
- prepare-query-ctx: 8.768us
- __MAX_OF_prepare-query-ctx: 12.457us
- __MIN_OF_prepare-query-ctx: 6.27us
- prepare-runtime-state: 137.702us
- __MAX_OF_prepare-runtime-state: 175.9us
- __MIN_OF_prepare-runtime-state: 79.535us
- InstanceAllocatedMemoryUsage: 1.31 MB
- __MAX_OF_InstanceAllocatedMemoryUsage: 964.51 KB
- __MIN_OF_InstanceAllocatedMemoryUsage: 33.05 KB
- InstanceDeallocatedMemoryUsage: 865.17 KB
- __MAX_OF_InstanceDeallocatedMemoryUsage: 629.40 KB
- __MIN_OF_InstanceDeallocatedMemoryUsage: 14.05 KB
- InstanceNum: 3
- InstancePeakMemoryUsage: 943.68 KB
- __MAX_OF_InstancePeakMemoryUsage: 2.06 MB
- __MIN_OF_InstancePeakMemoryUsage: 10.58 KB
- QueryMemoryLimit: 32.00 GB
Pipeline (id=2):
- ActiveTime: 247.106us
- __MAX_OF_ActiveTime: 352.524us
- __MIN_OF_ActiveTime: 96.908us
- BlockByInputEmpty: 0
- BlockByOutputFull: 0
- BlockByPrecondition: 0
- DegreeOfParallelism: 1
- DriverPrepareTime: 669.528us
- __MAX_OF_DriverPrepareTime: 961.477us
- __MIN_OF_DriverPrepareTime: 218.615us
- DriverTotalTime: 6s403ms
- __MAX_OF_DriverTotalTime: 6s405ms
- __MIN_OF_DriverTotalTime: 6s401ms
- OverheadTime: 0ns
- PendingTime: 6s402ms
- __MAX_OF_PendingTime: 6s405ms
- __MIN_OF_PendingTime: 6s401ms
- InputEmptyTime: 6s402ms
- __MAX_OF_InputEmptyTime: 6s405ms
- __MIN_OF_InputEmptyTime: 6s400ms
- FirstInputEmptyTime: 6s402ms
- __MAX_OF_FirstInputEmptyTime: 6s405ms
- __MIN_OF_FirstInputEmptyTime: 6s400ms
- FollowupInputEmptyTime: 0ns
- OutputFullTime: 0ns
- PendingFinishTime: 633.256us
- __MAX_OF_PendingFinishTime: 1.3ms
- __MIN_OF_PendingFinishTime: 159.515us
- PreconditionBlockTime: 0ns
- ScheduleCount: 3
- __MAX_OF_ScheduleCount: 1
- __MIN_OF_ScheduleCount: 1
- ScheduleTime: 194.272us
- __MAX_OF_ScheduleTime: 300.11us
- __MIN_OF_ScheduleTime: 135.401us
- TotalDegreeOfParallelism: 3
- __MAX_OF_TotalDegreeOfParallelism: 1
- __MIN_OF_TotalDegreeOfParallelism: 1
- YieldByPreempt: 0
- YieldByTimeLimit: 0
EXCHANGE_SINK (plan_node_id=6):
CommonMetrics:
- CloseTime: 19.227us
- __MAX_OF_CloseTime: 21.529us
- __MIN_OF_CloseTime: 17.653us
- OperatorAllocatedMemoryUsage: 23.84 KB
- __MAX_OF_OperatorAllocatedMemoryUsage: 11.31 KB
- __MIN_OF_OperatorAllocatedMemoryUsage: 4.28 KB
- OperatorDeallocatedMemoryUsage: 12.47 KB
- __MAX_OF_OperatorDeallocatedMemoryUsage: 7.22 KB
- __MIN_OF_OperatorDeallocatedMemoryUsage: 1.09 KB
- OperatorPeakMemoryUsage: 3.79 KB
- __MAX_OF_OperatorPeakMemoryUsage: 4.09 KB
- __MIN_OF_OperatorPeakMemoryUsage: 3.19 KB
- OperatorTotalTime: 115.187us
- __MAX_OF_OperatorTotalTime: 140.945us
- __MIN_OF_OperatorTotalTime: 79.458us
- PrepareTime: 632.866us
- __MAX_OF_PrepareTime: 927.946us
- __MIN_OF_PrepareTime: 182.998us
- PullChunkNum: 0
- PullRowNum: 0
- PullTotalTime: 0ns
- PushChunkNum: 4
- __MAX_OF_PushChunkNum: 3
- __MIN_OF_PushChunkNum: 0
- PushRowNum: 4
- __MAX_OF_PushRowNum: 3
- __MIN_OF_PushRowNum: 0
- PushTotalTime: 16.368us
- __MAX_OF_PushTotalTime: 26.185us
- __MIN_OF_PushTotalTime: 0ns
- SetFinishedTime: 124ns
- __MAX_OF_SetFinishedTime: 167ns
- __MIN_OF_SetFinishedTime: 94ns
- SetFinishingTime: 79.467us
- __MAX_OF_SetFinishingTime: 97.13us
- __MIN_OF_SetFinishingTime: 60.792us
UniqueMetrics:
- DestID: 6
- DestFragments: 911072f7f94411ed-82d9a69c7b9835e3
- PartType: UNPARTITIONED
- ChannelNum: 1
- BytesPassThrough: 0.00
- BytesSent: 980.00 B
- __MAX_OF_BytesSent: 735.00 B
- __MIN_OF_BytesSent: 0.00
- CompressTime: 0ns
- NetworkBandwidth: 1.723062515258789 MB/sec
- __MAX_OF_NetworkBandwidth: 1.723062515258789 MB/sec
- __MIN_OF_NetworkBandwidth: 0.0 /sec
- NetworkTime: 176.450us
- __MAX_OF_NetworkTime: 406.805us
- __MIN_OF_NetworkTime: 0ns
- OverallThroughput: 957.9970703125 KB/sec
- __MAX_OF_OverallThroughput: 668.65234375 KB/sec
- __MIN_OF_OverallThroughput: 0.0 /sec
- OverallTime: 703.559us
- __MAX_OF_OverallTime: 1.73ms
- __MIN_OF_OverallTime: 210.321us
- RequestSent: 2
- __MAX_OF_RequestSent: 1
- __MIN_OF_RequestSent: 0
- RpcAvgTime: 20.509us
- __MAX_OF_RpcAvgTime: 203.402us
- __MIN_OF_RpcAvgTime: -264420ns
- RpcCount: 5
- __MAX_OF_RpcCount: 2
- __MIN_OF_RpcCount: 1
- SerializeChunkTime: 12.647us
- __MAX_OF_SerializeChunkTime: 20.373us
- __MIN_OF_SerializeChunkTime: 0ns
- ShuffleHashTime: 0ns
- UncompressedBytes: 980.00 B
- __MAX_OF_UncompressedBytes: 735.00 B
- __MIN_OF_UncompressedBytes: 0.00
- WaitTime: 702.910us
- __MAX_OF_WaitTime: 1.72ms
- __MIN_OF_WaitTime: 224.652us

#10 
    LOCAL_MERGE_SOURCE (plan_node_id=5):
      CommonMetrics:
         - CloseTime: 25.128us
           - __MAX_OF_CloseTime: 29.622us
           - __MIN_OF_CloseTime: 18.797us
         - OperatorAllocatedMemoryUsage: 107.30 KB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 77.50 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 2.36 KB
         - OperatorDeallocatedMemoryUsage: 133.37 KB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 86.21 KB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 11.05 KB
         - OperatorPeakMemoryUsage: 2.23 KB
           - __MAX_OF_OperatorPeakMemoryUsage: 2.26 KB
           - __MIN_OF_OperatorPeakMemoryUsage: 2.18 KB
         - OperatorTotalTime: 163.173us
           - __MAX_OF_OperatorTotalTime: 249.945us
           - __MIN_OF_OperatorTotalTime: 53.527us
         - PrepareTime: 17.231us
           - __MAX_OF_PrepareTime: 25.580us
           - __MIN_OF_PrepareTime: 11.374us
         - PullChunkNum: 4
           - __MAX_OF_PullChunkNum: 3
           - __MIN_OF_PullChunkNum: 0
         - PullRowNum: 4
           - __MAX_OF_PullRowNum: 3
           - __MIN_OF_PullRowNum: 0
         - PullTotalTime: 137.678us
           - __MAX_OF_PullTotalTime: 230.784us
           - __MIN_OF_PullTotalTime: 26.291us
         - PushChunkNum: 0
         - PushRowNum: 0
         - PushTotalTime: 0ns
         - SetFinishedTime: 101ns
           - __MAX_OF_SetFinishedTime: 130ns
           - __MIN_OF_SetFinishedTime: 83ns
         - SetFinishingTime: 265ns
           - __MAX_OF_SetFinishingTime: 336ns
           - __MIN_OF_SetFinishingTime: 187ns
      UniqueMetrics:
  Pipeline (id=1):
     - ActiveTime: 42.70us
       - __MAX_OF_ActiveTime: 199.893us
       - __MIN_OF_ActiveTime: 5.507us
     - BlockByInputEmpty: 0
     - BlockByOutputFull: 0
     - BlockByPrecondition: 0
     - DegreeOfParallelism: 8
     - DriverPrepareTime: 54.251us
       - __MAX_OF_DriverPrepareTime: 70.662us
       - __MIN_OF_DriverPrepareTime: 31.472us
     - DriverTotalTime: 6s402ms
       - __MAX_OF_DriverTotalTime: 6s402ms
       - __MIN_OF_DriverTotalTime: 6s401ms
     - OverheadTime: 216ns
       - __MAX_OF_OverheadTime: 5.191us
       - __MIN_OF_OverheadTime: 0ns
     - PendingTime: 6s400ms
       - __MAX_OF_PendingTime: 6s402ms
       - __MIN_OF_PendingTime: 6s400ms
       - InputEmptyTime: 6s400ms
         - __MAX_OF_InputEmptyTime: 6s402ms
         - __MIN_OF_InputEmptyTime: 6s400ms
         - FirstInputEmptyTime: 6s400ms
           - __MAX_OF_FirstInputEmptyTime: 6s402ms
           - __MIN_OF_FirstInputEmptyTime: 6s400ms
         - FollowupInputEmptyTime: 0ns
       - OutputFullTime: 0ns
       - PendingFinishTime: 0ns
       - PreconditionBlockTime: 0ns
     - ScheduleCount: 24
       - __MAX_OF_ScheduleCount: 1
       - __MIN_OF_ScheduleCount: 1
     - ScheduleTime: 1.320ms
       - __MAX_OF_ScheduleTime: 1.605ms
       - __MIN_OF_ScheduleTime: 688.368us
     - TotalDegreeOfParallelism: 24
       - __MAX_OF_TotalDegreeOfParallelism: 8
       - __MIN_OF_TotalDegreeOfParallelism: 8
     - YieldByPreempt: 0
     - YieldByTimeLimit: 0
    LOCAL_SORT_SINK (plan_node_id=5):
      CommonMetrics:
         - CloseTime: 5.822us
           - __MAX_OF_CloseTime: 13.550us
           - __MIN_OF_CloseTime: 3.581us
         - OperatorAllocatedMemoryUsage: 60.28 KB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 9.76 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 1.06 KB
         - OperatorDeallocatedMemoryUsage: 27.28 KB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 5.73 KB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 224.00 B
         - OperatorPeakMemoryUsage: 1.38 KB
           - __MAX_OF_OperatorPeakMemoryUsage: 4.03 KB
           - __MIN_OF_OperatorPeakMemoryUsage: 864.00 B
         - OperatorTotalTime: 26.48us
           - __MAX_OF_OperatorTotalTime: 98.317us
           - __MIN_OF_OperatorTotalTime: 6.866us
         - PrepareTime: 22.980us
           - __MAX_OF_PrepareTime: 41.940us
           - __MIN_OF_PrepareTime: 11.71us
         - PullChunkNum: 0
         - PullRowNum: 0
         - PullTotalTime: 0ns
         - PushChunkNum: 4
           - __MAX_OF_PushChunkNum: 1
           - __MIN_OF_PushChunkNum: 0
         - PushRowNum: 4
           - __MAX_OF_PushRowNum: 1
           - __MIN_OF_PushRowNum: 0
         - PushTotalTime: 1.940us
           - __MAX_OF_PushTotalTime: 18.441us
           - __MIN_OF_PushTotalTime: 0ns
         - RuntimeBloomFilterNum: 0
         - RuntimeInFilterNum: 0
         - SetFinishedTime: 56ns
           - __MAX_OF_SetFinishedTime: 82ns
           - __MIN_OF_SetFinishedTime: 32ns
         - SetFinishingTime: 18.228us
           - __MAX_OF_SetFinishingTime: 75.893us
           - __MIN_OF_SetFinishingTime: 2.688us
      UniqueMetrics:
         - SortKeys: 10: l_returnflag ASC, 11: l_linestatus ASC
         - SortType: All
         - MaxBufferedRows: 1024000
         - MaxBufferedBytes: 16777216
         - LateMaterialization: false
         - BuildingTime: 281ns
           - __MAX_OF_BuildingTime: 2.732us
           - __MIN_OF_BuildingTime: 0ns
         - InputRequiredMemory: 548.00 B
           - __MAX_OF_InputRequiredMemory: 137.00 B
           - __MIN_OF_InputRequiredMemory: 0.00 
         - MergingTime: 6.743us
           - __MAX_OF_MergingTime: 39.470us
           - __MIN_OF_MergingTime: 2.132us
         - NumSortedRuns: 4
           - __MAX_OF_NumSortedRuns: 1
           - __MIN_OF_NumSortedRuns: 0
         - OutputTime: 626ns
           - __MAX_OF_OutputTime: 4.16us
           - __MIN_OF_OutputTime: 0ns
         - SortingTime: 5.659us
           - __MAX_OF_SortingTime: 36.871us
           - __MIN_OF_SortingTime: 0ns
    AGGREGATE_BLOCKING_SOURCE (plan_node_id=4):
      CommonMetrics:
         - CloseTime: 16.470us
           - __MAX_OF_CloseTime: 62.932us
           - __MIN_OF_CloseTime: 883ns
         - JoinRuntimeFilterEvaluate: 0
         - JoinRuntimeFilterHashTime: 0ns
         - JoinRuntimeFilterInputRows: 0
         - JoinRuntimeFilterOutputRows: 0
         - JoinRuntimeFilterTime: 0ns
         - OperatorAllocatedMemoryUsage: 294.81 KB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 70.58 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 640.00 B
         - OperatorDeallocatedMemoryUsage: 264.20 KB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 82.48 KB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 64.00 B
         - OperatorPeakMemoryUsage: 11.60 KB
           - __MAX_OF_OperatorPeakMemoryUsage: 69.05 KB
           - __MIN_OF_OperatorPeakMemoryUsage: 0.00 
         - OperatorTotalTime: 33.557us
           - __MAX_OF_OperatorTotalTime: 127.396us
           - __MIN_OF_OperatorTotalTime: 7.669us
         - PrepareTime: 13.16us
           - __MAX_OF_PrepareTime: 23.118us
           - __MIN_OF_PrepareTime: 7.401us
         - PullChunkNum: 4
           - __MAX_OF_PullChunkNum: 1
           - __MIN_OF_PullChunkNum: 0
         - PullRowNum: 4
           - __MAX_OF_PullRowNum: 1
           - __MIN_OF_PullRowNum: 0
         - PullTotalTime: 16.889us
           - __MAX_OF_PullTotalTime: 112.374us
           - __MIN_OF_PullTotalTime: 0ns
         - PushChunkNum: 0
         - PushRowNum: 0
         - PushTotalTime: 0ns
         - RuntimeBloomFilterNum: 0
         - RuntimeInFilterNum: 0
         - SetFinishedTime: 107ns
           - __MAX_OF_SetFinishedTime: 437ns
           - __MIN_OF_SetFinishedTime: 38ns
         - SetFinishingTime: 89ns
           - __MAX_OF_SetFinishingTime: 144ns
           - __MIN_OF_SetFinishingTime: 45ns
      UniqueMetrics:
  Pipeline (id=0):
     - ActiveTime: 108.912us
       - __MAX_OF_ActiveTime: 629.890us
       - __MIN_OF_ActiveTime: 3.843us
     - BlockByInputEmpty: 6
       - __MAX_OF_BlockByInputEmpty: 2
       - __MIN_OF_BlockByInputEmpty: 0
     - BlockByOutputFull: 0
     - BlockByPrecondition: 0
     - DegreeOfParallelism: 8
     - DriverPrepareTime: 192.304us
       - __MAX_OF_DriverPrepareTime: 415.486us
       - __MIN_OF_DriverPrepareTime: 128.31us
     - DriverTotalTime: 6s402ms
       - __MAX_OF_DriverTotalTime: 6s403ms
       - __MIN_OF_DriverTotalTime: 6s401ms
     - OverheadTime: 6.407us
       - __MAX_OF_OverheadTime: 53.154us
       - __MIN_OF_OverheadTime: 0ns
     - PendingTime: 6s400ms
       - __MAX_OF_PendingTime: 6s400ms
       - __MIN_OF_PendingTime: 6s399ms
       - InputEmptyTime: 6s400ms
         - __MAX_OF_InputEmptyTime: 6s400ms
         - __MIN_OF_InputEmptyTime: 6s399ms
         - FirstInputEmptyTime: 6s329ms
           - __MAX_OF_FirstInputEmptyTime: 6s400ms
           - __MIN_OF_FirstInputEmptyTime: 5s975ms
         - FollowupInputEmptyTime: 70.704ms
           - __MAX_OF_FollowupInputEmptyTime: 424.257ms
           - __MIN_OF_FollowupInputEmptyTime: 0ns
       - OutputFullTime: 0ns
       - PendingFinishTime: 0ns
       - PreconditionBlockTime: 0ns
     - ScheduleCount: 30
       - __MAX_OF_ScheduleCount: 3
       - __MIN_OF_ScheduleCount: 1
     - ScheduleTime: 2.16ms
       - __MAX_OF_ScheduleTime: 2.939ms
       - __MIN_OF_ScheduleTime: 921.853us
     - TotalDegreeOfParallelism: 24
       - __MAX_OF_TotalDegreeOfParallelism: 8
       - __MIN_OF_TotalDegreeOfParallelism: 8
     - YieldByPreempt: 0
     - YieldByTimeLimit: 0
    AGGREGATE_BLOCKING_SINK (plan_node_id=4):
      CommonMetrics:
         - CloseTime: 14.414us
           - __MAX_OF_CloseTime: 50.97us
           - __MIN_OF_CloseTime: 8.130us
         - OperatorAllocatedMemoryUsage: 88.13 KB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 17.19 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 992.00 B
         - OperatorDeallocatedMemoryUsage: 236.27 KB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 118.44 KB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 96.00 B
         - OperatorPeakMemoryUsage: 3.32 KB
           - __MAX_OF_OperatorPeakMemoryUsage: 16.16 KB
           - __MIN_OF_OperatorPeakMemoryUsage: 32.00 B
         - OperatorTotalTime: 61.390us
           - __MAX_OF_OperatorTotalTime: 318.881us
           - __MIN_OF_OperatorTotalTime: 9.20us
         - PrepareTime: 133.28us
           - __MAX_OF_PrepareTime: 169.993us
           - __MIN_OF_PrepareTime: 100.976us
         - PullChunkNum: 0
         - PullRowNum: 0
         - PullTotalTime: 0ns
         - PushChunkNum: 96
           - __MAX_OF_PushChunkNum: 24
           - __MIN_OF_PushChunkNum: 0
         - PushRowNum: 96
           - __MAX_OF_PushRowNum: 24
           - __MIN_OF_PushRowNum: 0
         - PushTotalTime: 45.373us
           - __MAX_OF_PushTotalTime: 287.960us
           - __MIN_OF_PushTotalTime: 0ns
         - RuntimeBloomFilterNum: 0
         - RuntimeInFilterNum: 0
         - SetFinishedTime: 30ns
           - __MAX_OF_SetFinishedTime: 45ns
           - __MIN_OF_SetFinishedTime: 22ns
         - SetFinishingTime: 1.571us
           - __MAX_OF_SetFinishingTime: 4.184us
           - __MIN_OF_SetFinishingTime: 699ns
      UniqueMetrics:
         - GroupingKeys: 10: l_returnflag, 11: l_linestatus
         - AggregateFunctions: 10: l_returnflag, 11: l_linestatus
         - AggComputeTime: 25.764us
           - __MAX_OF_AggComputeTime: 163.396us
           - __MIN_OF_AggComputeTime: 0ns
         - AggFuncComputeTime: 14.981us
           - __MAX_OF_AggFuncComputeTime: 92.766us
           - __MIN_OF_AggFuncComputeTime: 0ns
         - ExprComputeTime: 10.310us
           - __MAX_OF_ExprComputeTime: 66.716us
           - __MIN_OF_ExprComputeTime: 0ns
         - ExprReleaseTime: 15.249us
           - __MAX_OF_ExprReleaseTime: 115.909us
           - __MIN_OF_ExprReleaseTime: 0ns
         - GetResultsTime: 13.268us
           - __MAX_OF_GetResultsTime: 90.35us
           - __MIN_OF_GetResultsTime: 0ns
         - HashTableMemoryUsage: 2.02 MB
           - __MAX_OF_HashTableMemoryUsage: 516.06 KB
           - __MIN_OF_HashTableMemoryUsage: 24.00 B
         - HashTableSize: 4
           - __MAX_OF_HashTableSize: 1
           - __MIN_OF_HashTableSize: 0
         - InputRowCount: 96
           - __MAX_OF_InputRowCount: 24
           - __MIN_OF_InputRowCount: 0
         - PassThroughRowCount: 0
         - ResultAggAppendTime: 1.82us
           - __MAX_OF_ResultAggAppendTime: 14.58us
           - __MIN_OF_ResultAggAppendTime: 0ns
         - ResultGroupByAppendTime: 331ns
           - __MAX_OF_ResultGroupByAppendTime: 2.941us
           - __MIN_OF_ResultGroupByAppendTime: 0ns
         - ResultIteratorTime: 8.143us
           - __MAX_OF_ResultIteratorTime: 66.695us
           - __MIN_OF_ResultIteratorTime: 0ns
         - RowsReturned: 0
         - StateAllocate: 0ns
         - StateDestroy: 0ns
         - StreamingTime: 0ns
    EXCHANGE_SOURCE (plan_node_id=3):
      CommonMetrics:
         - CloseTime: 7.854us
           - __MAX_OF_CloseTime: 15.875us
           - __MIN_OF_CloseTime: 376ns
         - JoinRuntimeFilterEvaluate: 0
         - JoinRuntimeFilterHashTime: 0ns
         - JoinRuntimeFilterInputRows: 0
         - JoinRuntimeFilterOutputRows: 0
         - JoinRuntimeFilterTime: 0ns
         - OperatorAllocatedMemoryUsage: 238.75 KB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 56.56 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 640.00 B
         - OperatorDeallocatedMemoryUsage: 103.70 KB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 26.78 KB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 64.00 B
         - OperatorPeakMemoryUsage: 5.63 KB
           - __MAX_OF_OperatorPeakMemoryUsage: 32.28 KB
           - __MIN_OF_OperatorPeakMemoryUsage: 464.00 B
         - OperatorTotalTime: 55.622us
           - __MAX_OF_OperatorTotalTime: 294.78us
           - __MIN_OF_OperatorTotalTime: 5.669us
         - PrepareTime: 37.623us
           - __MAX_OF_PrepareTime: 220.875us
           - __MIN_OF_PrepareTime: 7.142us
         - PullChunkNum: 96
           - __MAX_OF_PullChunkNum: 24
           - __MIN_OF_PullChunkNum: 0
         - PullRowNum: 96
           - __MAX_OF_PullRowNum: 24
           - __MIN_OF_PullRowNum: 0
         - PullTotalTime: 45.854us
           - __MAX_OF_PullTotalTime: 292.183us
           - __MIN_OF_PullTotalTime: 0ns
         - PushChunkNum: 0
         - PushRowNum: 0
         - PushTotalTime: 0ns
         - RuntimeBloomFilterNum: 0
         - RuntimeInFilterNum: 0
         - SetFinishedTime: 95ns
           - __MAX_OF_SetFinishedTime: 231ns
           - __MIN_OF_SetFinishedTime: 23ns
         - SetFinishingTime: 1.818us
           - __MAX_OF_SetFinishingTime: 10.573us
           - __MIN_OF_SetFinishingTime: 349ns
      UniqueMetrics:
         - BufferUnplugCount: 4
           - __MAX_OF_BufferUnplugCount: 3
           - __MIN_OF_BufferUnplugCount: 0
         - BytesPassThrough: 8.25 KB
           - __MAX_OF_BytesPassThrough: 6.19 KB
           - __MIN_OF_BytesPassThrough: 0.00 
         - BytesReceived: 17.50 KB
           - __MAX_OF_BytesReceived: 13.13 KB
           - __MIN_OF_BytesReceived: 0.00 
         - ClosureBlockCount: 0
         - ClosureBlockTime: 0ns
         - DecompressChunkTime: 0ns
         - DeserializeChunkTime: 233.270us
           - __MAX_OF_DeserializeChunkTime: 514.277us
           - __MIN_OF_DeserializeChunkTime: 0ns
         - ReceiverProcessTotalTime: 301.575us
           - __MAX_OF_ReceiverProcessTotalTime: 611.631us
           - __MIN_OF_ReceiverProcessTotalTime: 7.305us
         - RequestReceived: 35
           - __MAX_OF_RequestReceived: 17
           - __MIN_OF_RequestReceived: 1
         - SenderTotalTime: 299.738us
           - __MAX_OF_SenderTotalTime: 609.481us
           - __MIN_OF_SenderTotalTime: 6.893us
         - SenderWaitLockTime: 8.244us
           - __MAX_OF_SenderWaitLockTime: 16.647us
           - __MIN_OF_SenderWaitLockTime: 366ns
#11 
Fragment 2:
   - BackendAddresses: starrockscluster-sample-be-1.starrockscluster-sample-be-search.starrocks.svc.cluster.local:9060,starrockscluster-sample-be-2.starrockscluster-sample-be-search.starrocks.svc.cluster.local:9060,starrockscluster-sample-be-0.starrockscluster-sample-be-search.starrocks.svc.cluster.local:9060
   - BackendNum: 3
   - FragmentInstancePrepareTime: 1.199ms
     - __MAX_OF_FragmentInstancePrepareTime: 1.264ms
     - __MIN_OF_FragmentInstancePrepareTime: 1.163ms
     - prepare-fragment-ctx: 31.903us
       - __MAX_OF_prepare-fragment-ctx: 34.527us
       - __MIN_OF_prepare-fragment-ctx: 28.425us
     - prepare-pipeline-driver: 251.355us
       - __MAX_OF_prepare-pipeline-driver: 267.638us
       - __MIN_OF_prepare-pipeline-driver: 236.341us
     - prepare-query-ctx: 5.119us
       - __MAX_OF_prepare-query-ctx: 5.949us
       - __MIN_OF_prepare-query-ctx: 4.558us
     - prepare-runtime-state: 251.355us
       - __MAX_OF_prepare-runtime-state: 267.638us
       - __MIN_OF_prepare-runtime-state: 236.341us
   - InstanceAllocatedMemoryUsage: 172.92 GB
     - __MAX_OF_InstanceAllocatedMemoryUsage: 59.04 GB
     - __MIN_OF_InstanceAllocatedMemoryUsage: 56.29 GB
   - InstanceDeallocatedMemoryUsage: 172.84 GB
     - __MAX_OF_InstanceDeallocatedMemoryUsage: 59.01 GB
     - __MIN_OF_InstanceDeallocatedMemoryUsage: 56.26 GB
   - InstanceNum: 3
   - InstancePeakMemoryUsage: 934.52 MB
     - __MAX_OF_InstancePeakMemoryUsage: 963.29 MB
     - __MIN_OF_InstancePeakMemoryUsage: 904.64 MB
   - QueryMemoryLimit: 32.00 GB
  Pipeline (id=3):
     - ActiveTime: 391.796us
       - __MAX_OF_ActiveTime: 1.25ms
       - __MIN_OF_ActiveTime: 180.503us
     - BlockByInputEmpty: 0
     - BlockByOutputFull: 0
     - BlockByPrecondition: 0
     - DegreeOfParallelism: 8
     - DriverPrepareTime: 1.102ms
       - __MAX_OF_DriverPrepareTime: 1.384ms
       - __MIN_OF_DriverPrepareTime: 832.537us
     - DriverTotalTime: 5s983ms
       - __MAX_OF_DriverTotalTime: 6s393ms
       - __MIN_OF_DriverTotalTime: 5s589ms
     - OverheadTime: 1.93us
       - __MAX_OF_OverheadTime: 7.913us
       - __MIN_OF_OverheadTime: 0ns
     - PendingTime: 5s952ms
       - __MAX_OF_PendingTime: 6s344ms
       - __MIN_OF_PendingTime: 5s588ms
       - InputEmptyTime: 5s929ms
         - __MAX_OF_InputEmptyTime: 6s343ms
         - __MIN_OF_InputEmptyTime: 5s578ms
         - FirstInputEmptyTime: 5s929ms
           - __MAX_OF_FirstInputEmptyTime: 6s343ms
           - __MIN_OF_FirstInputEmptyTime: 5s578ms
         - FollowupInputEmptyTime: 0ns
       - OutputFullTime: 0ns
       - PendingFinishTime: 22.768ms
         - __MAX_OF_PendingFinishTime: 42.409ms
         - __MIN_OF_PendingFinishTime: 397.188us
       - PreconditionBlockTime: 0ns
     - ScheduleCount: 24
       - __MAX_OF_ScheduleCount: 1
       - __MIN_OF_ScheduleCount: 1
     - ScheduleTime: 30.875ms
       - __MAX_OF_ScheduleTime: 49.249ms
       - __MIN_OF_ScheduleTime: 826.700us
     - TotalDegreeOfParallelism: 24
       - __MAX_OF_TotalDegreeOfParallelism: 8
       - __MIN_OF_TotalDegreeOfParallelism: 8
     - YieldByPreempt: 0
     - YieldByTimeLimit: 0
    EXCHANGE_SINK (plan_node_id=3):
      CommonMetrics:
         - CloseTime: 3.7us
           - __MAX_OF_CloseTime: 24.36us
           - __MIN_OF_CloseTime: 154ns
         - OperatorAllocatedMemoryUsage: 637.61 KB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 37.83 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 20.19 KB
         - OperatorDeallocatedMemoryUsage: 321.61 KB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 22.85 KB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 4.66 KB
         - OperatorPeakMemoryUsage: 15.71 KB
           - __MAX_OF_OperatorPeakMemoryUsage: 24.03 KB
           - __MIN_OF_OperatorPeakMemoryUsage: 10.13 KB
         - OperatorTotalTime: 281.128us
           - __MAX_OF_OperatorTotalTime: 929.329us
           - __MIN_OF_OperatorTotalTime: 101.158us
         - PrepareTime: 1.50ms
           - __MAX_OF_PrepareTime: 1.340ms
           - __MIN_OF_PrepareTime: 781.546us
         - PullChunkNum: 0
         - PullRowNum: 0
         - PullTotalTime: 0ns
         - PushChunkNum: 24
           - __MAX_OF_PushChunkNum: 1
           - __MIN_OF_PushChunkNum: 1
         - PushRowNum: 96
           - __MAX_OF_PushRowNum: 4
           - __MIN_OF_PushRowNum: 4
         - PushTotalTime: 62.320us
           - __MAX_OF_PushTotalTime: 113.623us
           - __MIN_OF_PushTotalTime: 46.302us
         - SetFinishedTime: 90ns
           - __MAX_OF_SetFinishedTime: 143ns
           - __MIN_OF_SetFinishedTime: 49ns
         - SetFinishingTime: 215.708us
           - __MAX_OF_SetFinishingTime: 880.221us
           - __MIN_OF_SetFinishingTime: 45.410us
      UniqueMetrics:
         - DestID: 3
         - DestFragments: 911072f7f94411ed-82d9a69c7b9835e0, 911072f7f94411ed-82d9a69c7b9835e1, 911072f7f94411ed-82d9a69c7b9835e2
         - PartType: HASH_PARTITIONED
         - ChannelNum: 3
         - ShuffleNumPerChannel: 8
         - TotalShuffleNum: 24
         - PipelineLevelShuffle: Yes
         - BytesPassThrough: 8.25 KB
           - __MAX_OF_BytesPassThrough: 792.00 B
           - __MIN_OF_BytesPassThrough: 0.00 
         - BytesSent: 17.50 KB
           - __MAX_OF_BytesSent: 8.75 KB
           - __MIN_OF_BytesSent: 2.19 KB
         - CompressTime: 0ns
         - NetworkBandwidth: 3.349614143371582 MB/sec
           - __MAX_OF_NetworkBandwidth: 1.6365766525268555 MB/sec
           - __MIN_OF_NetworkBandwidth: 202.595703125 KB/sec
         - NetworkTime: 6.784ms
           - __MAX_OF_NetworkTime: 10.797ms
           - __MIN_OF_NetworkTime: 3.915ms
         - OverallThroughput: 734.42578125 KB/sec
           - __MAX_OF_OverallThroughput: 349.0966796875 KB/sec
           - __MIN_OF_OverallThroughput: 154.4345703125 KB/sec
         - OverallTime: 25.677ms
           - __MAX_OF_OverallTime: 42.493ms
           - __MIN_OF_OverallTime: 9.473ms
         - RequestSent: 32
           - __MAX_OF_RequestSent: 16
           - __MIN_OF_RequestSent: 8
         - RpcAvgTime: 649.786us
           - __MAX_OF_RpcAvgTime: 1.125ms
           - __MIN_OF_RpcAvgTime: 296.63us
         - RpcCount: 37
           - __MAX_OF_RpcCount: 17
           - __MIN_OF_RpcCount: 10
         - SerializeChunkTime: 23.283us
           - __MAX_OF_SerializeChunkTime: 52.943us
           - __MIN_OF_SerializeChunkTime: 8.248us
         - ShuffleHashTime: 4.662us
           - __MAX_OF_ShuffleHashTime: 58.284us
           - __MIN_OF_ShuffleHashTime: 1.319us
         - UncompressedBytes: 17.50 KB
           - __MAX_OF_UncompressedBytes: 1.09 KB
           - __MIN_OF_UncompressedBytes: 280.00 B
         - WaitTime: 5.127ms
           - __MAX_OF_WaitTime: 9.192ms
           - __MIN_OF_WaitTime: 1.497ms
    AGGREGATE_STREAMING_SOURCE (plan_node_id=2):
      CommonMetrics:
         - CloseTime: 21.132us
           - __MAX_OF_CloseTime: 51.147us
           - __MIN_OF_CloseTime: 8.903us
         - JoinRuntimeFilterEvaluate: 0
         - JoinRuntimeFilterHashTime: 0ns
         - JoinRuntimeFilterInputRows: 0
         - JoinRuntimeFilterOutputRows: 0
         - JoinRuntimeFilterTime: 0ns
         - OperatorAllocatedMemoryUsage: 1.66 MB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 71.04 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 71.04 KB
         - OperatorDeallocatedMemoryUsage: 1.94 MB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 82.91 KB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 82.91 KB
         - OperatorPeakMemoryUsage: 69.16 KB
         - OperatorTotalTime: 121.941us
           - __MAX_OF_OperatorTotalTime: 479.505us
           - __MIN_OF_OperatorTotalTime: 73.54us
         - PrepareTime: 19.358us
           - __MAX_OF_PrepareTime: 32.428us
           - __MIN_OF_PrepareTime: 11.49us
         - PullChunkNum: 24
           - __MAX_OF_PullChunkNum: 1
           - __MIN_OF_PullChunkNum: 1
         - PullRowNum: 96
           - __MAX_OF_PullRowNum: 4
           - __MIN_OF_PullRowNum: 4
         - PullTotalTime: 100.504us
           - __MAX_OF_PullTotalTime: 457.77us
           - __MIN_OF_PullTotalTime: 56.670us
         - PushChunkNum: 0
         - PushRowNum: 0
         - PushTotalTime: 0ns
         - RuntimeBloomFilterNum: 0
         - RuntimeInFilterNum: 0
         - SetFinishedTime: 192ns
           - __MAX_OF_SetFinishedTime: 778ns
           - __MIN_OF_SetFinishedTime: 69ns
         - SetFinishingTime: 110ns
           - __MAX_OF_SetFinishingTime: 143ns
           - __MIN_OF_SetFinishingTime: 65ns
      UniqueMetrics:
#12 
  Pipeline (id=2):
     - ActiveTime: 5s604ms
       - __MAX_OF_ActiveTime: 6s184ms
       - __MIN_OF_ActiveTime: 5s22ms
     - BlockByInputEmpty: 1.336K (1336)
       - __MAX_OF_BlockByInputEmpty: 155
       - __MIN_OF_BlockByInputEmpty: 3
     - BlockByOutputFull: 0
     - BlockByPrecondition: 0
     - DegreeOfParallelism: 8
     - DriverPrepareTime: 172.553us
       - __MAX_OF_DriverPrepareTime: 206.799us
       - __MIN_OF_DriverPrepareTime: 129.460us
     - DriverTotalTime: 5s966ms
       - __MAX_OF_DriverTotalTime: 6s394ms
       - __MIN_OF_DriverTotalTime: 5s588ms
     - OverheadTime: 102.792ms
       - __MAX_OF_OverheadTime: 184.390ms
       - __MIN_OF_OverheadTime: 65.896ms
     - PendingTime: 197.656ms
       - __MAX_OF_PendingTime: 414.107ms
       - __MIN_OF_PendingTime: 33.702ms
       - InputEmptyTime: 202.967ms
         - __MAX_OF_InputEmptyTime: 419.839ms
         - __MIN_OF_InputEmptyTime: 33.714ms
         - FirstInputEmptyTime: 20.472ms
           - __MAX_OF_FirstInputEmptyTime: 32.468ms
           - __MIN_OF_FirstInputEmptyTime: 3.341ms
         - FollowupInputEmptyTime: 182.495ms
           - __MAX_OF_FollowupInputEmptyTime: 395.170ms
           - __MIN_OF_FollowupInputEmptyTime: 8.169ms
       - OutputFullTime: 0ns
       - PendingFinishTime: 0ns
       - PreconditionBlockTime: 0ns
     - ScheduleCount: 2.553K (2553)
       - __MAX_OF_ScheduleCount: 199
       - __MIN_OF_ScheduleCount: 58
     - ScheduleTime: 163.573ms
       - __MAX_OF_ScheduleTime: 303.182ms
       - __MIN_OF_ScheduleTime: 82.274ms
     - TotalDegreeOfParallelism: 24
       - __MAX_OF_TotalDegreeOfParallelism: 8
       - __MIN_OF_TotalDegreeOfParallelism: 8
     - YieldByPreempt: 0
     - YieldByTimeLimit: 53
    AGGREGATE_STREAMING_SINK (plan_node_id=2):
      CommonMetrics:
         - CloseTime: 11.272us
           - __MAX_OF_CloseTime: 25.466us
           - __MIN_OF_CloseTime: 6.791us
         - OperatorAllocatedMemoryUsage: 6.35 MB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 285.06 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 262.56 KB
         - OperatorDeallocatedMemoryUsage: 58.04 GB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 2.61 GB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 2.26 GB
         - OperatorPeakMemoryUsage: 16.27 KB
         - OperatorTotalTime: 5s450ms
           - __MAX_OF_OperatorTotalTime: 6s25ms
           - __MIN_OF_OperatorTotalTime: 4s896ms
         - PrepareTime: 137.521us
           - __MAX_OF_PrepareTime: 171.824us
           - __MIN_OF_PrepareTime: 105.527us
         - PullChunkNum: 0
         - PullRowNum: 0
         - PullTotalTime: 0ns
         - PushChunkNum: 124.567K (124567)
           - __MAX_OF_PushChunkNum: 5.192K (5192)
           - __MIN_OF_PushChunkNum: 5.189K (5189)
         - PushRowNum: 510.045617M (510045617)
           - __MAX_OF_PushRowNum: 21.259756M (21259756)
           - __MIN_OF_PushRowNum: 21.242248M (21242248)
         - PushTotalTime: 5s450ms
           - __MAX_OF_PushTotalTime: 6s25ms
           - __MIN_OF_PushTotalTime: 4s896ms
         - RuntimeBloomFilterNum: 0
         - RuntimeInFilterNum: 0
         - SetFinishedTime: 101ns
           - __MAX_OF_SetFinishedTime: 217ns
           - __MIN_OF_SetFinishedTime: 33ns
         - SetFinishingTime: 379ns
           - __MAX_OF_SetFinishingTime: 858ns
           - __MIN_OF_SetFinishingTime: 151ns
      UniqueMetrics:
         - GroupingKeys: 10: l_returnflag, 11: l_linestatus
         - AggregateFunctions: 10: l_returnflag, 11: l_linestatus
         - AggComputeTime: 4s473ms
           - __MAX_OF_AggComputeTime: 5s105ms
           - __MIN_OF_AggComputeTime: 4s75ms
         - AggFuncComputeTime: 1s522ms
           - __MAX_OF_AggFuncComputeTime: 1s842ms
           - __MIN_OF_AggFuncComputeTime: 1s304ms
         - ExprComputeTime: 58.318ms
           - __MAX_OF_ExprComputeTime: 102.427ms
           - __MIN_OF_ExprComputeTime: 39.639ms
         - ExprReleaseTime: 939.828ms
           - __MAX_OF_ExprReleaseTime: 1s154ms
           - __MIN_OF_ExprReleaseTime: 680.983ms
         - GetResultsTime: 77.55us
           - __MAX_OF_GetResultsTime: 434.959us
           - __MIN_OF_GetResultsTime: 38.192us
         - HashTableMemoryUsage: 18.10 MB
           - __MAX_OF_HashTableMemoryUsage: 772.20 KB
           - __MIN_OF_HashTableMemoryUsage: 772.20 KB
         - HashTableSize: 96
           - __MAX_OF_HashTableSize: 4
           - __MIN_OF_HashTableSize: 4
         - InputRowCount: 510.045617M (510045617)
           - __MAX_OF_InputRowCount: 21.259756M (21259756)
           - __MIN_OF_InputRowCount: 21.242248M (21242248)
         - PassThroughRowCount: 0
         - ResultAggAppendTime: 4.655us
           - __MAX_OF_ResultAggAppendTime: 7.202us
           - __MIN_OF_ResultAggAppendTime: 2.440us
         - ResultGroupByAppendTime: 1.706us
           - __MAX_OF_ResultGroupByAppendTime: 2.747us
           - __MIN_OF_ResultGroupByAppendTime: 1.215us
         - ResultIteratorTime: 48.873us
           - __MAX_OF_ResultIteratorTime: 349.436us
           - __MIN_OF_ResultIteratorTime: 13.24us
         - RowsReturned: 0
         - StateAllocate: 0ns
         - StateDestroy: 0ns
         - StreamingTime: 0ns
    LOCAL_EXCHANGE_SOURCE (pseudo_plan_node_id=-100):
      CommonMetrics:
         - CloseTime: 442ns
           - __MAX_OF_CloseTime: 1.143us
           - __MIN_OF_CloseTime: 146ns
         - OperatorAllocatedMemoryUsage: 27.00 KB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 1.13 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 1.13 KB
         - OperatorDeallocatedMemoryUsage: 2.88 MB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 122.94 KB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 122.77 KB
         - OperatorPeakMemoryUsage: 0.00 
         - OperatorTotalTime: 51.715ms
           - __MAX_OF_OperatorTotalTime: 106.871ms
           - __MIN_OF_OperatorTotalTime: 22.805ms
         - PrepareTime: 13.666us
           - __MAX_OF_PrepareTime: 19.561us
           - __MIN_OF_PrepareTime: 8.557us
         - PullChunkNum: 124.567K (124567)
           - __MAX_OF_PullChunkNum: 5.192K (5192)
           - __MIN_OF_PullChunkNum: 5.189K (5189)
         - PullRowNum: 510.045617M (510045617)
           - __MAX_OF_PullRowNum: 21.259756M (21259756)
           - __MIN_OF_PullRowNum: 21.242248M (21242248)
         - PullTotalTime: 51.711ms
           - __MAX_OF_PullTotalTime: 106.835ms
           - __MIN_OF_PullTotalTime: 22.802ms
         - PushChunkNum: 0
         - PushRowNum: 0
         - PushTotalTime: 0ns
         - SetFinishedTime: 3.879us
           - __MAX_OF_SetFinishedTime: 34.978us
           - __MIN_OF_SetFinishedTime: 1.342us
         - SetFinishingTime: 231ns
           - __MAX_OF_SetFinishingTime: 684ns
           - __MIN_OF_SetFinishingTime: 103ns
      UniqueMetrics:
  Pipeline (id=1):
     - ActiveTime: 5s115ms
       - __MAX_OF_ActiveTime: 5s778ms
       - __MIN_OF_ActiveTime: 4s604ms
     - BlockByInputEmpty: 697
       - __MAX_OF_BlockByInputEmpty: 113
       - __MIN_OF_BlockByInputEmpty: 1
     - BlockByOutputFull: 1.83K (1830)
       - __MAX_OF_BlockByOutputFull: 123
       - __MIN_OF_BlockByOutputFull: 36
     - BlockByPrecondition: 0
     - DegreeOfParallelism: 8
     - DriverPrepareTime: 80.561us
       - __MAX_OF_DriverPrepareTime: 127.383us
       - __MIN_OF_DriverPrepareTime: 52.166us
     - DriverTotalTime: 5s814ms
       - __MAX_OF_DriverTotalTime: 6s370ms
       - __MIN_OF_DriverTotalTime: 5s202ms
     - OverheadTime: 258.572ms
       - __MAX_OF_OverheadTime: 374.674ms
       - __MIN_OF_OverheadTime: 151.77ms
     - PendingTime: 402.45ms
       - __MAX_OF_PendingTime: 620.880ms
       - __MIN_OF_PendingTime: 160.749ms
       - InputEmptyTime: 227.942ms
         - __MAX_OF_InputEmptyTime: 559.228ms
         - __MIN_OF_InputEmptyTime: 17.190ms
         - FirstInputEmptyTime: 25.772ms
           - __MAX_OF_FirstInputEmptyTime: 47.554ms
           - __MIN_OF_FirstInputEmptyTime: 6.833ms
         - FollowupInputEmptyTime: 202.169ms
           - __MAX_OF_FollowupInputEmptyTime: 535.540ms
           - __MIN_OF_FollowupInputEmptyTime: 0ns
       - OutputFullTime: 200.803ms
         - __MAX_OF_OutputFullTime: 335.79ms
         - __MIN_OF_OutputFullTime: 64.713ms
       - PendingFinishTime: 0ns
       - PreconditionBlockTime: 0ns
     - ScheduleCount: 3.425K (3425)
       - __MAX_OF_ScheduleCount: 183
       - __MIN_OF_ScheduleCount: 98
     - ScheduleTime: 297.656ms
       - __MAX_OF_ScheduleTime: 452.552ms
       - __MIN_OF_ScheduleTime: 148.863ms
     - TotalDegreeOfParallelism: 24
       - __MAX_OF_TotalDegreeOfParallelism: 8
       - __MIN_OF_TotalDegreeOfParallelism: 8
     - YieldByPreempt: 0
     - YieldByTimeLimit: 37
    LOCAL_EXCHANGE_SINK (pseudo_plan_node_id=-100):
      CommonMetrics:
         - CloseTime: 278ns
           - __MAX_OF_CloseTime: 478ns
           - __MIN_OF_CloseTime: 145ns
         - OperatorAllocatedMemoryUsage: 1.91 MB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 98.16 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 68.50 KB
         - OperatorDeallocatedMemoryUsage: 4.16 KB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 672.00 B
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 0.00 
         - OperatorPeakMemoryUsage: 81.20 KB
           - __MAX_OF_OperatorPeakMemoryUsage: 98.09 KB
           - __MIN_OF_OperatorPeakMemoryUsage: 68.50 KB
         - OperatorTotalTime: 205.490ms
           - __MAX_OF_OperatorTotalTime: 354.867ms
           - __MIN_OF_OperatorTotalTime: 82.794ms
         - PrepareTime: 14.735us
           - __MAX_OF_PrepareTime: 29.15us
           - __MIN_OF_PrepareTime: 6.722us
         - PullChunkNum: 0
         - PullRowNum: 0
         - PullTotalTime: 0ns
         - PushChunkNum: 124.567K (124567)
           - __MAX_OF_PushChunkNum: 5.194K (5194)
           - __MIN_OF_PushChunkNum: 5.188K (5188)
         - PushRowNum: 510.045617M (510045617)
           - __MAX_OF_PushRowNum: 21.267386M (21267386)
           - __MIN_OF_PushRowNum: 21.240542M (21240542)
         - PushTotalTime: 205.489ms
           - __MAX_OF_PushTotalTime: 354.867ms
           - __MIN_OF_PushTotalTime: 82.793ms
         - SetFinishedTime: 29ns
           - __MAX_OF_SetFinishedTime: 43ns
           - __MIN_OF_SetFinishedTime: 23ns
         - SetFinishingTime: 451ns
           - __MAX_OF_SetFinishingTime: 2.650us
           - __MIN_OF_SetFinishingTime: 112ns
      UniqueMetrics:
         - Type: Passthrough
         - ShuffleNum: 8
         - LocalExchangePeakMemoryUsage: 24.03 GB
           - __MAX_OF_LocalExchangePeakMemoryUsage: 1.00 GB
           - __MIN_OF_LocalExchangePeakMemoryUsage: 1.00 GB
    PROJECT (plan_node_id=1):
      CommonMetrics:
         - CloseTime: 7.71us
           - __MAX_OF_CloseTime: 26.365us
           - __MIN_OF_CloseTime: 3.957us
         - OperatorAllocatedMemoryUsage: 69.20 GB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 2.89 GB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 2.88 GB
         - OperatorDeallocatedMemoryUsage: 30.09 GB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 1.25 GB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 1.25 GB
         - OperatorPeakMemoryUsage: 1.63 GB
           - __MAX_OF_OperatorPeakMemoryUsage: 1.63 GB
           - __MIN_OF_OperatorPeakMemoryUsage: 1.63 GB
         - OperatorTotalTime: 4s84ms
           - __MAX_OF_OperatorTotalTime: 4s760ms
           - __MIN_OF_OperatorTotalTime: 3s683ms
         - PrepareTime: 15.549us
           - __MAX_OF_PrepareTime: 50.296us
           - __MIN_OF_PrepareTime: 8.589us
         - PullChunkNum: 124.567K (124567)
           - __MAX_OF_PullChunkNum: 5.194K (5194)
           - __MIN_OF_PullChunkNum: 5.188K (5188)
         - PullRowNum: 510.045617M (510045617)
           - __MAX_OF_PullRowNum: 21.267386M (21267386)
           - __MIN_OF_PullRowNum: 21.240542M (21240542)
         - PullTotalTime: 3.656ms
           - __MAX_OF_PullTotalTime: 24.839ms
           - __MIN_OF_PullTotalTime: 1.9ms
         - PushChunkNum: 124.567K (124567)
           - __MAX_OF_PushChunkNum: 5.194K (5194)
           - __MIN_OF_PushChunkNum: 5.188K (5188)
         - PushRowNum: 510.045617M (510045617)
           - __MAX_OF_PushRowNum: 21.267386M (21267386)
           - __MIN_OF_PushRowNum: 21.240542M (21240542)
         - PushTotalTime: 4s80ms
           - __MAX_OF_PushTotalTime: 4s758ms
           - __MIN_OF_PushTotalTime: 3s682ms
         - RuntimeBloomFilterNum: 0
         - RuntimeInFilterNum: 0
         - SetFinishedTime: 43ns
           - __MAX_OF_SetFinishedTime: 56ns
           - __MIN_OF_SetFinishedTime: 32ns
         - SetFinishingTime: 131ns
           - __MAX_OF_SetFinishingTime: 261ns
           - __MIN_OF_SetFinishingTime: 86ns
      UniqueMetrics:
         - CommonSubExprComputeTime: 1s820ms
           - __MAX_OF_CommonSubExprComputeTime: 2s79ms
           - __MIN_OF_CommonSubExprComputeTime: 1s593ms
         - ExprComputeTime: 2s211ms
           - __MAX_OF_ExprComputeTime: 2s716ms
           - __MIN_OF_ExprComputeTime: 1s987ms
    CHUNK_ACCUMULATE (plan_node_id=0):
      CommonMetrics:
         - CloseTime: 540ns
           - __MAX_OF_CloseTime: 1.39us
           - __MIN_OF_CloseTime: 204ns
         - OperatorAllocatedMemoryUsage: 0.00 
         - OperatorDeallocatedMemoryUsage: 973.18 KB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 40.58 KB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 40.53 KB
         - OperatorPeakMemoryUsage: 0.00 
         - OperatorTotalTime: 10.254ms
           - __MAX_OF_OperatorTotalTime: 34.557ms
           - __MIN_OF_OperatorTotalTime: 2.971ms
         - PrepareTime: 12.676us
           - __MAX_OF_PrepareTime: 23.480us
           - __MIN_OF_PrepareTime: 6.520us
         - PullChunkNum: 124.567K (124567)
           - __MAX_OF_PullChunkNum: 5.194K (5194)
           - __MIN_OF_PullChunkNum: 5.188K (5188)
         - PullRowNum: 510.045617M (510045617)
           - __MAX_OF_PullRowNum: 21.267386M (21267386)
           - __MIN_OF_PullRowNum: 21.240542M (21240542)
         - PullTotalTime: 2.739ms
           - __MAX_OF_PullTotalTime: 14.821ms
           - __MIN_OF_PullTotalTime: 730.185us
         - PushChunkNum: 124.567K (124567)
           - __MAX_OF_PushChunkNum: 5.194K (5194)
           - __MIN_OF_PushChunkNum: 5.188K (5188)
         - PushRowNum: 510.045617M (510045617)
           - __MAX_OF_PushRowNum: 21.267386M (21267386)
           - __MIN_OF_PushRowNum: 21.240542M (21240542)
         - PushTotalTime: 7.514ms
           - __MAX_OF_PushTotalTime: 31.394ms
           - __MIN_OF_PushTotalTime: 1.729ms
         - SetFinishedTime: 369ns
           - __MAX_OF_SetFinishedTime: 1.804us
           - __MIN_OF_SetFinishedTime: 191ns
         - SetFinishingTime: 288ns
           - __MAX_OF_SetFinishingTime: 1.109us
           - __MIN_OF_SetFinishingTime: 190ns
      UniqueMetrics:
    OLAP_SCAN (plan_node_id=0):
      CommonMetrics:
         - CloseTime: 851.907us
           - __MAX_OF_CloseTime: 1.492ms
           - __MIN_OF_CloseTime: 486.348us
         - JoinRuntimeFilterEvaluate: 0
         - JoinRuntimeFilterHashTime: 0ns
         - JoinRuntimeFilterInputRows: 0
         - JoinRuntimeFilterOutputRows: 0
         - JoinRuntimeFilterTime: 0ns
         - OperatorAllocatedMemoryUsage: 119.50 GB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 5.50 GB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 4.49 GB
         - OperatorDeallocatedMemoryUsage: 73.62 GB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 3.46 GB
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 2.69 GB
         - OperatorPeakMemoryUsage: 1.94 GB
           - __MAX_OF_OperatorPeakMemoryUsage: 2.11 GB
           - __MIN_OF_OperatorPeakMemoryUsage: 1.70 GB
         - OperatorTotalTime: 556.475ms
           - __MAX_OF_OperatorTotalTime: 954.802ms
           - __MIN_OF_OperatorTotalTime: 265.781ms
         - PrepareTime: 20.293us
           - __MAX_OF_PrepareTime: 66.878us
           - __MIN_OF_PrepareTime: 11.397us
         - PullChunkNum: 124.567K (124567)
           - __MAX_OF_PullChunkNum: 5.194K (5194)
           - __MIN_OF_PullChunkNum: 5.188K (5188)
         - PullRowNum: 510.045617M (510045617)
           - __MAX_OF_PullRowNum: 21.267386M (21267386)
           - __MIN_OF_PullRowNum: 21.240542M (21240542)
         - PullTotalTime: 555.622ms
           - __MAX_OF_PullTotalTime: 953.681ms
           - __MIN_OF_PullTotalTime: 265.132ms
         - PushChunkNum: 0
         - PushRowNum: 0
         - PushTotalTime: 0ns
         - RuntimeBloomFilterNum: 0
         - RuntimeInFilterNum: 0
         - SetFinishedTime: 193ns
           - __MAX_OF_SetFinishedTime: 355ns
           - __MIN_OF_SetFinishedTime: 93ns
         - SetFinishingTime: 636ns
           - __MAX_OF_SetFinishingTime: 856ns
           - __MIN_OF_SetFinishingTime: 525ns
      UniqueMetrics:
         - MorselQueueType: fixed_morsel_queue
         - BufferUnplugThreshold: 32
         - SharedScan: False
         - Predicates: 1: l_shipdate &lt;= '1998-12-01'
         - Rollup: lineitem
         - Table: lineitem
         - BufferUnplugCount: 133
           - __MAX_OF_BufferUnplugCount: 12
           - __MIN_OF_BufferUnplugCount: 1
         - BytesRead: 21.97 GB
           - __MAX_OF_BytesRead: 234.78 MB
           - __MIN_OF_BytesRead: 233.92 MB
         - CachedPagesNum: 136.568K (136568)
           - __MAX_OF_CachedPagesNum: 2.0K (2000)
           - __MIN_OF_CachedPagesNum: 528
         - ChunkBufferCapacity: 512
         - CompressedBytesRead: 1.02 GB
           - __MAX_OF_CompressedBytesRead: 22.71 MB
           - __MIN_OF_CompressedBytesRead: 3.35 MB
         - CreateSegmentIter: 19.446us
           - __MAX_OF_CreateSegmentIter: 38.548us
           - __MIN_OF_CreateSegmentIter: 10.573us
         - DefaultChunkBufferCapacity: 512
         - DictDecode: 860.845ms
           - __MAX_OF_DictDecode: 1s139ms
           - __MIN_OF_DictDecode: 639.200ms
         - GetDelVec: 87ns
           - __MAX_OF_GetDelVec: 411ns
           - __MIN_OF_GetDelVec: 20ns
         - GetRowsets: 233ns
           - __MAX_OF_GetRowsets: 794ns
           - __MIN_OF_GetRowsets: 85ns
         - IOTaskExecTime: 1s926ms
           - __MAX_OF_IOTaskExecTime: 2s469ms
           - __MIN_OF_IOTaskExecTime: 1s397ms
         - IOTaskWaitTime: 414.507ms
           - __MAX_OF_IOTaskWaitTime: 986.351ms
           - __MIN_OF_IOTaskWaitTime: 153.899ms
         - IOTime: 34.832ms
           - __MAX_OF_IOTime: 156.497ms
           - __MIN_OF_IOTime: 7.4ms
         - LateMaterialize: 1.641ms
           - __MAX_OF_LateMaterialize: 10.610ms
           - __MIN_OF_LateMaterialize: 491.506us
         - MorselsCount: 96
           - __MAX_OF_MorselsCount: 4
           - __MIN_OF_MorselsCount: 4
         - PeakChunkBufferSize: 512
         - PushdownPredicates: 1
         - RawRowsRead: 510.045617M (510045617)
           - __MAX_OF_RawRowsRead: 5.322652M (5322652)
           - __MIN_OF_RawRowsRead: 5.303307M (5303307)
         - ReadPagesNum: 343.001K (343001)
           - __MAX_OF_ReadPagesNum: 3.577K (3577)
           - __MIN_OF_ReadPagesNum: 3.566K (3566)
         - RowsRead: 510.045617M (510045617)
           - __MAX_OF_RowsRead: 5.322652M (5322652)
           - __MIN_OF_RowsRead: 5.303307M (5303307)
         - ScanTime: 2s341ms
           - __MAX_OF_ScanTime: 3s34ms
           - __MIN_OF_ScanTime: 1s695ms
         - SegmentInit: 439.752us
           - __MAX_OF_SegmentInit: 5.196ms
           - __MIN_OF_SegmentInit: 251.725us
           - BitmapIndexFilter: 0ns
           - BitmapIndexFilterRows: 0
           - BloomFilterFilterRows: 0
           - SegmentRuntimeZoneMapFilterRows: 0
           - SegmentZoneMapFilterRows: 0
           - ShortKeyFilterRows: 0
           - ZoneMapIndexFilterRows: 0
         - SegmentRead: 994.622ms
           - __MAX_OF_SegmentRead: 1s435ms
           - __MIN_OF_SegmentRead: 621.374ms
           - BlockFetch: 979.16ms
             - __MAX_OF_BlockFetch: 1s424ms
             - __MIN_OF_BlockFetch: 609.575ms
           - BlockFetchCount: 124.567K (124567)
             - __MAX_OF_BlockFetchCount: 1.3K (1300)
             - __MIN_OF_BlockFetchCount: 1.295K (1295)
           - BlockSeek: 195.586us
             - __MAX_OF_BlockSeek: 3.691ms
             - __MIN_OF_BlockSeek: 70.914us
           - BlockSeekCount: 342.233K (342233)
             - __MAX_OF_BlockSeekCount: 3.569K (3569)
             - __MIN_OF_BlockSeekCount: 3.558K (3558)
           - ChunkCopy: 154.128us
             - __MAX_OF_ChunkCopy: 2.70ms
             - __MIN_OF_ChunkCopy: 82.484us
           - DecompressT: 11.799ms
             - __MAX_OF_DecompressT: 53.143ms
             - __MIN_OF_DecompressT: 3.732ms
           - DelVecFilterRows: 0
           - PredFilter: 7.49ms
             - __MAX_OF_PredFilter: 39.693ms
             - __MIN_OF_PredFilter: 2.304ms
           - PredFilterRows: 0
           - RowsetsReadCount: 96
             - __MAX_OF_RowsetsReadCount: 1
             - __MIN_OF_RowsetsReadCount: 1
           - SegmentsReadCount: 96
             - __MAX_OF_SegmentsReadCount: 1
             - __MIN_OF_SegmentsReadCount: 1
           - TotalColumnsDataPageCount: 342.809K (342809)
             - __MAX_OF_TotalColumnsDataPageCount: 3.575K (3575)
             - __MIN_OF_TotalColumnsDataPageCount: 3.564K (3564)
         - SubmitTaskCount: 5.654K (5654)
           - __MAX_OF_SubmitTaskCount: 261
           - __MIN_OF_SubmitTaskCount: 189
         - TabletCount: 96
           - __MAX_OF_TabletCount: 32
           - __MIN_OF_TabletCount: 32
         - UncompressedBytesRead: 1.04 GB
           - __MAX_OF_UncompressedBytesRead: 22.93 MB
           - __MIN_OF_UncompressedBytesRead: 3.62 MB
  Pipeline (id=0):
     - ActiveTime: 46.687us
       - __MAX_OF_ActiveTime: 83.253us
       - __MIN_OF_ActiveTime: 15.509us
     - BlockByInputEmpty: 0
     - BlockByOutputFull: 0
     - BlockByPrecondition: 0
     - DegreeOfParallelism: 8
     - DriverPrepareTime: 60.567us
       - __MAX_OF_DriverPrepareTime: 99.632us
       - __MIN_OF_DriverPrepareTime: 40.796us
     - DriverTotalTime: 11.443ms
       - __MAX_OF_DriverTotalTime: 11.934ms
       - __MIN_OF_DriverTotalTime: 10.932ms
     - OverheadTime: 430ns
       - __MAX_OF_OverheadTime: 4.358us
       - __MIN_OF_OverheadTime: 0ns
     - PendingTime: 0ns
       - InputEmptyTime: 0ns
         - FirstInputEmptyTime: 0ns
         - FollowupInputEmptyTime: 0ns
       - OutputFullTime: 0ns
       - PendingFinishTime: 0ns
       - PreconditionBlockTime: 0ns
     - ScheduleCount: 24
       - __MAX_OF_ScheduleCount: 1
       - __MIN_OF_ScheduleCount: 1
     - ScheduleTime: 11.396ms
       - __MAX_OF_ScheduleTime: 11.870ms
       - __MIN_OF_ScheduleTime: 10.908ms
     - TotalDegreeOfParallelism: 24
       - __MAX_OF_TotalDegreeOfParallelism: 8
       - __MIN_OF_TotalDegreeOfParallelism: 8
     - YieldByPreempt: 0
     - YieldByTimeLimit: 0
    NOOP_SINK (plan_node_id=0):
      CommonMetrics:
         - CloseTime: 315ns
           - __MAX_OF_CloseTime: 600ns
           - __MIN_OF_CloseTime: 125ns
         - OperatorAllocatedMemoryUsage: 0.00 
         - OperatorDeallocatedMemoryUsage: 0.00 
         - OperatorPeakMemoryUsage: 0.00 
         - OperatorTotalTime: 455ns
           - __MAX_OF_OperatorTotalTime: 732ns
           - __MIN_OF_OperatorTotalTime: 204ns
         - PrepareTime: 14.339us
           - __MAX_OF_PrepareTime: 33.94us
           - __MIN_OF_PrepareTime: 9.248us
         - PullChunkNum: 0
         - PullRowNum: 0
         - PullTotalTime: 0ns
         - PushChunkNum: 0
         - PushRowNum: 0
         - PushTotalTime: 0ns
         - SetFinishedTime: 32ns
           - __MAX_OF_SetFinishedTime: 41ns
           - __MIN_OF_SetFinishedTime: 24ns
         - SetFinishingTime: 106ns
           - __MAX_OF_SetFinishingTime: 191ns
           - __MIN_OF_SetFinishingTime: 41ns
      UniqueMetrics:
    OLAP_SCAN_PREPARE (plan_node_id=0):
      CommonMetrics:
         - CloseTime: 9.813us
           - __MAX_OF_CloseTime: 39.865us
           - __MIN_OF_CloseTime: 3.510us
         - OperatorAllocatedMemoryUsage: 86.44 KB
           - __MAX_OF_OperatorAllocatedMemoryUsage: 3.60 KB
           - __MIN_OF_OperatorAllocatedMemoryUsage: 3.60 KB
         - OperatorDeallocatedMemoryUsage: 17.81 KB
           - __MAX_OF_OperatorDeallocatedMemoryUsage: 760.00 B
           - __MIN_OF_OperatorDeallocatedMemoryUsage: 760.00 B
         - OperatorPeakMemoryUsage: 2.86 KB
         - OperatorTotalTime: 50.231us
           - __MAX_OF_OperatorTotalTime: 85.691us
           - __MIN_OF_OperatorTotalTime: 16.398us
         - PrepareTime: 26.110us
           - __MAX_OF_PrepareTime: 53.703us
           - __MIN_OF_PrepareTime: 18.177us
         - PullChunkNum: 0
         - PullRowNum: 0
         - PullTotalTime: 40.235us
           - __MAX_OF_PullTotalTime: 77.682us
           - __MIN_OF_PullTotalTime: 12.806us
         - PushChunkNum: 0
         - PushRowNum: 0
         - PushTotalTime: 0ns
         - RuntimeBloomFilterNum: 0
         - RuntimeInFilterNum: 0
         - SetFinishedTime: 59ns
           - __MAX_OF_SetFinishedTime: 86ns
           - __MIN_OF_SetFinishedTime: 28ns
         - SetFinishingTime: 121ns
           - __MAX_OF_SetFinishingTime: 318ns
           - __MIN_OF_SetFinishingTime: 51ns
      UniqueMetrics:
#13

q1.profile (64.7 KB)

#14

好的, 我们这边跟进看看差异在哪儿.

#15

大佬，有发现问题在哪里吗？

#16

还需要点时间, 抱歉

#17

从profile看, q1实际只用到了三台机器, 也没有开启dict优化, 能跑一下以下SQL, 把结果发出来看一看不?

show create table lineitem;

admin show replica distribution from lineitem;

explain verbose <q1_sql>

京ICP备2022026421号-1