【详述】starrocks 明细表count统计缓慢
【背景】starrocks 明细表count统计缓慢
【业务影响】
【StarRocks版本】：1.19.1
【集群规模】例如：3fe（3 follower）+6be（fe与be混部）
【机器信息】CPU虚拟核/内存/网卡，48C/256G/千兆
【附件】
整体数据量为40亿的数据，是否有优化方案
整体数据量为40亿的数据，是否有优化方案
可以补充下profile
Query:
Summary:
- Query ID: c10a18fe-b63c-11ec-b2a6-ac1f6be1dfa4
- Start Time: 2022-04-07 14:34:19
- End Time: 2022-04-07 14:37:07
- Total: 2m48s
- Query Type: Query
- Query State: EOF
- StarRocks Version: 1.19.1
- User: root
- Default Db: default_cluster:intelligence_net_b
- Sql Statement: select count() from subdomain_dns_analysis_ip_uniq
Execution Profile c10a18fe-b63c-11ec-b2a6-ac1f6be1dfa4:(Active: 2m48s[168349826485ns], % non-child: 100.00%)
Fragment 0:
Instance c10a18fe-b63c-11ec-b2a6-ac1f6be1dfab (host=TNetworkAddress(hostname:10.0.5.25, port:9060)):(Active: 2m48s[168308430334ns], % non-child: 0.00%)
- AverageThreadTokens: 4607182418800017400.00
- MemoryLimit: 16.00 GB
- PeakMemoryUsage: 4.00 KB
- PeakReservation: 0.00
- PeakUsedReservation: 0.00
- RowsProduced: 1
BlockMgr:
- BlockWritesOutstanding: 0
- BlocksCreated: 0
- BlocksRecycled: 0
- BufferedPins: 0
- BytesWritten: 0.00
- MaxBlockSize: 8.00 MB
- TotalBufferWaitTime: 0ns
- TotalReadBlockTime: 0ns
DataBufferSender (dst_fragment_instance_id=c10a18fe-b63c-11ec-b2a6-ac1f6be1dfab):
- AppendBatchTime: 3.821us
- ResultRendTime: 55.2us
- TupleConvertTime: 1.188us
- NumSentRows: 1
AGGREGATION_NODE (id=3):(Active: 2m48s[168308340679ns], % non-child: 0.00%)
- AggregateFunctions: count(13: count())
- AggComputeTime: 11.250us
- ExprComputeTime: 23.668us
- ExprReleaseTime: 17.21us
- GetResultsTime: 14.276us
- HashTableSize: 0
- InputRowCount: 6
- PassThroughRowCount: 0
- PeakMemoryUsage: 4.00 KB
- ResultAggAppendTime: 0ns
- ResultGroupByAppendTime: 0ns
- ResultIteratorTime: 0ns
- RowsReturned: 1
- RowsReturnedRate: 0
- StreamingTime: 0ns
EXCHANGE_NODE (id=2):(Active: 2m48s[168308202759ns], % non-child: 99.98%)
- BytesReceived: 120.00 B
- ConvertRowBatchTime: 0ns
- DataArrivalWaitTime: 0ns
- DecompressRowBatchTimer: 0ns
- DeserializeChunkMetaTimer: 0ns
- DeserializeRowBatchTimer: 45.251us
- FirstBatchArrivalWaitTime: 0ns
- PeakMemoryUsage: 0.00
- RequestReceived: 6.00 B
- RowsReturned: 6
- RowsReturnedRate: 0
- SenderTotalTime: 925.326us
- SenderWaitLockTime: 1.133us
- SendersBlockedTotalTimer(): 0ns
Fragment 1:
Instance c10a18fe-b63c-11ec-b2a6-ac1f6be1dfaa (host=TNetworkAddress(hostname:10.0.5.20, port:9060)):(Active: 2m48s[168299938085ns], % non-child: 0.00%)
- AverageThreadTokens: 4607182418800017400.00
- MemoryLimit: 16.00 GB
- PeakMemoryUsage: 165.23 MB
- PeakReservation: 0.00
- PeakUsedReservation: 0.00
- RowsProduced: 1
BlockMgr:
- BlockWritesOutstanding: 0
- BlocksCreated: 0
- BlocksRecycled: 0
- BufferedPins: 0
- BytesWritten: 0.00
- MaxBlockSize: 8.00 MB
- TotalBufferWaitTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=2, dst_fragments=[c10a18feb63c11ec-b2a6ac1f6be1dfab]):(Active: 462.694us[462694ns], % non-child: 0.00%)
- PartType: UNPARTITIONED
- BytesSent: 20.00 B
- CompressTime: 9.203us
- IgnoreRows: 0
- OverallThroughput: 42.2119140625 KB/sec
- PeakMemoryUsage: 16.00 KB
- SendRequestTime: 64.532us
- SerializeBatchTime: 8.766us
- ShuffleDispatchTime: 0ns
- ShuffleHashTime: 0ns
- UncompressedBytes: 20.00 B
- WaitResponseTime: 325.792us
AGGREGATION_NODE (id=1):(Active: 2m48s[168299542480ns], % non-child: 0.06%)
- AggregateFunctions: count()
- AggComputeTime: 22.5ms
- ExprComputeTime: 7.318ms
- ExprReleaseTime: 11.722ms
- GetResultsTime: 24.277us
- HashTableSize: 0
- InputRowCount: 799.68413M (799684130)
- PassThroughRowCount: 0
- PeakMemoryUsage: 4.00 KB
- ResultAggAppendTime: 0ns
- ResultGroupByAppendTime: 0ns
- ResultIteratorTime: 0ns
- RowsReturned: 1
- RowsReturnedRate: 0
- StreamingTime: 0ns
OLAP_SCAN_NODE (id=0):(Active: 2m48s[168201943540ns], % non-child: 0.00%)
- Table: subdomain_dns_analysis_ip_uniq
- Rollup: subdomain_dns_analysis_ip_uniq
- BytesRead: 72.89 GB
- NumDiskAccess: 0
- PeakMemoryUsage: 165.21 MB
- PerReadThreadRawHdfsThroughput: 0.0 /sec
- RowsRead: 0
- RowsReturned: 799.68413M (799684130)
- RowsReturnedRate: 4.754309M /sec
- ScanTime: 2h1m
- ScannerThreadsInvoluntaryContextSwitches: 0
- ScannerThreadsTotalWallClockTime: 0ns
- MaterializeTupleTime(): 0ns
- ScannerThreadsSysTime: 0ns
- ScannerThreadsUserTime: 0ns
- ScannerThreadsVoluntaryContextSwitches: 0
- TabletCount : 342
- TotalRawReadTime(): 0ns
- TotalReadThroughput: 442.61815071105957 MB/sec
MERGE:
- aggr: 1h51m
- sort: 1h51m
SCAN:(Active: 1h59m[7140260729587ns], % non-child: 100.00%)
- CachedPagesNum: 0
- CompressedBytesRead: 43.92 GB
- CreateSegmentIter: 17.109ms
- IOTime: 1h55m
- PushdownPredicates: 0
- RawRowsRead: 815.217014M (815217014)
- SegmentInit: 2m32s
- BitmapIndexFilter: 0ns
- BitmapIndexFilterRows: 0
- BloomFilterFilterRows: 0
- ShortKeyFilterRows: 0
- ZoneMapIndexFilterRows: 0
- SegmentRead: 1h56m
- BlockFetch: 1h48m
- BlockFetchCount: 199.555K (199555)
- BlockSeek: 7m27s
- BlockSeekCount: 1.42K (1420)
- ChunkCopy: 0ns
- DecompressT: 8s293ms
- DelVecFilterRows: 0
- IndexLoad: 0ns
- PredFilter: 0ns
- PredFilterRows: 0
- TotalPagesNum: 1.069013M (1069013)
- UncompressedBytesRead: 59.35 GB
Instance c10a18fe-b63c-11ec-b2a6-ac1f6be1dfa6 (host=TNetworkAddress(hostname:10.0.5.23, port:9060)):(Active: 2m36s[156182801818ns], % non-child: 0.00%)
- AverageThreadTokens: 4607182418800017400.00
- MemoryLimit: 16.00 GB
- PeakMemoryUsage: 174.77 MB
- PeakReservation: 0.00
- PeakUsedReservation: 0.00
- RowsProduced: 1
BlockMgr:
- BlockWritesOutstanding: 0
- BlocksCreated: 0
- BlocksRecycled: 0
- BufferedPins: 0
- BytesWritten: 0.00
- MaxBlockSize: 8.00 MB
- TotalBufferWaitTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=2, dst_fragments=[c10a18feb63c11ec-b2a6ac1f6be1dfab]):(Active: 504.552us[504552ns], % non-child: 0.00%)
- PartType: UNPARTITIONED
- BytesSent: 20.00 B
- CompressTime: 8.688us
- IgnoreRows: 0
- OverallThroughput: 38.7099609375 KB/sec
- PeakMemoryUsage: 16.00 KB
- SendRequestTime: 88.596us
- SerializeBatchTime: 3.409us
- ShuffleDispatchTime: 0ns
- ShuffleHashTime: 0ns
- UncompressedBytes: 20.00 B
- WaitResponseTime: 280.956us
AGGREGATION_NODE (id=1):(Active: 2m36s[156182450880ns], % non-child: 0.09%)
- AggregateFunctions: count()
- AggComputeTime: 39.918ms
- ExprComputeTime: 12.282ms
- ExprReleaseTime: 16.653ms
- GetResultsTime: 10.760us
- HashTableSize: 0
- InputRowCount: 831.459737M (831459737)
- PassThroughRowCount: 0
- PeakMemoryUsage: 4.00 KB
- ResultAggAppendTime: 0ns
- ResultGroupByAppendTime: 0ns
- ResultIteratorTime: 0ns
- RowsReturned: 1
- RowsReturnedRate: 0
- StreamingTime: 0ns
OLAP_SCAN_NODE (id=0):(Active: 2m36s[156027564764ns], % non-child: 0.00%)
- Table: subdomain_dns_analysis_ip_uniq
- Rollup: subdomain_dns_analysis_ip_uniq
- BytesRead: 75.75 GB
- NumDiskAccess: 0
- PeakMemoryUsage: 174.75 MB
- PerReadThreadRawHdfsThroughput: 0.0 /sec
- RowsRead: 0
- RowsReturned: 831.459737M (831459737)
- RowsReturnedRate: 5.328928M /sec
- ScanTime: 1h55m
- ScannerThreadsInvoluntaryContextSwitches: 0
- ScannerThreadsTotalWallClockTime: 0ns
- MaterializeTupleTime(): 0ns
- ScannerThreadsSysTime: 0ns
- ScannerThreadsUserTime: 0ns
- ScannerThreadsVoluntaryContextSwitches: 0
- TabletCount : 342
- TotalRawReadTime(): 0ns
- TotalReadThroughput: 494.5065116882324 MB/sec
MERGE:
- aggr: 1h49m
- sort: 1h48m
SCAN:(Active: 1h52m[6722744655753ns], % non-child: 100.00%)
- CachedPagesNum: 0
- CompressedBytesRead: 45.52 GB
- CreateSegmentIter: 27.619ms
- IOTime: 1h48m
- PushdownPredicates: 0
- RawRowsRead: 847.200399M (847200399)
- SegmentInit: 2m14s
- BitmapIndexFilter: 0ns
- BitmapIndexFilterRows: 0
- BloomFilterFilterRows: 0
- ShortKeyFilterRows: 0
- ZoneMapIndexFilterRows: 0
- SegmentRead: 1h49m
- BlockFetch: 1h43m
- BlockFetchCount: 207.375K (207375)
- BlockSeek: 6m17s
- BlockSeekCount: 1.451K (1451)
- ChunkCopy: 0ns
- DecompressT: 10s795ms
- DelVecFilterRows: 0
- IndexLoad: 0ns
- PredFilter: 0ns
- PredFilterRows: 0
- TotalPagesNum: 1.111327M (1111327)
- UncompressedBytesRead: 61.67 GB
Instance c10a18fe-b63c-11ec-b2a6-ac1f6be1dfa9 (host=TNetworkAddress(hostname:10.0.5.22, port:9060)):(Active: 2m30s[150966341551ns], % non-child: 0.00%)
- AverageThreadTokens: 4607182418800017400.00
- MemoryLimit: 16.00 GB
- PeakMemoryUsage: 162.84 MB
- PeakReservation: 0.00
- PeakUsedReservation: 0.00
- RowsProduced: 1
BlockMgr:
- BlockWritesOutstanding: 0
- BlocksCreated: 0
- BlocksRecycled: 0
- BufferedPins: 0
- BytesWritten: 0.00
- MaxBlockSize: 8.00 MB
- TotalBufferWaitTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=2, dst_fragments=[c10a18feb63c11ec-b2a6ac1f6be1dfab]):(Active: 561.402us[561402ns], % non-child: 0.00%)
- PartType: UNPARTITIONED
- BytesSent: 20.00 B
- CompressTime: 8.803us
- IgnoreRows: 0
- OverallThroughput: 34.7900390625 KB/sec
- PeakMemoryUsage: 16.00 KB
- SendRequestTime: 64.946us
- SerializeBatchTime: 8.908us
- ShuffleDispatchTime: 0ns
- ShuffleHashTime: 0ns
- UncompressedBytes: 20.00 B
- WaitResponseTime: 411.903us
AGGREGATION_NODE (id=1):(Active: 2m30s[150965817658ns], % non-child: 0.09%)
- AggregateFunctions: count()
- AggComputeTime: 35.409ms
- ExprComputeTime: 11.516ms
- ExprReleaseTime: 16.452ms
- GetResultsTime: 24.648us
- HashTableSize: 0
- InputRowCount: 802.603963M (802603963)
- PassThroughRowCount: 0
- PeakMemoryUsage: 4.00 KB
- ResultAggAppendTime: 0ns
- ResultGroupByAppendTime: 0ns
- ResultIteratorTime: 0ns
- RowsReturned: 1
- RowsReturnedRate: 0
- StreamingTime: 0ns
OLAP_SCAN_NODE (id=0):(Active: 2m30s[150818562608ns], % non-child: 0.00%)
- Table: subdomain_dns_analysis_ip_uniq
- Rollup: subdomain_dns_analysis_ip_uniq
- BytesRead: 73.43 GB
- NumDiskAccess: 0
- PeakMemoryUsage: 162.82 MB
- PerReadThreadRawHdfsThroughput: 0.0 /sec
- RowsRead: 0
- RowsReturned: 802.603963M (802603963)
- RowsReturnedRate: 5.321652M /sec
- ScanTime: 1h46m
- ScannerThreadsInvoluntaryContextSwitches: 0
- ScannerThreadsTotalWallClockTime: 0ns
- MaterializeTupleTime(): 0ns
- ScannerThreadsSysTime: 0ns
- ScannerThreadsUserTime: 0ns
- ScannerThreadsVoluntaryContextSwitches: 0
- TabletCount : 341
- TotalRawReadTime(): 0ns
- TotalReadThroughput: 494.8872261047363 MB/sec
MERGE:
- aggr: 1h39m
- sort: 1h39m
SCAN:(Active: 1h43m[6215171210164ns], % non-child: 100.00%)
- CachedPagesNum: 0
- CompressedBytesRead: 44.25 GB
- CreateSegmentIter: 20.64ms
- IOTime: 1h40m
- PushdownPredicates: 0
- RawRowsRead: 821.303111M (821303111)
- SegmentInit: 2m
- BitmapIndexFilter: 0ns
- BitmapIndexFilterRows: 0
- BloomFilterFilterRows: 0
- ShortKeyFilterRows: 0
- ZoneMapIndexFilterRows: 0
- SegmentRead: 1h41m
- BlockFetch: 1h35m
- BlockFetchCount: 201.094K (201094)
- BlockSeek: 6m17s
- BlockSeekCount: 1.527K (1527)
- ChunkCopy: 0ns
- DecompressT: 10s79ms
- DelVecFilterRows: 0
- IndexLoad: 0ns
- PredFilter: 0ns
- PredFilterRows: 0
- TotalPagesNum: 1.073848M (1073848)
- UncompressedBytesRead: 59.60 GB
Instance c10a18fe-b63c-11ec-b2a6-ac1f6be1dfa5 (host=TNetworkAddress(hostname:10.0.5.25, port:9060)):(Active: 2m13s[133489627603ns], % non-child: 0.00%)
- AverageThreadTokens: 4607182418800017400.00
- MemoryLimit: 16.00 GB
- PeakMemoryUsage: 168.82 MB
- PeakReservation: 0.00
- PeakUsedReservation: 0.00
- RowsProduced: 1
BlockMgr:
- BlockWritesOutstanding: 0
- BlocksCreated: 0
- BlocksRecycled: 0
- BufferedPins: 0
- BytesWritten: 0.00
- MaxBlockSize: 8.00 MB
- TotalBufferWaitTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=2, dst_fragments=[c10a18feb63c11ec-b2a6ac1f6be1dfab]):(Active: 431.671us[431671ns], % non-child: 0.00%)
- PartType: UNPARTITIONED
- BytesSent: 20.00 B
- CompressTime: 8.276us
- IgnoreRows: 0
- OverallThroughput: 45.2451171875 KB/sec
- PeakMemoryUsage: 16.00 KB
- SendRequestTime: 63.368us
- SerializeBatchTime: 2.418us
- ShuffleDispatchTime: 0ns
- ShuffleHashTime: 0ns
- UncompressedBytes: 20.00 B
- WaitResponseTime: 276.975us
AGGREGATION_NODE (id=1):(Active: 2m13s[133489275199ns], % non-child: 0.05%)
- AggregateFunctions: count()
- AggComputeTime: 19.19ms
- ExprComputeTime: 7.200ms
- ExprReleaseTime: 10.637ms
- GetResultsTime: 14.807us
- HashTableSize: 0
- InputRowCount: 746.577323M (746577323)
- PassThroughRowCount: 0
- PeakMemoryUsage: 4.00 KB
- ResultAggAppendTime: 0ns
- ResultGroupByAppendTime: 0ns
- ResultIteratorTime: 0ns
- RowsReturned: 1
- RowsReturnedRate: 0
- StreamingTime: 0ns
OLAP_SCAN_NODE (id=0):(Active: 2m13s[133400494850ns], % non-child: 0.00%)
- Table: subdomain_dns_analysis_ip_uniq
- Rollup: subdomain_dns_analysis_ip_uniq
- BytesRead: 68.23 GB
- NumDiskAccess: 0
- PeakMemoryUsage: 168.80 MB
- PerReadThreadRawHdfsThroughput: 0.0 /sec
- RowsRead: 0
- RowsReturned: 746.577323M (746577323)
- RowsReturnedRate: 5.59651M /sec
- ScanTime: 1h33m
- ScannerThreadsInvoluntaryContextSwitches: 0
- ScannerThreadsTotalWallClockTime: 0ns
- MaterializeTupleTime(): 0ns
- ScannerThreadsSysTime: 0ns
- ScannerThreadsUserTime: 0ns
- ScannerThreadsVoluntaryContextSwitches: 0
- TabletCount : 342
- TotalRawReadTime(): 0ns
- TotalReadThroughput: 521.6710214614868 MB/sec
MERGE:
- aggr: 1h26m
- sort: 1h25m
SCAN:(Active: 1h30m[5441971329398ns], % non-child: 100.00%)
- CachedPagesNum: 0
- CompressedBytesRead: 41.43 GB
- CreateSegmentIter: 18.29ms
- IOTime: 1h28m
- PushdownPredicates: 0
- RawRowsRead: 763.096212M (763096212)
- SegmentInit: 1m56s
- BitmapIndexFilter: 0ns
- BitmapIndexFilterRows: 0
- BloomFilterFilterRows: 0
- ShortKeyFilterRows: 0
- ZoneMapIndexFilterRows: 0
- SegmentRead: 1h28m
- BlockFetch: 1h23m
- BlockFetchCount: 186.854K (186854)
- BlockSeek: 5m38s
- BlockSeekCount: 1.48K (1480)
- ChunkCopy: 0ns
- DecompressT: 7s660ms
- DelVecFilterRows: 0
- IndexLoad: 0ns
- PredFilter: 0ns
- PredFilterRows: 0
- TotalPagesNum: 998.989K (998989)
- UncompressedBytesRead: 55.49 GB
Instance c10a18fe-b63c-11ec-b2a6-ac1f6be1dfa7 (host=TNetworkAddress(hostname:10.0.5.24, port:9060)):(Active: 2m10s[130313109366ns], % non-child: 0.00%)
- AverageThreadTokens: 4607182418800017400.00
- MemoryLimit: 16.00 GB
- PeakMemoryUsage: 168.91 MB
- PeakReservation: 0.00
- PeakUsedReservation: 0.00
- RowsProduced: 1
BlockMgr:
- BlockWritesOutstanding: 0
- BlocksCreated: 0
- BlocksRecycled: 0
- BufferedPins: 0
- BytesWritten: 0.00
- MaxBlockSize: 8.00 MB
- TotalBufferWaitTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=2, dst_fragments=[c10a18feb63c11ec-b2a6ac1f6be1dfab]):(Active: 458.220us[458220ns], % non-child: 0.00%)
- PartType: UNPARTITIONED
- BytesSent: 20.00 B
- CompressTime: 8.92us
- IgnoreRows: 0
- OverallThroughput: 42.6240234375 KB/sec
- PeakMemoryUsage: 16.00 KB
- SendRequestTime: 49.178us
- SerializeBatchTime: 9.220us
- ShuffleDispatchTime: 0ns
- ShuffleHashTime: 0ns
- UncompressedBytes: 20.00 B
- WaitResponseTime: 349.221us
AGGREGATION_NODE (id=1):(Active: 2m10s[130312697170ns], % non-child: 0.06%)
- AggregateFunctions: count()
- AggComputeTime: 23.900ms
- ExprComputeTime: 7.922ms
- ExprReleaseTime: 11.477ms
- GetResultsTime: 20.205us
- HashTableSize: 0
- InputRowCount: 730.729327M (730729327)
- PassThroughRowCount: 0
- PeakMemoryUsage: 4.00 KB
- ResultAggAppendTime: 0ns
- ResultGroupByAppendTime: 0ns
- ResultIteratorTime: 0ns
- RowsReturned: 1
- RowsReturnedRate: 0
- StreamingTime: 0ns
OLAP_SCAN_NODE (id=0):(Active: 2m10s[130213280448ns], % non-child: 0.00%)
- Table: subdomain_dns_analysis_ip_uniq
- Rollup: subdomain_dns_analysis_ip_uniq
- BytesRead: 66.66 GB
- NumDiskAccess: 0
- PeakMemoryUsage: 168.89 MB
- PerReadThreadRawHdfsThroughput: 0.0 /sec
- RowsRead: 0
- RowsReturned: 730.729327M (730729327)
- RowsReturnedRate: 5.611788M /sec
- ScanTime: 1h30m
- ScannerThreadsInvoluntaryContextSwitches: 0
- ScannerThreadsTotalWallClockTime: 0ns
- MaterializeTupleTime(): 0ns
- ScannerThreadsSysTime: 0ns
- ScannerThreadsUserTime: 0ns
- ScannerThreadsVoluntaryContextSwitches: 0
- TabletCount : 342
- TotalRawReadTime(): 0ns
- TotalReadThroughput: 519.3775749206543 MB/sec
MERGE:
- aggr: 1h25m
- sort: 1h24m
SCAN:(Active: 1h28m[5303445467705ns], % non-child: 100.00%)
- CachedPagesNum: 0
- CompressedBytesRead: 40.66 GB
- CreateSegmentIter: 967.271ms
- IOTime: 1h26m
- PushdownPredicates: 0
- RawRowsRead: 745.542363M (745542363)
- SegmentInit: 1m37s
- BitmapIndexFilter: 0ns
- BitmapIndexFilterRows: 0
- BloomFilterFilterRows: 0
- ShortKeyFilterRows: 0
- ZoneMapIndexFilterRows: 0
- SegmentRead: 1h26m
- BlockFetch: 1h20m
- BlockFetchCount: 182.567K (182567)
- BlockSeek: 6m9s
- BlockSeekCount: 1.481K (1481)
- ChunkCopy: 0ns
- DecompressT: 8s268ms
- DelVecFilterRows: 0
- IndexLoad: 0ns
- PredFilter: 0ns
- PredFilterRows: 0
- TotalPagesNum: 978.751K (978751)
- UncompressedBytesRead: 54.38 GB
Instance c10a18fe-b63c-11ec-b2a6-ac1f6be1dfa8 (host=TNetworkAddress(hostname:10.0.5.21, port:9060)):(Active: 1m31s[91105299157ns], % non-child: 0.00%)
- AverageThreadTokens: 4607182418800017400.00
- MemoryLimit: 16.00 GB
- PeakMemoryUsage: 178.65 MB
- PeakReservation: 0.00
- PeakUsedReservation: 0.00
- RowsProduced: 1
BlockMgr:
- BlockWritesOutstanding: 0
- BlocksCreated: 0
- BlocksRecycled: 0
- BufferedPins: 0
- BytesWritten: 0.00
- MaxBlockSize: 8.00 MB
- TotalBufferWaitTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=2, dst_fragments=[c10a18feb63c11ec-b2a6ac1f6be1dfab]):(Active: 1.889ms[1889666ns], % non-child: 0.00%)
- PartType: UNPARTITIONED
- BytesSent: 20.00 B
- CompressTime: 9.110us
- IgnoreRows: 0
- OverallThroughput: 10.3349609375 KB/sec
- PeakMemoryUsage: 16.00 KB
- SendRequestTime: 66.986us
- SerializeBatchTime: 3.770us
- ShuffleDispatchTime: 0ns
- ShuffleHashTime: 0ns
- UncompressedBytes: 20.00 B
- WaitResponseTime: 1.759ms
AGGREGATION_NODE (id=1):(Active: 1m31s[91103458937ns], % non-child: 0.05%)
- AggregateFunctions: count()
- AggComputeTime: 17.896ms
- ExprComputeTime: 7.134ms
- ExprReleaseTime: 10.850ms
- GetResultsTime: 14.832us
- HashTableSize: 0
- InputRowCount: 792.303847M (792303847)
- PassThroughRowCount: 0
- PeakMemoryUsage: 4.00 KB
- ResultAggAppendTime: 0ns
- ResultGroupByAppendTime: 0ns
- ResultIteratorTime: 0ns
- RowsReturned: 1
- RowsReturnedRate: 0
- StreamingTime: 0ns
OLAP_SCAN_NODE (id=0):(Active: 1m31s[91016637184ns], % non-child: 0.00%)
- Table: subdomain_dns_analysis_ip_uniq
- Rollup: subdomain_dns_analysis_ip_uniq
- BytesRead: 72.16 GB
- NumDiskAccess: 0
- PeakMemoryUsage: 178.64 MB
- PerReadThreadRawHdfsThroughput: 0.0 /sec
- RowsRead: 0
- RowsReturned: 792.303847M (792303847)
- RowsReturnedRate: 8.705044M /sec
- ScanTime: 1h1m
- ScannerThreadsInvoluntaryContextSwitches: 0
- ScannerThreadsTotalWallClockTime: 0ns
- MaterializeTupleTime(): 0ns
- ScannerThreadsSysTime: 0ns
- ScannerThreadsUserTime: 0ns
- ScannerThreadsVoluntaryContextSwitches: 0
- TabletCount : 339
- TotalRawReadTime(): 0ns
- TotalReadThroughput: 804.7028121948242 MB/sec
MERGE:
- aggr: 56m9s
- sort: 55m30s
SCAN:(Active: 58m35s[3515307539493ns], % non-child: 100.00%)
- CachedPagesNum: 0
- CompressedBytesRead: 43.52 GB
- CreateSegmentIter: 1s523ms
- IOTime: 57m9s
- PushdownPredicates: 0
- RawRowsRead: 807.068823M (807068823)
- SegmentInit: 44s128ms
- BitmapIndexFilter: 0ns
- BitmapIndexFilterRows: 0
- BloomFilterFilterRows: 0
- ShortKeyFilterRows: 0
- ZoneMapIndexFilterRows: 0
- SegmentRead: 57m50s
- BlockFetch: 54m6s
- BlockFetchCount: 197.577K (197577)
- BlockSeek: 3m43s
- BlockSeekCount: 1.428K (1428)
- ChunkCopy: 0ns
- DecompressT: 8s565ms
- DelVecFilterRows: 0
- IndexLoad: 0ns
- PredFilter: 0ns
- PredFilterRows: 0
- TotalPagesNum: 1.05897M (1058970)
- UncompressedBytesRead: 58.80 GB
set new_planner_agg_stage =2 将这个参数设置成2执行以下。您可以吧profile放到一个文件中。
你好，我遇到一个问题，如何判断并且设置new_planner_agg_stage参数的大小？似乎并不是越大越好，谢谢期待您的回复
你这个不是明细表，应该是Unique表
Unqiue表就会有这个问题，在Compaction完成之前。
用Duplicate表或PrimaryKey表会快