StarRocks 高并发主键查询性能问题

💬 StarRocks 用户问答 性能相关
#1

【详述】主键模型表，通过主键查询，profile中显示NetworkTime耗时高
【背景】压测工具压测
【业务影响】
【是否存算分离】否
【StarRocks版本】3.2.4
【集群规模】3fe（1 follower+2observer）+ 3be
【机器信息】基于kubernetes部署
FE部署信息

image
image1617×222 16.9 KB

BE部署信息
image
image1623×179 16.8 KB

主机信息
一共6台物理机，3台用于fe，3台用于be
CPU虚拟核/内存/网卡/磁盘：40C/256G/万兆/nvme SSD
【联系方式】hongweijin1993@gmail.com
【附件】

  • Profile信息
    profile8.yml (34.9 KB)

  • 并行度：show variables like ‘%parallel_fragment_exec_instance_num%’;
    show variables like ‘%pipeline_dop%’;

  • pipeline是否开启：show variables like ‘%pipeline%’;

  • 执行计划：explain costs + sql

  • be节点cpu和内存使用率截图

image
image1421×116 1.23 KB

image
image1210×237 3.33 KB

image
image1857×786 109 KB

看profile的话，NetworkTime耗时比较高，但是怎么去优化这个NetworkTime目前还没有比较好的思路

image
image1920×967 87.6 KB

压测方式

# 需要查询所有列
mysqlslap -uusername -ppassword -h127.0.0.1 -P9030 --query="select * from table where ( pk='320571000011270003830220240415002113776040654207' ) " --concurrency=100 --iterations=20

当前表数据总量：30万

#3

用GDB调试了一下，看了一下 pip_wg_executor 线程池中的线程，以下是线程工作情况

(gdb) thread 233
                [Switching to thread 233 (Thread 0x7f22d6ec1640 (LWP 782))]
#0  0x00000000069cae7b in my_free ()
(gdb) bt
        #0  0x00000000069cae7b in my_free ()
#1  0x0000000003f050fa in starrocks::TabletSchema::_init_from_pb(starrocks::TabletSchemaPB const&) ()
#2  0x0000000003f0681b in starrocks::TabletSchema::copy_from(std::shared_ptr<starrocks::TabletSchema const> const&) ()
#3  0x0000000003f06cea in starrocks::TabletSchema::copy(std::shared_ptr<starrocks::TabletSchema const> const&) ()
#4  0x00000000065b49df in starrocks::pipeline::OlapChunkSource::_init_olap_reader(starrocks::RuntimeState*) ()
#5  0x00000000065b569d in starrocks::pipeline::OlapChunkSource::prepare(starrocks::RuntimeState*) ()
#6  0x0000000005f347ca in starrocks::pipeline::ScanOperator::_pickup_morsel(starrocks::RuntimeState*, int) ()
#7  0x0000000005f33c40 in starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan(starrocks::RuntimeState*) ()
#8  0x0000000005f33f76 in starrocks::pipeline::ScanOperator::pull_chunk(starrocks::RuntimeState*) ()
#9  0x0000000005f2ba4f in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) ()
#10 0x00000000067c67ee in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() ()
#11 0x0000000006b57b93 in starrocks::ThreadPool::dispatch_thread() ()
#12 0x0000000006b51e6a in starrocks::Thread::supervise_thread(void*) ()
#13 0x00007f2360dd6ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007f2360e67a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb) thread 234
                [Switching to thread 234 (Thread 0x7f22d66c0640 (LWP 783))]
#0  0x00000000065b3aa0 in starrocks::pipeline::OlapChunkSource::_init_reader_params(std::vector<std::unique_ptr<starrocks::OlapScanRange, std::default_delete<starrocks::OlapScanRange> >, std::allocator<std::unique_ptr<starrocks::OlapScanRange, std::default_delete<starrocks::OlapScanRange> > > > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> >&) ()
(gdb) bt
        #0  0x00000000065b3aa0 in starrocks::pipeline::OlapChunkSource::_init_reader_params(std::vector<std::unique_ptr<starrocks::OlapScanRange, std::default_delete<starrocks::OlapScanRange> >, std::allocator<std::unique_ptr<starrocks::OlapScanRange, std::default_delete<starrocks::OlapScanRange> > > > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> >&) ()
#1  0x00000000065b4b70 in starrocks::pipeline::OlapChunkSource::_init_olap_reader(starrocks::RuntimeState*) ()
#2  0x00000000065b569d in starrocks::pipeline::OlapChunkSource::prepare(starrocks::RuntimeState*) ()
#3  0x0000000005f347ca in starrocks::pipeline::ScanOperator::_pickup_morsel(starrocks::RuntimeState*, int) ()
#4  0x0000000005f33c40 in starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan(starrocks::RuntimeState*) ()
#5  0x0000000005f33f76 in starrocks::pipeline::ScanOperator::pull_chunk(starrocks::RuntimeState*) ()
#6  0x0000000005f2ba4f in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) ()
#7  0x00000000067c67ee in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() ()
#8  0x0000000006b57b93 in starrocks::ThreadPool::dispatch_thread() ()
#9  0x0000000006b51e6a in starrocks::Thread::supervise_thread(void*) ()
#10 0x00007f2360dd6ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007f2360e67a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb) thread 235
                [Switching to thread 235 (Thread 0x7f22d5ebf640 (LWP 784))]
#0  0x0000000003f1e90b in std::__detail::_Map_base<starrocks::LogicalType, std::pair<starrocks::LogicalType const, std::unique_ptr<starrocks::ScalarTypeInfo, std::default_delete<starrocks::ScalarTypeInfo> > >, std::allocator<std::pair<starrocks::LogicalType const, std::unique_ptr<starrocks::ScalarTypeInfo, std::default_delete<starrocks::ScalarTypeInfo> > > >, std::__detail::_Select1st, std::equal_to<starrocks::LogicalType>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](starrocks::LogicalType const&) ()
(gdb) bt
        #0  0x0000000003f1e90b in std::__detail::_Map_base<starrocks::LogicalType, std::pair<starrocks::LogicalType const, std::unique_ptr<starrocks::ScalarTypeInfo, std::default_delete<starrocks::ScalarTypeInfo> > >, std::allocator<std::pair<starrocks::LogicalType const, std::unique_ptr<starrocks::ScalarTypeInfo, std::default_delete<starrocks::ScalarTypeInfo> > > >, std::__detail::_Select1st, std::equal_to<starrocks::LogicalType>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](starrocks::LogicalType const&) ()
#1  0x0000000003f0fa3d in starrocks::get_type_info(starrocks::LogicalType) ()
#2  0x0000000003f23d63 in starrocks::ChunkHelper::convert_field(unsigned int, starrocks::TabletColumn const&) ()
#3  0x0000000003f03b7d in starrocks::TabletSchema::_init_schema() const ()
#4  0x00007f2360ddbee8 in __pthread_once_slow (once_control=0x7f219bd24e00, init_routine=0xa5e7090 <__once_proxy>) at ./nptl/pthread_once.c:116
#5  0x0000000003f00885 in starrocks::TabletSchema::schema() const ()
#6  0x0000000003f2208a in starrocks::ChunkHelper::convert_schema(std::shared_ptr<starrocks::TabletSchema const> const&, std::vector<unsigned int, std::allocator<unsigned int> > const&) ()
#7  0x00000000065b4bcd in starrocks::pipeline::OlapChunkSource::_init_olap_reader(starrocks::RuntimeState*) ()
#8  0x00000000065b569d in starrocks::pipeline::OlapChunkSource::prepare(starrocks::RuntimeState*) ()
#9  0x0000000005f347ca in starrocks::pipeline::ScanOperator::_pickup_morsel(starrocks::RuntimeState*, int) ()
#10 0x0000000005f33c40 in starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan(starrocks::RuntimeState*) ()
#11 0x0000000005f33f76 in starrocks::pipeline::ScanOperator::pull_chunk(starrocks::RuntimeState*) ()
#12 0x0000000005f2ba4f in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) ()
#13 0x00000000067c67ee in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() ()
#14 0x0000000006b57b93 in starrocks::ThreadPool::dispatch_thread() ()
#15 0x0000000006b51e6a in starrocks::Thread::supervise_thread(void*) ()
#16 0x00007f2360dd6ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#17 0x00007f2360e67a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb) thread 236
                [Switching to thread 236 (Thread 0x7f22d56be640 (LWP 785))]
#0  0x00000000070301d4 in starrocks::TColumnType::~TColumnType() ()
(gdb) bt
        #0  0x00000000070301d4 in starrocks::TColumnType::~TColumnType() ()
#1  0x0000000006f0e4e2 in starrocks::TColumn::~TColumn() ()
#2  0x0000000003f066f7 in starrocks::TabletColumn::init_from_thrift(starrocks::TColumn const&) ()
#3  0x0000000003f067a2 in starrocks::TabletColumn::TabletColumn(starrocks::TColumn const&) ()
#4  0x00000000065b521a in starrocks::pipeline::OlapChunkSource::_init_olap_reader(starrocks::RuntimeState*) ()
#5  0x00000000065b569d in starrocks::pipeline::OlapChunkSource::prepare(starrocks::RuntimeState*) ()
#6  0x0000000005f347ca in starrocks::pipeline::ScanOperator::_pickup_morsel(starrocks::RuntimeState*, int) ()
#7  0x0000000005f33c40 in starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan(starrocks::RuntimeState*) ()
#8  0x0000000005f33f76 in starrocks::pipeline::ScanOperator::pull_chunk(starrocks::RuntimeState*) ()
#9  0x0000000005f2ba4f in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) ()
#10 0x00000000067c67ee in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() ()
#11 0x0000000006b57b93 in starrocks::ThreadPool::dispatch_thread() ()
#12 0x0000000006b51e6a in starrocks::Thread::supervise_thread(void*) ()
#13 0x00007f2360dd6ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007f2360e67a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb) thread 237
                [Switching to thread 237 (Thread 0x7f22d4ebd640 (LWP 786))]
#0  0x0000000008ae0e96 in je_arena_cache_bin_fill_small ()
(gdb) bt
        #0  0x0000000008ae0e96 in je_arena_cache_bin_fill_small ()
#1  0x0000000008b6f2ae in je_tcache_alloc_small_hard ()
#2  0x0000000008ac9cb0 in je_malloc_default ()
#3  0x00000000069c9fe0 in my_malloc ()
#4  0x000000000a561c4c in operator new(unsigned long) ()
#5  0x0000000003effe63 in starrocks::TabletColumn::TabletColumn(starrocks::TabletColumn const&) ()
#6  0x0000000003f05050 in starrocks::TabletSchema::_init_from_pb(starrocks::TabletSchemaPB const&) ()
#7  0x0000000003f0681b in starrocks::TabletSchema::copy_from(std::shared_ptr<starrocks::TabletSchema const> const&) ()
#8  0x0000000003f06cea in starrocks::TabletSchema::copy(std::shared_ptr<starrocks::TabletSchema const> const&) ()
#9  0x00000000065b49df in starrocks::pipeline::OlapChunkSource::_init_olap_reader(starrocks::RuntimeState*) ()
#10 0x00000000065b569d in starrocks::pipeline::OlapChunkSource::prepare(starrocks::RuntimeState*) ()
#11 0x0000000005f347ca in starrocks::pipeline::ScanOperator::_pickup_morsel(starrocks::RuntimeState*, int) ()
#12 0x0000000005f33c40 in starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan(starrocks::RuntimeState*) ()
#13 0x0000000005f33f76 in starrocks::pipeline::ScanOperator::pull_chunk(starrocks::RuntimeState*) ()
#14 0x0000000005f2ba4f in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) ()
#15 0x00000000067c67ee in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() ()
#16 0x0000000006b57b93 in starrocks::ThreadPool::dispatch_thread() ()
#17 0x0000000006b51e6a in starrocks::Thread::supervise_thread(void*) ()
#18 0x00007f2360dd6ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f2360e67a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb) thread 238
                [Switching to thread 238 (Thread 0x7f22d46bc640 (LWP 787))]
#0  0x000000000a561b3c in std::_Hash_bytes(void const*, unsigned long, unsigned long) ()
(gdb) bt
        #0  0x000000000a561b3c in std::_Hash_bytes(void const*, unsigned long, unsigned long) ()
#1  0x000000000404e882 in std::pair<std::__detail::_Node_iterator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, unsigned long>, false, true>, bool> std::_Hashtable<std::basic_string_view<char, std::char_traits<char> >, std::pair<std::basic_string_view<char, std::char_traits<char> > const, unsigned long>, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, unsigned long> >, std::__detail::_Select1st, std::equal_to<std::basic_string_view<char, std::char_traits<char> > >, std::hash<std::basic_string_view<char, std::char_traits<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::basic_string_view<char, std::char_traits<char> >, unsigned long>(std::integral_constant<bool, true>, std::basic_string_view<char, std::char_traits<char> >&&, unsigned long&&) [clone .isra.0] ()
#2  0x000000000404e9cc in starrocks::Schema::append(std::shared_ptr<starrocks::Field> const&) ()
#3  0x0000000004093a6a in starrocks::ChunkIterator::init_encoded_schema(phmap::flat_hash_map<unsigned int, phmap::flat_hash_map<starrocks::Slice, int, starrocks::SliceHashWithSeed<(starrocks::PhmapSeed)0>, starrocks::SliceEqual, std::allocator<std::pair<starrocks::Slice const, int> > >*, phmap::Hash<unsigned int>, phmap::EqualTo<unsigned int>, std::allocator<std::pair<unsigned int const, phmap::flat_hash_map<starrocks::Slice, int, starrocks::SliceHashWithSeed<(starrocks::PhmapSeed)0>, starrocks::SliceEqual, std::allocator<std::pair<starrocks::Slice const, int> > >*> > >&) ()
#4  0x00000000056d7572 in starrocks::ProjectionIterator::init_encoded_schema(phmap::flat_hash_map<unsigned int, phmap::flat_hash_map<starrocks::Slice, int, starrocks::SliceHashWithSeed<(starrocks::PhmapSeed)0>, starrocks::SliceEqual, std::allocator<std::pair<starrocks::Slice const, int> > >*, phmap::Hash<unsigned int>, phmap::EqualTo<unsigned int>, std::allocator<std::pair<unsigned int const, phmap::flat_hash_map<starrocks::Slice, int, starrocks::SliceHashWithSeed<(starrocks::PhmapSeed)0>, starrocks::SliceEqual, std::allocator<std::pair<starrocks::Slice const, int> > >*> > >&) ()
#5  0x0000000004093d1c in starrocks::SegmentIteratorWrapper::init_encoded_schema(phmap::flat_hash_map<unsigned int, phmap::flat_hash_map<starrocks::Slice, int, starrocks::SliceHashWithSeed<(starrocks::PhmapSeed)0>, starrocks::SliceEqual, std::allocator<std::pair<starrocks::Slice const, int> > >*, phmap::Hash<unsigned int>, phmap::EqualTo<unsigned int>, std::allocator<std::pair<unsigned int const, phmap::flat_hash_map<starrocks::Slice, int, starrocks::SliceHashWithSeed<(starrocks::PhmapSeed)0>, starrocks::SliceEqual, std::allocator<std::pair<starrocks::Slice const, int> > >*> > >&) ()
#6  0x0000000005648475 in starrocks::TimedChunkIterator::init_encoded_schema(phmap::flat_hash_map<unsigned int, phmap::flat_hash_map<starrocks::Slice, int, starrocks::SliceHashWithSeed<(starrocks::PhmapSeed)0>, starrocks::SliceEqual, std::allocator<std::pair<starrocks::Slice const, int> > >*, phmap::Hash<unsigned int>, phmap::EqualTo<unsigned int>, std::allocator<std::pair<unsigned int const, phmap::flat_hash_map<starrocks::Slice, int, starrocks::SliceHashWithSeed<(starrocks::PhmapSeed)0>, starrocks::SliceEqual, std::allocator<std::pair<starrocks::Slice const, int> > >*> > >&) ()
#7  0x00000000042b2e76 in starrocks::TabletReader::_init_collector(starrocks::TabletReaderParams const&) ()
#8  0x00000000042b40e9 in starrocks::TabletReader::open(starrocks::TabletReaderParams const&) ()
#9  0x00000000065b50df in starrocks::pipeline::OlapChunkSource::_init_olap_reader(starrocks::RuntimeState*) ()
#10 0x00000000065b569d in starrocks::pipeline::OlapChunkSource::prepare(starrocks::RuntimeState*) ()
#11 0x0000000005f347ca in starrocks::pipeline::ScanOperator::_pickup_morsel(starrocks::RuntimeState*, int) ()
#12 0x0000000005f33c40 in starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan(starrocks::RuntimeState*) ()
#13 0x0000000005f33f76 in starrocks::pipeline::ScanOperator::pull_chunk(starrocks::RuntimeState*) ()
#14 0x0000000005f2ba4f in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) ()
#15 0x00000000067c67ee in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() ()
#16 0x0000000006b57b93 in starrocks::ThreadPool::dispatch_thread() ()
#17 0x0000000006b51e6a in starrocks::Thread::supervise_thread(void*) ()
#18 0x00007f2360dd6ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f2360e67a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb) thread 241
                [Switching to thread 241 (Thread 0x7f22d2eb9640 (LWP 790))]
#0  0x0000000005d56422 in starrocks::new_segment_iterator(std::shared_ptr<starrocks::Segment> const&, starrocks::Schema const&, starrocks::SegmentReadOptions const&) ()
(gdb) bt
        #0  0x0000000005d56422 in starrocks::new_segment_iterator(std::shared_ptr<starrocks::Segment> const&, starrocks::Schema const&, starrocks::SegmentReadOptions const&) ()
#1  0x0000000004097d55 in starrocks::Segment::_new_iterator(starrocks::Schema const&, starrocks::SegmentReadOptions const&) ()
#2  0x0000000004098073 in starrocks::Segment::new_iterator(starrocks::Schema const&, starrocks::SegmentReadOptions const&) ()
#3  0x000000000408e216 in starrocks::Rowset::get_segment_iterators(starrocks::Schema const&, starrocks::RowsetReadOptions const&, std::vector<std::shared_ptr<starrocks::ChunkIterator>, std::allocator<std::shared_ptr<starrocks::ChunkIterator> > >*) ()
#4  0x00000000042b2563 in starrocks::TabletReader::get_segment_iterators(starrocks::TabletReaderParams const&, std::vector<std::shared_ptr<starrocks::ChunkIterator>, std::allocator<std::shared_ptr<starrocks::ChunkIterator> > >*) ()
#5  0x00000000042b2995 in starrocks::TabletReader::_init_collector(starrocks::TabletReaderParams const&) ()
#6  0x00000000042b40e9 in starrocks::TabletReader::open(starrocks::TabletReaderParams const&) ()
#7  0x00000000065b50df in starrocks::pipeline::OlapChunkSource::_init_olap_reader(starrocks::RuntimeState*) ()
#8  0x00000000065b569d in starrocks::pipeline::OlapChunkSource::prepare(starrocks::RuntimeState*) ()
#9  0x0000000005f347ca in starrocks::pipeline::ScanOperator::_pickup_morsel(starrocks::RuntimeState*, int) ()
#10 0x0000000005f33c40 in starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan(starrocks::RuntimeState*) ()
#11 0x0000000005f33f76 in starrocks::pipeline::ScanOperator::pull_chunk(starrocks::RuntimeState*) ()
#12 0x0000000005f2ba4f in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) ()
#13 0x00000000067c67ee in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() ()
#14 0x0000000006b57b93 in starrocks::ThreadPool::dispatch_thread() ()
#15 0x0000000006b51e6a in starrocks::Thread::supervise_thread(void*) ()
#16 0x00007f2360dd6ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#17 0x00007f2360e67a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb) thread 242
                [Switching to thread 242 (Thread 0x7f22d26b8640 (LWP 791))]
#0  0x0000000006a45520 in starrocks::string_to_logical_type(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
(gdb) bt
        #0  0x0000000006a45520 in starrocks::string_to_logical_type(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#1  0x0000000003f036a9 in starrocks::TabletColumn::init_from_pb(starrocks::ColumnPB const&) ()
#2  0x0000000003f0668e in starrocks::TabletColumn::init_from_thrift(starrocks::TColumn const&) ()
#3  0x0000000003f067a2 in starrocks::TabletColumn::TabletColumn(starrocks::TColumn const&) ()
#4  0x00000000065b521a in starrocks::pipeline::OlapChunkSource::_init_olap_reader(starrocks::RuntimeState*) ()
#5  0x00000000065b569d in starrocks::pipeline::OlapChunkSource::prepare(starrocks::RuntimeState*) ()
#6  0x0000000005f347ca in starrocks::pipeline::ScanOperator::_pickup_morsel(starrocks::RuntimeState*, int) ()
#7  0x0000000005f33c40 in starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan(starrocks::RuntimeState*) ()
#8  0x0000000005f33f76 in starrocks::pipeline::ScanOperator::pull_chunk(starrocks::RuntimeState*) ()
#9  0x0000000005f2ba4f in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) ()
#10 0x00000000067c67ee in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() ()
#11 0x0000000006b57b93 in starrocks::ThreadPool::dispatch_thread() ()
#12 0x0000000006b51e6a in starrocks::Thread::supervise_thread(void*) ()
#13 0x00007f2360dd6ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007f2360e67a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb) thread 243
                [Switching to thread 243 (Thread 0x7f22d1eb7640 (LWP 792))]
#0  0x00000000069ca020 in my_malloc ()
(gdb) bt
        #0  0x00000000069ca020 in my_malloc ()
#1  0x000000000a561c4c in operator new(unsigned long) ()
#2  0x0000000005771ff2 in void std::vector<starrocks::SlotDescriptor*, std::allocator<starrocks::SlotDescriptor*> >::_M_realloc_insert<starrocks::SlotDescriptor* const&>(__gnu_cxx::__normal_iterator<starrocks::SlotDescriptor**, std::vector<starrocks::SlotDescriptor*, std::allocator<starrocks::SlotDescriptor*> > >, starrocks::SlotDescriptor* const&) ()
#3  0x00000000065b32d3 in starrocks::pipeline::OlapChunkSource::_init_scanner_columns(std::vector<unsigned int, std::allocator<unsigned int> >&) ()
#4  0x00000000065b4b32 in starrocks::pipeline::OlapChunkSource::_init_olap_reader(starrocks::RuntimeState*) ()
#5  0x00000000065b569d in starrocks::pipeline::OlapChunkSource::prepare(starrocks::RuntimeState*) ()
#6  0x0000000005f347ca in starrocks::pipeline::ScanOperator::_pickup_morsel(starrocks::RuntimeState*, int) ()
#7  0x0000000005f33c40 in starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan(starrocks::RuntimeState*) ()
#8  0x0000000005f33f76 in starrocks::pipeline::ScanOperator::pull_chunk(starrocks::RuntimeState*) ()
#9  0x0000000005f2ba4f in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) ()
#10 0x00000000067c67ee in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() ()
#11 0x0000000006b57b93 in starrocks::ThreadPool::dispatch_thread() ()
#12 0x0000000006b51e6a in starrocks::Thread::supervise_thread(void*) ()
#13 0x00007f2360dd6ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007f2360e67a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
#4

能获取个perf top吗

#5

应该是3.2引入的一个问题，可以试试3.1版本压测下

#6

好的，我找个时间压测一下3.1的版本，如果还有问题，就给个perf结果哈，感谢回复

#7

3.2, 3.1都帮获取下perf top

#8

好的，等我测一下

#9

大佬，这是已知问题嘛，如果是已知问题，预计啥时候修复呀，有没有issue链接呀

#10

看堆栈是3.2加了一个TabletSchema Copy导致，先看看perf top看，通过perf top先确认下是不是这个问题。

#11

perf top 查看并未发现有CPU占用特别高的地方（StarRocks版本 3.2.4）

# 执行脚本
perf top -p ${pid}

# 压测语句
mysqlslap -uusername -ppassword -hhost -P9030 --query="select * from db.table where ( id='320571000011270003830220240415002113776040654207' )" --concurrency=100 --iterations=2000

image
image1902×764 65 KB

#12

starrocks-3.2.4-perf.data (13.6 MB)

#13

好的收到 我们看下

#14

StarRocks 3.1.10 版本perf信息

这里搞了个表结构非常简单的表来测试

# 压测SQL
select `pk`,`value` from `pk` where ( pk='05fb3fc0-ea16-7c1f-e647-070062010000' ) and (VPCIsDelete = false) limit 1;

# 表结构

CREATE TABLE
  `pk` (
    `pk` varchar(1048576) NOT NULL COMMENT "pk",
    `value` varchar(1048576) NULL COMMENT "pk",
    `VPCIsDelete` boolean NULL COMMENT ""
  ) ENGINE = OLAP PRIMARY KEY (`pk`) DISTRIBUTED BY HASH (`pk`) PROPERTIES (
    "replication_num" = "3",
    "in_memory" = "false",
    "enable_persistent_index" = "true",
    "replicated_storage" = "true",
    "compression" = "LZ4"
  );

starrocks-3.1.10-perf.data (4.2 MB)

image
image1913×840 74.5 KB

有个现象就是vdso排名比较很高

image
image1920×838 72.7 KB

#15

你是所有都开启profile了？

#16

不是的，
我设置了 big_query_profile_threshold = ‘2s’，见：https://docs.starrocks.io/zh/docs/reference/System_variable/#big_query_profile_threshold

#17

今天换了个场景压测


-- 这个时间区间的数据总量是1500万
mysqlslap -uusername -ppassword -hhost -P9030 --create-schema=test_cos --query="SELECT count(1) FROM db.table  WHERE ShotTime >= '2024-04-09' AND ShotTime < '2024-04-12'" --concurrency=200 --iterations=20

效果依旧不是很理想，但是StarRocks集群BE的内存和CPU使用率都很低

image
image2878×670 63 KB

image
image2750×922 381 KB

image
image884×892 66.7 KB

image
image2876×1710 457 KB

这里还是咨询一下社区的大佬，该从什么地方分析呢？

#18

be conf brpc_max_connections_per_server 改成16试试

#19

这个之前调整过，那我再调整一下试试

#20

我试了一下，效果依旧不理想，还是有许多查询的NetWork耗时很高

image
image2864×1590 421 KB

image
image2854×994 451 KB

image
image2878×1626 636 KB

#21

这里想要咨询一下，社区这边是否有对于主键模型高并发的经验或者最佳实践呢？

京ICP备2022026421号-1