为了更快的定位您的问题,请提供以下信息,谢谢
【详述】问题详细描述
paimon查询必现以下问题,请问如何修复,我尝试着修改be的jvm参数,但是看起来没有效果
be.out日志如下:
Exception in thread “Thread-35” java.lang.OutOfMemoryError: Java heap space
at java.base/java.nio.HeapByteBuffer.(HeapByteBuffer.java:64)
at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:363)
at com.github.luben.zstd.RecyclingBufferPool.get(RecyclingBufferPool.java:42)
at com.github.luben.zstd.ZstdInputStreamNoFinalizer.(ZstdInputStreamNoFinalizer.java:65)
at com.github.luben.zstd.ZstdInputStream.(ZstdInputStream.java:34)
at org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstdDecompressorStream.(ZstdDecompressorStream.java:39)
at org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:94)
at org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:83)
at org.apache.paimon.shade.org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:112)
at org.apache.paimon.shade.org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader.readDictionaryPage(ColumnChunkPageReadStore.java:236)
at org.apache.paimon.format.parquet.reader.AbstractColumnReader.(AbstractColumnReader.java:110)
at org.apache.paimon.format.parquet.reader.IntColumnReader.(IntColumnReader.java:34)
at org.apache.paimon.format.parquet.reader.ParquetSplitReaderUtil.createColumnReader(ParquetSplitReaderUtil.java:105)
at org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.readNextRowGroup(ParquetReaderFactory.java:362)
at org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.nextBatch(ParquetReaderFactory.java:327)
at org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.readBatch(ParquetReaderFactory.java:309)
at org.apache.paimon.io.FileRecordReader.readBatch(FileRecordReader.java:47)
at org.apache.paimon.io.KeyValueDataFileRecordReader.readBatch(KeyValueDataFileRecordReader.java:48)
at org.apache.paimon.mergetree.compact.ConcatRecordReader.readBatch(ConcatRecordReader.java:66)
at org.apache.paimon.mergetree.compact.LoserTree$LeafIterator.advanceIfAvailable(LoserTree.java:315)
at org.apache.paimon.mergetree.compact.LoserTree.initializeIfNeeded(LoserTree.java:87)
at org.apache.paimon.mergetree.compact.SortMergeReaderWithLoserTree.readBatch(SortMergeReaderWithLoserTree.java:71)
at org.apache.paimon.mergetree.DropDeleteReader.readBatch(DropDeleteReader.java:44)
at org.apache.paimon.reader.RecordReader$1.readBatch(RecordReader.java:173)
at org.apache.paimon.table.source.KeyValueTableRead$1.readBatch(KeyValueTableRead.java:131)
at org.apache.paimon.reader.RecordReader$2.readBatch(RecordReader.java:194)
at org.apache.paimon.reader.RecordReaderIterator.(RecordReaderIterator.java:37)
at com.starrocks.paimon.reader.PaimonSplitScanner.initReader(PaimonSplitScanner.java:106)
at com.starrocks.paimon.reader.PaimonSplitScanner.open(PaimonSplitScanner.java:115)
【背景】做过哪些操作?
【业务影响】
【是否存算分离】
【StarRocks版本】例如:3.2.5
【集群规模】例如:3fe(1 follower+2observer)+5be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息,请补充下您的联系方式,例如:社区群4-小李或者邮箱,谢谢
【附件】
- fe.log/beINFO/相应截图
- 慢查询:
- Profile信息,获取Profile,通过Profile分析查询瓶颈
- 并行度:show variables like ‘%parallel_fragment_exec_instance_num%’;
- pipeline是否开启:show variables like ‘%pipeline%’;
- be节点cpu和内存使用率截图
- 查询报错:
- query_dump,怎么获取query_dump文件
- be crash
- be.out
- coredump,如何获取coredump
- 外表查询报错
- be.out和fe.warn.log