[编译] ARM 架构编译问题咨询

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】基于openEuler(ARM架构) 22.03 LTS版本使用 starrocks/dev-env-ubuntu:3.3-latest 编译StarRocks,无法编译
【背景】需要编译StarRocks ARM包

Docker 启动命令

nerdctl run  -it       \
--ulimit memlock=-1:-1     \
--ulimit nofile=1048576:1048576    \
 -m 16g         \
-v "/root/starrocks:/root/starrocks"       \
 -v "/root/.m2:/root/.m2"         \
--name starrocks-dev \
--privileged=true         \
starrocks/dev-env-ubuntu:3.3-latest

编译命令:

./build.sh --be -j 1

报错信息

Found python3, version: Python 3.10.12
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 11.0.23, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-arm64
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "5.10.0-60.18.0.50.oe2203.aarch64", arch: "aarch64", family: "unix"
Get params:
    BUILD_BE            -- 1
    BE_CMAKE_TYPE       -- Release
    BUILD_FE            -- 0
    BUILD_SPARK_DPP     -- 0
    BUILD_HIVE_UDF      -- 0
    CCACHE              -- ccache
    CLEAN               -- 0
    RUN_UT              -- 0
    WITH_GCOV           -- OFF
    WITH_BENCH          -- OFF
    WITH_CLANG_TIDY     -- OFF
    WITH_COMPRESS_DEBUG_SYMBOL  -- ON
    WITH_STARCACHE      -- ON
    ENABLE_SHARED_DATA  -- OFF
    USE_AVX2            -- OFF
    USE_AVX512          -- OFF
    USE_SSE4_2          -- OFF
    JEMALLOC_DEBUG      -- OFF
    PARALLEL            -- 1
    ENABLE_QUERY_DEBUG_TRACE -- OFF
    ENABLE_FAULT_INJECTION -- OFF
    BUILD_JAVA_EXT      -- ON
    OUTPUT_COMPILE_TIME   -- OFF

Build generated code
make -C script
make[1]: Entering directory '/root/starrocks/gensrc/script'
python3 /root/starrocks/gensrc/script/../../build-support/gen_build_version.py --cpp /root/starrocks/gensrc/script/../build//gen_cpp --java /root/starrocks/gensrc/script/../../fe/fe-core/target/generated-sources/build
gen_build_version.py /root/starrocks/gensrc/script/../build//gen_cpp/version.cpp: old fingerprint = 6d1192a6339b77b68e2a871478a38e2a, new fingerprint = 6d1192a6339b77b68e2a871478a38e2a
gen_build_version.py /root/starrocks/gensrc/script/../../fe/fe-core/target/generated-sources/build/com/starrocks/common/Version.java: old fingerprint = 87880dfaba4150774bb0fa6bb321e6c3, new fingerprint = 87880dfaba4150774bb0fa6bb321e6c3
make[1]: Leaving directory '/root/starrocks/gensrc/script'
make -C proto
make[1]: Entering directory '/root/starrocks/gensrc/proto'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/starrocks/gensrc/proto'
make -C thrift
make[1]: Entering directory '/root/starrocks/gensrc/thrift'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/starrocks/gensrc/thrift'
cmake version 3.22.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
Build Backend: Release
-- Build type is RELEASE
-- Build target arch is aarch64
-- make test: OFF
-- GCC version 11.4.0 is greater than 10.3.0, disable -Werror. Be careful with compile warnings.
-- defined PIC_LIB_PATH
-- build gensrc if necessary
make: Entering directory '/root/starrocks/gensrc'
make -C script
make[1]: Entering directory '/root/starrocks/gensrc/script'
python3 /root/starrocks/gensrc/script/../../build-support/gen_build_version.py --cpp /root/starrocks/gensrc/script/../build//gen_cpp --java /root/starrocks/gensrc/script/../../fe/fe-core/target/generated-sources/build
gen_build_version.py /root/starrocks/gensrc/script/../build//gen_cpp/version.cpp: old fingerprint = 6d1192a6339b77b68e2a871478a38e2a, new fingerprint = 6d1192a6339b77b68e2a871478a38e2a
gen_build_version.py /root/starrocks/gensrc/script/../../fe/fe-core/target/generated-sources/build/com/starrocks/common/Version.java: old fingerprint = 87880dfaba4150774bb0fa6bb321e6c3, new fingerprint = 87880dfaba4150774bb0fa6bb321e6c3
make[1]: Leaving directory '/root/starrocks/gensrc/script'
make -C proto
make[1]: Entering directory '/root/starrocks/gensrc/proto'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/starrocks/gensrc/proto'
make -C thrift
make[1]: Entering directory '/root/starrocks/gensrc/thrift'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/starrocks/gensrc/thrift'
make: Leaving directory '/root/starrocks/gensrc'
-- /var/local/thirdparty/installed/lib/libboost_thread.a/var/local/thirdparty/installed/lib/libboost_regex.a/var/local/thirdparty/installed/lib/libboost_program_options.a/var/local/thirdparty/installed/lib/libboost_filesystem.a/var/local/thirdparty/installed/lib/libboost_context.a/var/local/thirdparty/installed/lib/libboost_chrono.a/var/local/thirdparty/installed/lib/libboost_atomic.a
-- Using OpenSSL Root Dir: /var/local/thirdparty/installed/
-- Found AWS SDK for C++, Version: 1.11.267, Install Root:/var/local/thirdparty/installed, Platform Prefix:, Platform Dependent Libraries: pthread;crypto;ssl;z;curl
-- Components specified for AWSSDK: core;s3;s3-crt;transfer;identity-management;sts, application will be depending on libs: aws-cpp-sdk-s3-crt;aws-cpp-sdk-transfer;aws-cpp-sdk-identity-management;aws-cpp-sdk-cognito-identity;aws-cpp-sdk-sts;aws-cpp-sdk-s3;aws-cpp-sdk-core;aws-crt-cpp;aws-c-auth;aws-c-cal;aws-c-common;aws-c-compression;aws-c-event-stream;aws-c-http;aws-c-io;aws-c-mqtt;aws-c-s3;aws-checksums;aws-c-sdkutils;pthread;crypto;ssl;z;curl
-- Try finding aws-cpp-sdk-core
-- S2N found target: crypto
-- crypto Include Dir: crypto_INCLUDE_DIR-NOTFOUND
-- S2N found target: crypto
-- crypto Include Dir: crypto_INCLUDE_DIR-NOTFOUND
-- S2N found target: crypto
-- crypto Include Dir: crypto_INCLUDE_DIR-NOTFOUND
-- S2N found target: crypto
-- crypto Include Dir: crypto_INCLUDE_DIR-NOTFOUND
-- S2N found target: crypto
-- crypto Include Dir: crypto_INCLUDE_DIR-NOTFOUND
-- S2N found target: crypto
-- crypto Include Dir: crypto_INCLUDE_DIR-NOTFOUND
-- S2N found target: crypto
-- crypto Include Dir: crypto_INCLUDE_DIR-NOTFOUND
-- S2N found target: crypto
-- crypto Include Dir: crypto_INCLUDE_DIR-NOTFOUND
-- Found aws-cpp-sdk-core
-- Try finding aws-cpp-sdk-s3
-- Found aws-cpp-sdk-s3
-- Try finding aws-cpp-sdk-sts
-- Found aws-cpp-sdk-sts
-- Try finding aws-cpp-sdk-cognito-identity
-- Found aws-cpp-sdk-cognito-identity
-- Try finding aws-cpp-sdk-identity-management
-- Found aws-cpp-sdk-identity-management
-- Try finding aws-cpp-sdk-transfer
-- Found aws-cpp-sdk-transfer
-- Try finding aws-cpp-sdk-s3-crt
-- Found aws-cpp-sdk-s3-crt
-- link the starcache in directory: /var/local/thirdparty/installed//starcache
-- Compiler Flags:   -Wall -Wno-sign-compare -Wno-unknown-pragmas -pthread -Wno-register -Wno-strict-aliasing -fno-omit-frame-pointer -std=gnu++20 -D__STDC_FORMAT_MACROS -Wno-deprecated -Wno-vla -Wno-comment -ffile-prefix-map=/root/starrocks/be=be -fno-sized-deallocation -DBOOST_DATE_TIME_POSIX_TIME_STD_CONFIG -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -Werror=return-type -Werror=switch -DWITH_STARCACHE -march=armv8-a+crc  -Wno-attributes -DS2_USE_GFLAGS -DS2_USE_GLOG -faligned-new -gz=zlib  -fcoroutines -g -Wno-unused-local-typedefs -O3 -gdwarf-4 -DNDEBUG
use jemalloc
++++ <STARROCKS_HOME> is set. formats/orc/apache-orc is compiled in StarRocks env ++++
-- Build type: RELEASE
-- compiler GNU version 11.4.0
-- SNAPPY_HOME: /var/local/thirdparty/installed
-- Found the Snappy header: /var/local/thirdparty/installed/include/snappy.h
-- Found the Snappy library: /var/local/thirdparty/installed/lib/libsnappy.a
-- ZLIB_HOME: /var/local/thirdparty/installed
-- Found the ZLIB header: /var/local/thirdparty/installed/include/zlib.h
-- Found the ZLIB library: /var/local/thirdparty/installed/lib/libz.a
-- Found the ZLIB static library: /var/local/thirdparty/installed/lib/libz.a
-- ZSTD_HOME: /var/local/thirdparty/installed
-- Found the zstd header: /var/local/thirdparty/installed/include/zstd/zstd.h
-- Found the zstd library: /var/local/thirdparty/installed/lib
-- LZ4_HOME: /var/local/thirdparty/installed
-- Found the LZ4 header: /var/local/thirdparty/installed/include/lz4/lz4.h
-- Found the LZ4 library: /var/local/thirdparty/installed/lib
-- PROTOBUF_HOME: /var/local/thirdparty/installed
-- Found the Protobuf headers: /var/local/thirdparty/installed/include
-- Found the Protobuf library: /var/local/thirdparty/installed/lib/libprotobuf.a
-- Found the Protoc library: /var/local/thirdparty/installed/lib/libprotoc.a
-- Found the Protoc executable: /var/local/thirdparty/installed/bin/protoc
-- Found the Protobuf static library: /var/local/thirdparty/installed/lib/libprotobuf.a
-- Found the Protoc static library: /var/local/thirdparty/installed/lib/libprotoc.a
-- Configuring done
-- Generating done
-- Build files have been written to: /root/starrocks/be/build_Release
Consolidate compiler generated dependencies of target build_version
[  0%] Built target build_version
Consolidate compiler generated dependencies of target Agent
[  0%] Building CXX object src/agent/CMakeFiles/Agent.dir/heartbeat_server.cpp.o
In file included from /usr/include/c++/11/memory:76,
                 from /root/starrocks/be/src/agent/heartbeat_server.h:37,
                 from /root/starrocks/be/src/agent/heartbeat_server.cpp:35:
/usr/include/c++/11/bits/unique_ptr.h:596:11: internal compiler error: Segmentation fault
  596 |         { }
      |           ^
0xb500b3 internal_error(char const*, ...)
        ???:0
0x11e30e8 ggc_set_mark(void const*)
        ???:0
0x11dfd1b gt_ggc_mx_lang_tree_node(void*)
        ???:0
0x63a227 gt_ggc_mx_vec_cp_token_va_gc_(void*)
        ???:0
0x63a1a3 gt_ggc_mx_cp_lexer(void*)
        ???:0
0x63a0f7 gt_ggc_mx_cp_parser(void*)
        ???:0
0x14ee057 ggc_mark_roots()
        ???:0
0xcbd683 ggc_collect()
        ???:0
0xe0f10b expand_or_defer_fn_1(tree_node*)
        ???:0
0x1326b97 c_parse_file()
        ???:0
0x1311023 c_common_parse_file()
        ???:0
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <file:///usr/share/doc/gcc-11/README.Bugs> for instructions.
make[2]: *** [src/agent/CMakeFiles/Agent.dir/build.make:90: src/agent/CMakeFiles/Agent.dir/heartbeat_server.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:643: src/agent/CMakeFiles/Agent.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

real    0m7.216s
user    0m5.858s
sys     0m1.296s

论坛中未搜索到相似错误,在google中搜索有部分用户说internal compiler error: Segmentation fault 报错是因为 open files 配置太小,目前也调整了,但是并没有作用,这里想咨询一下官方是在什么机器上用docker做编译的?

尝试在amd架构的机器上用qemu模拟,倒是可以进行编译
执行操作

docker run --rm --privileged  multiarch/qemu-user-static:latest --reset -p yes

### 剩下流程和上述一致

没见过这个问题. 尝试用 starrocks/dev-env-centos7:3.3-latest看看?

好的,我试试

大佬可以帮忙看下这个问题嘛,在跑BE单测,加了–with-gcov (enable to build with gcov)参数
在链接的时候报错 reloaction out of range的错误

用的最新的代码,docker镜像是starrocks/dev-env-ubuntu:latest

@lvlouisaslia

原因: ASAN build和gcov build时, 编译器都会注入很多代码检查内存和覆盖率, 导致binary VM SIZE超过2G的elf格式限制

解决办法: BUILD_TYPE=ASAN和–gcov-build不要同时用.