Re: Build failure on the Reef release

[Yong Yuan <yycslab@xxxxxxxxx>]

Thanks, Kefu! I had bumped the memory of my container to 32GB in anticipation of the need for more memory, but I guess it was not enough. My machine has 8 cores, 64GB of physical memory, and had more than 36GB of available memory when I kicked off the build. I reduced the number of parallel jobs to 8 and reran the ninja job. This time the job failed for the errors below. It looks like the header file cephfs/libcephfs.h was missing? The file ./src/include/cephfs/libcephfs.h does exist, though. 

[root@bf1a404c20a3 build]# ninja -v -j 8
[1/567] cd /ceph/src/pybind/cephfs && env CC="/opt/rh/gcc-toolset-11/root/usr/bin/gcc" CFLAGS="" CPPFLAGS="-iquote/ceph/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG##0'" CXX="/opt/rh/gcc-toolset-11/root/usr/bin/g++" LDSHARED="/opt/rh/gcc-toolset-11/root/usr/bin/gcc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/ceph/build/lib CYTHON_BUILD_DIR=/ceph/build/src/pybind/cephfs CEPH_LIBDIR=/ceph/build/lib /usr/bin/python3.6 /ceph/src/pybind/cephfs/setup.py build --build-base /ceph/build/lib/cython_modules --build-platlib /ceph/build/lib/cython_modules/lib.3
FAILED: lib/cython_modules/lib.3/cephfs.cpython-36m-x86_64-linux-gnu.so
cd /ceph/src/pybind/cephfs && env CC="/opt/rh/gcc-toolset-11/root/usr/bin/gcc" CFLAGS="" CPPFLAGS="-iquote/ceph/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG##0'" CXX="/opt/rh/gcc-toolset-11/root/usr/bin/g++" LDSHARED="/opt/rh/gcc-toolset-11/root/usr/bin/gcc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/ceph/build/lib CYTHON_BUILD_DIR=/ceph/build/src/pybind/cephfs CEPH_LIBDIR=/ceph/build/lib /usr/bin/python3.6 /ceph/src/pybind/cephfs/setup.py build --build-base /ceph/build/lib/cython_modules --build-platlib /ceph/build/lib/cython_modules/lib.3
as: unrecognized option '--gdwarf-4'
/ceph/src/pybind/cephfs/tmpog18e3jq/cephfs_dummy.c:9:1: fatal error: error writing to -: Broken pipe
    9 | }
      | ^
compilation terminated.

Compile Error: Ceph FS development headers not found
ninja: build stopped: subcommand failed.

On Sun, Dec 3, 2023 at 8:00 PM kefu chai <tchaikov@xxxxxxxxx> wrote:

On Mon, Dec 4, 2023 at 9:29 AM Yong Yuan <yycslab@xxxxxxxxx> wrote:

My bad. I'm still developing the right mental model for C++'s or Ceph's build toolchain. Please see attached for the entire output of the run with ninja -v. The file dot_ninja.log.gzip is the copy of the file .ninja_log in the same build directory. In case it helps,  I used the container quay.io/centos/centos:stream8, and below is the os configuration of the container instance. Thanks,  

i am quoting the command line which failed:

[755/2207] /usr/bin/ccache /opt/rh/gcc-toolset-11/root/usr/bin/g++ -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_HAS_IO_URING -DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT -DHAVE_CONFIG_H -DTEST_LIBRBD_INTERNALS -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -Isrc/include -I../src -isystem boost/include -isystem include -isystem ../src/xxHash -isystem ../src/fmt/include -isystem ../src/googletest/googlemock/include -isystem ../src/googletest/googlemock -isystem ../src/googletest/googletest/include -isystem ../src/googletest/googletest -isystem ../src/jaegertracing/opentelemetry-cpp/api/include -isystem ../src/jaegertracing/opentelemetry-cpp/exporters/jaeger/include -isystem ../src/jaegertracing/opentelemetry-cpp/ext/include -isystem ../src/jaegertracing/opentelemetry-cpp/sdk/include -Og -g -fPIE -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DBOOST_PHOENIX_STL_TUPLE_H_ -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -DCEPH_DEBUG_MUTEX -fstack-protector-strong -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -std=c++2a -MD -MT src/test/librbd/CMakeFiles/unittest_librbd.dir/migration/test_mock_HttpClient.cc.o -MF src/test/librbd/CMakeFiles/unittest_librbd.dir/migration/test_mock_HttpClient.cc.o.d -o src/test/librbd/CMakeFiles/unittest_librbd.dir/migration/test_mock_HttpClient.cc.o -c ../src/test/librbd/migration/test_mock_HttpClient.cc
FAILED: src/test/librbd/CMakeFiles/unittest_librbd.dir/migration/test_mock_HttpClient.cc.o
/usr/bin/ccache /opt/rh/gcc-toolset-11/root/usr/bin/g++ -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_HAS_IO_URING -DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT -DHAVE_CONFIG_H -DTEST_LIBRBD_INTERNALS -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -Isrc/include -I../src -isystem boost/include -isystem include -isystem ../src/xxHash -isystem ../src/fmt/include -isystem ../src/googletest/googlemock/include -isystem ../src/googletest/googlemock -isystem ../src/googletest/googletest/include -isystem ../src/googletest/googletest -isystem ../src/jaegertracing/opentelemetry-cpp/api/include -isystem ../src/jaegertracing/opentelemetry-cpp/exporters/jaeger/include -isystem ../src/jaegertracing/opentelemetry-cpp/ext/include -isystem ../src/jaegertracing/opentelemetry-cpp/sdk/include -Og -g -fPIE -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DBOOST_PHOENIX_STL_TUPLE_H_ -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -DCEPH_DEBUG_MUTEX -fstack-protector-strong -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -std=c++2a -MD -MT src/test/librbd/CMakeFiles/unittest_librbd.dir/migration/test_mock_HttpClient.cc.o -MF src/test/librbd/CMakeFiles/unittest_librbd.dir/migration/test_mock_HttpClient.cc.o.d -o src/test/librbd/CMakeFiles/unittest_librbd.dir/migration/test_mock_HttpClient.cc.o -c ../src/test/librbd/migration/test_mock_HttpClient.cc
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.

this error is sitting right at the 17th line from the end of the file.

please check your building environment. my hunch is that your OOM killer kicked in and killed the compiler process with SIGTERM. if that's the case, probably you could lower the number of jobs, by passing "-j <num-of-jobs>" to ninja or to cmake. you could also help by providing more details on your environment, like the number of cores, the total physical memory, and the available memory when compiling ceph, etc. we try to figure out a reasonable level of parallelism in cmake/modules/LimitJobs.cmake based on previous experiments, but it might not work well under some settings.

[root@bf1a404c20a3 build]# cat /etc/os-release
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

On Sun, Dec 3, 2023 at 12:50 AM kefu chai <tchaikov@xxxxxxxxx> wrote:

Le dim. 3 déc. 2023 à 16:27, Yong Yuan <yycslab@xxxxxxxxx> a écrit :

Thanks, Kefu! When I ran ninja -v, I didn't see any other error message other than "ninja: build stopped: subcommand failed." The step and the output right before this message is below. The file .ninja_log does not contain any error message, either. I also reran the step below and it simply completed successfully. Any suggestion on what I can do to gather more detailed error messages? 

The snippet does not help. Again, please collect the *full* output of the run which failed, if the size of output is too large, you could use some pastebin service, or upload it somewhere so one can download.

[767/2207] /usr/bin/ccache /opt/rh/gcc-toolset-11/root/usr/bin/g++ -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_HAS_IO_URING -DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -Isrc/include -I../src -isystem boost/include -isystem include -isystem ../src/xxHash -isystem ../src/fmt/include -isystem ../src/jaegertracing/opentelemetry-cpp/api/include -isystem ../src/jaegertracing/opentelemetry-cpp/exporters/jaeger/include -isystem ../src/jaegertracing/opentelemetry-cpp/ext/include -isystem ../src/jaegertracing/opentelemetry-cpp/sdk/include -Og -g -fPIC -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DBOOST_PHOENIX_STL_TUPLE_H_ -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -DCEPH_DEBUG_MUTEX -fstack-protector-strong -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -std=c++2a -MD -MT src/mds/CMakeFiles/mds.dir/CInode.cc.o -MF src/mds/CMakeFiles/mds.dir/CInode.cc.o.d -o src/mds/CMakeFiles/mds.dir/CInode.cc.o -c ../src/mds/CInode.cc

On Sat, Dec 2, 2023 at 5:37 PM kefu chai <tchaikov@xxxxxxxxx> wrote:

On Sun, Dec 3, 2023 at 5:00 AM Yong Yuan <yycslab@xxxxxxxxx> wrote:

Hi, 

I'm trying to build a DEBUG version of Ceph's reef-release branch on a virtual Ubuntu-LTS 22.04 running on Lima by following the README on Ceph's github repo. The build failed and the last CMake error was ""g++-11: error: unrecognized command-line option '-Wimplicit-const-int-float-conversion'". Could anyone please help me figure out I can do to track down and fix the compilation error? Below are more details. 

The system configuration is as follows: 

> uname -a
Linux lima-ceph-dev 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

I followed the instructions on the README in Ceph's github repo: https://github.com/ceph/ceph, and the command ./do_cmake.sh failed at step [137/2150] that builds frontend dashboard with the error message "ninja: build stopped: subcommand failed." The last error logged in the file CMakeError.log has to do with "g++-11: error: unrecognized command-line option '-Wimplicit-const-int-float-conversion'". 

Below the last error message on the CMakeError.log: 

Performing C++ SOURCE FILE Test COMPILER_SUPPORTS_WARN_IMPLICIT_CONST_INT_FLOAT_CONVERSION failed with the following output:
Change Dir: /home/dyuan.linux/ceph/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/ninja cmTC_bab6d && [1/2] Building CXX object CMakeFiles/cmTC_bab6d.dir/src.cxx.o
FAILED: CMakeFiles/cmTC_bab6d.dir/src.cxx.o
/usr/bin/g++-11 -DCOMPILER_SUPPORTS_WARN_IMPLICIT_CONST_INT_FLOAT_CONVERSION  -fPIE   -Wimplicit-const-int-float-conversion -std=c++20 -o CMakeFiles/cmTC_bab6d.dir/src.cxx.o -c /home/dyuan.linux/ceph/build/CMakeFiles/CMakeTmp/src.cxx
g++-11: error: unrecognized command-line option '-Wimplicit-const-int-float-conversion'
ninja: build stopped: subcommand failed.

I think the error from CMake is not fatal. What it implies is that the compiler does not support this warning option. see the change which introduced this check: https://github.com/ceph/ceph/commit/658ecaec8ebade868799f2535bd195853f07f7f9 .  

if the build fails, the root cause could be in the cmake script which generates the building system. but the error message does not prove this hypothesis. please paste the full error from ninja, not the one in the CMakeError.log. 

I also tried to build ceph on the docker quay.io/centos/centos and got the same g++ failure. 

_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx

--

Regards
Kefu Chai

--

Regards
Kefu Chai

_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx