What I was getting at was to see if this was somehow related to the bug described here https://www.youtube.com/watch?v=_4HUR00oCGo It should not be given the version of ceph you are using but the CRC error you are seeing is similar. Respectfully, *Wes Dillingham* wes@xxxxxxxxxxxxxxxxx LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Mon, Nov 27, 2023 at 2:19 PM Anthony D'Atri <aad@xxxxxxxxxxxxxx> wrote: > The options Wes listed are for data, not RocksDB. > > > On Nov 27, 2023, at 1:59 PM, Denis Polom <denispolom@xxxxxxxxx> wrote: > > > > Hi, > > > > no we don't: > > > > "bluestore_rocksdb_options": > "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", > > > > thx > > > > On 11/27/23 19:17, Wesley Dillingham wrote: > >> Curious if you are using bluestore compression? > >> > >> Respectfully, > >> > >> *Wes Dillingham* > >> wes@xxxxxxxxxxxxxxxxx > >> LinkedIn <http://www.linkedin.com/in/wesleydillingham> > >> > >> > >> On Mon, Nov 27, 2023 at 10:09 AM Denis Polom <denispolom@xxxxxxxxx> > wrote: > >> > >> Hi > >> > >> we have issue to start some OSDs on one node on our Ceph Quincy > >> 17.2.7 > >> cluster. Some OSDs on that node are running fine, but some failing > >> to start. > >> > >> Looks like crc32 checksum error, and failing to get OSD map. I > >> found a > >> some discussions on that but nothing helped. > >> > >> I've also tried to insert current OSD map but that ends with error: > >> > >> # CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool > >> --data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap > >> osdmap (#-1:20684533:::osdmap.931991:0#) does not exist. > >> > >> Log is bellow > >> > >> Any ideas please? > >> > >> Thank you > >> > >> > >> From log file: > >> > >> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public > >> interface > >> > >> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 > >> bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 > >> checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, > >> device > >> location [0x10000~1000], logical extent 0x0~1000, object > >> #-1:7b3f43c4:::osd_superblock:0# > >> > >> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load > >> OSD map for epoch 927580, got 0 bytes > >> > >> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef > >> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time > >> 2023-11-27T16:01:51.443522+0100 > >> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) > >> ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > >> quincy > >> (stable) > >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > >> const*)+0x14f) [0x561ad07d2624] > >> 2: ceph-osd(+0xc2e836) [0x561ad07d2836] > >> 3: (OSD::init()+0x4026) [0x561ad08e5a86] > >> 4: main() > >> 5: __libc_start_main() > >> 6: _start() > >> *** Caught signal (Aborted) ** > >> in thread 7f3f17aa13c0 thread_name:ceph-osd > >> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 > >> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef > >> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time > >> 2023-11-27T16:01:51.443522+0100 > >> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) > >> > >> ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > >> quincy > >> (stable) > >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > >> const*)+0x14f) [0x561ad07d2624] > >> 2: ceph-osd(+0xc2e836) [0x561ad07d2836] > >> 3: (OSD::init()+0x4026) [0x561ad08e5a86] > >> 4: main() > >> 5: __libc_start_main() > >> 6: _start() > >> > >> > >> ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > >> quincy > >> (stable) > >> 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] > >> 2: gsignal() > >> 3: abort() > >> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char > >> const*)+0x1b7) [0x561ad07d268c] > >> 5: ceph-osd(+0xc2e836) [0x561ad07d2836] > >> 6: (OSD::init()+0x4026) [0x561ad08e5a86] > >> 7: main() > >> 8: __libc_start_main() > >> 9: _start() > >> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal > >> (Aborted) ** > >> in thread 7f3f17aa13c0 thread_name:ceph-osd > >> > >> ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > >> quincy > >> (stable) > >> 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] > >> 2: gsignal() > >> 3: abort() > >> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char > >> const*)+0x1b7) [0x561ad07d268c] > >> 5: ceph-osd(+0xc2e836) [0x561ad07d2836] > >> 6: (OSD::init()+0x4026) [0x561ad08e5a86] > >> 7: main() > >> 8: __libc_start_main() > >> 9: _start() > >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is > >> needed to interpret this. > >> > >> > >> -558> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to > >> public interface > >> > >> -5> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 > >> bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 > >> checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, > >> device > >> location [0x10000~1000], logical extent 0x0~1000, object > >> #-1:7b3f43c4:::osd_superblock:0# > >> > >> -2> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 > >> failed > >> to load OSD map for epoch 927580, got 0 bytes > >> > >> -1> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 > >> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef > >> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time > >> 2023-11-27T16:01:51.443522+0100 > >> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) > >> > >> ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > >> quincy > >> (stable) > >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > >> const*)+0x14f) [0x561ad07d2624] > >> 2: ceph-osd(+0xc2e836) [0x561ad07d2836] > >> 3: (OSD::init()+0x4026) [0x561ad08e5a86] > >> 4: main() > >> 5: __libc_start_main() > >> 6: _start() > >> > >> > >> 0> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught > >> signal > >> (Aborted) ** > >> in thread 7f3f17aa13c0 thread_name:ceph-osd > >> > >> ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > >> quincy > >> (stable) > >> 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] > >> 2: gsignal() > >> 3: abort() > >> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char > >> const*)+0x1b7) [0x561ad07d268c] > >> 5: ceph-osd(+0xc2e836) [0x561ad07d2836] > >> 6: (OSD::init()+0x4026) [0x561ad08e5a86] > >> 7: main() > >> 8: __libc_start_main() > >> 9: _start() > >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is > >> needed to interpret this. > >> > >> > >> -562> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to > >> public interface > >> > >> -9> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 > >> bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 > >> checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, > >> device > >> location [0x10000~1000], logical extent 0x0~1000, object > >> #-1:7b3f43c4:::osd_superblock:0# > >> > >> -6> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 > >> failed > >> to load OSD map for epoch 927580, got 0 bytes > >> > >> -5> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 > >> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef > >> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time > >> 2023-11-27T16:01:51.443522+0100 > >> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) > >> > >> ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > >> quincy > >> (stable) > >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > >> const*)+0x14f) [0x561ad07d2624] > >> 2: ceph-osd(+0xc2e836) [0x561ad07d2836] > >> 3: (OSD::init()+0x4026) [0x561ad08e5a86] > >> 4: main() > >> 5: __libc_start_main() > >> 6: _start() > >> > >> > >> -4> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught > >> signal > >> (Aborted) ** > >> in thread 7f3f17aa13c0 thread_name:ceph-osd > >> > >> ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > >> quincy > >> (stable) > >> 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] > >> 2: gsignal() > >> 3: abort() > >> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char > >> const*)+0x1b7) [0x561ad07d268c] > >> 5: ceph-osd(+0xc2e836) [0x561ad07d2836] > >> 6: (OSD::init()+0x4026) [0x561ad08e5a86] > >> 7: main() > >> 8: __libc_start_main() > >> 9: _start() > >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is > >> needed to interpret this. > >> > >> > >> Aborted > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |