Re: OSDs failing to start due to crc32 and osdmap error

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



What I was getting at was to see if this was somehow related to the bug
described here https://www.youtube.com/watch?v=_4HUR00oCGo

It should not be given the version of ceph you are using but the CRC error
you are seeing is similar.

Respectfully,

*Wes Dillingham*
wes@xxxxxxxxxxxxxxxxx
LinkedIn <http://www.linkedin.com/in/wesleydillingham>


On Mon, Nov 27, 2023 at 2:19 PM Anthony D'Atri <aad@xxxxxxxxxxxxxx> wrote:

> The options Wes listed are for data, not RocksDB.
>
> > On Nov 27, 2023, at 1:59 PM, Denis Polom <denispolom@xxxxxxxxx> wrote:
> >
> > Hi,
> >
> > no we don't:
> >
> > "bluestore_rocksdb_options":
> "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824",
> >
> > thx
> >
> > On 11/27/23 19:17, Wesley Dillingham wrote:
> >> Curious if you are using bluestore compression?
> >>
> >> Respectfully,
> >>
> >> *Wes Dillingham*
> >> wes@xxxxxxxxxxxxxxxxx
> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> >>
> >>
> >> On Mon, Nov 27, 2023 at 10:09 AM Denis Polom <denispolom@xxxxxxxxx>
> wrote:
> >>
> >>    Hi
> >>
> >>    we have issue to start some OSDs on one node on our Ceph Quincy
> >>    17.2.7
> >>    cluster. Some OSDs on that node are running fine, but some failing
> >>    to start.
> >>
> >>    Looks like crc32 checksum error, and failing to get OSD map. I
> >>    found a
> >>    some discussions on that but nothing helped.
> >>
> >>    I've also tried to insert current OSD map but that ends with error:
> >>
> >>    # CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool
> >>    --data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap
> >>    osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.
> >>
> >>    Log is bellow
> >>
> >>    Any ideas please?
> >>
> >>    Thank you
> >>
> >>
> >>     From log file:
> >>
> >>    2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public
> >>    interface
> >>
> >>    2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
> >>    bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
> >>    checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2,
> >>    device
> >>    location [0x10000~1000], logical extent 0x0~1000, object
> >>    #-1:7b3f43c4:::osd_superblock:0#
> >>
> >>    2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load
> >>    OSD map for epoch 927580, got 0 bytes
> >>
> >>    /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
> >>    OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
> >>    2023-11-27T16:01:51.443522+0100
> >>    /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
> >>      ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>    quincy
> >>    (stable)
> >>      1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>    const*)+0x14f) [0x561ad07d2624]
> >>      2: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>      3: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>      4: main()
> >>      5: __libc_start_main()
> >>      6: _start()
> >>    *** Caught signal (Aborted) **
> >>      in thread 7f3f17aa13c0 thread_name:ceph-osd
> >>    2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
> >>    /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
> >>    OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
> >>    2023-11-27T16:01:51.443522+0100
> >>    /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
> >>
> >>      ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>    quincy
> >>    (stable)
> >>      1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>    const*)+0x14f) [0x561ad07d2624]
> >>      2: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>      3: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>      4: main()
> >>      5: __libc_start_main()
> >>      6: _start()
> >>
> >>
> >>      ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>    quincy
> >>    (stable)
> >>      1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
> >>      2: gsignal()
> >>      3: abort()
> >>      4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>    const*)+0x1b7) [0x561ad07d268c]
> >>      5: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>      6: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>      7: main()
> >>      8: __libc_start_main()
> >>      9: _start()
> >>    2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal
> >>    (Aborted) **
> >>      in thread 7f3f17aa13c0 thread_name:ceph-osd
> >>
> >>      ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>    quincy
> >>    (stable)
> >>      1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
> >>      2: gsignal()
> >>      3: abort()
> >>      4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>    const*)+0x1b7) [0x561ad07d268c]
> >>      5: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>      6: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>      7: main()
> >>      8: __libc_start_main()
> >>      9: _start()
> >>      NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> >>    needed to interpret this.
> >>
> >>
> >>       -558> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to
> >>    public interface
> >>
> >>         -5> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
> >>    bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
> >>    checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2,
> >>    device
> >>    location [0x10000~1000], logical extent 0x0~1000, object
> >>    #-1:7b3f43c4:::osd_superblock:0#
> >>
> >>         -2> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0
> >>    failed
> >>    to load OSD map for epoch 927580, got 0 bytes
> >>
> >>         -1> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
> >>    /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
> >>    OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
> >>    2023-11-27T16:01:51.443522+0100
> >>    /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
> >>
> >>      ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>    quincy
> >>    (stable)
> >>      1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>    const*)+0x14f) [0x561ad07d2624]
> >>      2: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>      3: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>      4: main()
> >>      5: __libc_start_main()
> >>      6: _start()
> >>
> >>
> >>          0> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught
> >>    signal
> >>    (Aborted) **
> >>      in thread 7f3f17aa13c0 thread_name:ceph-osd
> >>
> >>      ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>    quincy
> >>    (stable)
> >>      1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
> >>      2: gsignal()
> >>      3: abort()
> >>      4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>    const*)+0x1b7) [0x561ad07d268c]
> >>      5: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>      6: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>      7: main()
> >>      8: __libc_start_main()
> >>      9: _start()
> >>      NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> >>    needed to interpret this.
> >>
> >>
> >>       -562> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to
> >>    public interface
> >>
> >>         -9> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
> >>    bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
> >>    checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2,
> >>    device
> >>    location [0x10000~1000], logical extent 0x0~1000, object
> >>    #-1:7b3f43c4:::osd_superblock:0#
> >>
> >>         -6> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0
> >>    failed
> >>    to load OSD map for epoch 927580, got 0 bytes
> >>
> >>         -5> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
> >>    /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
> >>    OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
> >>    2023-11-27T16:01:51.443522+0100
> >>    /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
> >>
> >>      ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>    quincy
> >>    (stable)
> >>      1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>    const*)+0x14f) [0x561ad07d2624]
> >>      2: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>      3: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>      4: main()
> >>      5: __libc_start_main()
> >>      6: _start()
> >>
> >>
> >>         -4> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught
> >>    signal
> >>    (Aborted) **
> >>      in thread 7f3f17aa13c0 thread_name:ceph-osd
> >>
> >>      ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>    quincy
> >>    (stable)
> >>      1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
> >>      2: gsignal()
> >>      3: abort()
> >>      4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>    const*)+0x1b7) [0x561ad07d268c]
> >>      5: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>      6: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>      7: main()
> >>      8: __libc_start_main()
> >>      9: _start()
> >>      NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> >>    needed to interpret this.
> >>
> >>
> >>    Aborted
> >>    _______________________________________________
> >>    ceph-users mailing list -- ceph-users@xxxxxxx
> >>    To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux