I'm suspicious of cross contamination of devices here. I was on CentOS for
eons until Red Hat shenanigans pinned me to CentOS 7 and nautilus. I had
very well defined udev rules that ensured dm devices were statically set
and owned correctly and survived reboots.
I seem to be struggling with this in the openSuSe world. Ownership on my
devices flip back to root despite my long standing udev rules bing migrated
over.
I know my paths are correct though. The osd root dirs are also lv's with
filesystem labels. the block, db, wal links are correct. db and wal are
lv's named appropriately (yea yea, per Sage that ship has sailed. I LV'd
too). The osd drives get partitions with osd number labels. bluestore tool
also confirms.
On Fri, Feb 11, 2022 at 9:14 AM Mazzystr <mazzystr@xxxxxxxxx> wrote:
> I forgot to mention I freeze the cluster with 'ceph osd set
> no{down,out,backfill}'. Then I zyp up all hosts and reboot them. Only
> when everything is backup do I unset.
>
> My client IO patterns allow me to do this since it's a worm data store
> with long spans of time between writes and reads. I have plenty of time to
> work with the community and get my store back online.
>
> This thread is really for documentation for the next person that comes
> along with the same problem
>
> On Fri, Feb 11, 2022 at 9:08 AM Mazzystr <mazzystr@xxxxxxxxx> wrote:
>
>> My clusters are self rolled. My start command is as follows
>>
>> podman run -it --privileged --pid=host --cpuset-cpus 0,1 --memory 2g
>> --name ceph_osd0 --hostname ceph_osd0 -v /dev:/dev -v
>> /etc/localtime:/etc/localtime:ro -v /etc/ceph:/etc/ceph/ -v
>> /var/lib/ceph/osd/ceph-0:/var/lib/ceph/osd/ceph-0 -v
>> /var/log/ceph:/var/log/ceph -v /run/udev/:/run/udev/
>> ceph/ceph:v16.2.7-20220201 ceph-osd --id 0 -c /etc/ceph/ceph.conf --cluster
>> ceph -f
>>
>>
>> I jumped from the octopus img to the 16.2.7 img. I've been running well
>> for awhile with no issues. The cluster was clean, no backfills in
>> progressor etc This latest zyp up and reboot and now I have osds that
>> don't start.
>>
>> podman image ls
>> quay.io/ceph/ceph v16.2.7
>> 231fd40524c4 9 days ago 1.39 GB
>> quay.io/ceph/ceph
>> v16.2.7-20220201 231fd40524c4 9 days ago 1.39 GB
>>
>>
>> bluefs fails to mount up, I guess? The headers are still readable via
>> bluestore tool
>>
>> ceph-bluestore-tool show-label --dev /dev/mapper/ceph-0block
>> {
>> "/dev/mapper/ceph-0block": {
>> "osd_uuid": "1234abcd-1234-abcd-1234-1234 abcd1234",
>> "size": 6001171365888,
>> "btime": "2019-04-11T08:46:36.013428-0700",
>> "description": "main",
>> "bfm_blocks": "1465129728",
>> "bfm_blocks_per_key": "128",
>> "bfm_bytes_per_block": "4096",
>> "bfm_size": "6001171365888",
>> "bluefs": "1",
>> "ceph_fsid": "1234abcd-1234-abcd-1234-1234 abcd1234",
>> "kv_backend": "rocksdb",
>> "magic": "ceph osd volume v026",
>> "mkfs_done": "yes",
>> "ready": "ready",
>> "require_osd_release": "16",
>> "whoami": "0"
>> }
>> }
>>
>>
>> On Fri, Feb 11, 2022 at 1:06 AM Eugen Block <eblock@xxxxxx> wrote:
>>
>>> Can you share some more information how exactly you upgraded? It looks
>>> like a cephadm managed cluster. Did you intall OS updates on all nodes
>>> without waiting for the first one to recover? Maybe I'm misreading so
>>> please clarify what your update process looked like.
>>>
>>>
>>> Zitat von Mazzystr <mazzystr@xxxxxxxxx>:
>>>
>>> > I applied latest os updates and rebooted my hosts. Now all my osds
>>> fail to
>>> > start.
>>> >
>>> > # cat /etc/os-release
>>> > NAME="openSUSE Tumbleweed"
>>> > # VERSION="20220207"
>>> > ID="opensuse-tumbleweed"
>>> > ID_LIKE="opensuse suse"
>>> > VERSION_ID="20220207"
>>> >
>>> > # uname -a
>>> > Linux cube 5.16.5-1-default #1 SMP PREEMPT Thu Feb 3 05:26:48 UTC 2022
>>> > (1af4009) x86_64 x86_64 x86_64 GNU/Linux
>>> >
>>> > container image: v16.2.7 / v16.2.7-20220201
>>> >
>>> > osd debug log shows the following
>>> > -11> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 bluefs
>>> add_block_device
>>> > bdev 0 path /var/lib/ceph/osd/ceph-0/block.wal size 50 GiB
>>> > -10> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb
>>> option
>>> > max_total_wal_size = 1073741824
>>> > -9> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb
>>> option
>>> > compaction_readahead_size = 2097152
>>> > -8> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb
>>> option
>>> > max_write_buffer_number = 4
>>> > -7> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb
>>> option
>>> > max_background_compactions = 2
>>> > -6> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb
>>> option
>>> > compression = kNoCompression
>>> > -5> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb
>>> option
>>> > writable_file_max_buffer_size = 0
>>> > -4> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb
>>> option
>>> > min_write_buffer_number_to_merge = 1
>>> > -3> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb
>>> option
>>> > recycle_log_file_num = 4
>>> > -2> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 set rocksdb
>>> option
>>> > write_buffer_size = 268435456
>>> > -1> 2022-02-10T19:14:48.383-0800 7ff1be4c3080 1 bluefs mount
>>> > 0> 2022-02-10T19:14:48.387-0800 7ff1be4c3080 -1 *** Caught signal
>>> > (Aborted) **
>>> > in thread 7ff1be4c3080 thread_name:ceph-osd
>>> >
>>> > ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific
>>> > (stable)
>>> > 1: /lib64/libpthread.so.0(+0x12c20) [0x7ff1bc465c20]
>>> > 2: gsignal()
>>> > 3: abort()
>>> > 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff1bba7c09b]
>>> > 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff1bba8253c]
>>> > 6: /lib64/libstdc++.so.6(+0x96597) [0x7ff1bba82597]
>>> > 7: /lib64/libstdc++.so.6(+0x967f8) [0x7ff1bba827f8]
>>> > 8: ceph-osd(+0x56301f) [0x559ff6d6301f]
>>> > 9: (BlueFS::_open_super()+0x18c) [0x559ff745f08c]
>>> > 10: (BlueFS::mount()+0xeb) [0x559ff748085b]
>>> > 11: (BlueStore::_open_bluefs(bool, bool)+0x94) [0x559ff735e464]
>>> > 12: (BlueStore::_prepare_db_environment(bool, bool,
>>> > std::__cxx11::basic_string<char, std::char_traits<char>,
>>> > std::allocator<char> >*, std::__cxx11::basic_string<char,
>>> > std::char_traits<char>, std::allocator<char> >*)+0x6d9)
>>> [0x559ff735f5b9]
>>> > 13: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x559ff73608b5]
>>> > 14: (BlueStore::_open_db_and_around(bool, bool)+0x273)
>>> [0x559ff73cba33]
>>> > 15: (BlueStore::_mount()+0x204) [0x559ff73ce974]
>>> > 16: (OSD::init()+0x380) [0x559ff6ea2400]
>>> > 17: main()
>>> > 18: __libc_start_main()
>>> > 19: _start()
>>> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>>> needed
>>> > to interpret this.
>>> >
>>> >
>>> > The process log shows the following
>>> > 2022-02-10T19:33:31.852-0800 7f22869e8080 -1 WARNING: the following
>>> > dangerous and experimental features are enabled: bluestore,rocksdb
>>> > 2022-02-10T19:33:31.852-0800 7f22869e8080 -1 WARNING: the following
>>> > dangerous and experimental features are enabled: bluestore,rocksdb
>>> > 2022-02-10T19:33:31.852-0800 7f22869e8080 -1 WARNING: the following
>>> > dangerous and experimental features are enabled: bluestore,rocksdb
>>> > terminate called after throwing an instance of
>>> > 'ceph::buffer::v15_2_0::malformed_input'
>>> > what(): void
>>> > bluefs_super_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no
>>> > longer understand old encoding version 2 < 143: Malformed input
>>> > *** Caught signal (Aborted) **
>>> > in thread 7f22869e8080 thread_name:ceph-osd
>>> > ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific
>>> > (stable)
>>> > 1: /lib64/libpthread.so.0(+0x12c20) [0x7f228498ac20]
>>> > 2: gsignal()
>>> > 3: abort()
>>> > 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f2283fa109b]
>>> > 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f2283fa753c]
>>> > 6: /lib64/libstdc++.so.6(+0x96597) [0x7f2283fa7597]
>>> > 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f2283fa77f8]
>>> > 8: ceph-osd(+0x56301f) [0x55e6faf6301f]
>>> > 9: (BlueFS::_open_super()+0x18c) [0x55e6fb65f08c]
>>> > 10: (BlueFS::mount()+0xeb) [0x55e6fb68085b]
>>> > 11: (BlueStore::_open_bluefs(bool, bool)+0x94) [0x55e6fb55e464]
>>> > 12: (BlueStore::_prepare_db_environment(bool, bool,
>>> > std::__cxx11::basic_string<char, std::char_traits<char>,
>>> > std::allocator<char> >*, std::__cxx11::basic_string<char,
>>> > std::char_traits<char>, std::allocator<char> >*)+0x6d9)
>>> [0x55e6fb55f5b9]
>>> > 13: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x55e6fb5608b5]
>>> > 14: (BlueStore::_open_db_and_around(bool, bool)+0x273)
>>> [0x55e6fb5cba33]
>>> > 15: (BlueStore::_mount()+0x204) [0x55e6fb5ce974]
>>> > 16: (OSD::init()+0x380) [0x55e6fb0a2400]
>>> > 17: main()
>>> > 18: __libc_start_main()
>>> > 19: _start()
>>> > 2022-02-10T19:33:34.620-0800 7f22869e8080 -1 *** Caught signal
>>> (Aborted) **
>>> > in thread 7f22869e8080 thread_name:ceph-osd
>>> >
>>> >
>>> > Doesn't anyone have any ideas what could be going on here?
>>> >
>>> > Thanks,
>>> > /Chris
>>> > _______________________________________________
>>> > ceph-users mailing list -- ceph-users@xxxxxxx
>>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>
>>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
