On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus <andrius.jurkus@xxxxxxxxxx> wrote: > > Hello, I will try to keep it sad and short :) :( PS sorry if this > dublicate I tried post it from web also. > > Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds. > After data migration for few hours, 1 SSD failed, then another and > another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same > host has SSD and HDD, but only SSD's are failing so I think this has to > be balancing refiling or something bug and probably not upgrade bug. > > Cluster has been in pause for 4 hours and no more OSD's are failing. > > full trace > https://pastebin.com/UxbfFYpb This looks very similar to https://tracker.ceph.com/issues/50656. Adding Igor for more ideas. Neha > > Now I m googling and learning but, Is there a way how to easily test > lets say 15.2.XX version on osd without losing anything? > > Any help would be appreciated. > > Error start like this > > May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for > 4e01640b-951b-4f75-8dca-0bad4faf1b11... > May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > 16:58:53.057836433 +0000 UTC m=+0.454352919 container create > 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > GIT_BRANCH=HEAD, maintainer=D > May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun container. > May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > 16:58:53.3394116 +0000 UTC m=+0.735928098 container init > 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > maintainer=Dimitri Savineau <dsav > May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > 16:58:53.446921192 +0000 UTC m=+0.843437626 container start > 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > GIT_BRANCH=HEAD, org.label-sch > May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > 16:58:53.447050119 +0000 UTC m=+0.843566553 container attach > 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > org.label-schema.name=CentOS > May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 > May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev > /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 > --path /var/lib/ceph/osd/ceph-2 --no-mon-config > May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > /usr/bin/ln -snf > /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 > /var/lib/ceph/osd/ceph-2/block > May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block > May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > /usr/bin/chown -R ceph:ceph /dev/dm-1 > May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 > May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm > activate successful for osd ID: 2 > May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > 16:58:53.8147653 +0000 UTC m=+1.211281741 container died > 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate) > May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14 > 16:58:55.044964534 +0000 UTC m=+2.441480996 container remove > 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > CEPH_POINT_RELEASE=-16.2.4, R > May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 > 16:58:55.594265612 +0000 UTC m=+0.369978347 container create > 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da > (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD, > org.label-schema.build-d > May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 > 16:58:55.864589286 +0000 UTC m=+0.640302021 container init > 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da > (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, > org.label-schema.schema-version=1.0, GIT > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167 > (ceph:ceph) > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4 > (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process > ceph-osd, pid 2 > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore empty > --pid-file > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 > /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 > /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, > 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 > bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size > 3221225472 meta 0.45 kv 0.45 data 0.06 > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 > /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 > /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, > 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device bdev > 1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB > May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 > /var/lib/ceph/osd/ceph-2/block) close > May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 > 16:58:55.972267166 +0000 UTC m=+0.747979911 container start > 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da > (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True, > GIT_REPO=https://github.com/ > May 14 16:58:55 dragon-ball-radar bash[113558]: > 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da > May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for > 4e01640b-951b-4f75-8dca-0bad4faf1b11. > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 > /var/lib/ceph/osd/ceph-2/block) close > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0 > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data > /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 > address in networks '10.0.199.0/24' interfaces '' > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 > address in networks '172.16.199.0/24' interfaces '' > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc > load: isa > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 > /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 > /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, > 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 > bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size > 3221225472 meta 0.45 kv 0.45 data 0.06 > May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 > /var/lib/ceph/osd/ceph-2/block) close > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |