Hi Stefan, Thanks for the report. 9 hours fsck is the longest I've heard about yet -- and on NVMe, that's quite surprising! Which firmware are you running on those Samsung's? For a different reason Mark and we have been comparing performance of that drive between what's in his lab vs what we have in our data centre. We have no obvious perf issues running EDA5702Q; Mark has some issue with the Quincy RC running FW EDA53W0Q. I'm not sure if it's related, but worth checking... In any case, I'm also surprised you decided to drain the boxes before fsck. Wouldn't 9 hours of down osds, with noout set, going to be less invasive? Cheers, Dan On Mon, Apr 11, 2022 at 10:56 AM Stefan Kooman <stefan@xxxxxx> wrote: > > Hi All, > > Last week we succesfully upgraded our 14.2.18 cluster to 15.2.16. > According to "ceph crash ls" we did not have a single crash while > running Nautilus \o/. Unlike releases before Nautilus we occasionally > had issues with MDS (hitting bugs) but since Nautilus this is no longer > the case. And hopefully it stays like this. So kudos to all Ceph devs > and contributors! > > One thing that took *way* longer than expected was the bluestore fsck. > We did a "offline" and a "online" approach on one host. Both took the > same amount of time (unlike previous releases, where online fsck would > take way longer) ... about *9 hours* on NVMe disks (Samsung PM-983, > SAMSUNG MZQLB3T8HALS-00007). > Note that we have a relatively big CephFS workload (with lots of small > files and deep directory hierarchies), so your mileage may vary. Also > note that "online" does not mean that our OSDs are UP ... they are not. > They only "boot" when this process has finished. So the > "bluestore_fsck_quick_fix_on_mount" parameter is misleading here. > We decided to not proceed with bluestore fsck and first upgrade all > storage nodes. We are now planning on redeploying the remaining OSDs and > use "pgremapper" to drain hosts to new storage servers one by one: less > risk (no degraded data for a prolonged period of time) and potentially > even quicker. > > FYI, > > Gr. Stefan > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |