On Fri, 12 Jul 2019, Nathan Fish wrote: > Thanks. Speaking of 14.2.2, is there a timeline for it? We really want > some of the fixes in it as soon as possible. I think it's basically ready now... probably Monday? sage > > On Fri, Jul 12, 2019 at 11:22 AM Sage Weil <sweil@xxxxxxxxxx> wrote: > > > > Hi everyone, > > > > All current Nautilus releases have an issue where deploying a single new > > (Nautilus) BlueStore OSD on an upgraded cluster (i.e. one that was > > originally deployed pre-Nautilus) breaks the pool utilization stats > > reported by ``ceph df``. Until all OSDs have been reprovisioned or > > updated (via ``ceph-bluestore-tool repair``), the pool stats will show > > values that are lower than the true value. A fix is in the works but will > > not appear until 14.2.3. Users who have upgraded to Nautilus (or are > > considering upgrading) may want to delay provisioning new OSDs until the > > fix is available in the next release. > > > > This issue will only affect you if: > > > > - You started with a pre-nautilus cluster and upgraded > > - You then provision one or more new BlueStore OSDs, or run > > 'ceph-bluestore-tool repair' on an upgraded OSD. > > > > The symptom is that the pool stats from 'ceph df' are too small. For > > example, the pre-upgrade stats on our test cluster were > > > > ... > > POOLS: > > POOL ID STORED OBJECTS USED %USED MAX AVAIL > > data 0 63 TiB 44.59M 63 TiB 30.21 48 TiB > > ... > > > > but when one OSD was updated it changed to > > > > POOLS: > > POOL ID STORED OBJECTS USED %USED MAX AVAIL > > data 0 558 GiB 43.50M 1.7 TiB 1.22 45 TiB > > > > The root cause is that, starting with Nautilus, BlueStore maintains > > per-pool usage stats, but it requires a slight on-disk format change; > > upgraded OSDs won't have the new stats until you run a ceph-bluestore-tool > > repair. The problem is that the mon starts using the new stats as soon as > > *any* OSDs are reporting per-pool stats (instead of waiting until *all* > > OSDs are doing so). > > > > To avoid the issue, either > > > > - do not provision new BlueStore OSDs after the upgrade, or > > - update all OSDs to keep new per-pool stats. An existing BlueStore > > OSD can be converted with > > > > systemctl stop ceph-osd@$N > > ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-$N > > systemctl start ceph-osd@$N > > > > Note that FileStore does not support the new per-pool stats at all, so > > if there are filestore OSDs in your cluster there is no workaround > > that doesn't involve replacing the filestore OSDs with bluestore. > > > > A fix[1] is working it's way through QA and will appear in 14.2.3; it > > won't quite make the 14.2.2 release. > > > > sage > > > > > > [1] https://github.com/ceph/ceph/pull/28978 > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |