Hi Igor,
Thanks for trouble shooting with us. It is acceptable that the
'ceph-bluestore-tool repair' is nonreversible as once "ceph osd
require-osd-release nautilus" was set after upgrade, downgrade to
pre-naulitus is impossible.
But seems we can either make the "ceph-bluestore-tool repair" run
automatically after require_naulitus flag was set, and/or ,clearly
state in the release note that make "ceph-bluestore-tool repair" a
step of upgrade, just like ceph-volume import. Everyone upgrade to
Naulitus will finally have a day to hit into this issue, either when
cap add or break fix.
Xiaoxi
Igor Fedotov <ifedotov@xxxxxxx> 于2019年4月10日周三 下午11:06写道：
>
> Hi Xiaoxi,
>
> as we learned offline currently you have a mixture of new OSDs created
> by Nautilus and old ones created by earlier releases.
>
> New OSDs provide per-pool statistics in a different manner than old
> ones. Merging both together is hardly doable so once your cluster
> contains any OSD with new format 'df' report starts to show pool
> statistics using new OSDs only.
>
> To fix the issue one has to perform 'ceph-bluestore-tool repair' command
> for any old OSDs.
>
> Please note that repair is nonreversible OSD upgrade, one wouldn't be
> able to downgrade to prior to Nautilus releases after that.
>
>
> Thanks,
>
> Igor
>
> On 4/4/2019 11:48 AM, Xiaoxi Chen wrote:
> > Hi list,
> >
> >
> > The fs_data pool was under backfilling , 1 out of 16 hosts are
> > rebuild with same OSD id. After doing that the fs_data stored size is
> > not correct though it is increasing.
> >
> > RAW STORAGE:
> > CLASS SIZE AVAIL USED RAW USED %RAW USED
> > hdd 1.2 PiB 653 TiB 547 TiB 547 TiB 45.60
> > meta 25 TiB 25 TiB 40 GiB 107 GiB 0.42
> > ssd 219 TiB 147 TiB 72 TiB 73 TiB 33.11
> > TOTAL 1.4 PiB 824 TiB 619 TiB 620 TiB 42.92
> >
> > POOLS:
> > POOL ID STORED OBJECTS USED %USED
> > MAX AVAIL
> > cache_tier 3 8.0 TiB 9.94M 24 TiB 16.80
> > 40 TiB
> > fs_data 4 1.6 TiB 63.63M 4.8 TiB 1.12
> > 143 TiB
> > fs_meta 5 35 GiB 343.66k 40 GiB 0.18
> > 7.0 TiB
> >
> >
> > The RAW STORAGE by class is correct.
> > Any insight?
> >
> > Xiaoxi
