Re: ceph df: pool stored vs bytes_used -- raw or not?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Here's a different cluster we upgraded luminous -> nautilus in October:

2020-10-14 13:22:51.860 7f78e3d20a80  0 ceph version 14.2.11
(f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable), process
ceph-osd, pid 119714
...
2020-10-14 13:27:50.368 7f78e3d20a80  1
bluestore(/var/lib/ceph/osd/ceph-136) _fsck_on_open checking
shared_blobs
2020-10-14 13:27:50.368 7f78e3d20a80  1
bluestore(/var/lib/ceph/osd/ceph-136) _fsck_on_open checking
pool_statfs
2020-10-14 13:27:50.368 7f78e3d20a80 -1
bluestore(/var/lib/ceph/osd/ceph-136) fsck error: legacy statfs record
found, removing
2020-10-14 13:27:50.368 7f78e3d20a80 -1
bluestore(/var/lib/ceph/osd/ceph-136) fsck error: missing Pool StatFS
record for pool 1
2020-10-14 13:27:50.368 7f78e3d20a80 -1
bluestore(/var/lib/ceph/osd/ceph-136) fsck error: missing Pool StatFS
record for pool 2
2020-10-14 13:27:50.368 7f78e3d20a80 -1
bluestore(/var/lib/ceph/osd/ceph-136) fsck error: missing Pool StatFS
record for pool 5
2020-10-14 13:27:50.368 7f78e3d20a80 -1
bluestore(/var/lib/ceph/osd/ceph-136) fsck error: missing Pool StatFS
record for pool ffffffffffffffff

and ceph df right now showing stored == used in the pool stats:

# ceph df
RAW STORAGE:
    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED
    hdd       3.2 PiB     2.3 PiB     842 TiB      843 TiB         26.01
    TOTAL     3.2 PiB     2.3 PiB     842 TiB      843 TiB         26.00

POOLS:
    POOL                ID     STORED      OBJECTS     USED
%USED     MAX AVAIL
    cephfs_data          1     252 TiB     270.90M     252 TiB
20.48       326 TiB
    cephfs_metadata      2      72 GiB      22.03M      72 GiB
0       326 TiB
    test                 5     9.3 GiB      10.96k     9.3 GiB
0       326 TiB

Cheers, Dan

On Thu, Nov 26, 2020 at 6:00 PM Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote:
>
> Hi Igor,
>
> No BLUESTORE_LEGACY_STATFS warning, and
> bluestore_warn_on_legacy_statfs is the default true on this (and all)
> clusters.
> I'm quite sure we did the statfs conversion during one of the recent
> upgrades (I forget which one exactly).
>
> # ceph tell osd.* config get bluestore_warn_on_legacy_statfs | grep -v true
> #
>
> Is there a command to see the statfs reported by an individual OSD ?
> We have a mix of ~year old and recently recreated OSDs, so I could try
> to see if they differ.
>
> Thanks!
>
> Dan
>
>
> On Thu, Nov 26, 2020 at 5:50 PM Igor Fedotov <ifedotov@xxxxxxx> wrote:
> >
> > Hi Dan
> >
> > don't you have BLUESTORE_LEGACY_STATFS alert raised (might be silenced
> > by bluestore_warn_on_legacy_statfs param) for the older cluster?
> >
> >
> > Thanks,
> >
> > Igor
> >
> >
> > On 11/26/2020 7:29 PM, Dan van der Ster wrote:
> > > Hi,
> > >
> > > Depending on which cluster I look at (all running v14.2.11), the
> > > bytes_used is reporting raw space or stored bytes variably.
> > >
> > > Here's a 7 year old cluster:
> > >
> > > # ceph df -f json | jq .pools[0]
> > > {
> > >    "name": "volumes",
> > >    "id": 4,
> > >    "stats": {
> > >      "stored": 1229308190855881,
> > >      "objects": 294401604,
> > >      "kb_used": 1200496280133,
> > >      "bytes_used": 1229308190855881,
> > >      "percent_used": 0.4401889145374298,
> > >      "max_avail": 521125025021952
> > >    }
> > > }
> > >
> > > Note that stored == bytes_used for that pool. (this is a 3x replica pool).
> > >
> > > But here's a newer cluster (installed recently with nautilus)
> > >
> > > # ceph df -f json  | jq .pools[0]
> > > {
> > >    "name": "volumes",
> > >    "id": 1,
> > >    "stats": {
> > >      "stored": 680977600893041,
> > >      "objects": 163155803,
> > >      "kb_used": 1995736271829,
> > >      "bytes_used": 2043633942351985,
> > >      "percent_used": 0.23379847407341003,
> > >      "max_avail": 2232457428467712
> > >    }
> > > }
> > >
> > > In the second cluster, bytes_used is 3x stored.
> > >
> > > Does anyone know why these are not reported consistently?
> > > Noticing this just now, I'll update our monitoring to plot stored
> > > rather than bytes_used from now on.
> > >
> > > Thanks!
> > >
> > > Dan
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux