Re: disk usage reported incorrectly

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Fix is on its way too...

See https://github.com/ceph/ceph/pull/28978

On 7/17/2019 8:55 PM, Paul Mezzanini wrote:
Oh my.  That's going to hurt with 788 OSDs.   Time for some creative shell scripts and stepping through the nodes.  I'll report back.

--
Paul Mezzanini
Sr Systems Administrator / Engineer, Research Computing
Information & Technology Services
Finance & Administration
Rochester Institute of Technology
o:(585) 475-3245 | pfmeec@xxxxxxx

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.
------------------------

________________________________________
From: Igor Fedotov <ifedotov@xxxxxxx>
Sent: Wednesday, July 17, 2019 11:33 AM
To: Paul Mezzanini; ceph-users@xxxxxxxxxxxxxx
Subject: Re:  disk usage reported incorrectly

Forgot to provide a workaround...

If that's the case then you need to repair each OSD with corresponding
command in ceph-objectstore-tool...

Thanks,

Igor.


On 7/17/2019 6:29 PM, Paul Mezzanini wrote:
Sometime after our upgrade to Nautilus our disk usage statistics went off the rails wrong.  I can't tell you exactly when it broke but I know that after the initial upgrade it worked at least for a bit.

Correct numbers should be something similar to: (These are copy/pasted from the autoscale-status report)

POOL    SIZE
cephfs_metadata     327.1G
cold-ec    98.36T
ceph-bulk-3r    142.6T
cephfs_data    31890G
ceph-hot-2r    5276G
kgcoe-cinder    103.2T
rbd   3098


Instead, we now show:

POOL     SIZE
cephfs_metadata    362.9G     (correct)
cold-ec    607.2G    (wrong)
ceph-bulk-3r    5186G (wrong)
cephfs_data    1654G (wrong)
ceph-hot-2r    5884G (correct I think)
kgcoe-cinder    5761G   (wrong)
rbd    128.0k


`ceph fs status` reports similar numbers.  cold-ec, ceph-hot-2r and cephfs_data are all cephfs data pools and cephfs_metadata is unsurprisingly, cephfs metadata.  The remaining pools are all used for rbd.


Interestingly, the `ceph df` outpool for raw storage feels correct for each drive class while the pool usage is wrong:

RAW STORAGE:
      CLASS         SIZE        AVAIL       USED        RAW USED     %RAW USED
      hdd           6.3 PiB     5.2 PiB     1.1 PiB      1.1 PiB         17.08
      nvme          175 TiB     161 TiB      14 TiB       14 TiB          7.82
      nvme-meta      14 TiB      11 TiB     2.2 TiB      2.5 TiB         18.45
      TOTAL         6.5 PiB     5.4 PiB     1.1 PiB      1.1 PiB         16.84

POOLS:
      POOL                ID     STORED      OBJECTS     USED        %USED     MAX AVAIL
      kgcoe-cinder        24     1.9 TiB      29.49M     5.6 TiB      0.32       582 TiB
      ceph-bulk-3r        32     1.7 TiB      88.28M     5.1 TiB      0.29       582 TiB
      cephfs_data         35     518 GiB     135.68M     1.6 TiB      0.09       582 TiB
      cephfs_metadata     36     363 GiB       5.63M     363 GiB      3.35       3.4 TiB
      rbd                 37       931 B           5     128 KiB         0       582 TiB
      ceph-hot-2r         50     5.7 TiB      18.63M     5.7 TiB      3.72        74 TiB
      cold-ec             51     417 GiB     105.23M     607 GiB      0.02       2.1 PiB


Everything is on "ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)" and kernel 5.0.21 or 5.0.9.  I'm actually doing the patching now to pull the ceph cluster up to 5.0.21, same as the clients.  I'm not really sure where to dig into this one.  Everything is working fine except disk usage reporting.  This also completely blows up the autoscaler.

I feel like the question is obvious but I'll state it anyway.  How do I get this issue resolved?

Thanks
-paul

--
Paul Mezzanini
Sr Systems Administrator / Engineer, Research Computing
Information & Technology Services
Finance & Administration
Rochester Institute of Technology
o:(585) 475-3245 | pfmeec@xxxxxxx

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.
------------------------
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux