Oh my. That's going to hurt with 788 OSDs. Time for some creative shell scripts and stepping through the nodes. I'll report back. -- Paul Mezzanini Sr Systems Administrator / Engineer, Research Computing Information & Technology Services Finance & Administration Rochester Institute of Technology o:(585) 475-3245 | pfmeec@xxxxxxx CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information. ------------------------ ________________________________________ From: Igor Fedotov <ifedotov@xxxxxxx> Sent: Wednesday, July 17, 2019 11:33 AM To: Paul Mezzanini; ceph-users@xxxxxxxxxxxxxx Subject: Re: disk usage reported incorrectly Forgot to provide a workaround... If that's the case then you need to repair each OSD with corresponding command in ceph-objectstore-tool... Thanks, Igor. On 7/17/2019 6:29 PM, Paul Mezzanini wrote: > Sometime after our upgrade to Nautilus our disk usage statistics went off the rails wrong. I can't tell you exactly when it broke but I know that after the initial upgrade it worked at least for a bit. > > Correct numbers should be something similar to: (These are copy/pasted from the autoscale-status report) > > POOL SIZE > cephfs_metadata 327.1G > cold-ec 98.36T > ceph-bulk-3r 142.6T > cephfs_data 31890G > ceph-hot-2r 5276G > kgcoe-cinder 103.2T > rbd 3098 > > > Instead, we now show: > > POOL SIZE > cephfs_metadata 362.9G (correct) > cold-ec 607.2G (wrong) > ceph-bulk-3r 5186G (wrong) > cephfs_data 1654G (wrong) > ceph-hot-2r 5884G (correct I think) > kgcoe-cinder 5761G (wrong) > rbd 128.0k > > > `ceph fs status` reports similar numbers. cold-ec, ceph-hot-2r and cephfs_data are all cephfs data pools and cephfs_metadata is unsurprisingly, cephfs metadata. The remaining pools are all used for rbd. > > > Interestingly, the `ceph df` outpool for raw storage feels correct for each drive class while the pool usage is wrong: > > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 6.3 PiB 5.2 PiB 1.1 PiB 1.1 PiB 17.08 > nvme 175 TiB 161 TiB 14 TiB 14 TiB 7.82 > nvme-meta 14 TiB 11 TiB 2.2 TiB 2.5 TiB 18.45 > TOTAL 6.5 PiB 5.4 PiB 1.1 PiB 1.1 PiB 16.84 > > POOLS: > POOL ID STORED OBJECTS USED %USED MAX AVAIL > kgcoe-cinder 24 1.9 TiB 29.49M 5.6 TiB 0.32 582 TiB > ceph-bulk-3r 32 1.7 TiB 88.28M 5.1 TiB 0.29 582 TiB > cephfs_data 35 518 GiB 135.68M 1.6 TiB 0.09 582 TiB > cephfs_metadata 36 363 GiB 5.63M 363 GiB 3.35 3.4 TiB > rbd 37 931 B 5 128 KiB 0 582 TiB > ceph-hot-2r 50 5.7 TiB 18.63M 5.7 TiB 3.72 74 TiB > cold-ec 51 417 GiB 105.23M 607 GiB 0.02 2.1 PiB > > > Everything is on "ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)" and kernel 5.0.21 or 5.0.9. I'm actually doing the patching now to pull the ceph cluster up to 5.0.21, same as the clients. I'm not really sure where to dig into this one. Everything is working fine except disk usage reporting. This also completely blows up the autoscaler. > > I feel like the question is obvious but I'll state it anyway. How do I get this issue resolved? > > Thanks > -paul > > -- > Paul Mezzanini > Sr Systems Administrator / Engineer, Research Computing > Information & Technology Services > Finance & Administration > Rochester Institute of Technology > o:(585) 475-3245 | pfmeec@xxxxxxx > > CONFIDENTIALITY NOTE: The information transmitted, including attachments, is > intended only for the person(s) or entity to which it is addressed and may > contain confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon this > information by persons or entities other than the intended recipient is > prohibited. If you received this in error, please contact the sender and > destroy any copies of this information. > ------------------------ > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |