Hi,
I can't seem to find the threads I was looking for, this has been
discussed before. Anyway, IIRC it could be a MGR issue which fails to
update the stats. Maybe a MGR failover clears things up? If that
doesn't help I would try a compaction on one OSD and see if the stats
are corrected (and if they are, do it for the rest as well). If not,
someone else might have more ideas. :-)
Regards,
Eugen
Zitat von Andras Pataki <apataki@xxxxxxxxxxxxxxxxxxxxx>:
Here is a strange problem that I don't seem to be able to figure
out. Some of our OSDs that have zero weight, and no PGs have lots
of allocated space:
[root@cephosd0032 ~]# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
META AVAIL %USE VAR PGS STATUS
2925 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2
MiB 1021 MiB 984 GiB 11.32 0.15 0 up
2926 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2
MiB 1021 MiB 984 GiB 11.33 0.15 0 up
2927 ssd 0 1.00000 1.1 TiB 125 GiB 124 GiB 3.2
MiB 1021 MiB 984 GiB 11.31 0.15 0 up
2928 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2
MiB 1021 MiB 984 GiB 11.32 0.15 0 up
so 120GB+ allocated, but no PGs. The cluster is clean, all PGs are
active+clean, no rebalancing/recovery/etc. happening. There has
been no rebalancing/recovery for at least a couple of days (after
having added some nodes to the cluster).
Doing a perf dump on the OSD confirms the space usage - that it has
130GB or so allocated:
[root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok perf dump
....
"bluestore_allocated": 133277466624,
"bluestore_stored": 133014427663,
...
Bringing the OSD down - ceph-objectstore-tool claims that there are
no objects stored in the OSD:
[root@cephosd0032 ~]# ceph-objectstore-tool --data-path
/var/lib/ceph/osd/ceph-2925 --op list
[root@cephosd0032 ~]#
No fragmentation on the OSD either:
[root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok
bluestore allocator fragmentation block
{
"fragmentation_rating": 1.2165147867554808e-08
}
BlueFS also claims not to use significant space:
[root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok
bluefs stats
1 : device size 0x11548000000 : own 0x[8518510000~b175d0000] =
0xb175d0000 : using 0xd1e0000(210 MiB) : bluestore has
0xeb3b630000(941 GiB) available
wal_total:0, db_total:1131368205516, slow_total:0
So the puzzle is, what is stored in that 130GB allocated space?
The cluster is Octopus 15.2.17. I came to looking at this by a
mismatch in free space on our OSDs that are in service. After
adding some nodes and waiting for the rebalance to finish, the free
space didn't increase by the amount of space that was added,
although the total space in the cluster did increase correctly (and
data in pools didn't increase by a significant amount to explain
this). So there is a bit of a mystery regarding what a bunch of
space is allocated for.
Any ideas/pointers would be appreciated.
Andras
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
