Hi Sebastian,
Thank you for your quick answer.
I'm aware with your points, and maybe I was not really clear in my
previous email (written in a hurry!)
The problematic pool is the metadata one. All its OSDs (x3) are full.
The associated data pool is OK and no OSD is full on the data pool.
The problem is that metadata pool suddenly increases a lot and
continiously from 3% to 100% in 5 hours (from 5 am to 10 am, then crash)
And we don't understand the reason, since there was no specific
activities on the data pool ?
This cluster runs perfectly with the current configuration since many years.
Thanks again,
Hervé
Le 01/06/2021 à 13:42, Sebastian Knust a écrit :
Hi Hervé,
On 01.06.21 13:15, Hervé Ballans wrote:
# ceph status
cluster:
id: 838506b7-e0c6-4022-9e17-2d1cf9458be6
health: HEALTH_ERR
1 filesystem is degraded
3 full osd(s)
1 pool(s) full
1 daemons have recently crashed
You have full OSDs and therefore a full pool. The "fullness" of a pool
is limited by the fullest OSD, i.e. a single full OSDs can block your
pool. Take a look at `ceph osd df` and you will notice a very
non-uniform osd usage (both with numbers of PG / size as well as usage
%).
osd: 126 osds: 126 up (since 5m), 126 in (since 5M)
pgs: 1662 active+clean
The PG/osd ratio seems to be very low for me. The general
recommendation is 100 PG / osd post-replication (and power of 2 for
each pool). In my cluster I actually run with ~200 PG / osd for my SSD
which contain the cephfs metadata.
Thanks a lot if you have some ways for trying to solve this...
You have to get your OSDs to rebalance, which probably includes
increasing the number of PGs in some pools. Details depend on which
Ceph version you are running and your CRUSH rules (maybe your cephfs
metadata pool is residing only on NVMe?). Take a look at the balancer
module [1] and the autoscaler [2] (`ceph osd pool autoscale-status` is
most interesting).
Theoretically, you could (temporarilly!) increase the full_ratio.
However, this is a very dangerous operation which you should not do
unless you know *exactly* what you are doing.
Cheers & Best of luck
Sebastian
[1] https://docs.ceph.com/en/latest/rados/operations/balancer/
[2] https://docs.ceph.com/en/latest/rados/operations/placement-groups/
Replace latest in the URIs with your Ceph version string (i.e.
octopus, nautilus) for version specific documentation
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
