Re: cephfs full, 2/3 Raw capacity used

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 26-08-19 11:16, Dan van der Ster wrote:
Hi,

Which version of ceph are you using? Which balancer mode?

Nautilus (14.2.2), balancer is in upmap mode.

The balancer score isn't a percent-error or anything humanly usable.
`ceph osd df tree` can better show you exactly which osds are
over/under utilized and by how much.


Aha, I ran this and sorted on the %full column:

81 hdd 10.81149 1.00000 11 TiB 5.2 TiB 5.1 TiB 4 KiB 14 GiB 5.6 TiB 48.40 0.73 96 up osd.81 48 hdd 10.81149 1.00000 11 TiB 5.3 TiB 5.2 TiB 15 KiB 14 GiB 5.5 TiB 49.08 0.74 95 up osd.48 154 hdd 10.81149 1.00000 11 TiB 5.5 TiB 5.4 TiB 2.6 GiB 15 GiB 5.3 TiB 50.95 0.76 96 up osd.154 129 hdd 10.81149 1.00000 11 TiB 5.5 TiB 5.4 TiB 5.1 GiB 16 GiB 5.3 TiB 51.33 0.77 96 up osd.129 42 hdd 10.81149 1.00000 11 TiB 5.6 TiB 5.5 TiB 2.6 GiB 14 GiB 5.2 TiB 51.81 0.78 96 up osd.42 122 hdd 10.81149 1.00000 11 TiB 5.7 TiB 5.6 TiB 16 KiB 14 GiB 5.1 TiB 52.47 0.79 96 up osd.122 120 hdd 10.81149 1.00000 11 TiB 5.7 TiB 5.6 TiB 2.6 GiB 15 GiB 5.1 TiB 52.92 0.79 95 up osd.120 96 hdd 10.81149 1.00000 11 TiB 5.8 TiB 5.7 TiB 2.6 GiB 15 GiB 5.0 TiB 53.58 0.80 96 up osd.96 26 hdd 10.81149 1.00000 11 TiB 5.8 TiB 5.7 TiB 20 KiB 15 GiB 5.0 TiB 53.68 0.80 97 up osd.26
...
6 hdd 10.81149 1.00000 11 TiB 8.3 TiB 8.2 TiB 88 KiB 18 GiB 2.5 TiB 77.14 1.16 96 up osd.6 16 hdd 10.81149 1.00000 11 TiB 8.4 TiB 8.3 TiB 28 KiB 18 GiB 2.4 TiB 77.56 1.16 95 up osd.16 0 hdd 10.81149 1.00000 11 TiB 8.6 TiB 8.4 TiB 48 KiB 17 GiB 2.2 TiB 79.24 1.19 96 up osd.0 144 hdd 10.81149 1.00000 11 TiB 8.6 TiB 8.5 TiB 2.6 GiB 18 GiB 2.2 TiB 79.57 1.19 95 up osd.144 136 hdd 10.81149 1.00000 11 TiB 8.6 TiB 8.5 TiB 48 KiB 17 GiB 2.2 TiB 79.60 1.19 95 up osd.136 63 hdd 10.81149 1.00000 11 TiB 8.6 TiB 8.5 TiB 2.6 GiB 17 GiB 2.2 TiB 79.60 1.19 95 up osd.63 155 hdd 10.81149 1.00000 11 TiB 8.6 TiB 8.5 TiB 8 KiB 19 GiB 2.2 TiB 79.85 1.20 95 up osd.155 89 hdd 10.81149 1.00000 11 TiB 8.7 TiB 8.5 TiB 12 KiB 20 GiB 2.2 TiB 80.04 1.20 96 up osd.89 106 hdd 10.81149 1.00000 11 TiB 8.8 TiB 8.7 TiB 64 KiB 19 GiB 2.0 TiB 81.38 1.22 96 up osd.106 94 hdd 10.81149 1.00000 11 TiB 9.0 TiB 8.9 TiB 0 B 19 GiB 1.8 TiB 83.53 1.25 96 up osd.94 33 hdd 10.81149 1.00000 11 TiB 9.1 TiB 9.0 TiB 44 KiB 19 GiB 1.7 TiB 84.40 1.27 96 up osd.33 15 hdd 10.81149 1.00000 11 TiB 10 TiB 9.8 TiB 16 KiB 20 GiB 877 GiB 92.08 1.38 96 up osd.15 53 hdd 10.81149 1.00000 11 TiB 10 TiB 10 TiB 2.6 GiB 20 GiB 676 GiB 93.90 1.41 96 up osd.53 51 hdd 10.81149 1.00000 11 TiB 10 TiB 10 TiB 2.6 GiB 20 GiB 666 GiB 93.98 1.41 96 up osd.51 10 hdd 10.81149 1.00000 11 TiB 10 TiB 10 TiB 40 KiB 22 GiB 552 GiB 95.01 1.42 97 up osd.10

So the fullest one is at 95.01%, the emptiest one at 48.4%, so there's some balancing to be done.

You might be able to manually fix things by using `ceph osd reweight
...` on the most full osds to move data elsewhere.

I'll look into this, but I was hoping that the balancer module would take care of this...


Otherwise, in general, its good to setup monitoring so you notice and
take action well before the osds fill up.

Yes, I'm still working on this, I want to add some checks to our check_mk+icinga setup using native plugins, but my python skills are not quite up to the task, at least, not yet ;-)

Cheers

/Simon


Cheers, Dan

On Mon, Aug 26, 2019 at 11:09 AM Simon Oosthoek
<s.oosthoek@xxxxxxxxxxxxx> wrote:

Hi all,

we're building up our experience with our ceph cluster before we take it
into production. I've now tried to fill up the cluster with cephfs,
which we plan to use for about 95% of all data on the cluster.

The cephfs pools are full when the cluster reports 67% raw capacity
used. There are 4 pools we use for cephfs data, 3-copy, 4-copy, EC 8+3
and EC 5+7. The balancer module is turned on and `ceph balancer eval`
gives `current cluster score 0.013255 (lower is better)`, so well within
the default 5% margin. Is there a setting we can tweak to increase the
usable RAW capacity to say 85% or 90%, or is this the most we can expect
to store on the cluster?

[root@cephmon1 ~]# ceph df
RAW STORAGE:
      CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED
      hdd       1.8 PiB     605 TiB     1.2 PiB      1.2 PiB         66.71
      TOTAL     1.8 PiB     605 TiB     1.2 PiB      1.2 PiB         66.71

POOLS:
      POOL                    ID     STORED      OBJECTS     USED
%USED      MAX AVAIL
      cephfs_data              1     111 MiB      79.26M     1.2 GiB
100.00           0 B
      cephfs_metadata          2      52 GiB       4.91M      52 GiB
100.00           0 B
      cephfs_data_4copy        3     106 TiB      46.36M     428 TiB
100.00           0 B
      cephfs_data_3copy        8      93 TiB      42.08M     282 TiB
100.00           0 B
      cephfs_data_ec83        13     106 TiB      50.11M     161 TiB
100.00           0 B
      rbd                     14      21 GiB       5.62k      63 GiB
100.00           0 B
      .rgw.root               15     1.2 KiB           4       1 MiB
100.00           0 B
      default.rgw.control     16         0 B           8         0 B
      0           0 B
      default.rgw.meta        17       765 B           4       1 MiB
100.00           0 B
      default.rgw.log         18         0 B         207         0 B
      0           0 B
      scbench                 19     133 GiB      34.14k     400 GiB
100.00           0 B
      cephfs_data_ec57        20     126 TiB      51.84M     320 TiB
100.00           0 B
[root@cephmon1 ~]# ceph balancer eval
current cluster score 0.013255 (lower is better)


Being full at 2/3 Raw used is a bit too "pretty" to be accidental, it
seems like this could be a parameter for cephfs, however, I couldn't
find anything like this in the documentation for Nautilus.


The logs in the dashboard show this:
2019-08-26 11:00:00.000630
[ERR]
overall HEALTH_ERR 3 backfillfull osd(s); 1 full osd(s); 12 pool(s) full

2019-08-26 10:57:44.539964
[INF]
Health check cleared: POOL_BACKFILLFULL (was: 12 pool(s) backfillfull)

2019-08-26 10:57:44.539944
[WRN]
Health check failed: 12 pool(s) full (POOL_FULL)

2019-08-26 10:57:44.539926
[ERR]
Health check failed: 1 full osd(s) (OSD_FULL)

2019-08-26 10:57:44.539899
[WRN]
Health check update: 3 backfillfull osd(s) (OSD_BACKFILLFULL)

2019-08-26 10:00:00.000088
[WRN]
overall HEALTH_WARN 4 backfillfull osd(s); 12 pool(s) backfillfull

So it seems that ceph is completely stuck at 2/3 full, while we
anticipated being able to fill up the cluster to at least 85-90% of the
raw capacity. Or at least so that we would keep a functioning cluster
when we have a single osd node fail.

Cheers

/Simon
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux