OSD META usage growing without bounds

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi, I'm observing a strange behaviour on a small test cluster (13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable)). The cluster is up for about half a year and almost empty. We did a few rbd bench runs and created a file system, but there was zero client IO for at least 3 months. It looks like recently the OSD META usage of some OSDs started to increase for no apparent reason. One OSD already died with 100% usage and another is on its way. I can't see any obvious reason for this strange behaviour.

If anyone has an idea, please let me know.

Some diagnostic output:

[root@rit-tceph ~]# ceph status
  cluster:
    id:     bf1f51f5-b381-4cf7-b3db-88d044c1960c
    health: HEALTH_WARN
            1 nearfull osd(s)
            3 pool(s) nearfull
 
  services:
    mon: 3 daemons, quorum tceph-01,tceph-02,tceph-03
    mgr: tceph-01(active), standbys: tceph-02, tceph-03
    mds: testfs-1/1/1 up  {0=tceph-01=up:active}, 2 up:standby
    osd: 9 osds: 8 up, 8 in
 
  data:
    pools:   3 pools, 500 pgs
    objects: 24  objects, 2.3 KiB
    usage:   746 GiB used, 1.4 TiB / 2.2 TiB avail
    pgs:     500 active+clean
 
[root@rit-tceph ~]# ceph df 
GLOBAL:
    SIZE        AVAIL       RAW USED     %RAW USED 
    2.2 TiB     1.4 TiB      746 GiB         33.49 
POOLS:
    NAME                ID     USED        %USED     MAX AVAIL     OBJECTS 
    test                1         19 B         0        81 GiB           2 
    testfs_data         2          0 B         0        81 GiB           0 
    testfs_metadata     3      2.2 KiB         0        81 GiB          22 

[root@rit-tceph ~]# ceph osd df tree
ID CLASS WEIGHT  REWEIGHT SIZE    USE     DATA    OMAP   META    AVAIL   %USE  VAR  PGS TYPE NAME         
-1       2.44707        - 2.2 TiB 746 GiB 120 MiB 34 KiB 746 GiB 1.4 TiB 33.49 1.00   - root default      
-5       0.81569        - 557 GiB 195 GiB  30 MiB  3 KiB 195 GiB 362 GiB 35.04 1.05   -     host tceph-01 
 0   hdd 0.27190  1.00000 278 GiB  38 GiB  15 MiB  3 KiB  38 GiB 241 GiB 13.61 0.41 260         osd.0     
 3   hdd 0.27190        0     0 B     0 B     0 B    0 B     0 B     0 B     0    0   0         osd.3     
 8   hdd 0.27190  1.00000 278 GiB 157 GiB  15 MiB    0 B 157 GiB 121 GiB 56.47 1.69 240         osd.8     
-3       0.81569        - 835 GiB 113 GiB  45 MiB  3 KiB 113 GiB 723 GiB 13.48 0.40   -     host tceph-02 
 2   hdd 0.27190  1.00000 278 GiB  18 GiB  15 MiB    0 B  18 GiB 261 GiB  6.30 0.19 157         osd.2     
 4   hdd 0.27190  1.00000 278 GiB  48 GiB  15 MiB    0 B  48 GiB 231 GiB 17.21 0.51 172         osd.4     
 6   hdd 0.27190  1.00000 278 GiB  47 GiB  15 MiB  3 KiB  47 GiB 231 GiB 16.93 0.51 171         osd.6     
-7       0.81569        - 835 GiB 438 GiB  45 MiB 28 KiB 438 GiB 397 GiB 52.48 1.57   -     host tceph-03 
 1   hdd 0.27190  1.00000 278 GiB 238 GiB  15 MiB 25 KiB 238 GiB  41 GiB 85.35 2.55 171         osd.1     
 5   hdd 0.27190  1.00000 278 GiB 200 GiB  15 MiB  3 KiB 200 GiB  79 GiB 71.68 2.14 160         osd.5     
 7   hdd 0.27190  1.00000 278 GiB 1.1 GiB  15 MiB    0 B 1.1 GiB 277 GiB  0.40 0.01 169         osd.7     
                    TOTAL 2.2 TiB 746 GiB 120 MiB 34 KiB 746 GiB 1.4 TiB 33.49                            
MIN/MAX VAR: 0.01/2.55  STDDEV: 30.50

2 hours later:

[root@rit-tceph ~]# ceph status
  cluster:
    id:     bf1f51f5-b381-4cf7-b3db-88d044c1960c
    health: HEALTH_WARN
            1 nearfull osd(s)
            3 pool(s) nearfull
 
  services:
    mon: 3 daemons, quorum tceph-01,tceph-02,tceph-03
    mgr: tceph-01(active), standbys: tceph-02, tceph-03
    mds: testfs-1/1/1 up  {0=tceph-01=up:active}, 2 up:standby
    osd: 9 osds: 8 up, 8 in
 
  data:
    pools:   3 pools, 500 pgs
    objects: 24  objects, 2.3 KiB
    usage:   748 GiB used, 1.4 TiB / 2.2 TiB avail
    pgs:     500 active+clean

The usage is increasing surprisingly fast.

Thanks for any pointers!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux