Hi Team,
We have ceph setup with 6 OSD and we got alert with 2 OSD is near full . We faced issue like slow in accessing ceph from client. So i have added 7th OSD and still 2 OSD is showing near full ( OSD.0 and OSD.4) , I have restarted ceph service in osd.0 and osd.4 . Kindly check the below ceph osd status and please provide us the solutions.
# ceph health detail
HEALTH_WARN 46 pgs backfill_wait; 1 pgs backfilling; 32 pgs degraded; 50 pgs stuck unclean; 32 pgs undersized; recovery 1098780/40253637 objects degraded (2.730%); recovery 3401433/40253637 objects misplaced (8.450%); 2 near full osd(s); mds0: Client integ-hm3 failing to respond to cache pressure; mds0: Client integ-hm8 failing to respond to cache pressure; mds0: Client integ-hm2 failing to respond to cache pressure; mds0: Client integ-hm9 failing to respond to cache pressure; mds0: Client integ-hm5 failing to respond to cache pressure; mds0: Client integ-hm9-bkp failing to respond to cache pressure; mds0: Client me-build1-bkp failing to respond to cache pressure
pg 3.f6 is stuck unclean for 511223.069161, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 4.f6 is stuck unclean for 511232.770419, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 3.ec is stuck unclean for 510902.815668, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 3.eb is stuck unclean for 511285.576487, current state active+remapped+wait_backfill, last acting [3,0]
pg 4.17 is stuck unclean for 511235.326709, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 4.2f is stuck unclean for 511232.356371, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 4.3d is stuck unclean for 511300.446982, current state active+remapped, last acting [3,0]
pg 4.93 is stuck unclean for 511295.539229, current state active+undersized+degraded+remapped+wait_backfill, last acting [3]
pg 3.47 is stuck unclean for 511288.104965, current state active+remapped+wait_backfill, last acting [3,0]
pg 4.d5 is stuck unclean for 510916.509825, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 3.31 is stuck unclean for 511221.542878, current state active+remapped+wait_backfill, last acting [0,3]
pg 3.62 is stuck unclean for 511221.551662, current state active+undersized+degraded+remapped+wait_backfill, last acting [4]
pg 4.4d is stuck unclean for 511232.279602, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 4.48 is stuck unclean for 510911.095367, current state active+remapped+wait_backfill, last acting [5,4]
pg 3.4f is stuck unclean for 511226.712285, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 3.78 is stuck unclean for 511221.531199, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 3.24 is stuck unclean for 510903.483324, current state active+remapped+backfilling, last acting [1,2]
pg 4.8c is stuck unclean for 511231.668693, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 3.b4 is stuck unclean for 511222.612012, current state active+undersized+degraded+remapped+wait_backfill, last acting [0]
pg 4.41 is stuck unclean for 511287.031264, current state active+remapped+wait_backfill, last acting [3,2]
pg 3.d1 is stuck unclean for 510903.797329, current state active+remapped+wait_backfill, last acting [0,3]
pg 3.7f is stuck unclean for 511222.929722, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 4.af is stuck unclean for 511262.494659, current state active+undersized+degraded+remapped, last acting [0]
pg 3.66 is stuck unclean for 510903.296711, current state active+remapped+wait_backfill, last acting [3,0]
pg 3.76 is stuck unclean for 511224.615144, current state active+undersized+degraded+remapped+wait_backfill, last acting [3]
pg 4.57 is stuck unclean for 511234.514343, current state active+remapped, last acting [0,4]
pg 3.69 is stuck unclean for 511224.672085, current state active+undersized+degraded+remapped+wait_backfill, last acting [4]
pg 3.9a is stuck unclean for 510967.300000, current state active+remapped+wait_backfill, last acting [3,2]
pg 4.50 is stuck unclean for 510903.825565, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 4.53 is stuck unclean for 510921.975268, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 3.e7 is stuck unclean for 511221.530592, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 4.6a is stuck unclean for 510911.284877, current state active+undersized+degraded+remapped+wait_backfill, last acting [0]
pg 4.16 is stuck unclean for 511232.702762, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 3.2c is stuck unclean for 511222.443893, current state active+remapped+wait_backfill, last acting [2,3]
pg 4.89 is stuck unclean for 511228.846614, current state active+undersized+degraded+remapped+wait_backfill, last acting [4]
pg 4.39 is stuck unclean for 511239.544231, current state active+remapped+wait_backfill, last acting [3,2]
pg 4.ce is stuck unclean for 511232.294586, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 3.91 is stuck unclean for 511232.341380, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 3.96 is stuck unclean for 510904.043900, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 4.c0 is stuck unclean for 510904.253281, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 4.9c is stuck unclean for 511237.612850, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 3.ab is stuck unclean for 510960.756324, current state active+remapped+wait_backfill, last acting [3,2]
pg 4.aa is stuck unclean for 511229.307559, current state active+remapped+wait_backfill, last acting [0,3]
pg 3.ad is stuck unclean for 510903.764157, current state active+remapped+wait_backfill, last acting [0,3]
pg 3.b5 is stuck unclean for 511226.560774, current state active+undersized+degraded+remapped+wait_backfill, last acting [3]
pg 4.58 is stuck unclean for 510919.273667, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 4.b9 is stuck unclean for 511232.760066, current state active+remapped+wait_backfill, last acting [5,4]
pg 3.be is stuck unclean for 511224.422931, current state active+remapped+wait_backfill, last acting [0,4]
pg 4.d4 is stuck unclean for 510962.810416, current state active+undersized+degraded+remapped+wait_backfill, last acting [3]
pg 4.da is stuck unclean for 511259.506962, current state active+undersized+degraded+remapped+wait_backfill, last acting [2]
pg 4.8c is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 3.7f is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 3.78 is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 3.76 is active+undersized+degraded+remapped+wait_backfill, acting [3]
pg 4.6a is active+undersized+degraded+remapped+wait_backfill, acting [0]
pg 3.69 is active+undersized+degraded+remapped+wait_backfill, acting [4]
pg 3.66 is active+remapped+wait_backfill, acting [3,0]
pg 3.62 is active+undersized+degraded+remapped+wait_backfill, acting [4]
pg 4.58 is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 4.50 is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 4.53 is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 3.4f is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 4.48 is active+remapped+wait_backfill, acting [5,4]
pg 4.4d is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 3.47 is active+remapped+wait_backfill, acting [3,0]
pg 4.41 is active+remapped+wait_backfill, acting [3,2]
pg 3.31 is active+remapped+wait_backfill, acting [0,3]
pg 4.2f is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 3.24 is active+remapped+backfilling, acting [1,2]
pg 4.17 is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 4.16 is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 3.2c is active+remapped+wait_backfill, acting [2,3]
pg 4.39 is active+remapped+wait_backfill, acting [3,2]
pg 4.89 is active+undersized+degraded+remapped+wait_backfill, acting [4]
pg 3.91 is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 4.93 is active+undersized+degraded+remapped+wait_backfill, acting [3]
pg 3.96 is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 3.9a is active+remapped+wait_backfill, acting [3,2]
pg 4.9c is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 4.af is active+undersized+degraded+remapped, acting [0]
pg 3.ab is active+remapped+wait_backfill, acting [3,2]
pg 4.aa is active+remapped+wait_backfill, acting [0,3]
pg 3.ad is active+remapped+wait_backfill, acting [0,3]
pg 3.b4 is active+undersized+degraded+remapped+wait_backfill, acting [0]
pg 3.b5 is active+undersized+degraded+remapped+wait_backfill, acting [3]
pg 4.b9 is active+remapped+wait_backfill, acting [5,4]
pg 3.be is active+remapped+wait_backfill, acting [0,4]
pg 4.c0 is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 4.ce is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 3.d1 is active+remapped+wait_backfill, acting [0,3]
pg 4.d5 is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 4.d4 is active+undersized+degraded+remapped+wait_backfill, acting [3]
pg 4.da is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 3.e7 is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 3.eb is active+remapped+wait_backfill, acting [3,0]
pg 3.ec is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 4.f6 is active+undersized+degraded+remapped+wait_backfill, acting [2]
pg 3.f6 is active+undersized+degraded+remapped+wait_backfill, acting [2]
recovery 1098780/40253637 objects degraded (2.730%)
recovery 3401433/40253637 objects misplaced (8.450%)
osd.0 is near full at 85%
osd.4 is near full at 90%
mds0: Client integ-hm3 failing to respond to cache pressure(client_id: 733998)
mds0: Client integ-hm8 failing to respond to cache pressure(client_id: 843866)
mds0: Client integ-hm2 failing to respond to cache pressure(client_id: 844939)
mds0: Client integ-hm9 failing to respond to cache pressure(client_id: 845065)
mds0: Client integ-hm5 failing to respond to cache pressure(client_id: 845068)
mds0: Client integ-hm9-bkp failing to respond to cache pressure(client_id: 895898)
mds0: Client me-build1-bkp failing to respond to cache pressure(client_id: 888666)
hm ~]# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 22.92604 root default
-2 3.29749 host intcfs-osd1
0 3.29749 osd.0 up 1.00000 1.00000
-3 3.26869 host intcfs-osd2
1 3.26869 osd.1 up 1.00000 1.00000
-4 3.27339 host intcfs-osd3
2 3.27339 osd.2 up 1.00000 1.00000
-5 3.24089 host intcfs-osd4
3 3.24089 osd.3 up 1.00000 1.00000
-6 3.24089 host intcfs-osd5
4 3.24089 osd.4 up 1.00000 1.00000
-7 3.32669 host intcfs-osd6
5 3.32669 osd.5 up 1.00000 1.00000
-8 3.27800 host intcfs-osd7
6 3.27800 osd.6 up 1.00000 1.00000
hm5 ~]# ceph osd df
ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 3.29749 1.00000 3376G 2874G 502G 85.13 1.26 165
1 3.26869 1.00000 3347G 1922G 1424G 57.44 0.85 152
2 3.27339 1.00000 3351G 2009G 1342G 59.95 0.89 162
3 3.24089 1.00000 3318G 2130G 1188G 64.19 0.95 168
4 3.24089 1.00000 3318G 2996G 321G 90.30 1.34 176
5 3.32669 1.00000 3406G 2465G 940G 72.39 1.07 165
6 3.27800 1.00000 3356G 1435G 1921G 42.76 0.63 166
TOTAL 23476G 15834G 7641G 67.45
MIN/MAX VAR: 0.63/1.34 STDDEV: 15.29
Regards
Prabu GJ
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
