Colleagues, I have the update.
Starting from yestrerday the situation with ceph health is much worse than it was previously.
We found that
- ceph -s informs us that some PGs are in stale state
- almost all diagnostic ceph subcommands hang! For example, "ceph osd ls" , "ceph osd dump", "ceph osd tree", "ceph health detail" provide the output - but "ceph osd status", all the commands "ceph pg ..." and other ones hang.
So, it looks that the crashes of MDS daemons were the first signs of problems only.
I read that "stale" state for PGs means that all nodes storing this placement group may be down - but it's wrong, all osd daemons are up on all three nodes:
------- ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 68.05609 root default
-3 22.68536 host asrv-dev-stor-1
0 hdd 5.45799 osd.0 up 1.00000 1.00000
1 hdd 5.45799 osd.1 up 1.00000 1.00000
2 hdd 5.45799 osd.2 up 1.00000 1.00000
3 hdd 5.45799 osd.3 up 1.00000 1.00000
12 ssd 0.42670 osd.12 up 1.00000 1.00000
13 ssd 0.42670 osd.13 up 1.00000 1.00000
-5 22.68536 host asrv-dev-stor-2
4 hdd 5.45799 osd.4 up 1.00000 1.00000
5 hdd 5.45799 osd.5 up 1.00000 1.00000
6 hdd 5.45799 osd.6 up 1.00000 1.00000
7 hdd 5.45799 osd.7 up 1.00000 1.00000
14 ssd 0.42670 osd.14 up 1.00000 1.00000
15 ssd 0.42670 osd.15 up 1.00000 1.00000
-7 22.68536 host asrv-dev-stor-3
8 hdd 5.45799 osd.8 up 1.00000 1.00000
10 hdd 5.45799 osd.10 up 1.00000 1.00000
11 hdd 5.45799 osd.11 up 1.00000 1.00000
18 hdd 5.45799 osd.18 up 1.00000 1.00000
16 ssd 0.42670 osd.16 up 1.00000 1.00000
17 ssd 0.42670 osd.17 up 1.00000 1.00000
May it be the physical problem with our drives? "smartctl -a" informs nothing wrong. We started the surface check using dd command also but it will be 7 hours per drive at least...
What should we do also?
The output of "ceph health detail":
ceph health detail
HEALTH_ERR 1 MDSs report damaged metadata; insufficient standby MDS daemons available; Reduced data availability: 50 pgs stale; 90 daemons have recently crashed; 3 mgr modules have recently crashed
[ERR] MDS_DAMAGE: 1 MDSs report damaged metadata
mds.asrv-dev-stor-2(mds.0): Metadata damage detected
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
have 0; want 1 more
[WRN] PG_AVAILABILITY: Reduced data availability: 50 pgs stale
pg 5.0 is stuck stale for 67m, current state stale+active+clean, last acting [4,1,11]
pg 5.13 is stuck stale for 67m, current state stale+active+clean, last acting [4,0,10]
pg 5.18 is stuck stale for 67m, current state stale+active+clean, last acting [4,11,2]
pg 5.19 is stuck stale for 67m, current state stale+active+clean, last acting [4,3,10]
pg 5.1e is stuck stale for 10h, current state stale+active+clean, last acting [0,7,11]
pg 5.22 is stuck stale for 10h, current state stale+active+clean, last acting [0,6,18]
pg 5.26 is stuck stale for 67m, current state stale+active+clean, last acting [4,1,18]
pg 5.29 is stuck stale for 10h, current state stale+active+clean, last acting [0,11,6]
pg 5.2b is stuck stale for 10h, current state stale+active+clean, last acting [0,18,6]
pg 5.30 is stuck stale for 10h, current state stale+active+clean, last acting [0,8,7]
pg 5.37 is stuck stale for 67m, current state stale+active+clean, last acting [4,10,0]
pg 5.3c is stuck stale for 67m, current state stale+active+clean, last acting [4,10,3]
pg 5.43 is stuck stale for 10h, current state stale+active+clean, last acting [0,6,18]
pg 5.44 is stuck stale for 67m, current state stale+active+clean, last acting [4,2,11]
pg 5.45 is stuck stale for 67m, current state stale+active+clean, last acting [4,11,3]
pg 5.47 is stuck stale for 67m, current state stale+active+clean, last acting [4,10,1]
pg 5.48 is stuck stale for 10h, current state stale+active+clean, last acting [0,5,11]
pg 5.60 is stuck stale for 10h, current state stale+active+clean, last acting [0,10,7]
pg 7.2 is stuck stale for 67m, current state stale+active+clean, last acting [4,2,10]
pg 7.4 is stuck stale for 67m, current state stale+active+clean, last acting [4,18,3]
pg 7.f is stuck stale for 10h, current state stale+active+clean, last acting [0,4,8]
pg 7.13 is stuck stale for 10h, current state stale+active+clean, last acting [0,7,11]
pg 7.18 is stuck stale for 67m, current state stale+active+clean, last acting [4,0,10]
pg 7.1b is stuck stale for 67m, current state stale+active+clean, last acting [4,8,0]
pg 7.1f is stuck stale for 10h, current state stale+active+clean, last acting [0,5,11]
pg 7.2a is stuck stale for 10h, current state stale+active+clean, last acting [0,6,8]
pg 7.35 is stuck stale for 10h, current state stale+active+clean, last acting [0,6,10]
pg 7.36 is stuck stale for 67m, current state stale+active+clean, last acting [4,2,8]
pg 7.37 is stuck stale for 10h, current state stale+active+clean, last acting [0,8,7]
pg 7.38 is stuck stale for 10h, current state stale+active+clean, last acting [0,6,11]
pg 9.10 is stuck stale for 67m, current state stale+active+clean, last acting [4,0,8]
pg 9.16 is stuck stale for 10h, current state stale+active+clean, last acting [0,4,11]
pg 9.20 is stuck stale for 67m, current state stale+active+clean, last acting [4,3,8]
pg 9.2a is stuck stale for 67m, current state stale+active+clean, last acting [4,8,0]
pg 9.33 is stuck stale for 10h, current state stale+active+clean, last acting [0,18,5]
pg 9.3a is stuck stale for 10h, current state stale+active+clean, last acting [0,8,5]
pg 9.48 is stuck stale for 67m, current state stale+active+clean, last acting [4,2,11]
pg 9.4b is stuck stale for 10h, current state stale+active+clean, last acting [0,7,11]
pg 9.4f is stuck stale for 10h, current state stale+active+clean, last acting [0,6,10]
pg 9.52 is stuck stale for 67m, current state stale+active+clean, last acting [4,8,0]
pg 9.53 is stuck stale for 10h, current state stale+active+clean, last acting [0,11,7]
pg 9.56 is stuck stale for 10h, current state stale+active+clean, last acting [0,5,18]
pg 9.5a is stuck stale for 10h, current state stale+active+clean, last acting [0,7,8]
pg 9.5d is stuck stale for 10h, current state stale+active+clean, last acting [0,6,10]
pg 9.6b is stuck stale for 67m, current state stale+active+clean, last acting [4,11,0]
pg 9.6f is stuck stale for 67m, current state stale+active+clean, last acting [4,2,18]
pg 9.73 is stuck stale for 67m, current state stale+active+clean, last acting [4,2,10]
pg 9.76 is stuck stale for 67m, current state stale+active+clean, last acting [4,10,2]
pg 9.79 is stuck stale for 10h, current state stale+active+clean, last acting [0,6,8]
pg 9.7f is stuck stale for 10h, current state stale+active+clean, last acting [0,10,5]
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
