|
Hello. Could you please help me troubleshoot the issue. I have 3 nodes in a cluster.
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.02637 root default
-2 0.00879 host testk8s3
0 0.00879 osd.0 up 1.00000 1.00000
-3 0.00879 host testk8s1
1 0.00879 osd.1 down 0 1.00000
-4 0.00879 host testk8s2
2 0.00879 osd.2 up 1.00000 1.00000
So when all nodes are up, everything is fine.
When any of 3 nodes goes down, no matter if it shuts down gracefully or in a hard way, remaining nodes cannot read or write to the catalog where ceph storage is mounted. They also cannot unmount the volume. Every process touching
the catalog just hangs forever, going into uninterruptible sleep. When I try to strace that process, strace hangs too. When the failed node goes up, each hung process finishes successfully.
So what could cause the issue?
root@testk8s2:~# ps -eo pid,stat,cmd | grep ls
3700 D ls --color=auto /mnt/db
3997 S+ grep --color=auto ls
root@testk8s2:~# strace -p 3700&
[1] 4020
root@testk8s2:~# strace: Process 3700 attached
root@testk8s2:~# ps -eo pid,stat,cmd | grep strace
4020 S strace -p 3700
root@testk8s2:~# umount /mnt&
[2] 4084
root@testk8s2:~# ps -eo pid,state,cmd | grep umount
4084 D umount /mnt
root@testk8s2:~# ceph -v
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
root@testk8s2:~# ceph -s
cluster 0bcc00ec-731a-4734-8d76-599f70f06209
health HEALTH_ERR
80 pgs degraded
80 pgs stuck degraded
80 pgs stuck unclean
80 pgs stuck undersized
80 pgs undersized
recovery 1075/3225 objects degraded (33.333%)
mds rank 2 has failed
mds cluster is degraded
1 mons down, quorum 1,2 testk8s2,testk8s3
monmap e1: 3 mons at {testk8s1=10.105.6.116:6789/0,testk8s2=10.105.6.117:6789/0,testk8s3=10.105.6.118:6789/0}
election epoch 120, quorum 1,2 testk8s2,testk8s3
fsmap e14084: 2/3/3 up {0=testk8s2=up:active,1=testk8s3=up:active}, 1 failed
osdmap e9939: 3 osds: 2 up, 2 in; 80 remapped pgs
flags sortbitwise,require_jewel_osds
pgmap v17491: 80 pgs, 3 pools, 194 MB data, 1075 objects
1530 MB used, 16878 MB / 18408 MB avail
1075/3225 objects degraded (33.333%)
80 active+undersized+degraded
kind regards, Grigori
|
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
