Dear Cephers,
We're using the ceph file system with the fuse client, and lately some of
our processes are getting stuck seemingly waiting for fuse operations. At
the same time, the cluster is healthy, no slow requests, all OSDs up and
running, and both the MDS and the fuse client think that there are no
pending operations. The situation is semi-reproducible. When I run a
various cluster jobs, some get stuck after a few hours of correct operation.
The cluster is on ceph 10.2.5 and 10.2.6, the fuse clients are 10.2.6, but I
have tried 10.2.5 and 10.2.3, all of which have the same issue. This is on
CentOS (7.2 for the clients, 7.3 for the MDS/OSDs).
Here are some details:
The node with the stuck processes:
[root@worker1070 ~]# ps -auxwww | grep 30519
apataki 30519 39.8 0.9 8728064 5257588 ? Dl 12:11 60:50 ./Arepo
param.txt 2 6
[root@worker1070 ~]# cat /proc/30519/stack
[<ffffffffa0a1d7bb>] fuse_file_aio_write+0xbb/0x340 [fuse]
[<ffffffff811ddd3d>] do_sync_write+0x8d/0xd0
[<ffffffff811de55d>] vfs_write+0xbd/0x1e0
[<ffffffff811defff>] SyS_write+0x7f/0xe0
[<ffffffff816458c9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
[root@worker1070 ~]# ps -auxwww | grep 30533
apataki 30533 39.8 0.9 8795316 5261308 ? Sl 12:11 60:55 ./Arepo
param.txt 2 6
[root@worker1070 ~]# cat /proc/30533/stack
[<ffffffffa0a12241>] wait_answer_interruptible+0x91/0xe0 [fuse]
[<ffffffffa0a12653>] __fuse_request_send+0x253/0x2c0 [fuse]
[<ffffffffa0a126d2>] fuse_request_send+0x12/0x20 [fuse]
[<ffffffffa0a1b966>] fuse_send_write+0xd6/0x110 [fuse]
[<ffffffffa0a1d45d>] fuse_perform_write+0x2ed/0x590 [fuse]
[<ffffffffa0a1d9a1>] fuse_file_aio_write+0x2a1/0x340 [fuse]
[<ffffffff811ddd3d>] do_sync_write+0x8d/0xd0
[<ffffffff811de55d>] vfs_write+0xbd/0x1e0
[<ffffffff811defff>] SyS_write+0x7f/0xe0
[<ffffffff816458c9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
Presumably the second process is waiting on the first holding some lock ...
The fuse client on the node:
[root@worker1070 ~]# ceph daemon /var/run/ceph/ceph-client.admin.asok status
{
"metadata": {
"ceph_sha1": "656b5b63ed7c43bd014bcafd81b001959d5f089f",
"ceph_version": "ceph version 10.2.6
(656b5b63ed7c43bd014bcafd81b001959d5f089f)",
"entity_id": "admin",
"hostname": "worker1070",
"mount_point": "\/mnt\/ceph",
"root": "\/"
},
"dentry_count": 40,
"dentry_pinned_count": 23,
"inode_count": 123,
"mds_epoch": 19041,
"osd_epoch": 462327,
"osd_epoch_barrier": 462326
}
[root@worker1070 ~]# ceph daemon /var/run/ceph/ceph-client.admin.asok
mds_sessions
{
"id": 3616543,
"sessions": [
{
"mds": 0,
"addr": "10.128.128.110:6800\/909443124",
"seq": 338,
"cap_gen": 0,
"cap_ttl": "2017-03-13 14:47:37.575229",
"last_cap_renew_request": "2017-03-13 14:46:37.575229",
"cap_renew_seq": 12694,
"num_caps": 713,
"state": "open"
}
],
"mdsmap_epoch": 19041
}
[root@worker1070 ~]# ceph daemon /var/run/ceph/ceph-client.admin.asok
mds_requests
{}
The overall cluster health and the MDS:
[root@cephosd000 ~]# ceph -s
cluster d7b33135-0940-4e48-8aa6-1d2026597c2f
health HEALTH_WARN
noscrub,nodeep-scrub,require_jewel_osds flag(s) set
monmap e17: 3 mons at
{hyperv029=10.4.36.179:6789/0,hyperv030=10.4.36.180:6789/0,hyperv031=10.4.36.181:6789/0}
election epoch 29148, quorum 0,1,2 hyperv029,hyperv030,hyperv031
fsmap e19041: 1/1/1 up {0=cephosd000=up:active}
osdmap e462328: 624 osds: 624 up, 624 in
flags noscrub,nodeep-scrub,require_jewel_osds
pgmap v44458747: 42496 pgs, 6 pools, 924 TB data, 272 Mobjects
2154 TB used, 1791 TB / 3946 TB avail
42496 active+clean
client io 86911 kB/s rd, 556 MB/s wr, 227 op/s rd, 303 op/s wr
[root@cephosd000 ~]# ceph daemon /var/run/ceph/ceph-mds.cephosd000.asok ops
{
"ops": [],
"num_ops": 0
}
The odd thing is that if in this state I restart the MDS, the client process
wakes up and proceeds with its work without any errors. As if a request was
lost and somehow retransmitted/restarted when the MDS got restarted and the
fuse layer reconnected to it.
