Hi,
I'm currently investigating a lockup problem involving CephFS and SQLite
databases. Applications lock up if the same database is accessed from
multiple hosts.
I was able to narrow the problem down to two host:
host A:
sqlite3 <database file>
.schema
host B:
sqlite3 <database file>
.schema
If both .schema commands happen at the same time, both applications are
blocked. Client 1332420 is host A in the example above, client ID
1263969 is host B, the inode is the sqlite file:
ceph mds log:
2015-11-09 13:39:49.588024 7f6272805700 0 log_channel(cluster) log
[WRN] : client.1263969 isn't responding to mclientcaps(revoke), ino
10002c4e840 pending pAsLsXsFr issued pAsLsXsFscr, sent 245.303153
seconds ago
2015-11-09 13:39:49.588520 7f6272805700 0 log_channel(cluster) log
[WRN] : 1 slow requests, 1 included below; oldest blocked for >
245.301935 secs
2015-11-09 13:39:49.588527 7f6272805700 0 log_channel(cluster) log
[WRN] : slow request 245.301935 seconds old, received at 2015-11-09
13:35:44.286527: client_request(client.1332420:97 getattr pAsLsXsFs
#10002c4e840 2015-11-09 13:35:44.312820) currently failed to rdlock, waiting
ceph -s:
cluster 49098879-85ac-4c5d-aac0-e1a2658a680b
health HEALTH_WARN
mds0: Client <host B> failing to respond to capability release
mds0: Many clients (16) failing to respond to cache pressure
ceph mds cache dump (grepped for inode id):
inode 10002c4e840 [2,head]
/volumes/adm/temp/test/sqlite/uniprot_sprot.dat.idx auth v183 ap=2+0
s=53466112 n(v0 b53466112 1=1+0) (ifile sync->mix) (iversion lock)
cr={1263969=0-109051904@1,1332420=0-134217728@1}
caps={1263969=pAsLsXsFr/pAsLsXsFscr/pAsxXsxFsxcrwb@39,1332420=pAsLsXsFr/pAsxXsxFsxcrwb@63}
| ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=0 dirty=0 waiter=1
authpin=1 0x1480a3af8]
Cluster is running Hammer 0.94.5 on top of Ubuntu 14.04. Clients use
ceph-fuse with patches for improved page cache handling, but the problem
also occur with the official hammer packages from download.ceph.com
Any help with resolving this problem is appreciated.
Regards,
Burkhard
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
