This is quite strange. Given that you have a log, I think what you want to do is find one request in the log, trace it through its lifetime, and see where the time is elapsed. You may find a bifurcation, where some categories of requests happen instantly but other categories take a second or more; focus on the second, obviously.
-Greg
On Tue, Aug 21, 2018 at 1:31 PM Stefan Kooman <stefan@xxxxxx> wrote:
Hi,
I'm trying to find out why ceph-fuse client(s) are slow. Luminous 12.2.7
Ceph cluster, Mimic 13.2.1 ceph-fuse client. Ubuntu xenial, 4.13.0-38-generic kernel.
Test case:
25 curl requests directed at a single threaded apache process (apache2
-X).
When the requests are handled by ceph-kernel client it takes about 1.5
seconds for the first "GET /" (all fs /inode dentrie caches dropped
before hand). Subsequent requests only take ~ 0.4 seconds. So fs caches
seem to do their job.
With ceph-fuse it's a bit different. The first and subsequent requests
("GET /") will take around 4-5 seconds every single time. As if the
ceph-fuse / vfs cache does not work. ceph daemon client.id dump_cache
shows that all dentries and inodes are in the cache ~ 589 of them. Relevant
part of one of them:
"caps": [
{
"mds": 0,
"auth": 1,
"mds": 0,
"ino": "0x10000096a6a",
"cap_id": 50538956,
"issued": "pAsLsXsFscr",
"wanted": "-",
"seq": 1,
"issue_seq": 1,
"mseq": 0,
"gen": 0
}
],
"auth_cap": 0,
"dirty_caps": "-",
"shared_gen": 1,
"cache_gen": 1,
"hold_caps_until": "0.000000",
Not sure what the "issued" caps indicate. This client is currently the only
client for this directory, so cache should not be invalidated.
On the MDS side I hardly see any request from this client (I checked
"dump_ops_in_flight" every second). So, I guess they should come from the
cache. But why does it take so long? I have run ceph-fuse in debug mode
(--debug-client=20) but this of course results in a lot of output, and I'm not
sure what to look for.
Watching "mds_requests" on the client every second does not show any request.
I know the performance of ceph kernel client is (much) better than ceph-fuse,
but does this also apply to objects in cache?
Thanks for any hints.
Gr. Stefan
P.s. ceph-fuse luminous client 12.2.7 shows same result. the only active MDS server has 256 GB
cache and has hardly any load. So most inodes / dentries should be cached there also.
--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info@xxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
