Hi all,
During some ceph-fuse perf testing I've noticed that small buffered
write throughput is slowed down by the debug_client ldout logs.
At the default 0/5, the in mem log is filled with, e.g.:
-9760> 2017-08-09 18:16:10.487511 7f848c1a5700 3 client.407554315
xattr_permission 0x7f84a52c8a00 = 0
-9759> 2017-08-09 18:16:10.487515 7f848c1a5700 3 client.407554315
_getxattr(10003284f73, "security.capability", 0) = -61
-9758> 2017-08-09 18:16:10.487525 7f848c1a5700 3 client.407554315
ll_forget 10003284f73 1
-9757> 2017-08-09 18:16:10.487542 7f848b9a4700 3 client.407554315
ll_write 0x7f84a5147680 10003284f73 71824~1
-9756> 2017-08-09 18:16:10.487566 7f848b9a4700 3 client.407554315
ll_write 0x7f84a5147680 71824~1 = 1
-9755> 2017-08-09 18:16:10.487599 7f848c9a6700 3 client.407554315
ll_getxattr 10003284f73.head security.capability size 0
-9754> 2017-08-09 18:16:10.487609 7f848c9a6700 3 client.407554315
xattr_permission 0x7f84a52c8a00 = 0
-9753> 2017-08-09 18:16:10.487612 7f848c9a6700 3 client.407554315
_getxattr(10003284f73, "security.capability", 0) = -61
(so even tiny 1 byte buffered writes result in huge amounts of logging work).
In a simple test [1], I found that 100000 one byte writes takes
~10-11s with the default client log level, and ~5 to 6s with
debug_client=0/2.
I would therefore propose either increasing the ldout level for these
ll_* client ops to 6, or decreasing the default debug_client level to
0/2.
Any opinions?
Cheers, Dan
[1]
for i in xrange(100000):
f.write('x')
f.flush()
f.close()
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
