On Wed, Aug 9, 2017 at 6:42 PM, John Spray <jspray@xxxxxxxxxx> wrote:
> On Wed, Aug 9, 2017 at 5:32 PM, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote:
>> Hi all,
>>
>> During some ceph-fuse perf testing I've noticed that small buffered
>> write throughput is slowed down by the debug_client ldout logs.
>> At the default 0/5, the in mem log is filled with, e.g.:
>>
>> -9760> 2017-08-09 18:16:10.487511 7f848c1a5700 3 client.407554315
>> xattr_permission 0x7f84a52c8a00 = 0
>> -9759> 2017-08-09 18:16:10.487515 7f848c1a5700 3 client.407554315
>> _getxattr(10003284f73, "security.capability", 0) = -61
>> -9758> 2017-08-09 18:16:10.487525 7f848c1a5700 3 client.407554315
>> ll_forget 10003284f73 1
>> -9757> 2017-08-09 18:16:10.487542 7f848b9a4700 3 client.407554315
>> ll_write 0x7f84a5147680 10003284f73 71824~1
>> -9756> 2017-08-09 18:16:10.487566 7f848b9a4700 3 client.407554315
>> ll_write 0x7f84a5147680 71824~1 = 1
>> -9755> 2017-08-09 18:16:10.487599 7f848c9a6700 3 client.407554315
>> ll_getxattr 10003284f73.head security.capability size 0
>> -9754> 2017-08-09 18:16:10.487609 7f848c9a6700 3 client.407554315
>> xattr_permission 0x7f84a52c8a00 = 0
>> -9753> 2017-08-09 18:16:10.487612 7f848c9a6700 3 client.407554315
>> _getxattr(10003284f73, "security.capability", 0) = -61
>>
>> (so even tiny 1 byte buffered writes result in huge amounts of logging work).
>>
>> In a simple test [1], I found that 100000 one byte writes takes
>> ~10-11s with the default client log level, and ~5 to 6s with
>> debug_client=0/2.
>
> Wow, I had no idea the overhead was so high!
>
>> I would therefore propose either increasing the ldout level for these
>> ll_* client ops to 6, or decreasing the default debug_client level to
>> 0/2.
>
> Hmm. I'm not sure what would be left in the memory log at 0/2 though?
> For example in the case of a crash would we even have the message
> from the start of the operation that crashed?
AFAICT only mount/umount get logged at level 2.
-- Dan
>
> John
>
>>
>> Any opinions?
>>
>> Cheers, Dan
>>
>> [1]
>> for i in xrange(100000):
>> f.write('x')
>> f.flush()
>> f.close()
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
