Hello, On Sun, 27 Mar 2016 02:59:30 +0300 Dzianis Kahanovich wrote: > New problem (unsure, but probably not observed in Hammer, but sure in > Infernalis): copying large (tens g) files into kernel cephfs (from > outside of cluster, iron - non-VM, preempt kernel) - make slow requests > on some of OSDs (repeated range) - mostly 3 Gbps channels (slow). > > All OSDs default threads numbers. Scheduler=noop. size=3 min_size=2 > > No same problem with fuse. > > Looks like broken or unbalanced congestion mechanism or I don't know how > to moderate it. write_congestion_kb trying low (=1) - nothing > interesting. > I think cause and effect are not quite what you think they are. Firstly let me state that I have no experience with CephFS at all, but what you're seeing isn't likely related to it all. Next lets establish some parameters. You're testing kernel and fuse from the same machine, right? What is the write speed (throughput) when doing this with fuse compared to the speed when doing this via the kernel module? What is the top speed of your cluster when doing a "rados -p <yourpoolname> bench 60 write -t 32" from your test machine? Does this result in slow requests as well? What I think is happening is that you're simply at the limits of your current cluster and that fuse is slower, thus not exposing this. The kernel module is likely fast AND also will use pagecache, thus creating very large writes (how much memory does your test machine have) when it gets flushed. Christian -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |