you need to pay attention to "fiemap" option, "ceph --show-config | grep fiemap" There exists some tricks about fiemap, please search it at tracker.ceph.com On Fri, Jun 27, 2014 at 12:27 AM, huang jun <hjwsm1989@xxxxxxxxx> wrote: > hi,all > We migrate vmware EXSI vm from ceph cluster via NFS to local disk,the > image file size is about 250GB. we have 8 OSDs on 2 hosts. > But the migrate speed is pretty slow, we test it in 3 way: > 1) NFS + kclient + ceph > during the first 5 minutes, read speed is about 40MB/s, > but after that, down to 20MB/s, the whole migrate cost 3 hours. > Check the osd log, we find there are many read request at 4KB size, > and OSD return -2, which means those requests are for the hole in the > vm image file. > we also find that NFS send random read request to OSDs while checking > the client log, > > 2) NFS + ceph-fuse + ceph > avg read speed is about 40MB/s, the whole migrate cost 2 hours. > After check the osd log, the read request size is 128KB. > > 2.1) set the client readahead size like below: > [client] > client_readahead_min = 4194304 > client_readahead_max = 41943040 > > but is doesn't make big difference, the migration cost 1 hour 58 minutes. > > 2.2) we change the client code to always read ahead objects when we do > every read request. > unfortunately, the migration time does reduce much. > > 3) NFS + rbd + ceph > the read speed can reach 117MB/s (reading holes in sparse file) > and decrease to 40MB/s when reading the data parts of the vm image file. > > i have questions: > 1) is there any way to improve the through when reading sparse files? > 2) can i change the read size of nfs(may be asked on nfs mailing list)? > > 3) any other params to set on ceph fuse client to improve read speed? > > -- > thanks > huangjun > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
