On Tue, Apr 16, 2013 at 8:47 AM, Elso Andras <elso.andras@xxxxxxxxx> wrote: > Hi, > > We are serving mp4 videos from cephfs, with lots of random read. > > Here is a graph: http://elbandi.net/ceph/ceph_bandwidth.png > The issue is: there are +50% incoming bandwidth (osd -> cephfs mount, blue > line) compared outgoing bandwidth (webserver -> internet, green area). the > labels are swaped by the switch. The CephFS client engages in prefetching by default; it's possible that your workload is such that the prefetching isn't helping (ie, the prefetched data is cycling through the cache before it's requested, and then getting re-read). That's just speculation, though. (You can fidget with these values by looking at the client_readahead_* config options.) > > on the osd, there is a raid5 with 512k full stripe size. > > Files layout on cephfs: > stripe_unit: 65536 > stripe_count: 1 > object_size: 524288 (== raid5 full stripe size, so one object live in one > raid "block"). These settings are a bit silly. I think what you've got there is logically equivalent to having the stripe_unit and object_size both set to 512KB, but I'm not certain. 512KB is also a bit small for an FS object; why did you change it from the defaults at all? With streaming media you're probably better off with larger than smaller chunks... -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |