On Fri, Feb 15, 2013 at 10:46 AM, Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx> wrote: > On 02/15/2013 12:00 PM, Gregory Farnum wrote: > >> That's a lot more of a slowdown than I'd expect to see, but there > isn't much hint about where the slow-down is actually happening. I don't > recall precisely what's happening in the kernel client when you do a > bunch of mmaps — Sage? Does it require a network round-trip when you do > that or will it cache and pre-read appropriately? > >> >> But more generally you'll need to describe your workload pattern a >> bit more, and do some benchmarks at lower layers of the stack to see >> what kind of bandwidth is available to begin with. Look at the rados >> bench stuff to get some data on disk and then do a bunch of >> simultaneous read benchmarks to see how fast your OSDs can serve data >> up under a fairly reasonable streaming workload; check out >> smalliobenchrados to do some IO that more closely mimics your >> application, etc. > > This *was* the benchmark. Each host has the data on local hard drive, > that is the kind of bandwidth that's available to begin with. All other > things are equal, > - baseline test mmaps from /dev/sdb1 mounted as ext4, > - ceph test mmaps from /dev/sdb1 mounted as cephfs -> (hopefully > local) osd. > The slowdown is an order of magnitude and change. Right, but that's an inexplicably large slowdown, so I suggest you decompose the test a bit in order to see where it's being introduced. :) > As for workload pattern: > > Last I looked at mmap (been a while) it'd place data in shared memory > and COW it if needed. Since the application is now writing, that's never > needed -- so cephfs's mounted read-only. There should not be any caches > to invalidate or resync. This is my initial guess — are you sure the files are actually being opened read-only? If they were being opened with write permissions as well then the cluster would go into synchronous mode (in which all IO is synchronously mediated by the MDS) even if nobody is actually doing writes. -Greg _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |