On Wed, Oct 10, 2018 at 8:55 PM Spencer Melnick <smelnick97@xxxxxxxxx> wrote: > > Hi Jason, > > Thank you for the reply. Based on testing several different options, I > believe that the modification is occurring due to the page cache. > > Running our test program initially produces read requests at the > ImageCache level. > A second run of the test program produces no read requests in > ImageCache; however, by flushing the page cache using > > sync; echo 3 > /proc/sys/vm/drop_caches > > and re-running the test program we again see read requests in ImageCache. > Checking the scheduler for our virtual block device, nbd0 via > > cat /sys/block/nbd0/queue/scheduler > > produces [none]. > Likewise, > > cat /sys/block/nbd0/queue/read_ahead_kb > > produces 0. > Because of this I am assuming that the sole modification of the > requests is due to the page cache. > To fix this, we've attempted to mount the virtual block device both > with and without the -osync flag, and modifying the test program to > read a file using the O_DIRECT flag, but neither seems to have any > effect. > > I apologize, as this is somewhat unrelated to Ceph, but do you know of > any other methods to disable the page cache, if possible solely for > the virtual block device? I would have expected that if you open the "/dev/nbdX" block device w/ the O_DIRECT flag, all subsequent reads and writes would bypass the page cache. > > Thanks, > ----------- > Spencer Melnick > > On Sat, Oct 6, 2018 at 9:04 AM Jason Dillaman <jdillama@xxxxxxxxxx> wrote: > > > > On Fri, Oct 5, 2018 at 7:00 PM Spencer Melnick <smelnick97@xxxxxxxxx> wrote: > > > > > > Hello everyone, > > > > > > I am currently working on implementing a custom caching algorithm in > > > Ceph's librbd using the ImageCache interface, but I have found that > > > not all of the requests we expect are being processed by the > > > ImageCache::aio_read() function. > > > > > > Specifically, when we create a rbd image, map it to an nbd device, and > > > mount this device on a Linux filesystem, we run a series of file read > > > requests to test the algorithm. However, we find that the number of > > > requests dispatched to ImageCache::aio_read() is not the same as the > > > number of requests made by our test program. > > > > > > Normally this would not be a problem; however, our algorithm relies on > > > having a complete data stream to perform some predictive caching. Is > > > there some kind of caching happening at a higher level of Ceph that > > > must be turned off first? > > > > If you are using rbd-nbd (via the nbd block device), then I suspect > > you are just seeing the Linux kernel IO scheduler and/or page cache > > altering your requests. From the point of view of rbd-nbd, any read > > request received from the kernel will be passed unmodified to > > "ImageCache::aio_read". > > > > > Thanks, > > > ----------- > > > Spencer Melnick > > > > > > > > -- > > Jason -- Jason
