Hi Jason, Thank you for the reply. Based on testing several different options, I believe that the modification is occurring due to the page cache. Running our test program initially produces read requests at the ImageCache level. A second run of the test program produces no read requests in ImageCache; however, by flushing the page cache using sync; echo 3 > /proc/sys/vm/drop_caches and re-running the test program we again see read requests in ImageCache. Checking the scheduler for our virtual block device, nbd0 via cat /sys/block/nbd0/queue/scheduler produces [none]. Likewise, cat /sys/block/nbd0/queue/read_ahead_kb produces 0. Because of this I am assuming that the sole modification of the requests is due to the page cache. To fix this, we've attempted to mount the virtual block device both with and without the -osync flag, and modifying the test program to read a file using the O_DIRECT flag, but neither seems to have any effect. I apologize, as this is somewhat unrelated to Ceph, but do you know of any other methods to disable the page cache, if possible solely for the virtual block device? Thanks, ----------- Spencer Melnick On Sat, Oct 6, 2018 at 9:04 AM Jason Dillaman <jdillama@xxxxxxxxxx> wrote: > > On Fri, Oct 5, 2018 at 7:00 PM Spencer Melnick <smelnick97@xxxxxxxxx> wrote: > > > > Hello everyone, > > > > I am currently working on implementing a custom caching algorithm in > > Ceph's librbd using the ImageCache interface, but I have found that > > not all of the requests we expect are being processed by the > > ImageCache::aio_read() function. > > > > Specifically, when we create a rbd image, map it to an nbd device, and > > mount this device on a Linux filesystem, we run a series of file read > > requests to test the algorithm. However, we find that the number of > > requests dispatched to ImageCache::aio_read() is not the same as the > > number of requests made by our test program. > > > > Normally this would not be a problem; however, our algorithm relies on > > having a complete data stream to perform some predictive caching. Is > > there some kind of caching happening at a higher level of Ceph that > > must be turned off first? > > If you are using rbd-nbd (via the nbd block device), then I suspect > you are just seeing the Linux kernel IO scheduler and/or page cache > altering your requests. From the point of view of rbd-nbd, any read > request received from the kernel will be passed unmodified to > "ImageCache::aio_read". > > > Thanks, > > ----------- > > Spencer Melnick > > > > -- > Jason
