On Tue, Jul 02, 2013 at 08:55:14PM -0400, Mathieu Desnoyers wrote: > * Mathieu Desnoyers (mathieu.desnoyers@xxxxxxxxxxxx) wrote: > > * Dave Chinner (david@xxxxxxxxxxxxx) wrote: > > > On Thu, Jun 20, 2013 at 08:20:16AM -0400, Mathieu Desnoyers wrote: > > > > * Rob van der Heij (rvdheij@xxxxxxxxx) wrote: > > > > > Wouldn't you batch the calls to drop the pages from cache rather than drop > > > > > one packet at a time? > > > > > > > > By default for kernel tracing, lttng's trace packets are 1MB, so I > > > > consider the call to fadvise to be already batched by applying it to 1MB > > > > packets rather than indivitual pages. Even there, it seems that the > > > > extra overhead added by the lru drain on each CPU is noticeable. > > > > > > > > Another reason for not batching this in larger chunks is to limit the > > > > impact of the tracer on the kernel page cache. LTTng limits itself to > > > > its own set of buffers, and use the page cache for what is absolutely > > > > needed to perform I/O, but no more. > > > > > > I think you are doing it wrong. This is a poster child case for > > > using Direct IO and completely avoiding the page cache altogether.... > > > > I just tried replacing my sync_file_range()+fadvise() calls and instead > > pass the O_DIRECT flag to open(). Unfortunately, I must be doing > > something very wrong, because I get only 1/3rd of the throughput, and > > the page cache fills up. Any idea why ? > > Since O_DIRECT does not seem to provide acceptable throughput, it may be > interesting to investigate other ways to lessen the latency impact of > the fadvise DONTNEED hint. > There are cases where O_DIRECT falls back to buffered IO which is why you might have found that page cache was still filling up. There are a few reasons why this can happen but I would guess the common cause is that the range of pages being written was in the page cache already and could not be invalidated for some reason. I'm guessing this is the common case for page cache filling even with O_DIRECT but would not bet money on it as it's not a problem I investigated before. > Given it is just a hint, we should be allowed to perform page > deactivation lazily. Is there any fundamental reason to wait for worker > threads on each CPU to complete their lru drain before returning from > fadvise() to user-space ? > Only to make sure they pages are actually dropped as requested. The reason the wait was introduced in the first place was that page cache was filling up even with the fadvise calls and causing disruption. In 3.11 disruption due to this sort of parallel IO should be reduced but making fadvise work properly is reasonable in itself. Was that patch I posted ever tested or did I manage to miss it? -- Mel Gorman SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe stable" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html