> > On Tue, Jan 21, 2025 at 3:34 AM Mateusz Guzik <mjguzik@xxxxxxxxx> wrote: > Whether things work out or not I definitely encourage you to keep digging. > > Worst case scenario you had some fun, best case you made pipes on Linux faster. Thanks for your encouragement! > > Now to the point: in contrast to a regular fifo, the area backing the > total capacity of the pipe is not necessarily fully allocated -- it is > being backed by page-sized buffers allocated and freed as needed. > > I think the default capacity is 64K. Only ever using what's needed > (rounded up to a multiple of page size) saves memory for most pipes > and I presume this property will have to stay. > > Sample problem this poses for the lockless approach: after finishing > up a buffer, the reader will need to free it, but this needs to be > synchronized against a writer which may just happen to have started > writing there. > This is one of the points I was curious about. Is this on-demand allocation and freeing of pages necessary? What are some reasons that the full pipe capacity shouldn't be allocated for the lifetime of a pipe? Some of my results indicate that just this on-demand page management contributes something like ~25% overhead when continuously writing and reading multi-page buffers. But I'm not sure if there are some memory management concerns that are deemed more important than the cost of this overhead.
