On Thu, Oct 20, 2022 at 03:03:56PM +0100, David Howells wrote: > > What block file systems do is to take the pages from the iter and some flags > > on what is pinned. We can generalize this to store all extra state in a > > flags word, or byte the bullet and allow cloning of the iter in one form or > > another. > > Yeah, I know. A list of pages is not an ideal solution. It can only handle > contiguous runs of pages, possibly with a partial page at either end. A bvec > iterator would be of more use as it can handle a series of partial pages. > > Note also that I would need to turn the pages *back* into an iterator in order > to commune with sendmsg() in the nether reaches of some network filesystems. Yes. So I think the right thing here is to make sure we can send the iter through the whole stack without a convesion. > It would be nice to be able to pass an iterator to the crypto layer. I'm not > sure what the crypto people think of that. Let's ask them.. > On the other hand, if you think the RDMA API should be taking scatterlists > rather than sge lists, that would be fine. Even better if I can just pass an > iterator in directly - though neither scatterlist nor iterator has a place to > put the RDMA local_dma_key - though I wonder if that's actually necessary for > each sge element, or whether it could be handed through as part of the request > as a hole. Well, in the long run it should not take scatterlists either, as they are a bad data structure. But what should happen in the long run is that the DMA mapping is only done in the hardware drivers, not the ULPs, which is a really nasty layering violation. This requires the strange ib_dma_* stubs to disable DMA mapping for the software drivers, and it also does complete unneeded DMA mappings for sends that are inline in the SQE as supported by some Mellanox / Nvidia hardware. > That's fine in principle. However, I have some extraction code that can > convert an iterator to another iterator, an sglist or an rdma sge list, using > a common core of code to do all three. So I think the iterator to iterator is a really bad idea and we should not have it at all. It just works around the issue about not being able to easily keeping state after an iter based get_user_pages, but that is beeing addressed at the moment. The iter to ib_sge/scatterlist are very much RDMA specific at the moment, so I guess that might be a good place to keep them. In fact I suspect the scatterlist conversion should not be a public API at all, but hidden in rw.c and only be used internally for the DMA mapping.
