The motivation for the application (fio) to allocate physical pages is really about performance. If the IO size is more extreme, say 1MB, then descriptors of 512B each vs 1MB each means potentially many DMAs. All of those DMAs can really change performance. The iomem_align seems workable. My understanding now is that in the case of: bssplit=512B/3.5KB/4KB/8KB iomem_align=8k then all bs buffers are 8KB aligned, and there is a best effort (up to kernel) to have the 8KB buffer have two physically contiguous pages (assuming 4KB physical page size on host). In the case above, the 512B/3.5KB/4KB bs only use (DMA) one page of the 8KB iomem_align. In the case of the extreme 1MB IO size, we can use bssplit and iomem_align to cause 512B DMAs. However, we have probably low probability that we could cause a 1MB DMA. I'm guessing the odds of the kernel allocating a 1MB region of physically contiguous pages (4KB each) are small, especially for iodepth times. But, I think that's more of the extreme usage. Thanks for the help. Regards, Jeff -----Original Message----- From: fio-owner@xxxxxxxxxxxxxxx [mailto:fio-owner@xxxxxxxxxxxxxxx] On Behalf Of Jens Axboe Sent: Wednesday, September 21, 2016 4:26 PM To: Jeff Furlong <jeff.furlong@xxxxxxxx>; fio@xxxxxxxxxxxxxxx Subject: Re: fio SGL support On 09/21/2016 05:14 PM, Jeff Furlong wrote: > I believe kernel 4.8 offers an NVMe over fabrics driver that supports > SGL descriptors. This is just NVMe being a bit different from everyone else, basically all storage drivers use SG lists. This is completely detached from any application level interface, it's just an artifact of how physical memory is mapped for DMA and that information is passed to the driver. > In the case of a 16KB IO that is described as 512B, 3.5KB, 4KB, and > 8KB, there would be separate data transfers to the device under test > (such as NVMe over fabrics device). In the case of a 16KB IO > allocated as contiguous pages, one large data transfer could be > sufficient. Is it possible to force one way or the other with fio > (assuming the driver has support)? We could simply use bssplit as > 512B/3.5KB/4KB/8KB, but then the 8KB might actually be on > noncontiguous pages, in which case it could really end up as > 512B/3.5KB/4KB/4KB/4KB. I'm still confused, honestly I don't see the point of this at all. Fio makes no attempt to allocate (nor can it) physical pages, so we can't control how this memory ends up being laid out. The best we can do is align buffers to the same offset as their length, assuming we then get pages mapped that are physical contiguous. This will reduce the required number of SG elements on the driver side. 'iomem_align' will align the IO buffers to the given alignment. If you do: bssplit=512B/3.5KB/4KB/8KB iomem_align=8k then ANY IO buffer will be aligned to 8k, and 8k in length, even if we only do shorter transfers to it. I hope that explains it. -- Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
