> On Mar 23, 2021, at 4:13 PM, Jason Gunthorpe <jgg@xxxxxxxx> wrote: > > On Tue, Mar 23, 2021 at 12:41:51PM -0700, Aruna Ramakrishna wrote: >> There is a far greater possibility of an order-8 allocation failing, >> esp. with the addition of __GFP_NORETRY , and the code would have to >> fall back to a lower order allocation more often than not (esp. on a >> long running system). Unless the performance gains from using order-8 >> pages is significant (and it does not seem that way to me), we can just >> skip this step and directly go to the lower order allocation. > > Do not send HTML mails. I apologize; I’ve fixed the setting now. > > Do you have benchmarks that show the performance of the high order > pages is not relavent? I'm a bit surprised to hear that > I guess my point was more to the effect that an order-8 alloc will fail more often than not, in this flow. For instance, when we were debugging the latency spikes here, this was the typical buddyinfo output on that system: Node 0, zone DMA 0 1 1 2 3 0 1 0 1 1 3 Node 0, zone DMA32 7 7 7 6 10 2 6 7 6 2 306 Node 0, zone Normal 3390 51354 17574 6556 1586 26 2 1 0 0 0 Node 1, zone Normal 11519 23315 23306 9738 73 2 0 1 0 0 0 I think this level of fragmentation is pretty normal on long running systems. Here, in the reg_mr flow, the first try (order-8) alloc will probably fail 9 times out of 10 (esp. after the addition of GFP_NORETRY flag), and then as fallback, the code tries to allocate a lower order, and if that too fails, it allocates a page. I think it makes sense to just avoid trying an order-8 alloc here. Thanks, Aruna > This code really needs some attention to use a proper > scatter/gather. I understand the chip can do it, just some of the > software layers need to be stripped away so it can form the right SGL > in the HW. > > Jason
