On 21/01/2022 04:24, Tony Lu wrote: > On Thu, Jan 20, 2022 at 03:50:26PM +0100, Karsten Graul wrote: >> On 20/01/2022 15:09, Tony Lu wrote: >>> When analyzed memory usage of SMC, we found that the size of struct >>> smc_link_group is 16048 bytes, which is too big for a busy machine to >>> allocate contiguous memory. Using kvzalloc instead that falls back to >>> vmalloc if there has not enough contiguous memory. >> >> I am wondering where the needed contiguous memory for the required RMB buffers should come from when >> you don't even get enough storage for the initial link group? > > Yes, this is what I want to talking about. The RMB buffers size inherits > from TCP, we cannot assume that RMB is always larger than 16k bytes, the > tcp_mem can be changed on the fly, and it can be tuned to very small for > saving memory. Also, If we freed existed link group or somewhere else, > we can allocate enough contiguous memory for the new link group. The lowest size for an RMB is 16kb, smaller inherited tcp sizes do not apply here. > >> The idea is that when the system is so low on contiguous memory then a link group creation should fail >> early, because most of the later buffer allocations will also fail then later. > > IMHO, it is not a "pre-checker" for allocating buffer, it is a reminder > for us to save contiguous memory, this is a precious resource, and a > possible way to do this. This patch is not the best approach to solve > this problem, but the simplest one. A possible approach to allocate > link array in link group with a pointer to another memory. Glad to hear > your advice. I am still not fully convinced of this change. It does not harm and the overhead of a vmalloc() is acceptable because a link group is not created so often. But since kvzmalloc() will first try to use normal kmalloc() and if that fails switch to the (more expensive) vmalloc() this will not _save_ any contiguous memory. And for the subsequent required allocations of at least one RMB we need another 16KB. Did this change had any measurable advantages in your tests?
