On Thu, Mar 14, 2024 at 03:34:03PM -0400, Pasha Tatashin wrote: > On Thu, Mar 14, 2024 at 3:29 PM Kent Overstreet > <kent.overstreet@xxxxxxxxx> wrote: > > > > On Thu, Mar 14, 2024 at 03:23:08PM -0400, Pasha Tatashin wrote: > > > > > > > > > > My point is that what matters is total memory use, not just memory used in > > > > > the kernel. Amdahl's law. > > > > > > > > If userspace is running a few processes with many threads and the > > > > userspace stacks are small, kernel stacks could end up dominating. > > > > > > > > I'd like to see some numbers though. > > > > > > The unused kernel stack pages occupy petabytes of memory across the fleet [1]. > > > > Raw number doesn't mean much here (I know how many machines Google has, > > of course it's going to be petabytes ;), percentage of system memory > > would be better. > > > > What I'd _really_ like to see is raw output from memory allocation > > profiling, so we can see how much memory is going to kernel stacks vs. > > other kernel allocations. > > I've heard there is memory profiling working that can help with that... I heard you've tried it out, too :) > While I do not have the data you are asking for, the other kernel > allocations might be useful, but this particular project is targeted > to help with reducing overhead where the memory is not used, or used > in very extreme rare cases. Well, do you think you could gather it? We shouldn't be blindly applying performance optimizations; we need to know where to focus our efforts. e.g. on my laptop I've currently got 356 processes for < 6M of kernel stack out of 32G total ram, so clearly this isn't much use to me. If the ratio is similar on your servers - nah, don't want it. I expect the ratio is not similar and you are burning proportially more memory on kernel stacks, but we still need to gather the data and do the math :) > > > Number of kernel threads vs. number of user threads would also be good > > to know - I've been seeing ps output lately where we've got a lot more > > workqueue workers than we should, perhaps that's something that could be > > addressed. > > Yes, doing other optimizations make sense, reducing the total number > kernel threads if possible might help as well. I will look into this > as well to see how many user threads vs kernel threads we have. Great, that will help too.
