Am 10.06.22 um 16:16 schrieb Michal Hocko:
[...]
The primary question is whether it actually helps much or what kind of
scenarios it can help with and whether we can actually do better for
those.
Well, it does help massively with a standard Linux desktop and GPU workloads
(e.g. games).
See what currently happens is that when games allocate for example textures
the memory for that is not accounted against that game. Instead it's usually
the display server (X or Wayland) which most of the shared resources
accounts to because it needs to compose a desktop from it and usually also
mmaps it for fallback CPU operations.
Let me try to understand some more. So the game (or the entity to be
responsible for the resource) doesn't really allocate the memory but it
relies on somebody else (from memcg perspective living in a different
resource domain - i.e. a different memcg) to do that on its behalf.
Correct? If that is the case then that is certainly not fitting into the
memcg model then.
More or less: yes, that is one possible use case. But we could leave
that one out since it is not the primary use case.
What happens more is that 99% of the resources are only allocated per
process, but around 1% are shared with somebody else.
But see two comments below of a better description of the problem I'm
facing.
I am not really sure there is any reasonable model where you cannot
really tell who is responsible for the resource.
Well it would be fine with me to leave out those 1% of resources shared
with different memcgs.
What breaks my neck are those 99% which are allocated by a game and
could potentially be shared but are most of the time not.
So what happens when a games over allocates texture resources is that your
whole desktop restarts because the compositor is killed. This obviously also
kills the game, but it would be much nice if we would be more selective
here.
For hardware rendering DMA-buf and GPU drivers are used, but for the
software fallback shmem files is what is used under the hood as far as I
know. And the underlying problem is the same for both.
For shmem files the end user of the buffer can preallocate and so own
the buffer and be accounted for it.
The problem is just that it can easily happen that one process is
allocating the resource and a different one freeing it.
So just imaging the following example: Process opens X window, get
reference to the handle of the buffer backing this window for drawing,
tells X to close the window again and then a bit later closes the buffer
handle.
In this example the X server would be charged allocating the buffer and
the client (which is most likely in a different memcg group) is charged
freeing it.
I could of course add something to struct page to track which memcg (or
process) it was charged against, but extending struct page is most
likely a no-go.
Alternative I could try to track the "owner" of a buffer (e.g. a shmem
file), but then it can happen that one processes creates the object and
another one is writing to it and actually allocating the memory.
Also do not forget that shared file memory is not the only thing
to care about. What about the kernel memory used on behalf of processes?
Yeah, I'm aware of that as well. But at least inside the GPU drivers we try
to keep that in a reasonable ratio.
Just consider the above mentioned memcg driven model. It doesn't really
require to chase specific files and do some arbitrary math to share the
responsibility. It has a clear accounting and responsibility model.
Ok, how does that work then?
The memory is accounted to whoever faults that memory in or to the
allocating context if that is a kernel memory (in most situations).
That's what I had in mind as well. Problem with this approach is that
file descriptors are currently not informed that they are shared between
processes.
So to make this work we would need something like attach/detach to
process in struct file_operations.
And as I noted, this happens rather often. For example a game which
renders 120 frames per second needs to transfer 120 buffers per second
between client and X.
So this is not something which could take a lot of time and the file
descriptor tracking structures in the Linux kernel are not made for this
either.
I think for now I will try something like this specific for DRM drivers.
That doesn't solve the shmem file problem, but it at least gives me
something at hand for the accelerated Linux desktop case.
Regards,
Christian.
