I don't think fork() would work with userspace where all buffers are shared. It certainly doesn't work now. The driver needs to be notified that a buffer or texture is shared to ensure data coherency between processes, and the driver must execute decompression and other render passes when a buffer or texture is being shared for the first time. Those aren't called when fork() is called.
Marek
On Mon, Jan 17, 2022 at 9:34 AM Felix Kuehling <felix.kuehling@xxxxxxx> wrote:
Am 2022-01-17 um 9:21 a.m. schrieb Christian König:
> Am 17.01.22 um 15:17 schrieb Felix Kuehling:
>> Am 2022-01-17 um 6:44 a.m. schrieb Christian König:
>>> Am 14.01.22 um 18:40 schrieb Felix Kuehling:
>>>> Am 2022-01-14 um 12:26 p.m. schrieb Christian König:
>>>>> Am 14.01.22 um 17:44 schrieb Daniel Vetter:
>>>>>> Top post because I tried to catch up on the entire discussion here.
>>>>>>
>>>>>> So fundamentally I'm not opposed to just close this fork() hole
>>>>>> once and
>>>>>> for all. The thing that worries me from a upstream/platform pov is
>>>>>> really
>>>>>> only if we don't do it consistently across all drivers.
>>>>>>
>>>>>> So maybe as an idea:
>>>>>> - Do the original patch, but not just for ttm but all gem rendernode
>>>>>> drivers at least (or maybe even all gem drivers, no idea), with
>>>>>> the
>>>>>> below discussion cleaned up as justification.
>>>>> I know of at least one use case which this will break.
>>>>>
>>>>> A couple of years back we had a discussion on the Mesa mailing list
>>>>> because (IIRC) Marek introduced a background thread to push command
>>>>> submissions to the kernel.
>>>>>
>>>>> That broke because some compositor used to initialize OpenGL and then
>>>>> do a fork(). This indeed worked previously (no GPUVM at that time),
>>>>> but with the addition of the backround thread obviously broke.
>>>>>
>>>>> The conclusion back then was that the compositor is broken and needs
>>>>> fixing, but it still essentially means that there could be people out
>>>>> there with really old userspace where this setting would just break
>>>>> the desktop.
>>>>>
>>>>> I'm not really against that change either, but at least in theory we
>>>>> could make fork() work perfectly fine even with VMs and background
>>>>> threads.
>>>> You may regret this if you ever try to build a shared virtual address
>>>> space between GPU and CPU. Then you have two processes (parent and
>>>> child) sharing the same render context and GPU VM address space.
>>>> But the
>>>> CPU address spaces are different. You can't maintain consistent shared
>>>> virtual address spaces for both processes when the GPU address
>>>> space is
>>>> shared between them.
>>> That's actually not much of a problem.
>>>
>>> All you need to do is to use pthread_atfork() and do the appropriate
>>> action in parent/child to clean up your context:
>>> https://man7.org/linux/man-pages/man3/pthread_atfork.3.html
>> Thunk already does that. However, it's not foolproof. pthread_atfork
>> hanlders aren't called when the process is forked with a clone call.
>
> Yeah, but that's perfectly intentional. clone() is usually used to
> create threads.
Clone can be used to create new processes. Maybe not the common use today.
>
>>> The rest is just to make sure that all shared and all private data are
>>> kept separate all the time. Sharing virtual memory is already done for
>>> decades this way, it's just that nobody ever did it with a statefull
>>> device like GPUs.
>> My concern is not with sharing or not sharing data. It's with sharing
>> the address space itself. If you share the render node, you share GPU
>> virtual address space. However CPU address space is not shared between
>> parent and child. That's a fundamental mismatch between the CPU world
>> and current GPU driver implementation.
>
> Correct, but even that is easily solvable. As I said before you can
> hang this state on a VMA and let it be cloned together with the CPU
> address space.
I'm not following. The address space I'm talking about is struct
amdgpu_vm. It's associated with the render node file descriptor.
Inheriting and using that file descriptor in the child inherits the
amdgpu_vm. I don't see how you can hang that state on any one VMA.
To be consistent with the CPU, you'd need to clone the GPU address space
(struct amdgpu_vm) in the child process. That means you need a new
render node file descriptor that imports all the BOs from the parent
address space. It's a bunch of extra work to fork a process, that you're
proposing to immediately undo with an atfork handler. So I really don't
see the point.
Regards,
Felix
>
> Since VMAs are informed about their cloning (in opposite to file
> descriptors) it's trivial to even just clone kernel data on first access.
>
> Regards,
> Christian.
>
>>
>> Regards,
>> Felix
>>
>>
>>> Regards,
>>> Christian.
>>>
>>>> Regards,
>>>> Felix
>>>>
>
