Am 02.06.22 um 16:21 schrieb Felix Kuehling:
[SNIP]
In other words the free is not waiting for the unmap to complete,
but causes command submissions through the kernel to depend on
the unmap.
I guess I don't understand that dependency. The next command
submission obviously cannot use the memory that was unmapped. But
why does it need to synchronize with the unmap operation?
Because of the necessary TLB flush, only after that one is executed
we can be sure that nobody has access to the memory any more and
actually free it.
So freeing the memory has to wait for the TLB flush. Why does the
next command submission need to wait?
Because that's the one triggering the TLB flush. The issue is that
flushing the TLB while the VMID is in use is really unreliable on
most hardware generations.
It's been working well enough with ROCm. With user mode command
submission there is no way to block GPU work while a TLB flush is in
progress.
Yeah, but at least on Navi 1x that's so horrible broken that the SDMA
could write anywhere when we would try this.
User mode submissions are completely unrelated to that.
I mention user mode command submission because there is no way to
enforce the synchronization you describe here on a user mode
queue. So this approach is not very future proof.
With user mode queues you need to wait for the work on the queue to
finish anyway or otherwise you run into VM faults if you just unmap
or free the memory.
If the next command submission doesn't use the unmapped/freed
memory, why does it need to wait for the TLB flush?
Because it could potentially use it. When userspace lies to the
kernel and still accesses the mapping we would allow access to freed
up memory and create a major security problem.
I'm aware of the potential security problem. That's why I'm
recommending you don't actually free the memory until the TLB flush is
done. So a bogus access will either harmlessly access memory that's
not freed yet, or it will VM fault. It will never access memory that's
already freed and potentially allocated by someone else.
Yes, that's the idea. The question is just when we can do the TLB flush.
If it is using the unmapped/freed memory, that's a user mode bug.
But waiting for the TLB flush won't fix that. It will only turn a
likely VM fault into a certain VM fault.
Yeah, exactly that's the intention here.
The guarantee you need to give is, that the memory is not freed and
reused by anyone else until the TLB flush is done. This dependency
requires synchronization of the "free" operation with the TLB flush.
It does not require synchronization with any future command
submissions in the context that freed the memory.
See above, the future command submission is what triggers the TLB
flush because only then we can easily execute it without to much hassle.
That seems to be a limitation of your current command submission
model. User mode command submission will not be able to trigger a TLB
flush. Unmapping or freeing memory should be the trigger in that case.
That's how it works with KFD. That said, our TLB flushes aren't as
well pipelined (which could probably be improved), and your strategy
can probably batch more TLB flushes, so I see where you're coming from.
Well the mapping/unmapping IOCTL should certainly trigger the TLB
flushes for the user mode queues, but as I said this is completely
independent to this here.
The limitation is on the kernel CS IOCTL, not the VM IOCTL. So that is
completely unrelated to this.
For Vega and Navi 2x we could use async TLB flushes and on gfx6, gfx7
and gfx8 we could use double TLB flushes with grace time, but Navi 1x
is so horrible broken regarding this that I don't see how else we
could do that.
We're using heavy-weight TLB flushes on SOC15 GPUs. On Vega20 with
XGMI we need double flushes to be safe.
I'm raising my concerns because I don't think making user mode wait is
a good strategy long-term. And I believe this explicit sync and
explicit VM update should be designed with an eye for future user-mode
command submission models.
Yeah, but as already discussed with Daniel and Jason that will never
ever work correctly. IOCTLs can't depend on user mode queues in any way.
So user space can only block or rather call the map, unmap, free
functions at the right time.
If you need short-term workarounds for broken hardware, that's another
issue. But it would be good if that could be kept out of the API.
Well as I said that is completely unrelated to user mode queues. The
restriction is on the CS API, not the VM API.
Regards,
Christian.
Regards,
Felix
Regards,
Christian.
Regards,
Felix
The signal that TLB flush is completed comes from the MES in this
case.
Regards,
Christian.
Regards,
Felix
Regards,
Christian.
Regards,
Felix
3. All VM operations requested by userspace will still be
executed in order, e.g. we can't run unmap + map in parallel or
something like this.
Is that something you guys can live with? As far as I can see
it should give you the maximum freedom possible, but is still
doable.
Regards,
Christian.
