|
[AMD Official Use Only - Internal Distribution Only] From: Grodzovsky, Andrey <Andrey.Grodzovsky@xxxxxxx>
Thanks Ma, this was very helpful as I am sill not able to setup XGMI hive with latest FW and VBIOS. I traced the workqueue subsystem (full log attached). Specifically here is the life cycle of our 2 work items executing amdgpu_device_xgmi_reset_func bellow [Le]: Thanks Andrey for the deep debug. Your feedback gave me a more profound understanding on this case. My comments split as below. You were right to note they both run on came CPU (32) but they are executed by different threads. Also as you see by workqueue_execute_start/end timestamps they actually ran in parallel and not one after another even while being
assigned to the same CPU and that because of thread preemption (there is at least psp_v11_0_mode1_reset->msleep(500)) which yields the CPU and hence allows the second work to run + I am sure that on preemptive kernel one reset work would be preempted at some
point anyway and let the other run. [Le]: Yes, from the trace log, the xgmi_reset_func items are assigned to different work threads bound to one same CPU. And you are right that cpu preemption will happen when msleep called which yield the CPU to
allow second work to run. That’s a great founding😊. But it’s not a
real parallel run to me because second work can only preempt to run when first work go to sleep. I made an experiment here to change this unique msleep to udelay, then second work item will run after first item finished in a serial execuation. Now you had issues with BACO reset while the test I ran on your system is mode1 reset and so I assumed that maybe BACO has some non preempt-able busy wait which doesn't give a chance to second work item's thread to run on that CPU
before the first finished - but from looking in the code I see smu_v11_0_baco_enter->msleep(10) so even in that case the first reset work item was supposed to yield CPU after BACO ENTER sent to SMU and let the other reset work do the same to the second card
and so i don't see how even in this case there is a serial execution ? [Le]: VG20 uses old powerplay framework (smu_v11_0_baco_enter->msleep(10) in swSMU framework), so no msleep and no CPU preemption.
BACO reset has Enter/Exit 2 phases. We expect all the XGMI nodes enter BACO simultaneously instead of one after one as a serial execution, then exit BACO simultaneously. P.S How you solution solves the case where the XGMI hive is bigger then number of CPUs on the system ? Assuming that what you say is correct and there is a serial execution when on the same CPU, if they hive is bigger then number
of CPUs you will eventually get back to sending reset work to a CPU already executing BACO ENTER (or EXIT) for another device and will get the serialization problem anyway.
[Le]: Yeah, I also considered the situation that XGMI hive bigger than CPU NR. I think it’s an extreme situation and should not exist. However, assuming it exists, many work items scatter in several CPUs will
be executed faster than bound to one same CPU, isn’t it ? cat-3002 [032] d... 33153.791829: workqueue_queue_work: work struct=00000000e43c1ebb function=amdgpu_device_xgmi_reset_func [amdgpu] workqueue=0000000080331d91 req_cpu=8192 cpu=32 Andrey On 12/3/19 5:06 AM, Ma, Le wrote:
|
_______________________________________________ amd-gfx mailing list amd-gfx@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/amd-gfx
