Re: About kexec issues in AWS nitro instances (RH bz 1758323)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello Guilherme,

On Fri, Mar 20, 2020 at 9:10 PM Guilherme G. Piccoli
<gpiccoli@xxxxxxxxxxxxx> wrote:

Thanks for writing again. I was caught up in trying several other
suggestions/code-snippets to further debug this.
I tried several combinations - turning iommu off, turning off swiotlb
in the kexec kernel and testing various combinations with
retain_initrd added to the kexec kernel's bootargs.

But nothing seems to fix the nested repetitive kexec reboot attempts
on the aws t3 machines I have. It just becomes better on few instances
(i.e. the kexec reboots would survive around 10 nested repetitive
attempts), while on the other(s) the failure can be seen quite
frequently (approx ~3 kexec reboot attempts).

> Well, it seems we have some good results with this patch [0] - the idea
> behind the issue is that ena network driver has no PCI shutdown()
> handler, which would be called to gently quiesce the device before the
> kexec. The PCI stack in this case clears the master bit of the device
> configuration space, effectively stopping all the DMA transactions. But
> then, when the system boots the kexec'ed kernel, the network device
> firmware may send a memory write regarding that stopped DMA transaction
> (that is now invalid), corrupting some random kernel memory area.
>
> I've ran 1000 kexecs tests with mainline (5.6-rc5) + this patch and no
> failures were observed. Also, I'm running a test with Ubuntu 5.3 kernel
> + this patch and achieved > 450 runs now, with no failures (test is
> ongoing).
>
> I've tried to dump the initrd content (could be useful now to identify
> the corruption signature, maybe matching some ena admin queue periodic
> task) but I had trouble collecting the dmesg in case of failure. It gets
> huge and requires a big ramoops allocation, which unfortunately prevents
> the issue from happening (I guess the corruption ends-up happening in
> the ramoops reserved area, not initrd area anymore).

This is a really good debug and resulting patch.
I ran almost ~60 kexec repetitive attempts last night and also
repeated the same today morning and
the issue seems to get fixed for me with upstream kernel 5.6.0-rc6+
with this patch.

I am leaving a test running with RHEL kernel + this patch overnight
and will have more updates to share by tomorrow morning.

> Bhupesh, I've noticed that suddenly the Red Hat bugzilla got private -

Oops. I will check.

> is it okay to add me in CC list so I can see it?

Sure. I tried doing it, but seems Bugzilla is not happy as it keeps
complaining that you are not registered on BZ,
I will try to find out internally how to get around the issue.

> Thanks for all the collaboration, I hope the issue was figured and solved!

Sure. Thanks a lot for your inputs and trying the suggestions I posted
on the Bugzilla ticket.
I will soon share an update with RHEL/Fedora kernel kexec tests with
this patch applied and also reply with a Tested-by for the upstream
patch in the relevant thread.

Thanks,
Bhupesh


_______________________________________________
kexec mailing list
kexec@xxxxxxxxxxxxxxxxxxx
http://lists.infradead.org/mailman/listinfo/kexec



[Index of Archives]     [LM Sensors]     [Linux Sound]     [ALSA Users]     [ALSA Devel]     [Linux Audio Users]     [Linux Media]     [Kernel]     [Gimp]     [Yosemite News]     [Linux Media]

  Powered by Linux