Re: "default" watchdog device - ?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


On 29/03/2022 20:25, Nir Soffer wrote:
On Wed, Mar 16, 2022 at 1:55 PM lejeczek <peljasz@xxxxxxxxxxx> wrote:

On 15/03/2022 11:21, Daniel P. Berrangé wrote:
On Tue, Mar 15, 2022 at 10:39:50AM +0000, lejeczek wrote:
Hi guys.

Without explicitly, manually using watchdog device for a VM, the VM (centOS
8 Stream 4.18.0-365.el8.x86_64) shows '/dev/watchdog' exists.
To double check - 'dumpxml' does not show any such device - what kind of a
'watchdog' that is?
The kernel can always provide a pure software watchdog IIRC. It can be
useful if a userspace app wants a watchdog. The limitation is that it
relies on the kernel remaining functional, as there's no hardware
backing it up.

On a related note - with 'i6300esb' watchdog which I tested
and I believe is working.
I get often in my VMs from 'dmesg':
watchdog: BUG: soft lockup - CPU#0 stuck for xxxs! [swapper/0:0]
rcu: INFO: rcu_sched self-detected stall on CPU
This above is from Ubuntu and CentOS alike and when this
happens, console via VNC responds to until first 'enter'
then is non-resposive.
This happens after VM(s) was migrated between hosts, but
I do not see what I expected from 'watchdog' - there is no
action whatsoever, which should be 'reset'. VM remains in
such 'frozen' state forever.

any & all shared thoughts much appreciated.
You need to run some userspace tool that will open the watchdog
device, and pet it periodically, telling the kernel that userspace is alive.

If this tool will stop petting the watchdog, maybe because of a soft lockup
or other trouble, the watchdog device will reset the VM.

watchdog(8) may be the tool you need.

See also


I do not think that 'i6300esb' watchog works under those soft-lockups, whether it's qemu or OS end I cannot say.
    <watchdog model='i6300esb' action='reset'/>
in dom xml OS sees:
-> $ llr /dev/watchdog*
crw-------. 1 root root  10, 130 Apr  5 16:59 /dev/watchdog
crw-------. 1 root root 248,   0 Apr  5 16:59 /dev/watchdog0
crw-------. 1 root root 248,   1 Apr  5 16:59 /dev/watchdog1
-> $ wdctl
Device:        /dev/watchdog
Identity:      i6300ESB timer [version 0]
Timeout:       30 seconds
Pre-timeout:    0 seconds
KEEPALIVEPING  Keep alive ping reply          1           0
MAGICCLOSE     Supports magic close char      0           0
SETTIMEOUT     Set timeout (in seconds)       0           0

If it worked, the HW watchdog, then 'i6300esb' should reset the VM if nothing is pinging the watchdog - I read that it's possible to exit 'software' watchdog and not to cause HW watchdog take action. I do not know it that's happening here when I just 'systemclt stop watchdog' In '/etc/watchdog.conf' I do not point to any specific device, which I believe makes watchdogd do its things.
Simple test:
-> $ cat >> /dev/watchdog
& 'Enter' press twice
does invoke 'reset' action and I was to believe 'wdctl' that is HW watchdog working. But!... The main issue I have are those "soft lockups" where VM's OS becomes frozen, but nothing from the watchdog, no action - though, as VM is in such frozen state host shows high CPU for the VM.

I do not anything fancy so I really wonder if what I see is that rare. Soft-lockup occur I think usually, cannot say that uniquely though, during or after VM live-migration.

thanks, L.

[Index of Archives]     [Virt Tools]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux