On 4/15/2019 10:26 PM, TomK wrote:
On 4/15/2019 3:35 PM, Laurence Oberman wrote:
On Mon, 2019-04-15 at 08:39 -0700, Bart Van Assche wrote:
On Mon, 2019-04-15 at 08:55 -0400, Laurence Oberman wrote:
On Sun, 2019-04-14 at 23:25 -0400, TomK wrote:
Hey All,
I'm getting a kernel panic on an Gigabyte GA-890XA-UD3
motherboard
that
I've got a QLE2464 card in as a target (FC). The kernel
has
been
crashing / panicking in the last 1-2 months about once a
week. Before
that, it was rock solid for 4-5 years. I've upgraded to
kernel
4.18.19
but that hasn't made much of a difference. Since the
message
includes
qla2x00_request_irqs I thought I would try here first.
Tried to get more info on this but:
1) Keyboard doesn't work and locks up when the panic
occurs. No
USB
ports work. Tried the PS/2 port but nothing.
2) Unable to capture a kdump. Can't get to the kdump
vmcore due
to
1).
The two screenshots is pretty much all I can capture.
Tried
things
like
clocksource=rtc in the kernel parms and disabling hpet1
but
apparently I
haven't disabled it everywhere since it still shows up.
Wondering if anyone recognizes these messages or has any
idea
what
could
be the issue here? Even a hint would be appreciated.
Hello Tom
I have had similar issues and reported them to
Himanshu@Cavium
I have kept all my target servers at kernel 4.5 as it been
the only
version that has always been stable.
If your motherboard has an NMI (virtual or physical) set all
of
these
in /etc/sysctl.conf
Run sysctl -a;dracut -f and reboot
kernel.nmi_watchdog = 1
kernel.panic_on_io_nmi = 1
kernel.panic_on_unrecovered_nmi =
kernel.unknown_nmi_panic = 1
When the issue shows up press the virtual/physical NMI
This is with the assumption that generic kdump is properly
setup
and
dmesg | grep crash shows memory resrved by the crashkernel
and that
you
have tested kdump manually.
Other options are use a USB serial port to capture the full
log if
you
cannot get kdump to work.
That approach may provide further evidence about kernel bugs
but it
is not
guaranteed that that approach will lead to a solution. It would
help
if
either or both of you could do the following on a test system:
* Check out branch qla2xxx-for-next of my kernel repo on
github
(https://github.com/bvanassche/linux/tree/qla2xxx-for-next).
* Enable lockdep and KASAN in the kernel config
(CONFIG_PROVE_LOCKING
and
CONFIG_KASAN).
* Build and install that kernel.
* Run your favorite workload.
Please note that the qla2xxx-for-next branch is based on the
v5.1-rc1
kernel
and hence should not be installed on any production system.
Thanks,
Bart.
Hello Bart
OK, I will get to this by Thursday, wont be able to change the
targetserver kernel until then.
Regards
Laurence
Same. I'll try this out closer to the weekend.
Not an NMI motherboard. This is a 9-10 year old AMD board meant as
a desktop or home server.
I'll have to read more about the USB Serial port to capture further
info. That's interesting.
For the time being, I've disabled HPET in BIOS. ( Appears the
kernel boot parameter method wasn't enough. )
Hey Guy's,
Did some of what you suggested, including the USB serial setup:
1) One of DB9 RS232 Serial Null Modem Cable F/F
2) Two of USB to RS232 Serial Port DB9 9 Pin Male
however, when the kernel came down it took the USB support with it
and so minicom went offline:
CTRL-A Z for help |115200 8N1 | NOR | Minicom 2.6.2 | VT102 |
Offline
But I did enable full logging for the QLA module:
echo 0x7fffffff >
/sys/module/qla2xxx/parameters/ql2xextended_error_logging
Did all that, minus the Kernel v5.1-rc1 implementation, and this is
what was picked up from the minicom USB to Serial capture before
things went south:
1235905 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-
bufflen=512, cmd->sg_cnt=1, cmd-
dma_data_directi
on=1
se_cmd[0000
00009c9ea758]
qp
0
1235906 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-
bufflen=4096, cmd->sg_cnt=0,
cmd-
dma_data_direct
ion=2
se_cmd[000
0000096ae11b7]
q
p 0
1235907 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-
bufflen=20480, cmd->sg_cnt=0,
cmd
->dma_data_direc
tion=2
se_cmd[00
0000001738f793]
qp 0
1235908 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-
bufflen=20480, cmd->sg_cnt=0,
cmd
->dma_data_direc
tion=2
se_cmd[00
000000e8160a90]
qp 0
1235909 ^MDetected MISCOMPARE for addr: 0000000033045258 buf:
00000000f9849912
1235910 ^MTarget/fileio: Send MISCOMPARE check condition and sense
1235911 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-
bufflen=512, cmd->sg_cnt=0, cmd-
dma_data_directi
on=2
se_cmd[0000
0000363ae214]
qp
0
1235912 ^Mqla2xxx [0000:04:00.0]-e817: Skipping EXPLICIT_CONFORM and
CTIO7_FLAGS_CONFORM_REQ
fo
r FCP READ w/
no
n GOOD status
1235913 ^Mqla2xxx [0000:04:00.0]-e874:2: qlt_free_cmd:
se_cmd[000000001db805fd] ox_id 00c8
1235914 ^Mqla2xxx [0000:04:00.0]-e872:2: qlt_24xx_atio_pkt_all_vps:
qla_target(0): type 6
ox_id
00db
1235915 ^Mqla2xxx [0000:04:00.0]-e872:2: qlt_24xx_atio_pkt_all_vps:
qla_target(0): type 6
ox_id
00dc
1235916 ^Mqla2xxx [0000:04:00.0]-e874:2: qlt_free_cmd:
se_cmd[00000000f67a701f] ox_id 00c9
1235917 ^Mqla2xxx [0000:04:00.0]-e872:2: qlt_24xx_atio_pkt_all_vps:
qla_target(0): type 6
ox_id
00dd
1235918 ^Mqla2xxx [0000:04:00.0]-e872:2: qlt_24xx_atio_pkt_all_vps:
qla_target(0): type 6
ox_id
00de
On an earlier crash, captured the attached image. This time there
was nothing on the monitor and the keyboard didn't refresh it. No
signal.
When looking this up, closest I could see online is the following:
They too run ESXi .
To read the file I used the AnsiEsc plugin for VIM:
https://www.vim.org/scripts/script.php?script_id=302
This started to occur once had a VMware based MySQL and PostgreSQL
cluster configured. Takes a few days for the issue to occur so from
that perspective, appears to be memory related.
Firmware that I'm using is:
supported_classes = "Class 3"
supported_speeds = "1 Gbit, 2 Gbit, 4 Gbit"
symbolic_name = "QLE2464 FW:v8.04.00 DVR:v10.00.00.05-k"
Targetcli, rtslib and configshell versions I'm using are:
# rpm -aq|grep -Ei "targetcli|rtslib|configshell"
python-rtslib-3.0.pre4.9~g6fd0bbf-1.el6.noarch
python-configshell-1.1.fb4-1.el6.noarch
targetcli-3.0.pre4.5~ga125182-1.el6.noarch
--
Thx,
TK.
