Re: Panic when rebooting target server testing srp on 5.0.0-rc2

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, 2019-03-20 at 11:11 -0400, Laurence Oberman wrote:
+AD4 On Wed, 2019-03-20 at 09:45 -0400, Laurence Oberman wrote:
+AD4 +AD4 Hello Bart, I hope all is well with you.
+AD4 +AD4 
+AD4 +AD4 Quick question
+AD4 +AD4 preparing to test v5.1-rc2 SRP, my usual method is first validate the
+AD4 +AD4 prior kernel I had in place.
+AD4 +AD4 This had passed tests previously (5.0.0-rc2) but I had not run the
+AD4 +AD4 target server reboot test, just the disconnect tests.
+AD4 +AD4 
+AD4 +AD4 Today with mapped SRP devices I rebooted the target server and the
+AD4 +AD4 client panicked.
+AD4 +AD4 
+AD4 +AD4 Its been a while and I have been so busy that have not kept up with
+AD4 +AD4 all
+AD4 +AD4 the fixes. Is this a known issue.
+AD4 +AD4 
+AD4 +AD4 Thanks
+AD4 +AD4 Laurence
+AD4 +AD4 
+AD4 +AD4 5414228.917507+AF0 scsi host2: ib+AF8-srp: Path record query failed: sgid
+AD4 +AD4 fe80:0000:0000:0000:7cfe:9003:0072:6ed3, dgid
+AD4 +AD4 fe80:0000:0000:0000:7cfe:9003:0072:6e4f, pkey 0xffff, service+AF8-id
+AD4 +AD4 0x7cfe900300726e4e
+AD4 +AD4 +AFs-5414229.014355+AF0 scsi host2: reconnect attempt 7 failed (-110)
+AD4 +AD4 +AFs-5414239.318161+AF0 scsi host2: ib+AF8-srp: Sending CM DREQ failed
+AD4 +AD4 +AFs-5414239.318165+AF0 scsi host2: ib+AF8-srp: Sending CM DREQ failed
+AD4 +AD4 +AFs-5414239.318167+AF0 scsi host2: ib+AF8-srp: Sending CM DREQ failed
+AD4 +AD4 +AFs-5414239.318168+AF0 scsi host2: ib+AF8-srp: Sending CM DREQ failed
+AD4 +AD4 +AFs-5414239.318170+AF0 scsi host2: ib+AF8-srp: Sending CM DREQ failed
+AD4 +AD4 +AFs-5414239.318172+AF0 scsi host2: ib+AF8-srp: Sending CM DREQ failed
+AD4 +AD4 +AFs-5414239.318173+AF0 scsi host2: ib+AF8-srp: Sending CM DREQ failed
+AD4 +AD4 +AFs-5414239.318175+AF0 scsi host2: ib+AF8-srp: Sending CM DREQ failed
+AD4 +AD4 +AFs-5414243.670072+AF0 scsi host2: ib+AF8-srp: Got failed path rec status -110
+AD4 +AD4 +AFs-5414243.702179+AF0 scsi host2: ib+AF8-srp: Path record query failed: sgid
+AD4 +AD4 fe80:0000:0000:0000:7cfe:9003:0072:6ed3, dgid
+AD4 +AD4 fe80:0000:0000:0000:7cfe:9003:0072:6e4f, pkey 0xffff, service+AF8-id
+AD4 +AD4 0x7cfe900300726e4e
+AD4 +AD4 +AFs-5414243.799313+AF0 scsi host2: reconnect attempt 8 failed (-110)
+AD4 +AD4 +AFs-5414247.510115+AF0 scsi host1: ib+AF8-srp: Sending CM REQ failed
+AD4 +AD4 +AFs-5414247.510140+AF0 scsi host1: reconnect attempt 1 failed (-104)
+AD4 +AD4 +AFs-5414247.849078+AF0 BUG: unable to handle kernel NULL pointer
+AD4 +AD4 dereference
+AD4 +AD4 at 00000000000000b8
+AD4 +AD4 +AFs-5414247.893793+AF0 +ACM-PF error: +AFs-normal kernel read fault+AF0
+AD4 +AD4 +AFs-5414247.921839+AF0 PGD 0 P4D 0 
+AD4 +AD4 +AFs-5414247.937280+AF0 Oops: 0000 +AFsAIw-1+AF0 SMP PTI
+AD4 +AD4 +AFs-5414247.958332+AF0 CPU: 4 PID: 7773 Comm: kworker/4:1H Kdump: loaded
+AD4 +AD4 Tainted: G          I       5.0.0-rc2+- +ACM-2
+AD4 +AD4 +AFs-5414248.012856+AF0 Hardware name: HP ProLiant DL380 G7, BIOS P67
+AD4 +AD4 08/16/2015
+AD4 +AD4 +AFs-5414248.026174+AF0 device-mapper: multipath: Failing path 8:48.
+AD4 +AD4 
+AD4 +AD4 
+AD4 +AD4 +AFs-5414248.050003+AF0 Workqueue: kblockd blk+AF8-mq+AF8-run+AF8-work+AF8-fn
+AD4 +AD4 +AFs-5414248.108378+AF0 RIP: 0010:blk+AF8-mq+AF8-dispatch+AF8-rq+AF8-list+-0xc9/0x590
+AD4 +AD4 +AFs-5414248.139724+AF0 Code: 0f 85 c2 04 00 00 83 44 24 28 01 48 8b 45 00
+AD4 +AD4 48
+AD4 +AD4 39 c5 0f 84 ea 00 00 00 48 8b 5d 00 80 3c 24 00 4c 8d 6b b8 4c 8b 63
+AD4 +AD4 c8
+AD4 +AD4 75 25 +ADw-49+AD4 8b 84 24 b8 00 00 00 48 8b 40 40 48 8b 40 10 48 85 c0 74
+AD4 +AD4 10
+AD4 +AD4 4c
+AD4 +AD4 +AFs-5414248.246176+AF0 RSP: 0018:ffffb1cd8760fd90 EFLAGS: 00010246
+AD4 +AD4 +AFs-5414248.275599+AF0 RAX: ffffa049d67a1308 RBX: ffffa049d67a1308 RCX:
+AD4 +AD4 0000000000000004
+AD4 +AD4 +AFs-5414248.316090+AF0 RDX: 0000000000000000 RSI: ffffb1cd8760fe20 RDI:
+AD4 +AD4 ffffa0552ca08000
+AD4 +AD4 +AFs-5414248.356884+AF0 RBP: ffffb1cd8760fe20 R08: 0000000000000000 R09:
+AD4 +AD4 8080808080808080
+AD4 +AD4 +AFs-5414248.397632+AF0 R10: 0000000000000001 R11: 0000000000000001 R12:
+AD4 +AD4 0000000000000000
+AD4 +AD4 +AFs-5414248.439323+AF0 R13: ffffa049d67a12c0 R14: 0000000000000000 R15:
+AD4 +AD4 ffffa0552ca08000
+AD4 +AD4 +AFs-5414248.481743+AF0 FS:  0000000000000000(0000)
+AD4 +AD4 GS:ffffa04a37880000(0000)
+AD4 +AD4 knlGS:0000000000000000
+AD4 +AD4 +AFs-5414248.528310+AF0 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
+AD4 +AD4 +AFs-5414248.561779+AF0 CR2: 00000000000000b8 CR3: 0000000e9d40e004 CR4:
+AD4 +AD4 00000000000206e0
+AD4 +AD4 +AFs-5414248.602420+AF0 Call Trace:
+AD4 +AD4 +AFs-5414248.616660+AF0  blk+AF8-mq+AF8-sched+AF8-dispatch+AF8-requests+-0x15c/0x180
+AD4 +AD4 +AFs-5414248.647066+AF0  +AF8AXw-blk+AF8-mq+AF8-run+AF8-hw+AF8-queue+-0x5f/0xf0
+AD4 +AD4 +AFs-5414248.672633+AF0  process+AF8-one+AF8-work+-0x171/0x370
+AD4 +AD4 +AFs-5414248.695443+AF0  worker+AF8-thread+-0x49/0x3f0
+AD4 +AD4 +AFs-5414248.716730+AF0  kthread+-0xf8/0x130
+AD4 +AD4 +AFs-5414248.735085+AF0  ? max+AF8-active+AF8-store+-0x80/0x80
+AD4 +AD4 +AFs-5414248.758569+AF0  ? kthread+AF8-bind+-0x10/0x10
+AD4 +AD4 +AFs-5414248.779953+AF0  ret+AF8-from+AF8-fork+-0x35/0x40
+AD4 +AD4 
+AD4 +AD4 +AFs-5414248.801005+AF0 Modules linked in: ib+AF8-isert iscsi+AF8-target+AF8-mod
+AD4 +AD4 target+AF8-core+AF8-mod ib+AF8-srp rpcrdma scsi+AF8-transport+AF8-srp rdma+AF8-ucm ib+AF8-iser
+AD4 +AD4 ib+AF8-ipoib ib+AF8-umad rdma+AF8-cm libiscsi iw+AF8-cm scsi+AF8-transport+AF8-iscsi ib+AF8-cm
+AD4 +AD4 sunrpc mlx5+AF8-ib ib+AF8-uverbs ib+AF8-core intel+AF8-powerclamp coretemp kvm+AF8-intel
+AD4 +AD4 kvm irqbypass ipmi+AF8-ssif crct10dif+AF8-pclmul crc32+AF8-pclmul
+AD4 +AD4 ghash+AF8-clmulni+AF8-intel aesni+AF8-intel iTCO+AF8-wdt crypto+AF8-simd cryptd gpio+AF8-ich
+AD4 +AD4 iTCO+AF8-vendor+AF8-support glue+AF8-helper joydev ipmi+AF8-si dm+AF8-service+AF8-time pcspkr
+AD4 +AD4 ipmi+AF8-devintf hpilo hpwdt sg ipmi+AF8-msghandler acpi+AF8-power+AF8-meter lpc+AF8-ich
+AD4 +AD4 i7core+AF8-edac pcc+AF8-cpufreq dm+AF8-multipath ip+AF8-tables xfs libcrc32c radeon
+AD4 +AD4 sd+AF8-mod i2c+AF8-algo+AF8-bit drm+AF8-kms+AF8-helper syscopyarea sysfillrect sysimgblt
+AD4 +AD4 fb+AF8-sys+AF8-fops ttm drm mlx5+AF8-core crc32c+AF8-intel serio+AF8-raw i2c+AF8-core hpsa
+AD4 +AD4 bnx2
+AD4 +AD4 scsi+AF8-transport+AF8-sas mlxfw devlink dm+AF8-mirror dm+AF8-region+AF8-hash dm+AF8-log
+AD4 +AD4 dm+AF8-mod
+AD4 +AD4 +AFs-5414249.199354+AF0 CR2: 00000000000000b8
+AD4 +AD4 
+AD4 +AD4 
+AD4 
+AD4 Looking at the vmcore
+AD4 
+AD4 PID: 7773   TASK: ffffa04a2c1e2b80  CPU: 4   COMMAND: +ACI-kworker/4:1H+ACI
+AD4  +ACM-0 +AFs-ffffb1cd8760fab0+AF0 machine+AF8-kexec at ffffffffaaa6003f
+AD4  +ACM-1 +AFs-ffffb1cd8760fb08+AF0 +AF8AXw-crash+AF8-kexec at ffffffffaab373ed
+AD4  +ACM-2 +AFs-ffffb1cd8760fbd0+AF0 crash+AF8-kexec at ffffffffaab385b9
+AD4  +ACM-3 +AFs-ffffb1cd8760fbe8+AF0 oops+AF8-end at ffffffffaaa31931
+AD4  +ACM-4 +AFs-ffffb1cd8760fc08+AF0 no+AF8-context at ffffffffaaa6eb59
+AD4  +ACM-5 +AFs-ffffb1cd8760fcb0+AF0 do+AF8-page+AF8-fault at ffffffffaaa6feb2
+AD4  +ACM-6 +AFs-ffffb1cd8760fce0+AF0 page+AF8-fault at ffffffffab2010ee
+AD4     +AFs-exception RIP: blk+AF8-mq+AF8-dispatch+AF8-rq+AF8-list+-201+AF0
+AD4     RIP: ffffffffaad90589  RSP: ffffb1cd8760fd90  RFLAGS: 00010246
+AD4     RAX: ffffa049d67a1308  RBX: ffffa049d67a1308  RCX: 0000000000000004
+AD4     RDX: 0000000000000000  RSI: ffffb1cd8760fe20  RDI: ffffa0552ca08000
+AD4     RBP: ffffb1cd8760fe20   R8: 0000000000000000   R9: 8080808080808080
+AD4     R10: 0000000000000001  R11: 0000000000000001  R12: 0000000000000000
+AD4     R13: ffffa049d67a12c0  R14: 0000000000000000  R15: ffffa0552ca08000
+AD4     ORIG+AF8-RAX: ffffffffffffffff  CS: 0010  SS: 0018
+AD4  +ACM-7 +AFs-ffffb1cd8760fe18+AF0 blk+AF8-mq+AF8-sched+AF8-dispatch+AF8-requests at
+AD4 ffffffffaad9570c
+AD4  +ACM-8 +AFs-ffffb1cd8760fe60+AF0 +AF8AXw-blk+AF8-mq+AF8-run+AF8-hw+AF8-queue at ffffffffaad8de3f
+AD4  +ACM-9 +AFs-ffffb1cd8760fe78+AF0 process+AF8-one+AF8-work at ffffffffaaab0ab1
+AD4 +ACM-10 +AFs-ffffb1cd8760feb8+AF0 worker+AF8-thread at ffffffffaaab11d9
+AD4 +ACM-11 +AFs-ffffb1cd8760ff10+AF0 kthread at ffffffffaaab6758
+AD4 +ACM-12 +AFs-ffffb1cd8760ff50+AF0 ret+AF8-from+AF8-fork at ffffffffab200215
+AD4 
+AD4 We were working on this request+AF8-queue for
+AD4  blk+AF8-mq+AF8-sched+AF8-dispatch+AF8-requests
+AD4 
+AD4 crash+AD4 dev -d +AHw grep ffffa0552ca08000
+AD4     8
+AD4 ffffa055c81b5800   sdd        ffffa0552ca08000       0     0     0 
+AD4 +AF0
+AD4 
+AD4 That device was no longer accessible 
+AD4 
+AD4 sdev+AF8-state +AD0 SDEV+AF8-TRANSPORT+AF8-OFFLINE,
+AD4 
+AD4 So it looks like we tried to process a no longer valid list entry in 
+AD4 blk+AF8-mq+AF8-dispatch+AF8-rq+AF8-list
+AD4 
+AD4 /home/loberman/rpmbuild/BUILD/kernel-5.0.0+AF8-rc2+-/block/blk-mq.h: 211
+AD4 0xffffffffaad90589
+AD4 +ADw-blk+AF8-mq+AF8-dispatch+AF8-rq+AF8-list+-201+AD4:       mov    0xb8(+ACU-r12),+ACU-rax
+AD4 
+AD4 R12 is NULL
+AD4 
+AD4 
+AD4 From
+AD4 static inline bool blk+AF8-mq+AF8-get+AF8-dispatch+AF8-budget(struct blk+AF8-mq+AF8-hw+AF8-ctx
+AD4 +ACo-hctx)
+AD4 +AHs
+AD4         struct request+AF8-queue +ACo-q +AD0 hctx-+AD4-queue+ADs
+AD4 
+AD4         if (q-+AD4-mq+AF8-ops-+AD4-get+AF8-budget)
+AD4                 return q-+AD4-mq+AF8-ops-+AD4-get+AF8-budget(hctx)+ADs
+AD4         return true+ADs
+AD4 +AH0
+AD4 
+AD4 Willw ait for a reply befaore i try the newer kernel, but looks like a
+AD4 use after free to me

Hi Laurence,

I don't think that any of the recent SRP initiator changes can be the root
cause of this crash. However, significant changes went upstream in the block
layer core during the v5.1-rc1 merge window, e.g. multi-page bvec support.
Is it possible for you to bisect this kernel oops?

Thanks,

Bart.




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [SCSI Target Devel]     [Linux SCSI Target Infrastructure]     [Kernel Newbies]     [IDE]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux ATA RAID]     [Linux IIO]     [Samba]     [Device Mapper]

  Powered by Linux