On Friday, March 03/20/20, 2020 at 13:49:25 -0700, Sagi Grimberg wrote: > > >>I assume this happens with iSCSI as well? There is nothing special > >>we are doing with respect to digest. > > > >I don't see this issue with iscsi-tcp. > > > >May be blk-mq is causing this issue? I assume iscsi-tcp does not have > >blk_mq support yet upstream to verify with blk_mq enabled. > >I tried on Ubuntu 19.10(which is based on Linux kernel 5.3), note that > >RHEL does not support DataDigest. > > > >The reason that I'm seeing this issue only with NVMe(tcp/softiwarp) & > >iSER(softiwarp) is becuase of NVMeF&ISER using blk-mq? > > > >Anyhow, I see the content of the page is being updated by upper layers > >while the tranport driver is computing CRC on that page content and > >this needs a fix. > > Krishna, do you happen to run with nvme multipath enabled? Yes Sagi, issue occurs with nvme multipath enabled also.. dmesg at initiator: [ +10.671996] EXT4-fs (nvme0n1): mounting ext3 file system using the ext4 subsystem [ +0.004643] EXT4-fs (nvme0n1): mounted filesystem with ordered data mode. Opts: (null) [ +15.955424] block nvme0n1: no usable path - requeuing I/O [ +0.000142] block nvme0n1: no usable path - requeuing I/O [ +0.000135] block nvme0n1: no usable path - requeuing I/O [ +0.000119] block nvme0n1: no usable path - requeuing I/O [ +0.000108] block nvme0n1: no usable path - requeuing I/O [ +0.000111] block nvme0n1: no usable path - requeuing I/O [ +0.000118] block nvme0n1: no usable path - requeuing I/O [ +0.000158] block nvme0n1: no usable path - requeuing I/O [ +0.000130] block nvme0n1: no usable path - requeuing I/O [ +0.000138] block nvme0n1: no usable path - requeuing I/O [ +0.011754] nvme nvme0: Reconnecting in 10 seconds... [ +10.261223] nvme_ns_head_make_request: 5 callbacks suppressed [ +0.000002] block nvme0n1: no usable path - requeuing I/O [ +0.000240] block nvme0n1: no usable path - requeuing I/O [ +0.000107] block nvme0n1: no usable path - requeuing I/O [ +0.000107] block nvme0n1: no usable path - requeuing I/O [ +0.000107] block nvme0n1: no usable path - requeuing I/O [ +0.000108] block nvme0n1: no usable path - requeuing I/O [ +0.000132] block nvme0n1: no usable path - requeuing I/O [ +0.000010] nvme nvme0: creating 12 I/O queues. [ +0.000110] block nvme0n1: no usable path - requeuing I/O [ +0.000232] block nvme0n1: no usable path - requeuing I/O [ +0.000122] block nvme0n1: no usable path - requeuing I/O [ +0.008407] nvme nvme0: Successfully reconnected (1 attempt) dmesg at target: [Mar21 09:24] nvmet_tcp: queue 3: cmd 38 pdu (6) data digest error: recv 0x21e59730 expected 0x2b88fed0 [ +0.000029] nvmet: ctrl 1 fatal error occurred! [ +10.280101] nvmet: creating controller 1 for subsystem nvme-ram0 for NQN nqn.2014-08.org.nvmexpress.chelsio.
