There are a number of NVMeOF tests which are failing of which nvme/013 is the first one. Are these test failures known? It's often not clear to me whether blktests problems are due to kernel bugs, kernel misconfiguration, out-of-date or missing userspace components, or test bugs. nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed ns) [failed] runtime ... 31.249s --- tests/nvme/013.out 2019-02-11 18:57:39.000000000 -0500 +++ /results/nodev/nvme/013.out.bad 2019-02-11 21:00:42.839324301 -0500 @@ -1,5 +1,9 @@ Running nvme/013 91fdba0d-f87b-4c25-b80f-db7be1418b9e uuid.91fdba0d-f87b-4c25-b80f-db7be1418b9e +fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error: write offset=216387584, buflen=4096 +fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error: write offset=207941632, buflen=4096 +fio exited with status 0 ... Found in syslog: Feb 11 21:00:11 xfstests-tytso-20190211202457 root: run blktests nvme/013 Feb 11 21:00:11 xfstests-tytso-20190211202457 kernel: [ 2065.131191] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 Feb 11 21:00:11 xfstests-tytso-20190211202457 kernel: [ 2065.144934] nvmet: creating controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:41c2b863-9912-450a-87ca-7996c9228743. Feb 11 21:00:11 xfstests-tytso-20190211202457 kernel: [ 2065.160868] nvme nvme0: ANA group 1: optimized. Feb 11 21:00:11 xfstests-tytso-20190211202457 kernel: [ 2065.167407] nvme nvme0: creating 2 I/O queues. Feb 11 21:00:11 xfstests-tytso-20190211202457 kernel: [ 2065.174566] nvme nvme0: new ctrl: "blktests-subsystem-1" Feb 11 21:00:11 xfstests-tytso-20190211202457 kernel: [ 2065.384041] print_req_error: 1452 callbacks suppressed Feb 11 21:00:11 xfstests-tytso-20190211202457 kernel: [ 2065.384044] print_req_error: I/O error, dev nvme0c1n1, sector 0 flags 1000801 Feb 11 21:00:11 xfstests-tytso-20190211202457 kernel: [ 2065.412748] XFS (nvme0n1): Mounting V5 Filesystem Feb 11 21:00:11 xfstests-tytso-20190211202457 kernel: [ 2065.446064] XFS (nvme0n1): Ending clean mount Feb 11 21:00:42 xfstests-tytso-20190211202457 kernel: [ 2095.710995] print_req_error: I/O error, dev nvme0c1n1, sector 1048648 flags 1021801 Feb 11 21:00:42 xfstests-tytso-20190211202457 kernel: [ 2095.718852] XFS (nvme0n1): metadata I/O error in "xlog_iodone" at daddr 0x100048 len 64 error 5 Feb 11 21:00:42 xfstests-tytso-20190211202457 kernel: [ 2095.727729] XFS (nvme0n1): xfs_do_force_shutdown(0x2) called from line 1271 of file fs/xfs/xfs_log.c. Return address = 00000000588c28c5 Feb 11 21:00:42 xfstests-tytso-20190211202457 kernel: [ 2095.740041] XFS (nvme0n1): Log I/O Error Detected. Shutting down filesystem ... (and things go downhill from here) Thanks, - Ted