On Thu, Feb 21, 2019 at 07:34:15AM -0800, Luis Chamberlain wrote:
> On Sat, Feb 16, 2019 at 01:28:35PM -0500, Theodore Y. Ts'o wrote:
> > The block/*, loop/* and scsi/* tests in blktests do seem to be in
> > pretty good shape. The nvme, nvmeof, and srp tests are *definitely*
> > not as mature.
>
> Can you say more about this later part. What would you like to see more
> of for nvme tests for instance?
>
> It sounds like a productive session would include tracking our:
>
> a) sour spots
> b) who's already working on these
> c) gather volutneers for these sour spots
I complained on another LSF/MM topic thread, but there are a lot of
failures where it's not clear whether it's because I guessed
incorrectly about which version of nvme-cli I should be using (debian
stable and head of nvme-cli both are apparently wrong answers), or
kernel bugs or kernel misconfiguration issues on my side.
Current nvme/* failures that I'm still seeing are attached below.
- Ted
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [failed]
runtime ... 100.265s
something found in dmesg:
[ 1857.188083] run blktests nvme/012 at 2019-02-12 01:11:33
[ 1857.437322] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[ 1857.456187] nvmet: creating controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:78dc695a-2c99-4841-968d-c2c16a49a02a.
[ 1857.458162] nvme nvme0: ANA group 1: optimized.
[ 1857.458257] nvme nvme0: creating 2 I/O queues.
[ 1857.460893] nvme nvme0: new ctrl: "blktests-subsystem-1"
[ 1857.720666] ============================================
[ 1857.726308] WARNING: possible recursive locking detected
[ 1857.731784] 5.0.0-rc3-xfstests-00014-g1236f7d60242 #843 Not tainted
...
(See '/results/nodev/nvme/012.dmesg' for the entire message)
nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed ns) [failed]
runtime ... 32.634s
--- tests/nvme/013.out 2019-02-11 18:57:39.000000000 -0500
+++ /results/nodev/nvme/013.out.bad 2019-02-12 01:13:46.708757206 -0500
@@ -1,5 +1,9 @@
Running nvme/013
91fdba0d-f87b-4c25-b80f-db7be1418b9e
uuid.91fdba0d-f87b-4c25-b80f-db7be1418b9e
+fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error: write offset=329326592, buflen=4096
+fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error: write offset=467435520, buflen=4096
+fio exited with status 0
+4;fio-3.2;verify;0;5;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;192672;6182;1546;31166;4;9044;63.332763;57.979218;482;29948;10268.332290;1421.459893;1.000000%=4145;5.000000%=9109;10.000000%=9502;20.000000%=9764;30.000000%=10027;40.000000%=10289;50.000000%=10420;60.000000%=10551;70.000000%=10682;80.000000%=10682;90.000000%=10944;95.000000%=11206;99.000000%=13172;99.500000%=16318;99.900000%=24510;99.950000%=27394;99.990000%=29229;0%=0;0%=0;0%=0;507;30005;10331.973087;1421.131712;6040;8232;100.000000%;6189.741935;267.091495;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;3.991657%;6.484839%;91142;0;1024;0.1%;0.1%;0.1%;0.1%;100.0%;0.0%;0.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.20%;0.34%;0.25%;0.19%;27.86%;70.89%;0.23%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;nvme0n1;0;0;0;0;0;0;0;0.00%
...
(Run 'diff -u tests/nvme/013.out /results/nodev/nvme/013.out.bad' to see the entire diff)
nvme/015 (unit test for NVMe flush for file backed ns) [failed]
runtime ... 8.914s
--- tests/nvme/015.out 2019-02-11 18:57:39.000000000 -0500
+++ /results/nodev/nvme/015.out.bad 2019-02-12 01:14:05.429328259 -0500
@@ -1,6 +1,6 @@
Running nvme/015
91fdba0d-f87b-4c25-b80f-db7be1418b9e
uuid.91fdba0d-f87b-4c25-b80f-db7be1418b9e
-NVMe Flush: success
+NVME IO command error:INTERNAL: The command was not completed successfully due to an internal error(6006)
NQN:blktests-subsystem-1 disconnected 1 controller(s)
Test complete
nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery)
runtime ...
nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery) [failed]
runtime ... 23.576s
--- tests/nvme/016.out 2019-02-11 18:57:39.000000000 -0500
+++ /results/nodev/nvme/016.out.bad 2019-02-12 01:14:29.173378854 -0500
@@ -1,11 +1,11 @@
Running nvme/016
-Discovery Log Number of Records 1, Generation counter 1
+Discovery Log Number of Records 1, Generation counter 5
=====Discovery Log Entry 0======
trtype: loop
adrfam: pci
...
(Run 'diff -u tests/nvme/016.out /results/nodev/nvme/016.out.bad' to see the entire diff)
nvme/017 (create/delete many file-ns and test discovery)
runtime ...
nvme/017 (create/delete many file-ns and test discovery) [failed]
runtime ... 23.592s
--- tests/nvme/017.out 2019-02-11 18:57:39.000000000 -0500
+++ /results/nodev/nvme/017.out.bad 2019-02-12 01:14:52.880762691 -0500
@@ -1,11 +1,11 @@
Running nvme/017
-Discovery Log Number of Records 1, Generation counter 1
+Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype: loop
adrfam: pci
...
(Run 'diff -u tests/nvme/017.out /results/nodev/nvme/017.out.bad' to see the entire diff)
