Hi-
I'm running Ubuntu 16.04.5 with kernel 4.18.6. I connected two NVMeoF volumes:
rsa@rsd23n03:~$ sudo nvme connect -t rdma -i 32 -a 10.5.0.8 -n joe-vm-1
rsa@rsd23n03:~$ sudo nvme connect -t rdma -i 32 -a 10.5.0.8 -n joe-vm-2
Looking in disk stats afterwards, I see nvme2n1 and nvme3n1 as I would expect, but also there are these nvme2c2n1 and nvme3c3n1 devices present. Note nvme0n1 and nvme1n1 are actual local NVMe devices, not NVMeoF connected.
rsa@rsd23n03:~$ cat /proc/diskstats
7 0 loop0 5 0 16 0 0 0 0 0 0 0 0
7 1 loop1 0 0 0 0 0 0 0 0 0 0 0
7 2 loop2 0 0 0 0 0 0 0 0 0 0 0
7 3 loop3 0 0 0 0 0 0 0 0 0 0 0
7 4 loop4 0 0 0 0 0 0 0 0 0 0 0
7 5 loop5 0 0 0 0 0 0 0 0 0 0 0
7 6 loop6 0 0 0 0 0 0 0 0 0 0 0
7 7 loop7 0 0 0 0 0 0 0 0 0 0 0
259 0 nvme0n1 15222 0 696826 0 861 1596 28938 0 0 213856 218132
259 1 nvme0n1p1 1180 0 14056 0 2 0 2 0 0 24 964
259 2 nvme0n1p2 13828 0 669770 0 855 1596 28936 0 0 1612 4800
259 3 nvme0n1p3 112 0 8056 0 0 0 0 0 0 8 8
259 4 nvme1n1 135 0 6504 0 0 0 0 0 0 8 8
11 0 sr0 0 0 0 0 0 0 0 0 0 0 0
0 0 nvme2c2n1 49 0 2208 0 0 0 0 0 0 8 8
259 6 nvme2n1 0 0 0 0 0 0 0 0 0 0 0
0 0 nvme3c3n1 49 0 2208 0 0 0 0 0 0 4 4
259 8 nvme3n1 0 0 0 0 0 0 0 0 0 0 0
I read 1GB from each NVMeoF volume with DD, and the activity seems to get tracked against the 'c' devices instead of nvme2n1 and nvme3n1:
rsa@rsd23n03:~$ cat /proc/diskstats
7 0 loop0 5 0 16 0 0 0 0 0 0 0 0
7 1 loop1 0 0 0 0 0 0 0 0 0 0 0
7 2 loop2 0 0 0 0 0 0 0 0 0 0 0
7 3 loop3 0 0 0 0 0 0 0 0 0 0 0
7 4 loop4 0 0 0 0 0 0 0 0 0 0 0
7 5 loop5 0 0 0 0 0 0 0 0 0 0 0
7 6 loop6 0 0 0 0 0 0 0 0 0 0 0
7 7 loop7 0 0 0 0 0 0 0 0 0 0 0
259 0 nvme0n1 15226 0 696986 0 889 1618 29338 0 0 312432 316708
259 1 nvme0n1p1 1180 0 14056 0 2 0 2 0 0 24 964
259 2 nvme0n1p2 13832 0 669930 0 883 1618 29336 0 0 1620 4808
259 3 nvme0n1p3 112 0 8056 0 0 0 0 0 0 8 8
259 4 nvme1n1 135 0 6504 0 0 0 0 0 0 8 8
11 0 sr0 0 0 0 0 0 0 0 0 0 0 0
0 0 nvme2c2n1 8241 0 2099616 0 0 0 0 0 0 2620 4032
259 6 nvme2n1 0 0 0 0 0 0 0 0 0 0 0
0 0 nvme3c3n1 8241 0 2099616 0 0 0 0 0 0 1316 1856
259 8 nvme3n1 0 0 0 0 0 0 0 0 0 0 0
Also, when I run lsblk, I get an error related to these devices:
rsa@rsd23n03:~$ lsblk
lsblk: nvme2c2n1: unknown device name
lsblk: nvme3c3n1: unknown device name
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 119.2G 0 disk
├─nvme0n1p3 259:3 0 977M 0 part [SWAP]
├─nvme0n1p1 259:1 0 512M 0 part /boot/efi
└─nvme0n1p2 259:2 0 117.8G 0 part /
nvme3n1 259:8 0 931.5G 0 disk
nvme2n1 259:6 0 931.5G 0 disk
sr0 11:0 1 1024M 0 rom
nvme1n1 259:4 0 931.5G 0 disk
They do not appear to exist in /dev/:
rsa@rsd23n03:~$ ls /dev/nvm*
/dev/nvme0 /dev/nvme0n1p1 /dev/nvme0n1p3 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
/dev/nvme0n1 /dev/nvme0n1p2 /dev/nvme1 /dev/nvme2 /dev/nvme3 /dev/nvme-fabrics
The dmesg entries around the connect seem OK (I also did a discovery here before the connect):
[ 179.240487] nvme nvme2: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.5.0.8:4420
[ 179.241120] nvme nvme2: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[ 200.304085] nvme nvme2: creating 32 I/O queues.
[ 200.943936] nvme nvme2: new ctrl: NQN "joe-vm-1", addr 10.5.0.8:4420
[ 200.949023] nvme2n1:
[ 207.190511] nvme nvme3: creating 32 I/O queues.
[ 207.819411] nvme nvme3: new ctrl: NQN "joe-vm-2", addr 10.5.0.8:4420
[ 207.824087] nvme3n1:
The drives in the target are Intel P4500. Target is also running Ubuntu 16.04 with 4.18.6 and configuration looks like this:
rsa@storage-1:~$ sudo nvmetcli ls
o- / ........................................................................................................ [...]
o- hosts .................................................................................................. [...]
o- ports .................................................................................................. [...]
| o- 1 ............................................................. [trtype=rdma, traddr=10.5.0.8, trsvcid=4420]
| o- referrals .......................................................................................... [...]
| o- subsystems ......................................................................................... [...]
| o- joe-vm-1 ......................................................................................... [...]
| o- joe-vm-2 ......................................................................................... [...]
| o- nqn01 ............................................................................................ [...]
| o- nqn02 ............................................................................................ [...]
| o- nqn03 ............................................................................................ [...]
| o- nqn04 ............................................................................................ [...]
| o- nqn05 ............................................................................................ [...]
| o- nqn06 ............................................................................................ [...]
| o- nqn07 ............................................................................................ [...]
| o- nqn08 ............................................................................................ [...]
| o- nqn09 ............................................................................................ [...]
| o- nqn10 ............................................................................................ [...]
| o- nqn11 ............................................................................................ [...]
| o- nqn12 ............................................................................................ [...]
| o- nqn13 ............................................................................................ [...]
o- subsystems ............................................................................................. [...]
o- joe-vm-1 ............................................... [version=1.3, allow_any=1, serial=b6564d74b81e2ca0]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 ............................. [path=/dev/nvme14n1, uuid=9f7bca77-33d1-49bc-b36f-2e8997474ffb, enabled]
o- joe-vm-2 ............................................... [version=1.3, allow_any=1, serial=b729202cea8db493]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 ............................. [path=/dev/nvme15n1, uuid=7b5d87da-52c8-47d6-91e8-5f246bae142a, enabled]
o- nqn01 .................................................. [version=1.3, allow_any=1, serial=85b4c9db8d69b115]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 .............................. [path=/dev/nvme1n1, uuid=c311956d-c198-4291-aa1a-5544b8c529af, enabled]
o- nqn02 .................................................. [version=1.3, allow_any=1, serial=769b3c0093f050fa]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 .............................. [path=/dev/nvme2n1, uuid=e30fa42f-52c7-4f56-b956-458f6e48a815, enabled]
o- nqn03 .................................................. [version=1.3, allow_any=1, serial=7c5ff700733288ac]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 .............................. [path=/dev/nvme3n1, uuid=2361f047-9c8b-4c8f-ba8d-7a61327dd9be, enabled]
o- nqn04 .................................................. [version=1.3, allow_any=1, serial=bbcdf5c1976d7b89]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 .............................. [path=/dev/nvme4n1, uuid=a103ea20-c678-4171-a256-6e8ab5f07aea, enabled]
o- nqn05 .................................................. [version=1.3, allow_any=1, serial=b5d6519ea10ee571]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 .............................. [path=/dev/nvme5n1, uuid=062ebdea-f68d-4f8b-8d33-483b13fdc120, enabled]
o- nqn06 .................................................. [version=1.3, allow_any=1, serial=36af7cd910e698ae]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 .............................. [path=/dev/nvme6n1, uuid=317b6d55-af11-460f-9730-e06d734aca97, enabled]
o- nqn07 .................................................. [version=1.3, allow_any=1, serial=9038280e2442e469]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 .............................. [path=/dev/nvme7n1, uuid=bec91592-2ebb-4310-804e-0eb336bc166a, enabled]
o- nqn08 .................................................. [version=1.3, allow_any=1, serial=8fcb1df6c2c22328]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 .............................. [path=/dev/nvme8n1, uuid=96e2d208-3239-4878-8bde-7fb9821684cc, enabled]
o- nqn09 .................................................. [version=1.3, allow_any=1, serial=ca2739e704a71b40]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 .............................. [path=/dev/nvme9n1, uuid=4a203979-2dc2-465b-8ce9-8e6a277ff1de, enabled]
o- nqn10 ................................................... [version=1.3, allow_any=1, serial=70093977d1ebc0e]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 ............................. [path=/dev/nvme10n1, uuid=14c6f66d-f3c1-4fe1-a639-f7fde5568e34, enabled]
o- nqn11 .................................................. [version=1.3, allow_any=1, serial=ee7551c6e84e45cb]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 ............................. [path=/dev/nvme11n1, uuid=a5f24c39-f3b2-4190-a7e5-f53cdf783261, enabled]
o- nqn12 .................................................. [version=1.3, allow_any=1, serial=67a051ebffd3e8fe]
| o- allowed_hosts ...................................................................................... [...]
| o- namespaces ......................................................................................... [...]
| o- 1 ............................. [path=/dev/nvme12n1, uuid=3d6075fa-6e4e-46d3-918c-f9a1e3221aa1, enabled]
o- nqn13 .................................................. [version=1.3, allow_any=1, serial=79327e3ce01b319a]
o- allowed_hosts ...................................................................................... [...]
o- namespaces ......................................................................................... [...]
o- 1 ............................. [path=/dev/nvme13n1, uuid=24059ddc-e748-451e-97f7-c91482937069, enabled]
In general, my question is whether this is expected behavior? If a bug, is it fixed in a newer kernel? We are trying to track some disk stats and the unexpected (at least from our perspective) behavior in /proc/diskstats is causing us some confusion. Thanks!
