Dear Linux SCSI folks,
On 11/08/16 11:07, Paul Menzel wrote:
Updating from Linux 4.4.X to Linux 4.8.4, we noticed that the 3ware
devices under `/dev` – `/dev/twa0`, `/dev/twa1`, … – map to the
controllers differently.
This unfortunately breaks quite a lot of our scripts, as we depend on
the fact that the first controller is also in the front.
$ dmesg | grep 3ware
[ 14.509238] 3ware 9000 Storage Controller device driver for Linux
v2.26.02.014.
[ 14.824274] scsi host8: 3ware 9000 Storage Controller
[ 14.824537] 3w-9xxx: scsi8: Found a 3ware 9000 Storage Controller
at 0xd0200000, IRQ: 17.
[ 15.508310] scsi host9: 3ware 9000 Storage Controller
[ 15.508569] 3w-9xxx: scsi9: Found a 3ware 9000 Storage Controller
at 0xda100000, IRQ: 17.
Tracing `twi_cli` it looks like the ordering of the devices in
`/sys/class/scsi_host` might have changed, as `getdents64` seems to be
used for the ordering of creating `/dev/twaX`.
$ find /sys/class/scsi_host/ -ls
6033 0 drwxr-xr-x 2 root system 0 Nov 8
10:58 /sys/class/scsi_host/
23125 0 lrwxrwxrwx 1 root system 0 Oct 27
17:41 /sys/class/scsi_host/host0 ->
../../devices/pci0000:00/0000:00:07.0/ata1/host0/scsi_host/host0
29893 0 lrwxrwxrwx 1 root system 0 Oct 27
18:03 /sys/class/scsi_host/host9 ->
../../devices/pci0000:80/0000:80:0e.0/0000:90:00.0/host9/scsi_host/host9
23878 0 lrwxrwxrwx 1 root system 0 Oct 27
17:41 /sys/class/scsi_host/host7 ->
../../devices/pci0000:80/0000:80:08.0/ata8/host7/scsi_host/host7
23640 0 lrwxrwxrwx 1 root system 0 Oct 27
17:41 /sys/class/scsi_host/host5 ->
../../devices/pci0000:80/0000:80:07.0/ata6/host5/scsi_host/host5
23402 0 lrwxrwxrwx 1 root system 0 Oct 27
17:41 /sys/class/scsi_host/host3 ->
../../devices/pci0000:00/0000:00:08.0/ata4/host3/scsi_host/host3
23164 0 lrwxrwxrwx 1 root system 0 Oct 27
17:41 /sys/class/scsi_host/host1 ->
../../devices/pci0000:00/0000:00:07.0/ata2/host1/scsi_host/host1
29851 0 lrwxrwxrwx 1 root system 0 Oct 27
18:03 /sys/class/scsi_host/host8 ->
../../devices/pci0000:00/0000:00:0e.0/0000:05:00.0/host8/scsi_host/host8
23839 0 lrwxrwxrwx 1 root system 0 Oct 27
17:41 /sys/class/scsi_host/host6 ->
../../devices/pci0000:80/0000:80:08.0/ata7/host6/scsi_host/host6
23601 0 lrwxrwxrwx 1 root system 0 Oct 27
17:41 /sys/class/scsi_host/host4 ->
../../devices/pci0000:80/0000:80:07.0/ata5/host4/scsi_host/host4
23363 0 lrwxrwxrwx 1 root system 0 Oct 27
17:41 /sys/class/scsi_host/host2 ->
../../devices/pci0000:00/0000:00:08.0/ata3/host2/scsi_host/host2
$ sudo -i tw_cli show
Ctl Model (V)Ports Drives Units NotOpt RRate VRate BBU
------------------------------------------------------------------------
c8 9650SE-8LPML 8 8 1 0 5 1 OK
c9 9690SA-8E 0 0 0 0 5 1 OK
Enclosure Slots Drives Fans TSUnits PSUnits Alarms
--------------------------------------------------------------
/c9/e0 16 0 3 1 2 1
So in this case `c8` is mapped to `/dev/twa1`, and `c9` to `/dev/twa0`.
As we do not know of a way, to use `tw_cli` to find the correct mapping,
or another place, we rely on the implicit ordering, which – according to
my colleagues – has worked for over 15 years [1].
Here is the excerpt from the manual page for smartctl [2].
> --- end of manual page excerpt ---
3ware,N - [FreeBSD and Linux only] the device consists of one or more ATA disks con‐
nected to a 3ware RAID controller. The non-negative integer N (in the range from 0 to
127 inclusive) denotes which disk on the controller is monitored. Use syntax such as:
smartctl -a -d 3ware,2 /dev/sda
smartctl -a -d 3ware,0 /dev/twe0
smartctl -a -d 3ware,1 /dev/twa0
smartctl -a -d 3ware,1 /dev/twl0
The first two forms, which refer to devices /dev/sda-z and /dev/twe0-15, may be used
with 3ware series 6000, 7000, and 8000 series controllers that use the 3x-xxxx driver.
Note that the /dev/sda-z form is deprecated starting with the Linux 2.6 kernel series
and may not be supported by the Linux kernel in the near future. The final form, which
refers to devices /dev/twa0-15, must be used with 3ware 9000 series controllers, which
use the 3w-9xxx driver.
The devices /dev/twl0-15 must be used with the 3ware/LSI 9750 series controllers which
use the 3w-sas driver.
Note that if the special character device nodes /dev/twl?, /dev/twa? and /dev/twe? do
not exist, or exist with the incorrect major or minor numbers, smartctl will recreate
them on the fly. Typically /dev/twa0 refers to the first 9000-series controller,
/dev/twa1 refers to the second 9000 series controller, and so on. The /dev/twl0
devices refers to the first 9750 series controller, /dev/twl1 resfers to the second
9750 series controller, and so on. Likewise /dev/twe0 refers to the first
6/7/8000-series controller, /dev/twe1 refers to the second 6/7/8000 series controller,
and so on.
Note that for the 6/7/8000 controllers, any of the physical disks can be queried or
examined using any of the 3ware's SCSI logical device /dev/sd? entries. Thus, if log‐
ical device /dev/sda is made up of two physical disks (3ware ports zero and one) and
logical device /dev/sdb is made up of two other physical disks (3ware ports two and
three) then you can examine the SMART data on any of the four physical disks using
either SCSI device /dev/sda or /dev/sdb. If you need to know which logical SCSI device
a particular physical disk (3ware port) is associated with, use the dmesg or SYSLOG
output to show which SCSI ID corresponds to a particular 3ware unit, and then use the
3ware CLI or 3dm tool to determine which ports (physical disks) correspond to particu‐
lar 3ware units.
If the value of N corresponds to a port that does not exist on the 3ware controller, or
to a port that does not physically have a disk attached to it, the behavior of smartctl
depends upon the specific controller model, firmware, Linux kernel and platform. In
some cases you will get a warning message that the device does not exist. In other
cases you will be presented with ´void´ data for a non-existent device.
Note that if the /dev/sd? addressing form is used, then older 3w-xxxx drivers do not
pass the "Enable Autosave" (´-S on´) and "Enable Automatic Offline" (´-o on´) commands
to the disk, and produce these types of harmless syslog error messages instead:
"3w-xxxx: tw_ioctl(): Passthru size (123392) too big". This can be fixed by upgrading
to version 1.02.00.037 or later of the 3w-xxxx driver, or by applying a patch to older
versions. Alternatively, use the character device /dev/twe0-15 interface.
The selective self-test functions (´-t select,A-B´) are only supported using the char‐
acter device interface /dev/twl0-15, /dev/twa0-15 and /dev/twe0-15. The necessary
WRITE LOG commands can not be passed through the SCSI interface.
> --- end of manual page excerpt ---
Do you know of a way, to either get the mapping “over an API” so we
don’t have to rely on the implicit ordering?
Otherwise, do you know, why the ordering has changed, and can this be
reverted?
Kind regards,
Paul Menzel
[1] https://www.thomas-krenn.com/de/wiki/Smartmontools_mit_3ware_RAID_Controller
(German)
[2] https://www.smartmontools.org/browser/trunk/smartmontools/smartctl.8.in
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
