Hi,
Back on this.
I finally found out a logic in the mapping.
So after taking the time to note all the disks serial numbers on 3 different machines and 2 different OSes, I now know that my specific LSI SAS 2008 cards (no reference on them, but I think those are LSI sas 9207-8i) map the disks of the MD1000 in the reverse alphabetic order :
sd{b..p} map to slot{14..0}
There is absolutely nothing else that appears usable, except the sas_address of the disks which seems associated with slots.
But even this one is different depending on machines, and the address <-> slot mapping does not seem very obvious at the very least...
Good thing is that I now know that fun tools exist in packages such as sg3_tils, smp_utils and others like mpt-status...
Next step is to try an md1200 ;)
Thanks again
Cheers
-----Message d'origine-----
De : JF Le Fillâtre [mailto:jean-francois.lefillatre@xxxxxx]
Envoyé : mercredi 19 novembre 2014 13:42
À : SCHAER Frederic
Cc : ceph-users@xxxxxxxxxxxxxx
Objet : Re: jbod + SMART : how to identify failing disks ?
Hello again,
So whatever magic allows the Dell MD1200 to report the slot position for
each disk isn't present in your JBODs. Time for something else.
There are two sides to your problem:
1) Identifying which disk is where in your JBOD
Quite easy. Again I'd go for a udev rule + script that will either
rename the disks entirely, or create a symlink with a name like
"jbodX-slotY" or something to figure out easily which is which. The
mapping end-device-to-slot can be static in the script, so you need to
identify once the order in which the kernel scans the slots and then you
can map.
But it won't survive a disk swap or a change of scanning order from a
kernel upgrade, so it's not enough.
2) Finding a way of identification independent of hot-plugs and scan order
That's the tricky part. If you remove a disk from your JBOD and replace
it with another one, the other one will get another "sdX" name, and in
my experience even another "end_device-..." name. But given that you
want the new disk to have the exact same name or symlink as the previous
one, you have to find something in the path of the device or (better) in
the udev attributes that is immutable.
If possible at all, it will depend on your specific hardware
combination, so you will have to try for yourself.
Suggested methodology:
1) write down the serial number of one drive in any slot, and figure out
its device name (sdX) with "smartctl -i /dev/sd..."
2) grab the detailed /sys path name and list of udev attributes:
readlink -f /sys/class/block/sdX
udevadm info --attribute-walk /dev/sdX
3) pull that disk and replace it. Check the logs to see which is its new
device name (sdY)
4) rerun the commands from #2 with sdY
5) compare the outputs and find something in the path or in the
attributes that didn't change and is unique to that disk (ie not a
common parent for example).
If you have something that really didn't change, you're in luck. Either
use the serial numbers or unplug and replug all disks one by one to
figure out the mapping slot number / immutable item.
Then write the udev rule. :)
Thanks!
JF
On 19/11/14 11:29, SCHAER Frederic wrote:
> Hi
>
> Thanks.
> I hoped it would be it, but no ;)
>
> With this mapping :
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdb -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:0/expander-1:1/port-1:1:0/end_device-1:1:0/target1:0:1/1:0:1:0/block/sdb
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdc -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:0/expander-1:1/port-1:1:1/end_device-1:1:1/target1:0:2/1:0:2:0/block/sdc
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdd -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:0/expander-1:1/port-1:1:2/end_device-1:1:2/target1:0:3/1:0:3:0/block/sdd
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sde -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:0/expander-1:1/port-1:1:3/end_device-1:1:3/target1:0:4/1:0:4:0/block/sde
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdf -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:0/expander-1:1/port-1:1:4/end_device-1:1:4/target1:0:5/1:0:5:0/block/sdf
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdg -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:0/expander-1:1/port-1:1:5/end_device-1:1:5/target1:0:6/1:0:6:0/block/sdg
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdh -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:0/expander-1:1/port-1:1:6/end_device-1:1:6/target1:0:7/1:0:7:0/block/sdh
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdi -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:0/expander-1:1/port-1:1:7/end_device-1:1:7/target1:0:8/1:0:8:0/block/sdi
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdj -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:1/expander-1:2/port-1:2:0/end_device-1:2:0/target1:0:9/1:0:9:0/block/sdj
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdk -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:1/expander-1:2/port-1:2:1/end_device-1:2:1/target1:0:10/1:0:10:0/block/sdk
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdl -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:1/expander-1:2/port-1:2:2/end_device-1:2:2/target1:0:11/1:0:11:0/block/sdl
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdm -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:1/expander-1:2/port-1:2:3/end_device-1:2:3/target1:0:12/1:0:12:0/block/sdm
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdn -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:1/expander-1:2/port-1:2:4/end_device-1:2:4/target1:0:13/1:0:13:0/block/sdn
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdo -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:1/expander-1:2/port-1:2:5/end_device-1:2:5/target1:0:14/1:0:14:0/block/sdo
> lrwxrwxrwx 1 root root 0 Nov 12 12:31 /sys/class/block/sdp -> ../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/host1/port-1:0/expander-1:0/port-1:0:1/expander-1:2/port-1:2:6/end_device-1:2:6/target1:0:15/1:0:15:0/block/sdp
>
> sdd was on physical slot 12, sdk was on slot 5, and sdg was on slot 9 (and I did not check the others)...
> so clearly this cannot be put in production as is and I'll have to find a way.
>
> Regards
>
>
> -----Message d'origine-----
> De : Carl-Johan Schenström [mailto:carl-johan.schenstrom@xxxxx]
> Envoyé : lundi 17 novembre 2014 14:14
> À : SCHAER Frederic; Scottix; Erik Logtenberg
> Cc : ceph-users@xxxxxxxxxxxxxx
> Objet : RE: jbod + SMART : how to identify failing disks ?
>
> Hi!
>
> I'm fairly sure that the link targets in /sys/class/block were correct the last time I had to change a drive on a system with a Dell HBA connected to an MD1000, but perhaps I was just lucky. =/
>
> I.e.,
>
> # ls -l /sys/class/block/sdj
> lrwxrwxrwx. 1 root root 0 17 nov 13.54 /sys/class/block/sdj -> ../../devices/pci0000:20/0000:20:0a.0/0000:21:00.0/host7/port-7:0/expander-7:0/port-7:0:1/expander-7:2/port-7:2:6/end_device-7:2:6/target7:0:7/7:0:7:0/block/sdj
>
> would be first port on HBA, first expander, 7th slot (6, starting from 0). Don't take my word for it, though!
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
