Sorry for top posting, but Outlook just screws it all up.
The cards I've used are a LSI Logic SAS 3800X (8 port External PCI-X card w/ 2 x SFF-8470 SAS connectors) and LSI SAS 3801E ( 8 Port External PCI-e card with 2 x SFF-8088 SAS connectors). Each connector has 4 SAS links.
The SAS protocol is downwardly compatible with SATA, so you can run SATA drives right on a SAS cable.
So, in my setup, I basically have 1 drive per SAS link. No expanders, or anything fancy. The issues I mentioned happens to the 4 drives on the same connector. When the driver is detecting the new drive, it looks like it redetects all of the drives on the connector (or it at least reports one new drive and the other existing drives). If you were in a directory from one of the mounted drives, you get IO Errors as it appears that the drive was removed, and then remounted, but in a way that was not clean.
This has happened with Default CentOS 5 kernels (2.6.18-*.el5), 2.6.26 vanilla, 2.6.30 vanilla, Fedora latest.
The issue appeared no matter what.
The udev rules used the ENV{ID_PATH} option to tie to the sysfs value that indicated which PCI ID + SAS phy on the SAS HBA used by the drives to the device detected by the kernel, and then create a symlink from the /dev/sd<X> entry to /dev/slot<Y>, where Y is the label on the slot of the hot swap bays (a-h). Here is an example of the rule:
KERNEL=="sd*", ENV{ID_PATH}=="pci-0000:04:00.0-sas-phy0:1*", SYMLINK+="slota%n"
I did this because the device ID number that the kernel reports increments every time a drive is swapped. So, even though you are using the same SAS channel, you do not have a consistent drive numbering. So I had to go down to the SAS phy to get something consistent. The SiI-3124/libata setup had consistent device ID's (the ID was tied to the SATA channel, and I used the device ID to do the mapping. Perhaps udev is the reason for the issues, but I tend to think it is the way the SAS/SCSI subsystem works as I have never seen the SATA/libata subsystem have this "rescan/remount" behavior.
-----Original Message-----
From: linux-scsi-owner@xxxxxxxxxxxxxxx [mailto:linux-scsi-owner@xxxxxxxxxxxxxxx] On Behalf Of Asdo
Sent: Friday, March 05, 2010 6:06 PM
To: Moore, Michael
Cc: James Bottomley; linux-scsi@xxxxxxxxxxxxxxx
Subject: Re: LSI SAS changes SCSI address and by-path on hot-swap
Moore, Michael wrote:
> I did this with a LSI-1068E HBA and 2 x 4 drive hot swap SATA bays. I was able to create udev rules to map the drive slots to consistent /dev entries. However, the bigger problem I had was that if I had a drive inserted and mounted (say in slot A) and then I added or swapped another drive on the same port ( 4 SAS channels per port on the external HBAs) it would cause some sort of reset on the bus that would end up unmounting the drive
> in slot A even though I never did anything to the drive in slot A. Now, this was with SATA drives connected directly to the 1068 which should work, but since I needed this to work, I had to revert to the older setup that used Silicon Image 3124 eSATA cards.
>
> I can try to dig up the udev rules I used if this would be helpful.
>
> - Mike
>
Michael, thanks for replying.
The problem you describe would really be a showstopper for us, so I'd
really like to understand it well!...
I tried to reread your post multiple times but I don't fully understand,
excuse my ignorance... this is because I am not really familiar with
SAS/SCSI terms.
You have an LSI-1068E attached directly to hot swap SATA bays, you don't
have expanders in the middle, right?
Then you swapped one of the 8 drives and another one got disconnected
because of that?
This doesn't seem to happen on my setup. I have a mainboard-integrated
LSI-1068E and kernel 2.6.24. I only have 4 drives connected and they all
belong to "port-5:x" (x is different for each drive) I didn't try to
swap all of them, but I definitely tried swapping one, and no other
drive was disconnected because of this. The other drives were part of an
md-raid, it would have been disastrous if it happened.
What kernel version do you have?
The following sentence is not clear to me:
"then I added or swapped another drive on the same port ( 4 SAS channels
per port on the external HBAs)"
what is the "external HBAs"? Is that an expander?
Also... are you sure the problem was not maybe due to the udev rule?
Sorry I don't know udev, I don't know if it even has the power to
unmount a drive...
Thank you
A.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
��.n��������+%������w��{.n�����{������ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
