On Fri, 2011-10-21 at 18:08 -0600, Michael Robbert wrote: > I want some technical advice from the SCSI/SAS/SATA experts. We are > trying to setup a low cost HA storage system with multiple servers > that have SAS HBAs, SAS JBOD, and desktop SATA disks. When we first > set it up everything appeared to work. I created RAID6 on one host and > put it into a corosync/pacemaker config. I was then able to migrate > the RAID from one host to another. A short while later a failover > failed and I noticed that some of the drives became inaccessible on > one of the hosts. The kernel was showing timeouts to the device. > > Oct 21 17:55:34 haraid-12-1 kernel: sd 1:0:3:0: timing out command, waited 10s > > So, my question is this, is this setup technically possible or are the > 2 HBAs going to conflict with each other when talking over the same > SAS bus to the SATA drives? Well, yes, but the devil is in the details. Firstly it's only possible with SAS; SATA controllers don't really do multi-initiator and SATA disks don't have the cluster commands that SAS disks do. It is theoretically possible with a SAS controller and SATA disk provided the cluster software doesn't use any of the SCSI commands for clustering. The reason your setup likely doesn't work is because of the expander. SAS expanders are complex beasts: Any port can be device, table or subtractively routed. You don't usually see it, but a JBOD ships with a subtractive port for the HBA connection and a device port for everything else. You can't just plug another HBA into a random device port because the routing won't get device replies back to you (hence the timeout). The way I've got this set up at home uses a 12x expander with 4x subtractive and 9x table routing phys which group in clusters of up to four for ports. I can plug the second HBA into one of the table routed ports because the SCSI transport class installs the correct route tables, so I know it works for aic9xxx and mvsas. James -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
