On 08/18/2009 04:14 PM, James Bottomley wrote:
On Tue, 2009-08-18 at 13:44 -0400, Ric Wheeler wrote:
On 08/18/2009 12:09 PM, James Bottomley wrote:
On Tue, 2009-08-18 at 11:53 -0400, Ric Wheeler wrote:
On 08/18/2009 10:57 AM, Greg KH wrote:
On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
On 08/18/2009 09:25 AM, Greg KH wrote:
On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
We have a new toy to test very large& slow storage with built up from 5
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
adapter daughter card in the disk sled).
The basic idea is to build a cheap& slow test bed for file& storage
system scalability. Collectively, we have about 120TB (raw) of capacity
to play with in one server.
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long"
like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too long
Odd, what is the sysfs tree for this device? You have expanders
attached to ports attached to expanders? How deep can you go?
thanks,
greg k-h
There are two dual-port SAS HBA's in the server that plug into the first of the
5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy
chained to the next shelf.... This test is running with just the first 4
shelves active (although that fifth shelf is plugged in, just not used/active).
Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both
loops.
We could break up the shelves into two independent sets of devices which would
limit the tree depth.
How can I get you the sysfs tree information in a useful way?
'tree /sys/devices/'
or
'find /sys/devices/'
would be good.
thanks,
greg k-h
Attached is the bzipped output from - the uncompressed output is quite
large. Note that this same server has CCIS controllers and fibre HBA's
as well,
It's perfectly legal the way you have it, but I will say you have an
inefficient configuration. The way a configuration like this is
supposed to look is that there should be a fanout expander at the top
going to the expander in each tray, for a routing depth of two for every
device.
The way you've got it: expander daisy chained off the next expander
gives an unnecessary routing delay to the disks furthest away in the
chain.
James
I understand that this configuration is not optimal, but from what I saw with
some commercial arrays, this is not an uncommon config (up to 4 shelves) when
going for capacity over performance.
The config is fine ... it's the way the daisy chain routing is done
which isn't. You can see that each shelf gets further away from the HBA
by an expander as you go up.
Do you have a particular SAS fanout expander in mind? I suppose that we could
always add more hba's as well which would have other benefits...
Not really ... I've never seen a real expander in the flesh. I've got a
set of experimental ones LSI gave me (as bare circuit boards).
If you don't have the expanders, you can likely rig the first expander
to act as a fanout since it must have table routed ports otherwise it
wouldn't work in the daisy chain.
Failing that, as you suggest, multiple HBAs subbing for the fanout
expander would be fine as well.
James
Adding more HBA's (one per shelf or pair of shelves) would probably be the
easiest way around this....
Thanks!
Ric
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
