Malahal, After your mentioning bio vs request based I attempted to determine if my kernel contains the request based mpath. It seems in 2.6.31 all mpath was switched to request based. I have a kernel 2.6.31+ (actually .35 and .38), so I believe I have requrest-based mpath. All, There also appears to be a new multipath configuration option documented in the RHEL 6 beta documentation: rr_min_io_rq Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1. http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6-Beta/html/DM_Multipath/config_file_multipath.html I have not tested using this setting vs rr_min_io yet or even if my system supports the configuration directive. If I trust some of the claims of several VMware ESX iscsi multipath setups, it is possible (possibly using different software) to gain a multiplicative throughput by adding additional Ethernet links. This makes me hopeful that we can do this with open-iscsi and dm-mulitpath as well. It could be something obvious I am missing, but it appears a lot of people experience this same issue. Thanks, Adam On Tue, May 3, 2011 at 6:12 AM, John A. Sullivan III <jsullivan@xxxxxxxxxxxxxxxxxxx> wrote: > On Mon, 2011-05-02 at 22:04 -0700, Malahal Naineni wrote: >> John A. Sullivan III [jsullivan@xxxxxxxxxxxxxxxxxxx] wrote: >> > I'm also very curious about your findings on rr_min_io. I cannot find >> > my benchmarks but we tested various settings heavily. I do not recall >> > if we saw more even scaling with 10 or 100. I remember being surprised >> > that performance with it set to 1 was poor. I would have thought that, >> > in a bonded environment, changing paths per iSCSI command would give >> > optimal performance. Can anyone explain why it does not? >> >> rr_min_io of 1 will give poor performance if your multipath kernel >> module doesn't support request based multipath. In those BIO based >> multipath, multipath receives 4KB requests. Such requests can't be >> coalesced if they are sent on different paths. > <snip> > Ah, that makes perfect sense and why 3 seems to be the magic number in > Linux (4000 / 1460 (or whatever IP payload is)). Does that change with > Jumbo frames? In fact, how would that be optimized in Linux? > > 9KB seems to be a reasonable common jumbo frame value for various > vendors and that should contain two pages but, I would guess, Linux > can't utilize it as each block must be independently acknowledged. Is > that correct? Thus a frame size of a little over 4KB would be optimal > for Linux? > > Would that mean that rr_min_io of 1 would become optimal? However, if > each block needs to be acknowledged before the next is sent, I would > think we are still latency bound, i.e., even if I can send four requests > down four separate paths, I cannot send the second until the first has > been acknowledged and since I can easily place four packets on the same > path within the latency period of four packets, multibus gives me > absolutely no performance advantage for a single iSCSI stream and only > proves useful as I start multiplexing multiple iSCSI streams. > > Is that analysis correct? If so, what constitutes a separate iSCSI > stream? Are two separate file requests from the same file systems to the > same iSCSI device considered two iSCSI streams and thus can be > multiplexed and benefit from multipath or are they considered all part > of the same iSCSI stream? If they are considered one, do they become two > if they reside on different partitions and thus different file systems? > If not, then do we only see multibus performance gains between a single > file system host and a single iSCSI host when we use virtualization each > with their own iSCSI connection (as opposed to using iSCSI connections > in the underlying host and exposing them to the virtual machines as > local storage)? > > I hope I'm not hijacking this thread and realize I've asked some > convoluted questions but optimizing multibus through bonded links for > single large hosts is still a bit of a mystery to me. Thanks - John > > -- > dm-devel mailing list > dm-devel@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/dm-devel > -- dm-devel mailing list dm-devel@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/dm-devel
