On Mon, 2011-05-02 at 22:04 -0700, Malahal Naineni wrote: > John A. Sullivan III [jsullivan@xxxxxxxxxxxxxxxxxxx] wrote: > > I'm also very curious about your findings on rr_min_io. I cannot find > > my benchmarks but we tested various settings heavily. I do not recall > > if we saw more even scaling with 10 or 100. I remember being surprised > > that performance with it set to 1 was poor. I would have thought that, > > in a bonded environment, changing paths per iSCSI command would give > > optimal performance. Can anyone explain why it does not? > > rr_min_io of 1 will give poor performance if your multipath kernel > module doesn't support request based multipath. In those BIO based > multipath, multipath receives 4KB requests. Such requests can't be > coalesced if they are sent on different paths. <snip> Ah, that makes perfect sense and why 3 seems to be the magic number in Linux (4000 / 1460 (or whatever IP payload is)). Does that change with Jumbo frames? In fact, how would that be optimized in Linux? 9KB seems to be a reasonable common jumbo frame value for various vendors and that should contain two pages but, I would guess, Linux can't utilize it as each block must be independently acknowledged. Is that correct? Thus a frame size of a little over 4KB would be optimal for Linux? Would that mean that rr_min_io of 1 would become optimal? However, if each block needs to be acknowledged before the next is sent, I would think we are still latency bound, i.e., even if I can send four requests down four separate paths, I cannot send the second until the first has been acknowledged and since I can easily place four packets on the same path within the latency period of four packets, multibus gives me absolutely no performance advantage for a single iSCSI stream and only proves useful as I start multiplexing multiple iSCSI streams. Is that analysis correct? If so, what constitutes a separate iSCSI stream? Are two separate file requests from the same file systems to the same iSCSI device considered two iSCSI streams and thus can be multiplexed and benefit from multipath or are they considered all part of the same iSCSI stream? If they are considered one, do they become two if they reside on different partitions and thus different file systems? If not, then do we only see multibus performance gains between a single file system host and a single iSCSI host when we use virtualization each with their own iSCSI connection (as opposed to using iSCSI connections in the underlying host and exposing them to the virtual machines as local storage)? I hope I'm not hijacking this thread and realize I've asked some convoluted questions but optimizing multibus through bonded links for single large hosts is still a bit of a mystery to me. Thanks - John -- dm-devel mailing list dm-devel@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/dm-devel
