On 2004-10-29T23:23:02, Alasdair G Kergon <agk@xxxxxxxxxx> wrote: Hi Alasdair and list members, I'd like to revisit the discussion on the "bypassed" flag to priority groups. I've come to the conclusion that this is a suboptimal and non-elegant way of reaching the desired goal and would like to hear your comments on these thoughts. (Which came while thinking how to configure this from user-space.) First, lets start with the goal definition. What motivated the bypassed flag? The idea was (at least on my side) to minimize unnecessary switches between priority groups by the kernel itself, as this might have undesireable side effects. (Performance or affecting other nodes.) In short, to move the policy decision of when to switch the priority group out of the kernel to user-space. The only time when the kernel was 'authorized' to switch PGs internally was in response to all paths failing or when the PG was "pulled away" from under us (in a non-fatal condition). In short, when immediate and unavoidable action was necessary to keep the IO moving. Admins may have different policies for controlling the switching of PGs. In a cluster, it may need to be coordinated across multiple nodes (either by actual communication or via the timing based model I've been discussing on the list). On a single node even, we may want to stick to the PG which has the most healthy paths, both for redundancy and/or load balancing. Or we may want to switch back to some PG immediately right away as soon as it becomes available again, because it has 10GB/s FC and the other one is on a 1GB/s link. I believe such policy decisions would complicate the kernel too much. Now, why do I think the bypassed flag isn't the way to go? - It makes switching PGs icky from user-space. If you want the kernel to switch to some PG, you need to set bypassed on all others and clear it on just that one. -> excessive communication. - It makes error recovery in the kernel annoying. When we _must_ switch PGs in response to errors, we'd eventually ignore the bypassed flag anyway and switch to the one which still has healthy paths, so why have it in the first place? - User-space can't tell us right now which PGs it might want bypassed when loading the table in the first place, but it has to send additional messages afterwards, when it might already be too late. (Unless we want user-space to have to sort the table differently on each load, which adds unnecessary complexity and also means that admins using dmsetup status to look at it will have to pay more attention, which they won't.) - Following Alasdair's description of how table loads should be very rare, and that the modus operandi should instead be changed by atomic messages, I think this can be implemented better. So, I believe the goals would be better served by making it easy for user-space to command us which PG to use, not having to flag which PGs _not_ to use, and having to clear those flags etc. (At least that is how my design sensors work. Yours may be different.) My proposal would thus be - drop the bypassed flag completely. - Allow user-space to send us a "switch_pg" message, specifying the number of the PG to switch to instead. - Report the number of the active PG in the status report. (Random remark: Maybe a good idea to have a multipath -q command to query the state of the multipath device in some more human readable format.) - To keep the mapping itself stable, allow user-space to specify the no of the PG which should be active initially at table load time. - The kernel keeps track of the "current_pg" already and never gratuitiously changes PGs. Only in response to errors (all paths failed), or when the error handler tells us to SWITCH_PG do we switch, and then we switch to the first other PG with healthy paths. And send an event to user-space to allow it to find out that we did, and then user-space can coordinate the switch-back at it's discretion. - (Actually, I briefly entertained the idea of querying user-space to tell us which "other" PG we should switch to, or figuring it out in some fancy way ourselves. But then it occured to me that in 95% of all scenarios, I believe we'll be dealing with exactly two PGs anyway, and thus there'll be just that other PG to switch to. If we eventually need more, we can always introduce a feature flag.) I believe this captures the goals better and will also be cleaner to implement. Comments? If you approve, I can do a patch for dm-mpath on Monday. Sincerely, Lars Marowsky-Brée <lmb@xxxxxxx> -- High Availability & Clustering SUSE Labs, Research and Development SUSE LINUX Products GmbH - A Novell Business