On 2017/9/19 3:51, Martin Wilck wrote: > On Mon, 2017-09-18 at 22:36 +0800, Guan Junxiong wrote: >> Hi Muneendra, >> >> Thanks for you feedback. My comments are incline below. >> >> On 2017/9/18 20:53, Muneendra Kumar M wrote: >>> Hi Guan, >>> This a good effort for detecting the intermittent IO error >>> accounting to improve reliability. >>> Your new algorithm is mutually exclusive with san_path_err_XXX. >>> It resolved the below issue which you have mentioned . >>>>> Even the san_path_err_threshold , san_path_err_forget_rate and >>>>> san_path_err_recovery_time is turned on, >>>>> the detect sample interval of that path checkers is so >>>>> big/coarse that it doesn't see what happens in the middle of >>>>> the sample interval. >>> >>> But I have some concerns. >>> >>> Correct me if my understanding on the below line is correct >>>>> On a particular path when a path failing events occur twice in >>>>> 60 second due to an IO error, multipathd will fail the path and >>>>> enqueue >>>>> this path into a queue of which each member is sent a couple of >>>>> continuous direct reading asynchronous io at a fixed sample >>>>> rate of 10HZ. >>> >>> Once we hit the above condition (2 errors in 60 secs) for a >>> path_io_err_sample_time we keeps on injecting the asynchronous io >>> at a fixed sample rate of 10HZ. >>> And during this path_io_err_sample_time if we hit the the >>> path_io_err_rate_threshold then we will not reinstantate this path >>> for a path_io_err_recovery_time. >>> Is this understanding correct? >>> >> >> Partial correct. >> If we hit the above condition (2 errors in 60 secs), we will fail the >> path first before injecting a couple of asynchronous IOs to keep the >> testing not affected by other IOs. >> And after this path_io_err_sample_time : >> (1) if we hit the the path_io_err_rate_threshold, the failed path >> will keep unchanged and then after the path_io_err_recovery_time >> (which is confusing, sorry, I will rename it to "recheck"), we will >> reschedule this IO error checking process again. >> (2) if we do NOT hit the path_io_err_rate_threshold, the failed path >> will reinstated by path checking thread in a tick (1 second) ASAP. >> >> >>> If the above understanding is correct then my concern is : >>> 1) On a particular path if we are seeing continuous errors but not >>> within 60 secs (may be for every 120 secs) of duration how do we >>> handle this. Still this a shaky link. >>> This is what our customers are pointing out. >>> And if i am not wrong the new algorithm will comes into place >>> only if a path failing events occur twice in 60 seconds. >>> >>> Then this will not solve the intermittent IO error issue which we >>> are seeing as the data is still going on the shaky path . >>> I think this is the place where we need to pull in in >>> san_path_err_forget_rate . >>> >> >> Yes . I have thought about using some adjustable parameters such as >> san_path_err_pre_check_time and san_path_err_threshold to cover ALL >> the scenarios the user encounters. >> In the above fixed example,san_path_err_pre_check_time is set to 60 >> seconds, san_path_err_threshold is set 2. >> However, if I adopt this, we have 5 parameters >> (san_path_err_pre_check_time and san_path_err_threshold + 3 >> path_io_err_XXXs ) to support this feature. You know, mulitpath.conf >> configuration is becoming more and more daunting as Martin pointed in >> the V1 of this patch. >> >> But now, maybe it is acceptable for users to use the 5 parameters if >> we set san_path_err_pre_check_time and san_path_err_threshold to >> some default values such as 60 second and 2 respectively. >> **Martin** , **Muneendra**, how about this a little compromising >> method? If it is OK , I will update in next version of patch. > > Hm, that sounds a lot like san_path_err_threshold and > san_path_err_forget_rate, which you were about to remove. > > Maybe we can simplify the algorithm by checking paths which fail in a > given time interval after they've been reinstated? That would be one > less additional parameter. > "san_path_double_fault_time" is great. One less additional parameter and still covering most scenarios are appreciated. > The big question is: how do administrators derive appropriate values > for these parameters for their environment? IIUC the values don't > depend on the storage array, but rather on the environment as a whole; > all kinds of things like switches, cabling, or even network load can > affect the behavior, so multipathd's hwtable will not help us provide > good defaults. Yet we have to assume that a very high percentage of > installations will just use default or vendor-recommended values. Even > if the documentation of the algorithm and its parameters was perfect > (which it currently isn't), most admins won't have a clue how to set > them. AFAICS we don't even have a test procedure to derive the optimal > settings experimentally, thus guesswork is going to be applied, with > questionable odds for success. > > IOW: the whole stuff is basically useless without good default values. > It would be up to you hardware guys to come up with them. > I agree. So let users to come up with those values. What we can do is to log the testing result such as path_io_err_rate in the given sample time. >> san_path_err_forget_rate is hard to understand, shall we use Regards >> san_path_err_pre_check_time > > A 'rate' would be something which is measured in Hz, which is not the > case here. Calling it a 'time' is more accurate. If we go with my > proposal above, we might call it "san_path_double_fault_time". > > Regards > Martin > Regards Guan -- dm-devel mailing list dm-devel@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/dm-devel
