On Mon, 2021-01-18 at 19:08 +0800, lixiaokeng wrote: > Hi > When we make IO stress test on multipath device, there will > be a metadata err because of wrong path. > > > Sequence of events: > (1)multipath -r， ip1 logout at same > The load table params of 36001405ca5165367d67447ea68108e1d is > "0 1 alua 1 1 service-time 0 1 1 8:128 1"（The reason no 128 may > be not long after ip2 login and path_discovery doesn't find sde). > However, domap failed because ip1 logout. The path of sdi is > still in gvecs->pathvec. verify_paths() would detect this. We do call verify_paths() in coalesce_paths() before calling domap(), but not immediately before. Perhaps we should move the verify_paths() call down to immediately before the domap() call. That would at least minimize the time window for this race. It's hard to avoid it entirely. The way multipathd is written, the vecs lock is held all the time during coalesce_paths(), and thus no uevents can be processed. We could also consider calling verify_paths() before *and* after domap(). Was this a map creation or a map reload? Was the map removed after the failure? Do you observe the message "ignoring map" or "removing map"? Do you observe a "remove" uevent for sdi? > > (2) multipathd add path sde > The load table params of 36001405ca5165367d67447ea68108e1d is > "0 1 alua 2 1 service-time 0 1 1 8:64 1 service-time 0 1 1 8:128 " > and domap successes. > At this time, 36001405ca5165367d67447ea68108e1d has two path (sde, > sdi), > but sdi is actually the path of 36001405b7679bd96b094bccbf971bc90. > > (3) metadata of 36001405ca5165367d67447ea68108e1d sync > The metadata of 36001405b7679bd96b094bccbf971bc90 will be covered Well, I suppose the "add" event for sde might come before the remove event for sdi (if any). I wonder if you'd see the issue also if you run the same test without the "multipath -F; multipath -r" loop, or with just one. Ok, one multipath_query() loop simulates an admin working on the system, but 2 parallel loops - 2 admins working in parallel, plus the intensive sequence of actions done in multipathd_query at the same time? The repeated "multipath -r" calls and multipathd commands will cause multipathd to spend a lot of time in reconfigure() and in cli_* calls holding the vecs lock, which makes it likely that uevents are missed or processed late. Don't get me wrong, I don't argue against tough testing. But we should be aware that there are always time intervals during which multipathd's picture of the present devices is different from what the kernel sees. There's definitely room for improvement in multipathd wrt locking and event processing in general, but that's a BIG piece of work. Thanks Martin -- dm-devel mailing list dm-devel@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/dm-devel
