On Tue, 2017-09-26 at 06:51 +0800, Ming Lei wrote: > On Mon, Sep 25, 2017 at 04:20:20PM +0000, Bart Van Assche wrote: > > So that means that you have not noticed that it is safe to leave out that > > smp_rmp() call because blk-mq queue freezing and unfreezing waits for a grace > > period and hence waits until all CPUs have executed a full memory barrier? > > No, memory barrier has to be pair, this barrier has to be there, another > one before unfreeze/freeze can be removed because it is implied inside > freeze/unfreeze. > > Actually it should have been smp_mb() between writing the percpu-refcount > and reading preempt_only flag, otherwise if the two op is reordered, > queue freeze/unfreeze may see the counter is zero, and this normal I/O > still can be observed even after queue is freezed and SCSI is put into > quiesced. That's wrong. Both memory barriers can be left out. Freezing + unfreezing a block layer queue causes call_rcu() to be invoked and also makes the code that performs the freezing + unfreezing to wait until a grace period has occurred on all CPUs. In other words, until all CPUs have performed a full memory barrier. This is why both memory barriers can be left out. A quote from "The RCU-barrier menagerie" (The RCU-barrier menagerie): "There are many issues with excessive use of memory barriers, one of which is their overhead. It turns out that RCU can be substituted for memory barriers in a great many cases, and this substitution allows one of the memory barriers of the pair to be removed. Unfortunately, in most cases, the other memory barrier must be replaced by an expensive grace-period-wait primitive such as synchronize_rcu(), but there are a few special cases where synchronize_rcu() is not needed." Bart.
