On Tue, 13 Apr 2021 20:24:06 +0200 Eric Farman <farman@xxxxxxxxxxxxx> wrote: > Hi Conny, Halil, > > Let's restart our discussion about the collision between interrupts for > START SUBCHANNEL and HALT/CLEAR SUBCHANNEL. It's been a quarter million > minutes (give or take), so here is the problematic scenario again: > > CPU 1 CPU 2 > 1 CLEAR SUBCHANNEL > 2 fsm_irq() > 3 START SUBCHANNEL > 4 vfio_ccw_sch_io_todo() > 5 fsm_irq() > 6 vfio_ccw_sch_io_todo() > > From the channel subsystem's point of view the CLEAR SUBCHANNEL (step 1) > is complete once step 2 is called, as the Interrupt Response Block (IRB) > has been presented and the TEST SUBCHANNEL was driven by the cio layer. > Thus, the START SUBCHANNEL (step 3) is submitted [1] and gets a cc=0 to > indicate the I/O was accepted. However, step 2 stacks the bulk of the > actual work onto a workqueue for when the subchannel lock is NOT held, > and is unqueued at step 4. That code misidentifies the data in the IRB > as being associated with the newly active I/O, and may release memory > that is actively in use by the channel subsystem and/or device. Eww. > > In this version... > > Patch 1 and 2 are defensive checks. Patch 2 was part of v3 [2], but I > would love a better option here to guard between steps 2 and 4. > > Patch 3 is a subset of the removal of the CP_PENDING FSM state in v3. > I've obviously gone away from this idea, but I thought this piece is > still valuable. > > Patch 4 collapses the code on the interrupt path so that changes to > the FSM state and the channel_program struct are handled at the same > point, rather than separated by a mutex boundary. Because of the > possibility of a START and HALT/CLEAR running concurrently, it does > not make sense to split them here. > > With the above patches, maybe it then makes sense to hold the io_mutex > across the entirety of vfio_ccw_sch_io_todo(). But I'm not completely > sure that would be acceptable. > > So... Thoughts? I believe we should address the concurrency, encapsulation and layering issues in the subchannel/ccw pass-through code (vfio-ccw) by taking a holistic approach as soon as possible. I find the current state of art very hard to reason about, and that adversely affects my ability to reason about attempts at partial improvements. I understand that such a holistic approach needs a lot of work, and we may have to stop some bleeding first. In the stop the bleeding phase we can take a pragmatic approach and accept changes that empirically seem to work towards stopping the bleeding. I.e. if your tests say it's better, I'm willing to accept that it is better. I have to admit, I don't understand how synchronization is done in the vfio-ccw kernel module (in the sense of avoiding data races). Regarding your patches, I have to admit, I have a hard time figuring out which one of these (or what combination of them) is supposed to solve the problem you described above. If I had to guess, I would guess it is either patch 4, because it has a similar scenario diagram in the commit message like the one in the problem statement. Is my guess right? If it is right I don't quite understand the mechanics of the fix, because what the patch seems to do is changing the content of step 4 in the above diagram. And I don't see how is change that code so that it does not "misidentifies the data in the IRB as being associated with the newly active I/O". Moreover patch 4 seems to rely on private->state which, AFAIR is still used in a racy fashion. But if strong empirical evidence shows that it performs better (stops the bleeding), I think we can go ahead with it. Regards, Halil
