On Wed, Apr 07, 2021 at 02:53:00PM +0800, Hillf Danton wrote:
> On Tue, 6 Apr 2021 Dan Schatzberg wrote:
> >On Sat, Apr 03, 2021 at 10:09:02AM +0800, Hillf Danton wrote:
> >> On Fri, 2 Apr 2021 12:16:32 Dan Schatzberg wrote:
> >> > +queue_work:
> >> > + if (worker) {
> >> > + /*
> >> > + * We need to remove from the idle list here while
> >> > + * holding the lock so that the idle timer doesn't
> >> > + * free the worker
> >> > + */
> >> > + if (!list_empty(&worker->idle_list))
> >> > + list_del_init(&worker->idle_list);
> >>
> >> Nit, only queue work if the worker is inactive - otherwise it is taking
> >> care of the cmd_list.
> >
> >By worker is inactive, you mean worker is on the idle_list? Yes, I
> >think you're right that queue_work() is unnecessary in that case since
> >each worker checks empty cmd_list then adds itself to idle_list under
> >the lock.
A couple other corner cases - When worker is just allocated, it needs
a queue_work() and rootcg always needs a queue_work() since it never
sits on the idle_list. It does add to the logic a bit rather than just
unconditionally invoking queue_work()
> >
> >>
> >> > + work = &worker->work;
> >> > + cmd_list = &worker->cmd_list;
> >> > + } else {
> >> > + work = &lo->rootcg_work;
> >> > + cmd_list = &lo->rootcg_cmd_list;
> >> > + }
> >> > + list_add_tail(&cmd->list_entry, cmd_list);
> >> > + queue_work(lo->workqueue, work);
> >> > + spin_unlock_irq(&lo->lo_work_lock);
> >> > }
> >> [...]
> >> > + /*
> >> > + * We only add to the idle list if there are no pending cmds
> >> > + * *and* the worker will not run again which ensures that it
> >> > + * is safe to free any worker on the idle list
> >> > + */
> >> > + if (worker && !work_pending(&worker->work)) {
> >>
> >> The empty cmd_list is a good enough reason for worker to become idle.
> >
> >This is only true with the above change to avoid a gratuitous
> >queue_work(), right?
>
> It is always true because of the empty cmd_list - the idle_list is the only
> place for the worker to go at this point.
>
> >Otherwise we run the risk of freeing a worker
> >concurrently with loop_process_work() being invoked.
>
> My suggestion is a minor optimization at most without any change to removing
> worker off the idle_list on queuing work - that cuts the risk for you.
If I just change this line from
if (worker && !work_pending(&worker->work)) {
to
if (worker) {
then the following sequence of events is possible:
1) loop_queue_work runs, adds a command to the worker list
2) loop_process_work runs, processes a single command and then drops
the lock and reschedules
3) loop_queue_work runs again, acquires the lock, adds to the list and
invokes queue_work() again
4) loop_process_work resumes, acquires lock, processes work, notices
list is empty and adds itself to the idle_list
5) idle timer fires and frees the worker
6) loop_process_work runs again (because of the queue_work in 3) and
accesses freed memory
The !work_pending... check prevents 4) from adding itself to the
idle_list so this is not possible. I believe we can only make this
change if we also make the other change you suggested to avoid
gratuitous queue_work()
