On 08/19/2019 03:36 PM, Ilya Dryomov wrote:
On Mon, Jul 29, 2019 at 11:43 AM Dongsheng Yang
<dongsheng.yang@xxxxxxxxxxxx> wrote:
Currently, if we share osdc in rbd device and journaling, they are
sharing the notify_wq in osdc to complete watch_cb. When we
need to close journal held with mutex of rbd device, we need
to flush the notify_wq. But we don't want to flush the watch_cb
of rbd_device, maybe some of it need to lock rbd mutex.
To solve this problem, this patch allow user to manage the notify
workqueue by themselves in watching.
What do you mean by "mutex of rbd device", rbd_dev->header_rwsem?
Did you actually encounter the resulting deadlock in testing?
Yes this patch is solving a real problem I met. But I wrote an uncorrect
commit message.
That's ->lock_rwsem, not mutex. The problem is if we dont have a special
notify_wq for journaler,
we have to call ceph_osdc_flush_notifies(journaler->osdc) to flush
notifies in ceph_journaler_close().
As journaler and rbd_device are shareing notify_wq in the same osdc,
this function will flush other
notifies to call rbd_watch_cb(), some of them need ->lock_rwsem.
There is a simple reproduce method:
(1) start a fio to write /dev/rbd0
(2) in the same time, run a script to add snapshot for this image:
for i in `seq 1 10`; do
rbd snap add test@snap$i &
done
So when there is another RBD_NOTIFY_OP_REQUEST_LOCK in notify_wq when we
are doing
ceph_osdc_flush_notifies(journaler->osdc). This flush will never finish.
[1]rbd_dev->task_wq [2]osdc->notify_wq
rbd_release_lock_work
- down_write(&rbd_dev->lock_rwsem);
rbd_watch_cb
- rbd_handle_request_lock
--
down_read(&rbd_dev->lock_rwsem); <-- this need to wait thread [1] to
release lock_rwsem
-- rbd_dev_close_journal()
--- ceph_osdc_flush_notifies() <-- this need to wait thread[2] to complete
Thanx
Thanks,
Ilya
