The OSDs will optionally take a "timeout" parameter on the watch request [1][2]. However, the kernel doesn't have this timeout field in its watch op [3] so perhaps it's defaulting to a random value. Ilya? [1] https://github.com/ceph/ceph/blob/v12.2.2/src/osd/PrimaryLogPG.cc#L6034 [2] https://github.com/ceph/ceph/blob/v12.2.2/src/include/rados.h#L577 [3] https://github.com/torvalds/linux/blob/v4.14/include/linux/ceph/rados.h#L487 On Wed, Dec 20, 2017 at 12:20 PM, Serguei Bezverkhi (sbezverk) <sbezverk@xxxxxxxxx> wrote: > It took 30 minutes for the Watcher to time out after ungraceful restart. Is there a way limit it to something a bit more reasonable? Like 1-3 minutes? > > On 2017-12-20, 12:01 PM, "Serguei Bezverkhi (sbezverk)" <sbezverk@xxxxxxxxx> wrote: > > Ok, here is what I found out. If I gracefully kill a pod then watcher gets properly cleared, but if it is done ungracefully, without “rbd unmap” then even after a node reboot Watcher stays up for a long time, it has been more than 20 minutes and it is still active (no any kubernetes services are running). > > I was wondering if you would accept the following solution. If in rbdStatus instead of checking just for watcher, we check for existence of /dev/rbd/{pool}/{image}. If it is not there, it would mean stale Watcher and it is safe to map this image. Appreciate your thoughts here. > > Thank you > Serguei > > On 2017-12-20, 11:32 AM, "Serguei Bezverkhi (sbezverk)" <sbezverk@xxxxxxxxx> wrote: > > > > On 2017-12-20, 11:17 AM, "Jason Dillaman" <jdillama@xxxxxxxxxx> wrote: > > On Wed, Dec 20, 2017 at 11:01 AM, Serguei Bezverkhi (sbezverk) > <sbezverk@xxxxxxxxx> wrote: > > Hello Jason, thank you for your prompt reply. > > > > My setup is very simple, I have 1 Centos 7.4 VM which is a storage node which is running latest 12.2.2 Luminous and 2nd VM is Ubuntu 16.04.3 192.168.80.235 where I run local kubernetes cluster based on the master. > > > > On client side I have ceph-common installed and I copied to /etc/ceph config and key rings from the storage. > > > > While running my PR I noticed that rmd map was failing on a just rebooted VM because rbdStatus was finding active Watcher. Even adding 30 seconds did not help as it was not timing out at all even with no any image mapping. > > OK -- but how did you get two different watchers listed? That implies > the first one timed out at some point in time. Does the watcher > eventually go away if you shut down all k8s processes on > > I cannot answer why there are two different watchers, I was just capturing info and until you pointed as I was not aware of that. I just checked VM and finally Watcher timed out. I cannot say how long it took, but I will run another set of tests to find out. > > 192.168.80.235? Are you overriding the "osd_client_watch_timeout" > configuration setting somewhere on the OSD host? > > No, no changes to default values were done. > > > As per your format 1 comment, I tried using format v2 and it was failing to map due to differences in capabilities as per rootfs suggestion I switched back to v1. Once Watcher issue is resolved I can switch back to v2 to show the exact issue I hit. > > > > Please let me know if you need any additional info. > > > > Thank you > > Serguei > > > > On 2017-12-20, 10:39 AM, "Jason Dillaman" <jdillama@xxxxxxxxxx> wrote: > > > > Can you please provide steps to repeat this scenario? What is/was the > > client running on the host at 192.168.80.235 and how did you shut down > > that client? In your PR [1], it showed a different client as a watcher > > ("192.168.80.235:0/34739158 client.64354 cookie=1"), so how did the > > previous entry get cleaned up? > > > > BTW -- unrelated, but k8s should be creating RBD image format 2 images > > [2]. Was that image created using an older version of k8s or did you > > override your settings to pick the deprecated v1 format? > > > > [1] https://github.com/kubernetes/kubernetes/pull/56651#issuecomment-352850884 > > [2] https://github.com/kubernetes/kubernetes/pull/51574 > > > > On Wed, Dec 20, 2017 at 10:24 AM, Serguei Bezverkhi (sbezverk) > > <sbezverk@xxxxxxxxx> wrote: > > > Hello, > > > > > > I hit an issue with latest Luminous when a Watcher is not timing out when the image is not mapped. It seems something similar was reported in 2016, here is the link: > > > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-August/012140.html > > > Has it been fixed? Appreciate some help here. > > > Thank you > > > Serguei > > > > > > date; sudo rbd status raw-volume --pool kubernetes > > > Wed Dec 20 10:04:19 EST 2017 > > > Watchers: > > > watcher=192.168.80.235:0/3789045165 client.64439 cookie=1 > > > date; sudo rbd status raw-volume --pool kubernetes > > > Wed Dec 20 10:04:51 EST 2017 > > > Watchers: > > > watcher=192.168.80.235:0/3789045165 client.64439 cookie=1 > > > date; sudo rbd status raw-volume --pool kubernetes > > > Wed Dec 20 10:05:14 EST 2017 > > > Watchers: > > > watcher=192.168.80.235:0/3789045165 client.64439 cookie=1 > > > > > > date; sudo rbd status raw-volume --pool kubernetes > > > Wed Dec 20 10:07:24 EST 2017 > > > Watchers: > > > watcher=192.168.80.235:0/3789045165 client.64439 cookie=1 > > > > > > sudo ls /dev/rbd* > > > ls: cannot access '/dev/rbd*': No such file or directory > > > > > > sudo rbd info raw-volume --pool kubernetes > > > rbd image 'raw-volume': > > > size 10240 MB in 2560 objects > > > order 22 (4096 kB objects) > > > block_name_prefix: rb.0.fafa.625558ec > > > format: 1 > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@xxxxxxxxxxxxxx > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > -- > > Jason > > > > > > > > -- > Jason > > > > > > -- Jason _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |