Hi Jason, I can reproduce the issue 100% Use standard ceph version 9.2.1 from repository Create a vm rbd version 2, in my example it is: vm-192-disk-1 enable these features: rbd feature enable $IMG exclusive-lock rbd feature enable $IMG object-map rbd feature enable $IMG fast-diff start the vm and inside the vm run some io, i ran bonnie++ in a loop then go ahead and create first snapshot /usr/bin/rbd snap create rbd/vm-192-disk-1@initial.20160321-130439 export the snapshot (don't know if it is necessary) /usr/bin/rbd export --rbd-concurrent-management-ops 20 vm-192-disk-1@initial.20160321-130439 -|pigz -b 512|/bin/dd of=/backups/ceph/vm-192-disk-1.initial.20160321-130439.gz.tmp && /bin/mv /backups/ceph/vm-192-disk-1.initial.20160321-130439.gz.tmp /backups/ceph/vm-192-disk-1.initial.20160321-130439.gz this is no problem, it will work then create the second snapshot: /usr/bin/rbd snap create rbd/vm-192-disk-1@incremental.20160321-130741 after a few seconds you see on the console: 2016-03-21 13:08:46.091526 7f8ab372a7c0 -1 librbd::ImageWatcher: 0x561d8a394150 no lock owners detected So it is not the export diff that is hanging, it is the rbd snap create operation on an additional snapshot Often the io in the vm is also hanging and sometimes load in the vm goes up to 800 or more. Even after stopping the vm I can see the image has an exclusive lock: # rbd lock ls vm-192-disk-1 There is 1 exclusive lock on this image. Locker ID Address client.71565451 auto 140269345641344 10.67.1.15:0/2701777604 Without the image features i do not have these problems. Can you reproduce this ? Greetings Christoph On Sun, Mar 20, 2016 at 10:57:16AM -0400, Jason Dillaman wrote: > Definitely not a known issue but from a quick test (running export-diff against an image being actively written) I wasn't able to recreate on v9.2.1. Are you able to recreate this reliably, and if so, can you share the steps you used? > > Thanks, > > -- > > Jason Dillaman > > > ----- Original Message ----- > > From: "Christoph Adomeit" <Christoph.Adomeit@xxxxxxxxxxx> > > To: "Jason Dillaman" <dillaman@xxxxxxxxxx> > > Cc: ceph-users@xxxxxxxx > > Sent: Friday, March 18, 2016 6:19:16 AM > > Subject: Re: Does object map feature lock snapshots ? > > > > Hi, > > > > I had no special logging activated. > > > > Today I re-enabled exclusive-lock object-map and fast-diff on an image in > > 9.2.1 > > > > As soon as I ran an rbd export-diff I had lots of these error messages on the > > console of the rbd export process: > > > > 2016-03-18 11:18:21.546658 7f77245d1700 1 heartbeat_map is_healthy > > 'librbd::thread_pool thread 0x7f77137fe700' had timed out after 60 > > 2016-03-18 11:18:26.546750 7f77245d1700 1 heartbeat_map is_healthy > > 'librbd::thread_pool thread 0x7f77137fe700' had timed out after 60 > > 2016-03-18 11:18:31.546840 7f77245d1700 1 heartbeat_map is_healthy > > 'librbd::thread_pool thread 0x7f77137fe700' had timed out after 60 > > 2016-03-18 11:18:36.546928 7f77245d1700 1 heartbeat_map is_healthy > > 'librbd::thread_pool thread 0x7f77137fe700' had timed out after 60 > > 2016-03-18 11:18:41.547017 7f77245d1700 1 heartbeat_map is_healthy > > 'librbd::thread_pool thread 0x7f77137fe700' had timed out after 60 > > 2016-03-18 11:18:46.547105 7f77245d1700 1 heartbeat_map is_healthy > > 'librbd::thread_pool thread 0x7f77137fe700' had timed out after 60 > > > > > > Is this a known issue ? > > > > > > > > > > > > On Tue, Mar 08, 2016 at 11:22:17AM -0500, Jason Dillaman wrote: > > > Is there anyway for you to provide debug logs (i.e. debug rbd = 20) from > > > your rbd CLI and qemu process when you attempt to create a snapshot? In > > > v9.2.0, there was an issue [1] where the cache flush writeback from the > > > snap create request was being blocked when the exclusive lock feature was > > > enabled, but that should have been fixed in v9.2.1. > > > > > > [1] http://tracker.ceph.com/issues/14542 > > > > > > -- > > > > > > Jason Dillaman > > > > > > > > > ----- Original Message ----- > > > > From: "Christoph Adomeit" <Christoph.Adomeit@xxxxxxxxxxx> > > > > To: ceph-users@xxxxxxxx > > > > Sent: Tuesday, March 8, 2016 11:13:04 AM > > > > Subject: Does object map feature lock snapshots ? > > > > > > > > Hi, > > > > > > > > i have installed ceph 9.21 on proxmox with kernel 4.2.8-1-pve. > > > > > > > > Afterwards I have enabled the features: > > > > > > > > rbd feature enable $IMG exclusive-lock > > > > rbd feature enable $IMG object-map > > > > rbd feature enable $IMG fast-diff > > > > > > > > > > > > During the night I have a cronjob which does a rbd snap create on each > > > > of my images and then an rbd export-diff > > > > > > > > I found out that my cronjob was hanging during the rbd snap create and > > > > does not create the snapshot. > > > > > > > > Also more worse, sometimes also the vms were hanging. > > > > > > > > What are your experiences with object maps ? For me it looks that they > > > > are not yet production ready. > > > > > > > > Thanks > > > > Christoph > > > > _______________________________________________ > > > > ceph-users mailing list > > > > ceph-users@xxxxxxxxxxxxxx > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > -- Christoph Adomeit GATWORKS GmbH Reststrauch 191 41199 Moenchengladbach Sitz: Moenchengladbach Amtsgericht Moenchengladbach, HRB 6303 Geschaeftsfuehrer: Christoph Adomeit, Hans Wilhelm Terstappen Christoph.Adomeit@xxxxxxxxxxx Internetloesungen vom Feinsten Fon. +49 2166 9149-32 Fax. +49 2166 9149-10 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |