On Mon, 13 May 2013, Mandell Degerness wrote: > Sorry. I should have mentioned, this is using the bobtail version of ceph. > > On Mon, May 13, 2013 at 1:13 PM, Mandell Degerness > <mandell@xxxxxxxxxxxxxxx> wrote: > > I know that there was another report of the bad behavior when deleting > > an RBD that is currently mounted on a host. My problem is related, > > but slightly different. > > > > We are using openstack and Grizzly Cinder to create a bootable ceph > > volume. The instance was booted and all was well. The server on > > which the instance had been booted was unplugged. The user deleted > > the instance - which amounts to a database update on the Nova side. > > They then tried to delete the volume, which failed with the following > > error: > > > > Traceback (most recent call last): > > 21262 File "/usr/lib64/python2.7/site-packages/cinder/volume/driver.py", > > line 90, in _try_execute > > 21263 self._execute(*command, **kwargs) > > 21264 File "/usr/lib64/python2.7/site-packages/cinder/utils.py", > > line 190, in execute > > 21265 cmd=' '.join(cmd)) > > 21266 ProcessExecutionError: Unexpected error while running command. > > 21267 Command: rbd rm --pool rbd volume-66e11621-1c38-4e2d-9d90-cc511013c290 > > 21268 Exit code: 16 > > 21269 Stdout: '\rRemoving image: 1% complete...\rRemoving image: 2% > > complete...\rRemoving image: 3% complete...\rRemoving image: 4% > > complete...\rRemoving image: 5% complete ...\rRemoving image: 6% > > complete...\rRemoving image: 7% complete...\rRemoving image: 8% > > complete...\rRemoving image: 9% complete...\rRemoving image: 10% > > complete...\r Removing image: 11% complete...\rRemoving image: > > 12% complete...\rRemoving image: 13% complete...\rRemoving image: 14% > > complete...\rRemoving image: 15% complete...\rR emoving image: > > 16% complete...\rRemoving image: 17% complete...\rRemoving image: 18% > > complete...\rRemoving image: 19% complete...\rRemoving image: 20% > > complete...\rRe moving image: 21% complete...\rRemoving image: > > 22% complete...\rRemoving image: 23% complete...\rRemoving image: 24% > > complete...\rRemoving image: 25% complete...\rRem oving image: > > 26% complete...\rRemoving image: 27% complete...\rRemoving image: 28% > > complete...\rRemoving image: 29% complete...\rRemoving image: 30% > > complete...\rRemo ving image: 31% complete...\rRemoving image: > > 32% complete...\rRemoving image: 33% complete...\rRemoving image: 34% > > complete...\rRemoving image: 35% complete...\rRemov ing image: > > 36% complete...\rRemoving image: 37% complete...\rRemoving image: 38% > > complete...\rRemoving image: 39% complete...\rRemoving image: 40% > > complete...\rRemovi ng image: 41% complete...\rRemoving image: > > 42% complete...\rRemoving image: 43% complete...\rRemoving image: 44% > > complete...\rRemoving image: 45% complete...\rRemovin g image: > > 46% complete...\rRemoving image: 47% complete...\rRemoving image: 48% > > complete...\rRemoving image: 49% complete...\rRemoving image: 50% > > complete...\rRemoving image: 51% complete...\rRemoving image: > > 52% complete...\rRemoving image: 53% complete...\rRemoving image: 54% > > complete...\rRemoving image: 55% complete...\rRemoving image: > > 56% complete...\rRemoving image: 57% complete...\rRemoving image: 58% > > complete...\rRemoving image: 59% complete...\rRemoving image: 60% > > complete...\rRemoving i mage: 61% complete...\rRemoving image: > > 62% complete...\rRemoving image: 63% complete...\rRemoving image: 64% > > complete...\rRemoving image: 65% complete...\rRemoving im age: > > 66% complete...\rRemoving image: 67% complete...\rRemoving image: 68% > > complete...\rRemoving image: 69% complete...\rRemoving image: 70% > > complete...\rRemoving ima ge: 71% complete...\rRemoving image: > > 72% complete...\rRemoving image: 73% complete...\rRemoving image: 74% > > complete...\rRemoving image: 75% complete...\rRemoving imag e: > > 76% complete...\rRemoving image: 77% complete...\rRemoving image: 78% > > complete...\rRemoving image: 79% complete...\rRemoving image: 80% > > complete...\rRemoving image : 81% complete...\rRemoving image: > > 82% complete...\rRemoving image: 83% complete...\rRemoving image: 84% > > complete...\rRemoving image: 85% complete...\rRemoving image: > > 86% complete...\rRemoving image: 87% complete...\rRemoving image: 88% > > complete...\rRemoving image: 89% complete...\rRemoving image: 90% > > complete...\rRemoving image: 91% complete...\rRemoving image: > > 92% complete...\rRemoving image: 93% complete...\rRemoving image: 94% > > complete...\rRemoving image: 95% complete...\rRemoving image: 9 > > 6% complete...\rRemoving image: 97% complete...\rRemoving image: 98% > > complete...\rRemoving image: 99% complete...\rRemoving image: 99% > > complete...failed.\n' > > 21270 Stderr: 'rbd: error: image still has watchers\nThis means the > > image is still open or the client using it crashed. Try again after > > closing/unmapping it or waiting 30s for the crashed client to > > timeout.\n2013-05-09 21:51:27.522986 7f8aca884780 -1 librbd: error > > removing header: (16) Device or resource busy\n' > > > > It appears to me that Ceph still believes the volume is mounted > > somewhere. Is there a way to tell Ceph to delete the RBD, in spite of > > it's belief that it is mounted? Currently you just have to wait >30 seconds and then try again. sage _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |