I'd imagine this is a side-effect of the fact that RBD keeps a watch/notify handle open on the header. Watch/notify is what allows you to manipulate an image from the rbd cli on a running image (i.e. take a snapshot, resize, etc). Even if the OSDs permitted this action, librbd would immediately race to re-establish the lost watch/notify connection before you could disassociate the cache tier. -- Jason Dillaman ----- Original Message ----- > From: "Robert LeBlanc" <robert@xxxxxxxxxxxxx> > To: ceph-users@xxxxxxxxxxxxxx > Sent: Monday, October 26, 2015 12:22:06 PM > Subject: Not possible to remove cache tier with RBDs open? > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > I'm trying to remove a cache tier while I have a VM running, but a few > of the objects will not evict. Do you have to stop all RBD access to > remove a cache tier? > > # ceph osd tier cache-mode ssd-pool forward > # rados -p ssd-pool cache-flush-evict-all > - --- lots of objects evicted, some failed with device or resource busy ----- > *** try again, just in case *** > # rados -p ssd-pool cache-try-flush-evict-all > rbd_header.2507f43faa7343 > failed to evict /rbd_header.2507f43faa7343: (16) Device or resource busy > rbd_header.2509da78c8b8fc > failed to evict /rbd_header.2509da78c8b8fc: (16) Device or resource busy > rbd_header.2505b7ad46de5 > failed to evict /rbd_header.2505b7ad46de5: (16) Device or resource busy > rbd_header.2509b9400aab77 > failed to evict /rbd_header.2509b9400aab77: (16) Device or resource busy > error from cache-try-flush-evict-all: (1) Operation not permitted > *** try to evict on manually *** > # rados -p ssd-pool cache-flush rbd_header.2507f43faa7343 > # rados -p ssd-pool cache-evict rbd_header.2507f43faa7343 > error from cache-evict rbd_header.2507f43faa7343: (16) Device or resource > busy > > It would be nice to not have to shut down VMs to remove a cache tier. > > Thanks, > > - ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > -----BEGIN PGP SIGNATURE----- > Version: Mailvelope v1.2.2 > Comment: https://www.mailvelope.com > > wsFcBAEBCAAQBQJWLlMrCRDmVDuy+mK58QAApVwP/2gvwlVbBT9lRjHfEzVI > 2pGN5JPPyvEwKnUPp4/UmCISITuWeNz9+E5vK6LCUolTnEFTAxcCy6ThD5T9 > 7Mj+p8+UukZ8upOJoMD+vnm5owXx4l7/UVifWLbebQemn7SkDyAUM28udme/ > l7yPUnANch5GuTC4hSPIyuet1ndEpLrn9Jfl5Yz+43p5esszlF3kMupd3hdu > jvJV+6jFL4wEuAJKkFgyyJFpKngBhuJAmrWitEnrmywkRrnfYKu66X1Wx3MC > XFsuA70TJ5CH+7PhJxP6xI5JgliTGT3LEDFYlHSzAzaWlfK1Inftc/1gPofv > WsDazwP22OqwqfSKiuo89UoTVcw0MIIKbf63n8A1IXLqZPFjiaGRvuijraji > GYNfXYGPXkq3soBRkM2O3lt3hDu9rI6DhIp9QeWRQYsh5+mDC5Zgnch50sOE > r3oAdU1Dkin5EDNXl23uiawLdc9yVqy8nAVEM5DeZPJQp8qq8r797cThNZRw > ptPxvZW5gpCjNrzkxJkKfef5nmKI6yYqFy74+d/Ptf/Zk93+WwAUnwkokhqI > eu/kQO9juqTiMjz2B548FpDyH4P7mJDoNahSvNA/BTRqyUkC7pTiiCOca44j > bq89/6eIbhCWBjn+s5ipUx/j7wV2vsCfCCKdKKjJEzqUpjFAGcHK/ga8s0C9 > zUQT > =2KGK > -----END PGP SIGNATURE----- > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |