Le mercredi 21 mai 2014 ? 18:20 -0700, Josh Durgin a ?crit :
> On 05/21/2014 03:03 PM, Olivier Bonvalet wrote:
> > Le mercredi 21 mai 2014 ? 08:20 -0700, Sage Weil a ?crit :
> >> You're certain that that is the correct prefix for the rbd image you
> >> removed? Do you see the objects lists when you do 'rados -p rbd ls - |
> >> grep <prefix>'?
> >
> > I'm pretty sure yes : since I didn't see a lot of space freed by the
> > "rbd snap purge" command, I looked at the RBD prefix before to do the
> > "rbd rm" (it's not the first time I see that problem, but previous time
> > without the RBD prefix I was not able to check).
> >
> > So :
> > - "rados -p sas3copies ls - | grep rb.0.14bfb5a.238e1f29" return nothing
> > at all
> > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026
> > error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.00000002f026: No such
> > file or directory
> > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.000000000000
> > error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.000000000000: No such
> > file or directory
> > - # ls -al /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9
> > -rw-r--r-- 1 root root 4194304 oct. 8 2013 /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9
> >
> >
> >> If the objects really are orphaned, teh way to clean them up is via 'rados
> >> -p rbd rm <objectname>'. I'd like to get to the bottom of how they ended
> >> up that way first, though!
> >
> > I suppose the problem came from me, by doing CTRL+C while "rbd snap
> > purge $IMG".
> > "rados rm -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026" don't remove
> > thoses files, and just answer with a "No such file or directory".
>
> Those files are all for snapshots, which are removed by the osds
> asynchronously in a process called 'snap trimming'. There's no
> way to directly remove them via rados.
>
> Since you stopped 'rbd snap purge' partway through, it may
> have removed the reference to the snapshot before removing
> the snapshot itself.
>
> You can get a list of snapshot ids for the remaining objects
> via the 'rados listsnaps' command, and use
> rados_ioctx_selfmanaged_snap_remove() (no convenient wrapper
> unfortunately) on each of those snapshot ids to be sure they are all
> scheduled for asynchronous deletion.
>
> Josh
>
Great : "rados listsnaps" see it :
# rados listsnaps -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026
rb.0.14bfb5a.238e1f29.00000002f026:
cloneid snaps size overlap
41554 35746 4194304 []
So, I have to write&compile a wrapper to
rados_ioctx_selfmanaged_snap_remove(), and find a way to obtain a list
of all "orphan" objects ?
I also try to recreate the object (rados put) then remove it (rados rm),
but snapshots still here.
Olivier
