Hi John,
On 10/08/2015 01:03 PM, John Spray wrote:
On Thu, Oct 8, 2015 at 11:41 AM, Burkhard Linke
<Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
*snipsnap*
Thanks for the fast reply. During the transfer of all files from the EC pool
to a standard replicated pool I've copied the file to a new file name,
removed the orignal one and renamed the copy. There might have been some
processed with open files at that time, which might explain the stray files
objects.
I've also been able to locate some processes that might be the reason for
these leftover files. I've terminated these processes, but the objects are
still present in the pool. How long does purging an inode usually take?
If nothing is holding a file open, it'll start purging within a couple
of journal-latencies of the unlink (i.e. pretty darn quick), and it'll
take as long to purge as there are objects in the file (again, pretty
darn quick for normal-sized files and a non-overloaded cluster).
Chances are if you're noticing strays, they're stuck for some reason.
You're probably on the right track looking for processes holding files
open.
You don't say what version you're running, so it's possible you're
running an older version (pre hammer, I think) where you're
experiencing either a bug holding up deletion (we've had a few) or a
bug preventing reintegration (we had one of those too). The bugs
holding up deletion can usually be worked around with some client
and/or mds restarts.
The cluster is running on hammer. I'm going to restart the mds to try to get
rid of these objects.
OK, let us know how it goes. You may find the num_strays,
num_strays_purging, num_strays_delayted performance counters (ceph
daemon mds.<foo> perf dump) useful.
The number of objects dropped to 7 after the mds restart. I was also
able to identify the application the objects belong to (some where perl
modules), but I've been unable to locate a running instance of this
application. The main user of this application is also not aware of any
running instance at the moment.
It isn't safe to remove the pool in this state. The MDS is likely to
crash if it eventually gets around to trying to purge these files.
That's bad. Does the mds provide a way to get more information about these
files, e.g. which client is blocking purging? We have about 3 hosts working
on CephFS, and checking every process might be difficult.
If a client has caps on an inode, you can find out about it by dumping
(the whole!) cache from a running MDS. We have tickets for adding a
more surgical version of this[1] but for now it's bit of a heavyweight
thing. You can do JSON ("ceph daemon mds.<id> dump cache > foo.json")
or plain text ("ceph daemon mds.<id> dump cache foo.txt"). The latter
version is harder to parse but is less likely to eat all the memory on
your MDS (JSON output builds the whole thing in memory before writing
it)!
Hammer 0.94.3 does not support a 'dump cache' mds command.
'dump_ops_in_flight' does not list any pending operations. Is there any
other way to access the cache?
'perf dump' stray information (after mds restart):
"num_strays": 2327,
"num_strays_purging": 0,
"num_strays_delayed": 0,
"strays_created": 33,
"strays_purged": 34,
The data pool is a combination of EC pool and cache tier. I've evicted
the cache pool resulting in 128 objects left (one per PG? hitset
information?). After restarting the MDS the number of objects increases
by 7 objects (the ones left in the data pool). So either the MDS rejoin
process promotes them back to the cache, or some ceph-fuse instance
insists on reading them.
Regards,
Burkhard
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
