Hi,
On 01/25/2016 01:05 PM, Yan, Zheng wrote:
On Mon, Jan 25, 2016 at 3:43 PM, Burkhard Linke
<Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
Hi,
there's a rogue file in our CephFS that we are unable to remove. Access to
the file (removal, move, copy, open etc.) results in the MDS starting to
spill out the following message to its log file:
2016-01-25 08:39:09.623398 7f472a0ee700 0 mds.0.cache
open_remote_dentry_finish bad remote dentry [dentry #1/<some file> [2,head]
auth REMOTE(reg) (dversion lock) pv=0 v=1779105 inode=0 0x24bb98c0]
I have to restart the MDS to keep it from filling up the log file partition.
How can I get rid of this file?
running following command can delete the file
first, flush mds journal
# ceph daemon mds.xxx flush journal
then, running following command
#rados -p metadata listomapkeys 1.00000000
the output should contain a entry named "<some file>_head"
finally run command
#rados -p metadata rmomapkey 1.00000000 <some file>_head
Before running above commands, please help us debug this issue. Set
debug_mds = 10, restart mds and access the bad file.
The situation is already partly resolved due to the restart of the MDS
service. This had to happen twice (after trying to remove the file and
after trying to move it to a different directory). The file was first
listed in the target directory of the move operation, but vanished after
some time. Nonetheless it is still available as RADOS object ('rados
getxattr <inode hex>.00000000 parent' contains the string 'stray', get
dumps its content).
But listing the now empty directory results in a single log entry:
2016-01-25 14:24:10.205747 7f9d9c8c3700 0 mds.0.cache
open_remote_dentry_finish bad remote dentry [dentry
#1/volumes/attic/<some filename> [2,head] auth REMOTE(reg) (dversion
lock) pv=0 v=183069 inode=0 0x10527d1c0]
After setting debug level to 10 (using ceph daemon ... config set ... ),
the following output refers to the broken file:
2016-01-25 14:25:23.204785 7f9d9afbf700 10 mds.0.cache
open_remote_dentry [dentry #1/volumes/attic/<some filename> [2,head]
auth REMOTE(reg) (dversion loc
k) pv=0 v=183069 inode=0 0x105cee640]
2016-01-25 14:25:23.204794 7f9d9afbf700 10 mds.0.cache open_ino
10002af7f78 pool -1 want_replica 1
2016-01-25 14:25:23.204802 7f9d9afbf700 10 mds.0.cache do_open_ino_peer
10002af7f78 active 0 all 0 checked 0
2016-01-25 14:25:23.204806 7f9d9afbf700 10 mds.0.cache all MDS peers
have been checked
2016-01-25 14:25:23.207729 7f9d9c8c3700 10 MDSIOContextBase::complete:
32C_IO_MDC_OpenInoBacktraceFetched
2016-01-25 14:25:23.207744 7f9d9c8c3700 10 mds.0.cache
_open_ino_backtrace_fetched ino 10002af7f78 errno -2
2016-01-25 14:25:23.207749 7f9d9c8c3700 10 mds.0.cache no object in
pool 7, retrying pool 8
2016-01-25 14:25:23.210403 7f9d9c8c3700 10 MDSIOContextBase::complete:
32C_IO_MDC_OpenInoBacktraceFetched
2016-01-25 14:25:23.210444 7f9d9c8c3700 10 mds.0.cache
_open_ino_backtrace_fetched ino 10002af7f78 errno -2
2016-01-25 14:25:23.210450 7f9d9c8c3700 10 mds.0.cache failed to open
ino 10002af7f78
2016-01-25 14:25:23.210452 7f9d9c8c3700 10 mds.0.cache open_ino_finish
ino 10002af7f78 ret -2
2016-01-25 14:25:23.210524 7f9d9c8c3700 10
MDSInternalContextBase::complete: 22C_MDC_OpenRemoteDentry
2016-01-25 14:25:23.210530 7f9d9c8c3700 0 mds.0.cache
open_remote_dentry_finish bad remote dentry [dentry
#1/volumes/attic/<some filename> [2,head] auth REMOTE(reg) (dversion
lock) pv=0 v=183069 inode=0 0x105cee640]
2016-01-25 14:25:23.210543 7f9d9c8c3700 10
MDSInternalContextBase::complete: 18C_MDS_RetryRequest
2016-01-25 14:25:23.210545 7f9d9c8c3700 7 mds.0.server
dispatch_client_request client_request(client.1297540:16260418 readdir
#100023ff2a0 2016-01-25 14:25:23.218538) v2
100023ff2a0 is the containing directory, the output is written during
running ls for that directory. Metadata pool id is 8, pool 7, 12 and 20
are datapools
Just send me a notice if you need more debug output (either mail or
IRC). The other commands mentioned above have not been run yet.
Regards,
Burkhard
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
