On Mon, Jan 25, 2016 at 10:40 PM, Burkhard Linke
<Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
> Hi,
>
>
> On 01/25/2016 03:27 PM, Yan, Zheng wrote:
>>
>> On Mon, Jan 25, 2016 at 9:43 PM, Burkhard Linke
>> <Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
>>>
>>> Hi,
>>>
>>> On 01/25/2016 01:05 PM, Yan, Zheng wrote:
>>>>
>>>> On Mon, Jan 25, 2016 at 3:43 PM, Burkhard Linke
>>>> <Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> there's a rogue file in our CephFS that we are unable to remove. Access
>>>>> to
>>>>> the file (removal, move, copy, open etc.) results in the MDS starting
>>>>> to
>>>>> spill out the following message to its log file:
>>>>>
>>>>> 2016-01-25 08:39:09.623398 7f472a0ee700 0 mds.0.cache
>>>>> open_remote_dentry_finish bad remote dentry [dentry #1/<some file>
>>>>> [2,head]
>>>>> auth REMOTE(reg) (dversion lock) pv=0 v=1779105 inode=0 0x24bb98c0]
>>>>>
>>>>> I have to restart the MDS to keep it from filling up the log file
>>>>> partition.
>>>>> How can I get rid of this file?
>>>>>
>>>> running following command can delete the file
>>>>
>>>> first, flush mds journal
>>>>
>>>> # ceph daemon mds.xxx flush journal
>>>>
>>>> then, running following command
>>>>
>>>> #rados -p metadata listomapkeys 1.00000000
>>>>
>>>> the output should contain a entry named "<some file>_head"
>>>>
>>>> finally run command
>>>>
>>>> #rados -p metadata rmomapkey 1.00000000 <some file>_head
>>>>
>>>>
>>>>
>>>> Before running above commands, please help us debug this issue. Set
>>>> debug_mds = 10, restart mds and access the bad file.
>>>
>>> The situation is already partly resolved due to the restart of the MDS
>>> service. This had to happen twice (after trying to remove the file and
>>> after
>>> trying to move it to a different directory). The file was first listed in
>>> the target directory of the move operation, but vanished after some time.
>>> Nonetheless it is still available as RADOS object ('rados getxattr <inode
>>> hex>.00000000 parent' contains the string 'stray', get dumps its
>>> content).
>>>
>> which pool does the Rados object live in ?
>
> Pool id 12.
>
> # rados -p cephfs_two_rep_data getxattr 10002af7f78.00000000 parent
> ex ¯ 10002af7f78øé! stray5Àï¶ # rados -p
> cephfs_two_rep_data get 10002af7f78.00000000 test.bin
> # wc -c test.bin
> 24984 test.bin
> # ceph mds dump | grep pool
> dumped mdsmap epoch 100701
> data_pools 7,12,20
> metadata_pool 8
I have a few questions
Which version of ceph are you using? When was the filesystem created?
Did you manually delete 10002af7f78.00000000 from pool 8?
Regards
Yan, Zheng
>
>
> Regards,
> Burkhard
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
