I'm guessing in this context that "write data" possibly means creating
a file (as opposed to writing to an existing file).
Indeed. Sorry for the confusion.
Ok. Is it somehow possible to have multiple FSs on the same ceph cluster?
a file (as opposed to writing to an existing file).
You've pretty much hit the limits of what the disaster recovery tools
are currently capable of. What I'd recommend you do at this stage is
mount your filesystem read-only, back it up, and then create a new
filesystem and restore from backup.
are currently capable of. What I'd recommend you do at this stage is
mount your filesystem read-only, back it up, and then create a new
filesystem and restore from backup.
On 19 November 2015 at 10:43, John Spray <jspray@xxxxxxxxxx> wrote:
On Wed, Nov 18, 2015 at 9:21 AM, Mykola Dvornik
<mykola.dvornik@xxxxxxxxx> wrote:
> Hi John,
>
> It turned out that mds triggers an assertion
>
> mds/MDCache.cc: 269: FAILED assert(inode_map.count(in->vino()) == 0)
>
> on any attempt to write data to the filesystem mounted via fuse.
I'm guessing in this context that "write data" possibly means creating
a file (as opposed to writing to an existing file).
Currently, cephfs-data-scan injects inodes well enough that you can
read them, but it's not updating the inode table to reflect that the
recovered inodes are in use. As a result, when new files are created
they are probably trying to take inode numbers that are already in use
(by the recovered files), and as a result you're hitting this
assertion. The ticket for updating the inotable after injection of
recovered inodes is http://tracker.ceph.com/issues/12131
> Deleting data is still OK.
>
> I cannot really follow why duplicated inodes appear.
>
> Are there any ways to flush/reset the MDS cache?
You've pretty much hit the limits of what the disaster recovery tools
are currently capable of. What I'd recommend you do at this stage is
mount your filesystem read-only, back it up, and then create a new
filesystem and restore from backup.
I'm writing a patch to handle the particular case where someone needs
to update their inode table to mark all inodes as used up to some
maximum, but the chances are that after that you'll still run into
some other issue, until we've finished the tools to make it all the
way through this path.
John
>
>
>
> On 17 November 2015 at 13:26, John Spray <jspray@xxxxxxxxxx> wrote:
>>
>> On Tue, Nov 17, 2015 at 12:17 PM, Mykola Dvornik
>> <mykola.dvornik@xxxxxxxxx> wrote:
>> > Dear John,
>> >
>> > Thanks for such a prompt reply!
>> >
>> > Seems like something happens on the mon side, since there are no
>> > mount-specific requests logged on the mds side (see below).
>> > FYI, some hours ago I've disabled auth completely, but it didn't help.
>> >
>> > The serialized metadata pool is 9.7G. I can try to compress it with 7z,
>> > then
>> > setup rssh account for you to scp/rsync it.
>> >
>> > debug mds = 20
>> > debug mon = 20
>>
>> Don't worry about the mon logs. That MDS log snippet appears to be
>> from several minutes earlier than the client's attempt to mount.
>>
>> In these cases it's generally simpler if you truncate all the logs,
>> then attempt the mount, then send all the logs in full rather than
>> snippets, so that we can be sure nothing is missing.
>>
>> Please also get the client log (use the fuse client with
>> --debug-client=20).
>>
>> John
>
>
>
>
> --
> Mykola
Mykola
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
