Thanks for the tip.
I will stay of the safe side and wait until it will be merged into master)
Many thanks for all your help.
-Mykola
On 19 November 2015 at 11:10, John Spray <jspray@xxxxxxxxxx> wrote:
On Thu, Nov 19, 2015 at 10:07 AM, Mykola Dvornik
<mykola.dvornik@xxxxxxxxx> wrote:
> I'm guessing in this context that "write data" possibly means creating
> a file (as opposed to writing to an existing file).
>
> Indeed. Sorry for the confusion.
>
> You've pretty much hit the limits of what the disaster recovery tools
> are currently capable of. What I'd recommend you do at this stage is
> mount your filesystem read-only, back it up, and then create a new
> filesystem and restore from backup.
>
> Ok. Is it somehow possible to have multiple FSs on the same ceph cluster?
No, we want to do this but it's not there yet. Your scenario is one
of the motivations :-)
(for the record multi-fs branch is
https://github.com/jcsp/ceph/commits/wip-multi-filesystems, which
works, but we'll probably go back and re-do the mon side of it before
finishing)
John
>
>
> On 19 November 2015 at 10:43, John Spray <jspray@xxxxxxxxxx> wrote:
>>
>> On Wed, Nov 18, 2015 at 9:21 AM, Mykola Dvornik
>> <mykola.dvornik@xxxxxxxxx> wrote:
>> > Hi John,
>> >
>> > It turned out that mds triggers an assertion
>> >
>> > mds/MDCache.cc: 269: FAILED assert(inode_map.count(in->vino()) == 0)
>> >
>> > on any attempt to write data to the filesystem mounted via fuse.
>>
>> I'm guessing in this context that "write data" possibly means creating
>> a file (as opposed to writing to an existing file).
>>
>> Currently, cephfs-data-scan injects inodes well enough that you can
>> read them, but it's not updating the inode table to reflect that the
>> recovered inodes are in use. As a result, when new files are created
>> they are probably trying to take inode numbers that are already in use
>> (by the recovered files), and as a result you're hitting this
>> assertion. The ticket for updating the inotable after injection of
>> recovered inodes is http://tracker.ceph.com/issues/12131
>>
>> > Deleting data is still OK.
>> >
>> > I cannot really follow why duplicated inodes appear.
>> >
>> > Are there any ways to flush/reset the MDS cache?
>>
>> You've pretty much hit the limits of what the disaster recovery tools
>> are currently capable of. What I'd recommend you do at this stage is
>> mount your filesystem read-only, back it up, and then create a new
>> filesystem and restore from backup.
>>
>> I'm writing a patch to handle the particular case where someone needs
>> to update their inode table to mark all inodes as used up to some
>> maximum, but the chances are that after that you'll still run into
>> some other issue, until we've finished the tools to make it all the
>> way through this path.
>>
>> John
>>
>> >
>> >
>> >
>> > On 17 November 2015 at 13:26, John Spray <jspray@xxxxxxxxxx> wrote:
>> >>
>> >> On Tue, Nov 17, 2015 at 12:17 PM, Mykola Dvornik
>> >> <mykola.dvornik@xxxxxxxxx> wrote:
>> >> > Dear John,
>> >> >
>> >> > Thanks for such a prompt reply!
>> >> >
>> >> > Seems like something happens on the mon side, since there are no
>> >> > mount-specific requests logged on the mds side (see below).
>> >> > FYI, some hours ago I've disabled auth completely, but it didn't
>> >> > help.
>> >> >
>> >> > The serialized metadata pool is 9.7G. I can try to compress it with
>> >> > 7z,
>> >> > then
>> >> > setup rssh account for you to scp/rsync it.
>> >> >
>> >> > debug mds = 20
>> >> > debug mon = 20
>> >>
>> >> Don't worry about the mon logs. That MDS log snippet appears to be
>> >> from several minutes earlier than the client's attempt to mount.
>> >>
>> >> In these cases it's generally simpler if you truncate all the logs,
>> >> then attempt the mount, then send all the logs in full rather than
>> >> snippets, so that we can be sure nothing is missing.
>> >>
>> >> Please also get the client log (use the fuse client with
>> >> --debug-client=20).
>> >>
>> >> John
>> >
>> >
>> >
>> >
>> > --
>> > Mykola
>
>
>
>
> --
> Mykola
Mykola
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
