Of coure, you’re right. After using the right name, the connection worked :) I tried to connect via a newer kernel client (under Ubuntu 16.04) and it worked as well. So the issue clearly seems to be related to our client kernel version. Thank you all very much for your time and help! Von: David Turner <drakonstein@xxxxxxxxx>
Gesendet: Dienstag, 19. Februar 2019 19:32
An: Hennen, Christian <christian.hennen@xxxxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Betreff: Re: CephFS: client hangs You're attempting to use mismatching client name and keyring. You want to use matching name and keyring. For your example, you would want to either use `--keyring /etc/ceph/ceph.client.admin.keyring --name client.admin` or `--keyring /etc/ceph/ceph.client.cephfs.keyring --name client.cephfs`. Mixing and matching does not work. Treat them like username and password. You wouldn't try to log into your computer under your account with the admin password. > sounds like network issue. are there firewall/NAT between nodes?
No, there is currently no firewall in place. Nodes and clients are on the same network. MTUs match, ports are opened according to nmap.
> try running ceph-fuse on the node that run mds, check if it works properly.
When I try to run ceph-fuse on either a client or cephfiler1 (MON,MGR,MDS,OSDs) I get
- "operation not permitted" when using the client keyring
- "invalid argument" when using the admin keyring
- "ms_handle_refused" when using the admin keyring and connecting to 127.0.0.1:6789
ceph-fuse --keyring /etc/ceph/ceph.client.admin.keyring --name client.cephfs -m 192.168.1.17:6789 /mnt/cephfs
-----Ursprüngliche Nachricht-----
Von: Yan, Zheng <ukernel@xxxxxxxxx>
Gesendet: Dienstag, 19. Februar 2019 11:31
An: Hennen, Christian <christian.hennen@xxxxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Betreff: Re: CephFS: client hangs
On Tue, Feb 19, 2019 at 5:10 PM Hennen, Christian <christian.hennen@xxxxxxxxxxxx> wrote:
>
> Hi!
>
> >mon_max_pg_per_osd = 400
> >
> >In the ceph.conf and then restart all the services / or inject the
> >config into the running admin
>
> I restarted each server (MONs and OSDs weren’t enough) and now the health warning is gone. Still no luck accessing CephFS though.
>
>
> > MDS show a client got evicted. Nothing else looks abnormal. Do new
> > cephfs clients also get evicted quickly?
>
> Aside from the fact that evicted clients don’t show up in ceph –s, we observe other strange things:
>
> · Setting max_mds has no effect
>
> · Ceph osd blacklist ls sometimes lists cluster nodes
>
sounds like network issue. are there firewall/NAT between nodes?
> The only client that is currently running is ‚master1‘. It also hosts a MON and a MGR. Its syslog (https://gitlab.uni-trier.de/snippets/78) shows messages like:
>
> Feb 13 06:40:33 master1 kernel: [56165.943008] libceph: wrong peer,
> want 192.168.1.17:6800/-2045158358, got 192.168.1.17:6800/1699349984
>
> Feb 13 06:40:33 master1 kernel: [56165.943014] libceph: mds1
> 192.168.1.17:6800 wrong peer at address
>
> The other day I did the update from 12.2.8 to 12.2.11, which can also be seen in the logs. Again, there appeared these messages. I assume that’s normal operations since ports can change and daemons have to find each other again? But what about Feb 13 in the morning? I didn’t do any restarts then.
>
> Also, clients are printing messages like the following on the console:
>
> [1026589.751040] ceph: handle_cap_import: mismatched seq/mseq: ino
> (10000994988.fffffffffffffffe) mds0 seq1 mseq 15 importer mds1 has
> peer seq 2 mseq 15
>
> [1352658.876507] ceph: build_path did not end path lookup where
> expected, namelen is 23, pos is 0
>
> Oh, and btw, the ceph nodes are running on Ubuntu 16.04, clients are on 14.04 with kernel 4.4.0-133.
>
try running ceph-fuse on the node that run mds, check if it works properly.
> For reference:
>
> > Cluster details: https://gitlab.uni-trier.de/snippets/77
>
> > MDS log:
> > https://gitlab.uni-trier.de/snippets/79?expanded=true&viewer=simple)
>
>
> Kind regards
> Christian Hennen
>
> Project Manager Infrastructural Services ZIMK University of Trier
> Germany
>
> Von: Ashley Merrick <singapore@xxxxxxxxxxxxxx>
> Gesendet: Montag, 18. Februar 2019 16:53
> An: Hennen, Christian <christian.hennen@xxxxxxxxxxxx>
> Cc: ceph-users@xxxxxxxxxxxxxx
> Betreff: Re: CephFS: client hangs
>
> Correct yes from my expirence OSD’s aswel.
>
> On Mon, 18 Feb 2019 at 11:51 PM, Hennen, Christian <christian.hennen@xxxxxxxxxxxx> wrote:
>
> Hi!
>
> >mon_max_pg_per_osd = 400
> >
> >In the ceph.conf and then restart all the services / or inject the
> >config into the running admin
>
> I restarted all MONs, but I assume the OSDs need to be restarted as well?
>
> > MDS show a client got evicted. Nothing else looks abnormal. Do new
> > cephfs clients also get evicted quickly?
>
> Yeah, it seems so. But strangely there is no indication of it in 'ceph
> -s' or 'ceph health detail'. And they don't seem to be evicted
> permanently? Right now, only 1 client is connected. The others are shut down since last week.
> 'ceph osd blacklist ls' shows 0 entries.
>
>
> Kind regards
> Christian Hennen
>
> Project Manager Infrastructural Services ZIMK University of Trier
> Germany
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
|
