> sounds like network issue. are there firewall/NAT between nodes?
No, there is currently no firewall in place. Nodes and clients are on the same network. MTUs match, ports are opened according to nmap.
> try running ceph-fuse on the node that run mds, check if it works properly.
When I try to run ceph-fuse on either a client or cephfiler1 (MON,MGR,MDS,OSDs) I get
- "operation not permitted" when using the client keyring
- "invalid argument" when using the admin keyring
- "ms_handle_refused" when using the admin keyring and connecting to 127.0.0.1:6789
ceph-fuse --keyring /etc/ceph/ceph.client.admin.keyring --name client.cephfs -m 192.168.1.17:6789 /mnt/cephfs
-----Ursprüngliche Nachricht-----
Von: Yan, Zheng <ukernel@xxxxxxxxx>
Gesendet: Dienstag, 19. Februar 2019 11:31
An: Hennen, Christian <christian.hennen@xxxxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Betreff: Re: CephFS: client hangs
On Tue, Feb 19, 2019 at 5:10 PM Hennen, Christian <christian.hennen@xxxxxxxxxxxx> wrote:
>
> Hi!
>
> >mon_max_pg_per_osd = 400
> >
> >In the ceph.conf and then restart all the services / or inject the
> >config into the running admin
>
> I restarted each server (MONs and OSDs weren’t enough) and now the health warning is gone. Still no luck accessing CephFS though.
>
>
> > MDS show a client got evicted. Nothing else looks abnormal. Do new
> > cephfs clients also get evicted quickly?
>
> Aside from the fact that evicted clients don’t show up in ceph –s, we observe other strange things:
>
> · Setting max_mds has no effect
>
> · Ceph osd blacklist ls sometimes lists cluster nodes
>
sounds like network issue. are there firewall/NAT between nodes?
> The only client that is currently running is ‚master1‘. It also hosts a MON and a MGR. Its syslog (https://gitlab.uni-trier.de/snippets/78) shows messages like:
>
> Feb 13 06:40:33 master1 kernel: [56165.943008] libceph: wrong peer,
> want 192.168.1.17:6800/-2045158358, got 192.168.1.17:6800/1699349984
>
> Feb 13 06:40:33 master1 kernel: [56165.943014] libceph: mds1
> 192.168.1.17:6800 wrong peer at address
>
> The other day I did the update from 12.2.8 to 12.2.11, which can also be seen in the logs. Again, there appeared these messages. I assume that’s normal operations since ports can change and daemons have to find each other again? But what about Feb 13 in the morning? I didn’t do any restarts then.
>
> Also, clients are printing messages like the following on the console:
>
> [1026589.751040] ceph: handle_cap_import: mismatched seq/mseq: ino
> (10000994988.fffffffffffffffe) mds0 seq1 mseq 15 importer mds1 has
> peer seq 2 mseq 15
>
> [1352658.876507] ceph: build_path did not end path lookup where
> expected, namelen is 23, pos is 0
>
> Oh, and btw, the ceph nodes are running on Ubuntu 16.04, clients are on 14.04 with kernel 4.4.0-133.
>
try running ceph-fuse on the node that run mds, check if it works properly.
> For reference:
>
> > Cluster details: https://gitlab.uni-trier.de/snippets/77
>
> > MDS log:
> > https://gitlab.uni-trier.de/snippets/79?expanded=true&viewer=simple)
>
>
> Kind regards
> Christian Hennen
>
> Project Manager Infrastructural Services ZIMK University of Trier
> Germany
>
> Von: Ashley Merrick <singapore@xxxxxxxxxxxxxx>
> Gesendet: Montag, 18. Februar 2019 16:53
> An: Hennen, Christian <christian.hennen@xxxxxxxxxxxx>
> Cc: ceph-users@xxxxxxxxxxxxxx
> Betreff: Re: CephFS: client hangs
>
> Correct yes from my expirence OSD’s aswel.
>
> On Mon, 18 Feb 2019 at 11:51 PM, Hennen, Christian <christian.hennen@xxxxxxxxxxxx> wrote:
>
> Hi!
>
> >mon_max_pg_per_osd = 400
> >
> >In the ceph.conf and then restart all the services / or inject the
> >config into the running admin
>
> I restarted all MONs, but I assume the OSDs need to be restarted as well?
>
> > MDS show a client got evicted. Nothing else looks abnormal. Do new
> > cephfs clients also get evicted quickly?
>
> Yeah, it seems so. But strangely there is no indication of it in 'ceph
> -s' or 'ceph health detail'. And they don't seem to be evicted
> permanently? Right now, only 1 client is connected. The others are shut down since last week.
> 'ceph osd blacklist ls' shows 0 entries.
>
>
> Kind regards
> Christian Hennen
>
> Project Manager Infrastructural Services ZIMK University of Trier
> Germany
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
