Hi!
>mon_max_pg_per_osd = 400
>
>In the ceph.conf and then restart all the services / or inject the config
>into the running admin
I restarted each server (MONs and OSDs weren’t enough) and now the health warning is gone. Still no luck accessing CephFS though.
> MDS show a client got evicted. Nothing else looks abnormal. Do new cephfs
> clients also get evicted quickly?
Aside from the fact that evicted clients don’t show up in ceph –s, we observe other strange things:
· Setting max_mds has no effect
· Ceph osd blacklist ls sometimes lists cluster nodes
The only client that is currently running is ‚master1‘. It also hosts a MON and a MGR. Its syslog (https://gitlab.uni-trier.de/snippets/78) shows messages like:
Feb 13 06:40:33 master1 kernel: [56165.943008] libceph: wrong peer, want 192.168.1.17:6800/-2045158358, got 192.168.1.17:6800/1699349984
Feb 13 06:40:33 master1 kernel: [56165.943014] libceph: mds1 192.168.1.17:6800 wrong peer at address
The other day I did the update from 12.2.8 to 12.2.11, which can also be seen in the logs. Again, there appeared these messages. I assume that’s normal operations since ports can change and daemons have to find each other again? But what about Feb 13 in the morning? I didn’t do any restarts then.
Also, clients are printing messages like the following on the console:
[1026589.751040] ceph: handle_cap_import: mismatched seq/mseq: ino (10000994988.fffffffffffffffe) mds0 seq1 mseq 15 importer mds1 has peer seq 2 mseq 15
[1352658.876507] ceph: build_path did not end path lookup where expected, namelen is 23, pos is 0
Oh, and btw, the ceph nodes are running on Ubuntu 16.04, clients are on 14.04 with kernel 4.4.0-133.
For reference:
> Cluster details: https://gitlab.uni-trier.de/snippets/77
> MDS log: https://gitlab.uni-trier.de/snippets/79?expanded=true&viewer=simple)
Kind regards
Christian Hennen
Project Manager Infrastructural Services ZIMK University of Trier
Germany
Von: Ashley Merrick <singapore@xxxxxxxxxxxxxx>
Gesendet: Montag, 18. Februar 2019 16:53
An: Hennen, Christian <christian.hennen@xxxxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Betreff: Re: CephFS: client hangs
Correct yes from my expirence OSD’s aswel.
On Mon, 18 Feb 2019 at 11:51 PM, Hennen, Christian <christian.hennen@xxxxxxxxxxxx> wrote:
Hi!
>mon_max_pg_per_osd = 400
>
>In the ceph.conf and then restart all the services / or inject the config
>into the running admin
I restarted all MONs, but I assume the OSDs need to be restarted as well?
> MDS show a client got evicted. Nothing else looks abnormal. Do new cephfs
> clients also get evicted quickly?
Yeah, it seems so. But strangely there is no indication of it in 'ceph -s' or
'ceph health detail'. And they don't seem to be evicted permanently? Right
now, only 1 client is connected. The others are shut down since last week.
'ceph osd blacklist ls' shows 0 entries.
Kind regards
Christian Hennen
Project Manager Infrastructural Services ZIMK University of Trier
Germany
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
