On Tue, Feb 19, 2019 at 5:10 PM Hennen, Christian <christian.hennen@xxxxxxxxxxxx> wrote: > > Hi! > > >mon_max_pg_per_osd = 400 > > > >In the ceph.conf and then restart all the services / or inject the config > >into the running admin > > I restarted each server (MONs and OSDs weren’t enough) and now the health warning is gone. Still no luck accessing CephFS though. > > > > MDS show a client got evicted. Nothing else looks abnormal. Do new cephfs > > clients also get evicted quickly? > > Aside from the fact that evicted clients don’t show up in ceph –s, we observe other strange things: > > · Setting max_mds has no effect > > · Ceph osd blacklist ls sometimes lists cluster nodes > sounds like network issue. are there firewall/NAT between nodes? > The only client that is currently running is ‚master1‘. It also hosts a MON and a MGR. Its syslog (https://gitlab.uni-trier.de/snippets/78) shows messages like: > > Feb 13 06:40:33 master1 kernel: [56165.943008] libceph: wrong peer, want 192.168.1.17:6800/-2045158358, got 192.168.1.17:6800/1699349984 > > Feb 13 06:40:33 master1 kernel: [56165.943014] libceph: mds1 192.168.1.17:6800 wrong peer at address > > The other day I did the update from 12.2.8 to 12.2.11, which can also be seen in the logs. Again, there appeared these messages. I assume that’s normal operations since ports can change and daemons have to find each other again? But what about Feb 13 in the morning? I didn’t do any restarts then. > > Also, clients are printing messages like the following on the console: > > [1026589.751040] ceph: handle_cap_import: mismatched seq/mseq: ino (10000994988.fffffffffffffffe) mds0 seq1 mseq 15 importer mds1 has peer seq 2 mseq 15 > > [1352658.876507] ceph: build_path did not end path lookup where expected, namelen is 23, pos is 0 > > Oh, and btw, the ceph nodes are running on Ubuntu 16.04, clients are on 14.04 with kernel 4.4.0-133. > try running ceph-fuse on the node that run mds, check if it works properly. > For reference: > > > Cluster details: https://gitlab.uni-trier.de/snippets/77 > > > MDS log: https://gitlab.uni-trier.de/snippets/79?expanded=true&viewer=simple) > > > Kind regards > Christian Hennen > > Project Manager Infrastructural Services ZIMK University of Trier > Germany > > Von: Ashley Merrick <singapore@xxxxxxxxxxxxxx> > Gesendet: Montag, 18. Februar 2019 16:53 > An: Hennen, Christian <christian.hennen@xxxxxxxxxxxx> > Cc: ceph-users@xxxxxxxxxxxxxx > Betreff: Re: CephFS: client hangs > > Correct yes from my expirence OSD’s aswel. > > On Mon, 18 Feb 2019 at 11:51 PM, Hennen, Christian <christian.hennen@xxxxxxxxxxxx> wrote: > > Hi! > > >mon_max_pg_per_osd = 400 > > > >In the ceph.conf and then restart all the services / or inject the config > >into the running admin > > I restarted all MONs, but I assume the OSDs need to be restarted as well? > > > MDS show a client got evicted. Nothing else looks abnormal. Do new cephfs > > clients also get evicted quickly? > > Yeah, it seems so. But strangely there is no indication of it in 'ceph -s' or > 'ceph health detail'. And they don't seem to be evicted permanently? Right > now, only 1 client is connected. The others are shut down since last week. > 'ceph osd blacklist ls' shows 0 entries. > > > Kind regards > Christian Hennen > > Project Manager Infrastructural Services ZIMK University of Trier > Germany > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |