(adding list back) The "clients failing to respond to capability release" messages can sometimes indicate a bug in the client code, so it's a good idea to make sure you've got the most recent fixes before investigating further. It's also useful to compare kernel vs. fuse clients to see if the issue occurs in one but not the other. The guidance on client choice and kernel versions is here: http://docs.ceph.com/docs/master/cephfs/best-practices/#which-client If you're happy running a non-LTS distro like Fedora, then I'd suggest running the latest Fedora release (28). John On Mon, Sep 10, 2018 at 3:17 PM marc-antoine desrochers <marc-antoine.desrochers@xxxxxxxxxxx> wrote: > > What Is the advantages of using ceph-fuse ? and if I stay on kernel client what kind of distro/kernel are you suggesting ? > > -----Message d'origine----- > De : John Spray [mailto:jspray@xxxxxxxxxx] > Envoyé : 10 septembre 2018 10:08 > À : marc-antoine.desrochers@xxxxxxxxxxx > Cc : ceph-users@xxxxxxxxxxxxxx > Objet : Re: Need help > > On Mon, Sep 10, 2018 at 1:40 PM marc-antoine desrochers <marc-antoine.desrochers@xxxxxxxxxxx> wrote: > > > > Hi, > > > > > > > > I am currently running a ceph cluster running in CEPHFS with 3 nodes each have 6 osd’s except 1 who got 5. I got 3 mds : 2 active and 1 standby, 3 mon. > > > > > > > > > > > > [root@ceph-n1 ~]# ceph -s > > > > cluster: > > > > id: 1d97aa70-2029-463a-b6fa-20e98f3e21fb > > > > health: HEALTH_WARN > > > > 3 clients failing to respond to capability release > > > > 2 MDSs report slow requests > > > > > > > > services: > > > > mon: 3 daemons, quorum ceph-n1,ceph-n2,ceph-n3 > > > > mgr: ceph-n1(active), standbys: ceph-n2, ceph-n3 > > > > mds: cephfs-2/2/2 up {0=ceph-n1=up:active,1=ceph-n2=up:active}, 1 > > up:standby > > > > osd: 17 osds: 17 up, 17 in > > > > > > > > data: > > > > pools: 2 pools, 1024 pgs > > > > objects: 541k objects, 42006 MB > > > > usage: 143 GB used, 6825 GB / 6969 GB avail > > > > pgs: 1024 active+clean > > > > > > > > io: > > > > client: 32980 B/s rd, 77295 B/s wr, 5 op/s rd, 14 op/s wr > > > > > > > > I’m using the cephFs as a mail storage. I currently have 3500 mailbox > > some of them are IMAP the others are POP3 the goal is to be able to > > migrate all mailbox from my old > > > > > > > > infrastructure so around 30 000 mailbox. > > > > > > > > I’m now facing a problem : > > > > MDS_CLIENT_LATE_RELEASE 3 clients failing to respond to capability > > release > > > > mdsceph-n1(mds.0): Client mda3.sogetel.net failing to respond to > > capability releaseclient_id: 1134426 > > > > mdsceph-n1(mds.0): Client mda2.sogetel.net failing to respond to > > capability releaseclient_id: 1172391 > > > > mdsceph-n2(mds.1): Client mda3.sogetel.net failing to respond to > > capability releaseclient_id: 1134426 > > > > MDS_SLOW_REQUEST 2 MDSs report slow requests > > > > mdsceph-n1(mds.0): 112 slow requests are blocked > 30 sec > > > > mdsceph-n2(mds.1): 323 slow requests are blocked > 30 sec > > > > > > > > I can’t figure out how to fix this… > > > > > > > > > > Here some information’s about my cluster : > > > > I’m running ceph luminous 12.2.5 on my 3 ceph nodes : ceph-n1, ceph-n2, ceph-n3. > > > > > > I have 3 client identical : > > > > LSB Version: :core-4.1-amd64:core-4.1-noarch > > > > Distributor ID: Fedora > > > > Description: Fedora release 25 (Twenty Five) > > > > Release: 25 > > > > Codename: TwentyFive > > > > I can't say for sure whether it would help, but I'd definitely suggest upgrading those nodes to latest Fedora if you're using the kernel client -- Fedora 25 hasn't received updates for quite some time. > > John > > > > > My ceph nodes : > > > > > > > > CentOS Linux release 7.5.1804 (Core) > > > > NAME="CentOS Linux" > > > > VERSION="7 (Core)" > > > > ID="centos" > > > > ID_LIKE="rhel fedora" > > > > VERSION_ID="7" > > > > PRETTY_NAME="CentOS Linux 7 (Core)" > > > > ANSI_COLOR="0;31" > > > > CPE_NAME="cpe:/o:centos:centos:7" > > > > HOME_URL="https://www.centos.org/"; > > > > BUG_REPORT_URL="https://bugs.centos.org/"; > > > > > > > > CENTOS_MANTISBT_PROJECT="CentOS-7" > > > > CENTOS_MANTISBT_PROJECT_VERSION="7" > > > > REDHAT_SUPPORT_PRODUCT="centos" > > > > REDHAT_SUPPORT_PRODUCT_VERSION="7" > > > > > > > > CentOS Linux release 7.5.1804 (Core) > > > > CentOS Linux release 7.5.1804 (Core) > > > > > > > > ceph daemon mds.ceph-n1 perf dump mds : > > > > > > > > > > > > "mds": { > > > > "request": 21968558, > > > > "reply": 21954801, > > > > "reply_latency": { > > > > "avgcount": 21954801, > > > > "sum": 100879.560315258, > > > > "avgtime": 0.004594874 > > > > }, > > > > "forward": 13627, > > > > "dir_fetch": 3327, > > > > "dir_commit": 162830, > > > > "dir_split": 1, > > > > "dir_merge": 0, > > > > "inode_max": 2147483647, > > > > "inodes": 68767, > > > > "inodes_top": 4524, > > > > "inodes_bottom": 56697, > > > > "inodes_pin_tail": 7546, > > > > "inodes_pinned": 62304, > > > > "inodes_expired": 1640159, > > > > "inodes_with_caps": 62192, > > > > "caps": 114126, > > > > "subtrees": 14, > > > > "traverse": 38309963, > > > > "traverse_hit": 37606227, > > > > "traverse_forward": 12189, > > > > "traverse_discover": 6634, > > > > "traverse_dir_fetch": 1769, > > > > "traverse_remote_ino": 6, > > > > "traverse_lock": 7731, > > > > "load_cent": 2196856701, > > > > "q": 0, > > > > "exported": 143, > > > > "exported_inodes": 291372, > > > > "imported": 125, > > > > "imported_inodes": 176509 > > > > > > > > > > > > Thanks for your help… > > > > > > > > Regards > > > > > > > > Marc-Antoine > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |