On Mon, Sep 10, 2018 at 1:40 PM marc-antoine desrochers
<marc-antoine.desrochers@xxxxxxxxxxx> wrote:
>
> Hi,
>
>
>
> I am currently running a ceph cluster running in CEPHFS with 3 nodes each have 6 osd’s except 1 who got 5. I got 3 mds : 2 active and 1 standby, 3 mon.
>
>
>
>
>
> [root@ceph-n1 ~]# ceph -s
>
> cluster:
>
> id: 1d97aa70-2029-463a-b6fa-20e98f3e21fb
>
> health: HEALTH_WARN
>
> 3 clients failing to respond to capability release
>
> 2 MDSs report slow requests
>
>
>
> services:
>
> mon: 3 daemons, quorum ceph-n1,ceph-n2,ceph-n3
>
> mgr: ceph-n1(active), standbys: ceph-n2, ceph-n3
>
> mds: cephfs-2/2/2 up {0=ceph-n1=up:active,1=ceph-n2=up:active}, 1 up:standby
>
> osd: 17 osds: 17 up, 17 in
>
>
>
> data:
>
> pools: 2 pools, 1024 pgs
>
> objects: 541k objects, 42006 MB
>
> usage: 143 GB used, 6825 GB / 6969 GB avail
>
> pgs: 1024 active+clean
>
>
>
> io:
>
> client: 32980 B/s rd, 77295 B/s wr, 5 op/s rd, 14 op/s wr
>
>
>
> I’m using the cephFs as a mail storage. I currently have 3500 mailbox some of them are IMAP the others are POP3 the goal is to be able to migrate all mailbox from my old
>
>
>
> infrastructure so around 30 000 mailbox.
>
>
>
> I’m now facing a problem :
>
> MDS_CLIENT_LATE_RELEASE 3 clients failing to respond to capability release
>
> mdsceph-n1(mds.0): Client mda3.sogetel.net failing to respond to capability releaseclient_id: 1134426
>
> mdsceph-n1(mds.0): Client mda2.sogetel.net failing to respond to capability releaseclient_id: 1172391
>
> mdsceph-n2(mds.1): Client mda3.sogetel.net failing to respond to capability releaseclient_id: 1134426
>
> MDS_SLOW_REQUEST 2 MDSs report slow requests
>
> mdsceph-n1(mds.0): 112 slow requests are blocked > 30 sec
>
> mdsceph-n2(mds.1): 323 slow requests are blocked > 30 sec
>
>
>
> I can’t figure out how to fix this…
>
>
>
>
> Here some information’s about my cluster :
>
> I’m running ceph luminous 12.2.5 on my 3 ceph nodes : ceph-n1, ceph-n2, ceph-n3.
>
>
> I have 3 client identical :
>
> LSB Version: :core-4.1-amd64:core-4.1-noarch
>
> Distributor ID: Fedora
>
> Description: Fedora release 25 (Twenty Five)
>
> Release: 25
>
> Codename: TwentyFive
>
I can't say for sure whether it would help, but I'd definitely suggest
upgrading those nodes to latest Fedora if you're using the kernel
client -- Fedora 25 hasn't received updates for quite some time.
John
>
> My ceph nodes :
>
>
>
> CentOS Linux release 7.5.1804 (Core)
>
> NAME="CentOS Linux"
>
> VERSION="7 (Core)"
>
> ID="centos"
>
> ID_LIKE="rhel fedora"
>
> VERSION_ID="7"
>
> PRETTY_NAME="CentOS Linux 7 (Core)"
>
> ANSI_COLOR="0;31"
>
> CPE_NAME="cpe:/o:centos:centos:7"
>
> HOME_URL="https://www.centos.org/";
>
> BUG_REPORT_URL="https://bugs.centos.org/";
>
>
>
> CENTOS_MANTISBT_PROJECT="CentOS-7"
>
> CENTOS_MANTISBT_PROJECT_VERSION="7"
>
> REDHAT_SUPPORT_PRODUCT="centos"
>
> REDHAT_SUPPORT_PRODUCT_VERSION="7"
>
>
>
> CentOS Linux release 7.5.1804 (Core)
>
> CentOS Linux release 7.5.1804 (Core)
>
>
>
> ceph daemon mds.ceph-n1 perf dump mds :
>
>
>
>
>
> "mds": {
>
> "request": 21968558,
>
> "reply": 21954801,
>
> "reply_latency": {
>
> "avgcount": 21954801,
>
> "sum": 100879.560315258,
>
> "avgtime": 0.004594874
>
> },
>
> "forward": 13627,
>
> "dir_fetch": 3327,
>
> "dir_commit": 162830,
>
> "dir_split": 1,
>
> "dir_merge": 0,
>
> "inode_max": 2147483647,
>
> "inodes": 68767,
>
> "inodes_top": 4524,
>
> "inodes_bottom": 56697,
>
> "inodes_pin_tail": 7546,
>
> "inodes_pinned": 62304,
>
> "inodes_expired": 1640159,
>
> "inodes_with_caps": 62192,
>
> "caps": 114126,
>
> "subtrees": 14,
>
> "traverse": 38309963,
>
> "traverse_hit": 37606227,
>
> "traverse_forward": 12189,
>
> "traverse_discover": 6634,
>
> "traverse_dir_fetch": 1769,
>
> "traverse_remote_ino": 6,
>
> "traverse_lock": 7731,
>
> "load_cent": 2196856701,
>
> "q": 0,
>
> "exported": 143,
>
> "exported_inodes": 291372,
>
> "imported": 125,
>
> "imported_inodes": 176509
>
>
>
>
>
> Thanks for your help…
>
>
>
> Regards
>
>
>
> Marc-Antoine
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
