Hi Eugen, Thanks for the reply, really appreciate The first command , just hang with no output # cephfs-journal-tool --rank=cephfs:0 --journal=mdlog journal inspect The second command # cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal inspect Overall journal integrity: OK root@rke-sh1-2:~# cephadm logs --fsid fcb373ce-7aaa-11eb-984f-e7c6e0038e87 --name mds.cephfs.rke-sh1-2.isqjza -- Logs begin at Fri 2024-02-23 04:49:32 UTC, end at Fri 2024-02-23 13:08:22 UTC. -- Feb 23 07:46:46 rke-sh1-2 bash[1058012]: ignoring --setuser ceph since I am not root Feb 23 07:46:46 rke-sh1-2 bash[1058012]: ignoring --setgroup ceph since I am not root Feb 23 07:46:46 rke-sh1-2 bash[1058012]: starting mds.cephfs.rke-sh1-2.isqjza at Feb 23 08:15:06 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:06.371+0000 7fbc17dd9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. Feb 23 08:15:13 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x1 object, errno -22 Feb 23 08:15:13 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 mds.0.12487 unhandled write error (22) Invalid argument, force readonly... Feb 23 10:20:36 rke-sh1-2 bash[1058012]: debug 2024-02-23T10:20:36.309+0000 7fbc17dd9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. root@rke-sh1-3:~# cephadm logs --fsid fcb373ce-7aaa-11eb-984f-e7c6e0038e87 --name mds.cephfs.rke-sh1-3.vdicdn -- Logs begin at Fri 2024-02-23 06:59:48 UTC, end at Fri 2024-02-23 13:09:18 UTC. -- Feb 23 07:46:46 rke-sh1-3 bash[2901]: ignoring --setuser ceph since I am not root Feb 23 07:46:46 rke-sh1-3 bash[2901]: ignoring --setgroup ceph since I am not root Feb 23 07:46:46 rke-sh1-3 bash[2901]: starting mds.cephfs.rke-sh1-3.vdicdn at Feb 23 10:25:51 rke-sh1-3 bash[2901]: ignoring --setuser ceph since I am not root Feb 23 10:25:51 rke-sh1-3 bash[2901]: ignoring --setgroup ceph since I am not root Feb 23 10:25:51 rke-sh1-3 bash[2901]: starting mds.cephfs.rke-sh1-3.vdicdn at debug2: channel 0: request window-change confirm 0 debug3: send packet: type 98 -- Logs begin at Fri 2024-02-23 00:24:42 UTC, end at Fri 2024-02-23 13:09:55 UTC. -- Feb 23 09:29:10 rke-sh1-1 bash[786820]: tcmalloc: large alloc 1073750016 bytes == 0x5598512de000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41da6> Feb 23 09:29:19 rke-sh1-1 bash[786820]: tcmalloc: large alloc 2147491840 bytes == 0x559891ae0000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41db3> Feb 23 09:29:26 rke-sh1-1 bash[786820]: tcmalloc: large alloc 2147491840 bytes == 0x559951ae4000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41da6> Feb 23 09:29:27 rke-sh1-1 bash[786820]: debug 2024-02-23T09:29:27.928+0000 7fb416d63700 -1 asok(0x5597c3904000) AdminSocket: error writing response length (32) Broken pipe Feb 23 12:35:53 rke-sh1-1 bash[786820]: ignoring --setuser ceph since I am not root Feb 23 12:35:53 rke-sh1-1 bash[786820]: ignoring --setgroup ceph since I am not root Feb 23 12:35:53 rke-sh1-1 bash[786820]: starting mds.cephfs.rke-sh1-1.ojmpnk at The logs of the MDS are in verbose 20 , do you want me to provide on a archive ? Is there a way to compact all the logs ? Best Regards, Edouard FAZENDA Technical Support Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 www.csti.ch -----Original Message----- From: Eugen Block <eblock@xxxxxx> Sent: vendredi, 23 février 2024 12:50 To: ceph-users@xxxxxxx Subject: Re: MDS in ReadOnly and 2 MDS behind on trimming Hi, the mds log should contain information why it goes into read-only mode. Just a few weeks ago I helped a user with a broken CephFS (MDS went into read-only mode because of missing objects in the journal). Can you check the journal status: # cephfs-journal-tool --rank=cephfs:0 --journal=mdlog journal inspect # cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal inspect and also share the logs. Thanks, Eugen Zitat von Edouard FAZENDA <e.fazenda@xxxxxxx>: > Dear Ceph Community, > > > > I am having an issue with my Ceph Cluster , there were several osd > crashing but now active and recovery finished and now the CephFS > filesystem cannot be access by clients in RW (K8S worklod) as the 1 > MDS is in Read-Only and 2 are being on trimming > > > > The cephfs seems to have volume OK > > > > The trimming process seems not going further, maybe stuck ? > > > > We are running 3 hosts using ceph Pacific version 16.2.1 > > > > Here some logs on the situation : > > > > ceph versions > > { > > "mon": { > > "ceph version 16.2.1 > (afb9061ab4117f798c858c741efa6390e48ccf10) > pacific (stable)": 3 > > }, > > "mgr": { > > "ceph version 16.2.1 > (afb9061ab4117f798c858c741efa6390e48ccf10) > pacific (stable)": 3 > > }, > > "osd": { > > "ceph version 16.2.1 > (afb9061ab4117f798c858c741efa6390e48ccf10) > pacific (stable)": 18 > > }, > > "mds": { > > "ceph version 16.2.1 > (afb9061ab4117f798c858c741efa6390e48ccf10) > pacific (stable)": 3 > > }, > > "rgw": { > > "ceph version 16.2.1 > (afb9061ab4117f798c858c741efa6390e48ccf10) > pacific (stable)": 6 > > }, > > "overall": { > > "ceph version 16.2.1 > (afb9061ab4117f798c858c741efa6390e48ccf10) > pacific (stable)": 33 > > } > > } > > > > ceph orch ps > > NAME HOST STATUS REFRESHED AGE > PORTS VERSION IMAGE ID CONTAINER ID > > crash.rke-sh1-1 rke-sh1-1 running (21h) 36s ago 21h - > 16.2.1 c757e4a3636b e8652edb2b49 > > crash.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 20M - > 16.2.1 c757e4a3636b a1249a605ee0 > > crash.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - > 16.2.1 c757e4a3636b 026667bc1776 > > mds.cephfs.rke-sh1-1.ojmpnk rke-sh1-1 running (18h) 36s ago 4M - > 16.2.1 c757e4a3636b 9b4c2b08b759 > > mds.cephfs.rke-sh1-2.isqjza rke-sh1-2 running (18h) 3m ago 23M - > 16.2.1 c757e4a3636b 71681a5f34d3 > > mds.cephfs.rke-sh1-3.vdicdn rke-sh1-3 running (17h) 36s ago 3M - > 16.2.1 c757e4a3636b e89946ad6b7e > > mgr.rke-sh1-1.qskoyj rke-sh1-1 running (21h) 36s ago 2y > *:8082 *:9283 16.2.1 c757e4a3636b 7ce7cfbb3e55 > > mgr.rke-sh1-2.lxmguj rke-sh1-2 running (21h) 3m ago 22M > *:8082 *:9283 16.2.1 c757e4a3636b 5a0025adfd46 > > mgr.rke-sh1-3.ckunvo rke-sh1-3 running (17h) 36s ago 6M > *:8082 *:9283 16.2.1 c757e4a3636b 2fcaf18f3218 > > mon.rke-sh1-1 rke-sh1-1 running (20h) 36s ago 20h - > 16.2.1 c757e4a3636b c0a90103cabc > > mon.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 3M - > 16.2.1 c757e4a3636b f4b32ba4466b > > mon.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - > 16.2.1 c757e4a3636b d5e44c245998 > > osd.0 rke-sh1-2 running (20h) 3m ago 2y - > 16.2.1 c757e4a3636b 7b0e69942c15 > > osd.1 rke-sh1-3 running (17h) 36s ago 2y - > 16.2.1 c757e4a3636b 4451654d9a2d > > osd.10 rke-sh1-3 running (17h) 36s ago 2y - > 16.2.1 c757e4a3636b 3f9d5f95e284 > > osd.11 rke-sh1-1 running (21h) 36s ago 2y - > 16.2.1 c757e4a3636b db1cc6d2e37f > > osd.12 rke-sh1-2 running (21h) 3m ago 2y - > 16.2.1 c757e4a3636b de416c1ef766 > > osd.13 rke-sh1-3 running (17h) 36s ago 2y - > 16.2.1 c757e4a3636b 25a281cc5a9b > > osd.14 rke-sh1-1 running (21h) 36s ago 2y - > 16.2.1 c757e4a3636b 62f25ba61667 > > osd.15 rke-sh1-2 running (21h) 3m ago 2y - > 16.2.1 c757e4a3636b d3514d823c45 > > osd.16 rke-sh1-3 running (17h) 36s ago 2y - > 16.2.1 c757e4a3636b bba857759bfe > > osd.17 rke-sh1-1 running (21h) 36s ago 2y - > 16.2.1 c757e4a3636b 59281d4bb3d0 > > osd.2 rke-sh1-1 running (21h) 36s ago 2y - > 16.2.1 c757e4a3636b 418041b5e60d > > osd.3 rke-sh1-2 running (21h) 3m ago 2y - > 16.2.1 c757e4a3636b 04a0e29d5623 > > osd.4 rke-sh1-1 running (20h) 36s ago 2y - > 16.2.1 c757e4a3636b 1cc78a5153d3 > > osd.5 rke-sh1-3 running (17h) 36s ago 2y - > 16.2.1 c757e4a3636b 39a4b11e31fb > > osd.6 rke-sh1-2 running (21h) 3m ago 2y - > 16.2.1 c757e4a3636b 2f218ffb566e > > osd.7 rke-sh1-1 running (20h) 36s ago 2y - > 16.2.1 c757e4a3636b cf761fbe4d5f > > osd.8 rke-sh1-3 running (17h) 36s ago 2y - > 16.2.1 c757e4a3636b f9f85480e800 > > osd.9 rke-sh1-2 running (21h) 3m ago 2y - > 16.2.1 c757e4a3636b 664c54ff46d2 > > rgw.default.rke-sh1-1.dgucwl rke-sh1-1 running (21h) 36s ago 22M > *:8000 16.2.1 c757e4a3636b f03212b955a7 > > rgw.default.rke-sh1-1.vylchc rke-sh1-1 running (21h) 36s ago 22M > *:8001 16.2.1 c757e4a3636b da486ce43fe5 > > rgw.default.rke-sh1-2.dfhhfw rke-sh1-2 running (21h) 3m ago 2y > *:8000 16.2.1 c757e4a3636b ef4089d0aef2 > > rgw.default.rke-sh1-2.efkbum rke-sh1-2 running (21h) 3m ago 2y > *:8001 16.2.1 c757e4a3636b 9e053d5a2f7b > > rgw.default.rke-sh1-3.krfgey rke-sh1-3 running (17h) 36s ago 9M > *:8001 16.2.1 c757e4a3636b 45cd3d75edd3 > > rgw.default.rke-sh1-3.pwdbmp rke-sh1-3 running (17h) 36s ago 9M > *:8000 16.2.1 c757e4a3636b e2710265a7f4 > > > > ceph health detail > > HEALTH_WARN 1 MDSs are read only; 2 MDSs behind on trimming > > [WRN] MDS_READ_ONLY: 1 MDSs are read only > > mds.cephfs.rke-sh1-2.isqjza(mds.0): MDS in read-only mode > > [WRN] MDS_TRIM: 2 MDSs behind on trimming > > mds.cephfs.rke-sh1-2.isqjza(mds.0): Behind on trimming (2149/128) > max_segments: 128, num_segments: 2149 > > mds.cephfs.rke-sh1-1.ojmpnk(mds.0): Behind on trimming (2149/128) > max_segments: 128, num_segments: 2149 > > > > root@rke-sh1-1:~# ceph fs status > > cephfs - 27 clients > > ====== > > RANK STATE MDS ACTIVITY DNS INOS > DIRS CAPS > > 0 active cephfs.rke-sh1-2.isqjza Reqs: 8 /s 85.2k 53.2k > 1742 101 > > 0-s standby-replay cephfs.rke-sh1-1.ojmpnk Evts: 0 /s 52.2k 20.2k > 1737 0 > > POOL TYPE USED AVAIL > > cephfs_metadata metadata 1109G 6082G > > cephfs_data data 8419G 6082G > > STANDBY MDS > > cephfs.rke-sh1-3.vdicdn > > MDS version: ceph version 16.2.1 > (afb9061ab4117f798c858c741efa6390e48ccf10) > pacific (stable) > > > > ceph status > > cluster: > > id: fcb373ce-7aaa-11eb-984f-e7c6e0038e87 > > health: HEALTH_WARN > > 1 MDSs are read only > > 2 MDSs behind on trimming > > > > services: > > mon: 3 daemons, quorum rke-sh1-2,rke-sh1-1,rke-sh1-3 (age 17h) > > mgr: rke-sh1-1.qskoyj(active, since 17h), standbys: > rke-sh1-2.lxmguj, rke-sh1-3.ckunvo > > mds: 1/1 daemons up, 1 standby, 1 hot standby > > osd: 18 osds: 18 up (since 17h), 18 in (since 20h) > > rgw: 6 daemons active (3 hosts, 1 zones) > > > > data: > > volumes: 1/1 healthy > > pools: 11 pools, 849 pgs > > objects: 10.10M objects, 5.3 TiB > > usage: 11 TiB used, 15 TiB / 26 TiB avail > > pgs: 849 active+clean > > > > io: > > client: 35 KiB/s rd, 1.0 MiB/s wr, 302 op/s rd, 165 op/s wr > > > > > > # ceph mds stat > > cephfs:1 {0=cephfs.rke-sh1-2.isqjza=up:active} 1 up:standby-replay 1 > up:standby > > > > Have you got an idea on what could be my next steps to bring the > cluster healthy ? > > > > Help will very be appreciated. > > > > Thank a lot for your feedback. > > > > Best Regards, > > > > Edouard FAZENDA > > Technical Support > > > > > > > > Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 > > > > <https://www.csti.ch/> www.csti.ch _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx