Hi Xiubo Thank you for the explanation! This won't be a issue for us, but made me think twice :) Kind regards, Sake > Op 04-06-2024 12:30 CEST schreef Xiubo Li <xiubli@xxxxxxxxxx>: > > > On 6/4/24 15:20, Sake Ceph wrote: > > Hi, > > > > A little break into this thread, but I have some questions: > > * How does this happen, that the filesystem gets into readonly modus > > The detail explanation you can refer to the ceph PR: > https://github.com/ceph/ceph/pull/55421. > > > * Is this avoidable? > > * How-to fix the issue, because I didn't see a workaround in the mentioned tracker (or I missed it) > Possibly avoid changing data pools or disable multiple data pools? > > * With this bug around, should you use cephfs with reef? > > This will happen in all the releases, so that doesn't matter. > > - Xiubo > > > > > Kind regards, > > Sake > > > >> Op 04-06-2024 04:04 CEST schreef Xiubo Li <xiubli@xxxxxxxxxx>: > >> > >> > >> Hi Nicolas, > >> > >> This is a known issue and Venky is working on it, please see > >> https://tracker.ceph.com/issues/63259. > >> > >> Thanks > >> - Xiubo > >> > >> On 6/3/24 20:04, nbarbier@xxxxxxxxxxxxxxx wrote: > >>> Hello, > >>> > >>> First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode". > >>> > >>> The logs are showing : > >>> > >>> Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 > >>> Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly... > >>> > >>> After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to. > >>> > >>> Please find below additional details if needed. Thanks a lot ! > >>> > >>> Nicolas > >>> > >>> --- > >>> > >>> # ceph osd dump > >>> [...] > >>> pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 > >>> [...] > >>> > >>> # ceph osd lspools > >>> 1 .mgr > >>> 4 cephfs_data > >>> 5 cephfs_metadata > >>> 18 ec-pool-001-data > >>> 19 ec-pool-001-metadata > >>> > >>> > >>> # ceph df > >>> --- RAW STORAGE --- > >>> CLASS SIZE AVAIL USED RAW USED %RAW USED > >>> hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 > >>> TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0 > >>> > >>> --- POOLS --- > >>> POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL > >>> .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB > >>> cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB > >>> cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB > >>> ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB > >>> ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB > >>> > >>> > >>> > >>> # ceph status > >>> cluster: > >>> id: f16f53e1-7028-440f-bf48-f99912619c33 > >>> health: HEALTH_WARN > >>> 1 MDSs are read only > >>> > >>> services: > >>> mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) > >>> mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V > >>> mds: 1/1 daemons up, 3 standby > >>> osd: 48 osds: 48 up (since 2d), 48 in (since 3d) > >>> > >>> data: > >>> volumes: 1/1 healthy > >>> pools: 5 pools, 129 pgs > >>> objects: 30.97k objects, 42 GiB > >>> usage: 61 GiB used, 633 TiB / 633 TiB avail > >>> pgs: 129 active+clean > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users@xxxxxxx > >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >>> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |