Hi Dan Thanks! ceph tell mds.ceph1 config set mds_bal_fragment_size_max 200000 got us running again. Cheers toby On 26/03/2019 16:56, Dan van der Ster wrote: > See http://tracker.ceph.com/issues/38849 > > As an immediate workaround you can increase `mds bal fragment size > max` to 200000 (which will increase the max number of strays to 2 > million.) > (Try injecting that option to the mds's -- I think it is read at runtime). > > And you don't need to stop the mds's and flush the journal/scan_links/etc... > Just find where the old rm'd hardlinks are and ls -l their directories > -- that should be enough to remove the files from stray. > > -- Dan > > > On Tue, Mar 26, 2019 at 5:50 PM Toby Darling <toby@xxxxxxxxxxxxxxxxx> wrote: >> >> Hi >> >> [root@ceph1 ~]# ceph version >> ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic >> (stable) >> >> We've run into a "No space left on device" issue when trying to delete a >> file, despite there being free space: >> >> [root@ceph1 ~]# ceph df >> GLOBAL: >> SIZE AVAIL RAW USED %RAW USED >> 3.2 PiB 871 TiB 2.3 PiB 73.42 >> POOLS: >> NAME ID USED %USED MAX AVAIL >> OBJECTS >> mds_nvme 3 2.7 MiB 0 527 GiB >> 2636225 >> compressed_ecpool 4 2.3 PiB 92.13 197 TiB >> 858679630 >> ecpool_comp_TEST_ONLY 5 0 B 0 197 TiB >> 0 >> >> Creating files/directories is fine. >> >> We do have 1M strays: >> >> [root@ceph1 ~]# ceph daemon mds.ceph1 perf dump | grep num_strays >> "num_strays": 1000000, >> "num_strays_delayed": 0, >> "num_strays_enqueuing": 0, >> >> and I found a post from 2016 >> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-October/013646.html) >> that suggests: >> >> run ' ceph daemon mds.xxx flush journal' to flush MDS journal >> stop all mds >> run 'cephfs-data-scan scan_links' >> restart mds >> run 'ceph daemon mds.x scrub_path / recursive repair' >> >> That was for jewel, is this still the recommended action for mimic? >> >> Cheers >> Toby >> -- >> Toby Darling, Scientific Computing (2N249) >> MRC Laboratory of Molecular Biology >> Francis Crick Avenue >> Cambridge Biomedical Campus >> Cambridge CB2 0QH >> Phone 01223 267070 >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |