See http://tracker.ceph.com/issues/38849 As an immediate workaround you can increase `mds bal fragment size max` to 200000 (which will increase the max number of strays to 2 million.) (Try injecting that option to the mds's -- I think it is read at runtime). And you don't need to stop the mds's and flush the journal/scan_links/etc... Just find where the old rm'd hardlinks are and ls -l their directories -- that should be enough to remove the files from stray. -- Dan On Tue, Mar 26, 2019 at 5:50 PM Toby Darling <toby@xxxxxxxxxxxxxxxxx> wrote: > > Hi > > [root@ceph1 ~]# ceph version > ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic > (stable) > > We've run into a "No space left on device" issue when trying to delete a > file, despite there being free space: > > [root@ceph1 ~]# ceph df > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 3.2 PiB 871 TiB 2.3 PiB 73.42 > POOLS: > NAME ID USED %USED MAX AVAIL > OBJECTS > mds_nvme 3 2.7 MiB 0 527 GiB > 2636225 > compressed_ecpool 4 2.3 PiB 92.13 197 TiB > 858679630 > ecpool_comp_TEST_ONLY 5 0 B 0 197 TiB > 0 > > Creating files/directories is fine. > > We do have 1M strays: > > [root@ceph1 ~]# ceph daemon mds.ceph1 perf dump | grep num_strays > "num_strays": 1000000, > "num_strays_delayed": 0, > "num_strays_enqueuing": 0, > > and I found a post from 2016 > (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-October/013646.html) > that suggests: > > run ' ceph daemon mds.xxx flush journal' to flush MDS journal > stop all mds > run 'cephfs-data-scan scan_links' > restart mds > run 'ceph daemon mds.x scrub_path / recursive repair' > > That was for jewel, is this still the recommended action for mimic? > > Cheers > Toby > -- > Toby Darling, Scientific Computing (2N249) > MRC Laboratory of Molecular Biology > Francis Crick Avenue > Cambridge Biomedical Campus > Cambridge CB2 0QH > Phone 01223 267070 > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |