Hi Magnus, Pacific can split the stray dirs so this "1 million deleted files in old snapshjots" limit no longer exists. Cheers, Dan On Thu, Oct 14, 2021 at 4:21 PM Magnus HAGDORN <Magnus.Hagdorn@xxxxxxxx> wrote: > > Hi all, > we seem to have recovered from our cephfs misadventure. Having said > that I would like to better understand what went wrong and if/how we > can avoid that in future. > > We have nautilus ceph cluster that provides cephfs to our school. We > keep nightly snapshots for one week. > > One user has a particularly deep directory structure with lots and lots > of tiny files (100s of millions). This user deleted a directory with ~2 > million entries in a deep directory structure. We noticed something > went wrong when we got 'No space left on device' messages when try to > delete a file. > > Eventually we figured out that we were exceeding the number of stray > files and once we set the mds_bal_fragment_size_max to 300000 we got > our service back (after lots of prodding that probably didn't help). > > So my understanding is that deleted files end up in a special directory > when snapshots are used. This directory is limited to 10^6 files by > default. > > I guess my main question is - does this problem still occur in more > recent versions of ceph than nautilus? If the problem does still occur, > can we mitigate it somehow? (we are wondering whether we should store > the millions of tiny files differently). > > Are snapshots the problem? Currently we snapshot at the top level. Does > it make sense to have multiple snapshots further down the tree? > > Cheers > magnus > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336. > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |