On Thu, Sep 15, 2016 at 2:20 PM, Burkhard Linke <Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote: > Hi, > > does CephFS impose an upper limit on the number of files in a directory? > > > We currently have one directory with a large number of subdirectories: > > $ ls | wc -l > 158141 > > Creating a new subdirectory fails: > > $ touch foo > touch: cannot touch 'foo': No space left on device This limit was added recently: it's a limit on the size of a directory fragment. Previously folks were hitting nasty OSD issues with very large directory fragments, so we added this limit to give a clean failure instead. Directory fragmentation (mds_bal_frag setting) is turned off by default in Jewel: I was planning to get this activated by default in Kraken, but haven't quite got there yet. Once fragmentation is enabled you should find that the threshold for splitting dirfrags is hit well before you hit the safety limit that gives you ENOSPC. Note that if you set mds_bal_frag then you also need to use the "ceph fs set <myfs> allow_dirfrags true" (that command from memory so check the help if it's wrong), or the MDSs will ignore the setting. John > > Creating files in a different directory does not show any problems. The last > message in the MDS log relates to the large directory: > > 2016-09-15 07:51:54.539216 7f24ef2a6700 0 mds.0.bal replicating dir [dir > 100008bf9a6 /volumes/biodb/ncbi_genomes/all/ [2,head] auth v=57751905 cv=0/0 > ap=0+2+2 state=1073741826|complete f(v0 m2016-08-22 08:51:34.714570 > 158141=3+158138) n(v182285 rc2016-08-22 11:59:34.976373 b3360569421156 > 2989670=2478235+511435) hs=158141+842,ss=0+0 | child=1 waiter=0 authpin=0 > 0x7f252ca05f00] pop 12842 .. rdp 7353.96 adj 0 > > Any hints what might go wrong in this case? MDS is taken from the current > jewel git branch due to some pending backports: > > # ceph-mds --version > ceph version 10.2.2-508-g9bfc0cf (9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b) > > CephFS is mounted via kernel implementation: > > # uname -a > Linux waas 4.6.6-040606-generic #201608100733 SMP Wed Aug 10 11:35:29 UTC > 2016 x86_64 x86_64 x86_64 GNU/Linux > > ceph-fuse from jewel is also affected. > > > Regards, > Burkhard > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |