On Wed, Apr 6, 2016 at 10:42 PM, Scottix <scottix@xxxxxxxxx> wrote:
> I have been running some speed tests in POSIX file operations and I noticed
> even just listing files can take a while compared to an attached HDD. I am
> wondering is there a reason it takes so long to even just list files.
If you're running comparisons, it would really be more instructive to
compare ceph with something like an NFS server, rather than a local
filesystem.
> Here is the test I ran
>
> time for i in {1..100000}; do touch $i; done
>
> Internal HDD:
> real 4m37.492s
> user 0m18.125s
> sys 1m5.040s
>
> Ceph Dir
> real 12m30.059s
> user 0m16.749s
> sys 0m53.451s
>
> ~300% faster on HDD
>
> *I am actually ok with this but nice to be quicker.
>
> When I am listing the directory it is taking a lot longer compared to an
> attached HDD
>
> time ls -1
>
> Internal HDD
> real 0m2.112s
> user 0m0.560s
> sys 0m0.440s
>
> Ceph Dir
> real 3m35.982s
> user 0m2.788s
> sys 0m4.580s
>
> ~1000% faster on HDD
>
> *I understand there is some time in the display so what is really making it
> odd is the following test.
>
> time ls -1 > /dev/null
>
> Internal HDD
> real 0m0.367s
> user 0m0.324s
> sys 0m0.040s
>
> Ceph Dir
> real 0m2.807s
> user 0m0.128s
> sys 0m0.052s
If the difference when sending to /dev/null is reproducible (not just
a cache artifact), I would suspect that your `ls` is noticing that
it's not talking to a tty, so it's not bothering to color things, so
it's not individually statting each file to decide what color to make
it. On network filesystems, "ls -l" (or colored ls) is often much
slower than a straight directory listing.
Cheers,
John
> ~700% faster on HDD
>
> My guess the performance issue is with the batch requests as you stated. So
> I am wondering if the file deletion of the 40M files is not just deleting
> the files but even just traversing that many files takes a while.
It's an unhappy feedback combination of listing them, sending N
individual unlink operations, and then the MDS getting bogged down in
the resulting purges while it's still trying to handle incoming unlink
requests.
> I am running this on 0.94.6 with Ceph Fuse Client
> And config
> fuse multithreaded = false
>
> Since multithreaded crashes in hammer.
>
> It would be interesting to see the performance on newer versions.
>
> Any thoughts or comments would be good.
>
> On Tue, Apr 5, 2016 at 9:22 AM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
>>
>> On Mon, Apr 4, 2016 at 9:55 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
>> > Deletes are just slow right now. You can look at the ops in flight on
>> > you
>> > client or MDS admin socket to see how far along it is and watch them to
>> > see
>> > how long stuff is taking -- I think it's a sync disk commit for each
>> > unlink
>> > though so at 40M it's going to be a good looong while. :/
>> > -Greg
>>
>> Oh good, I misremembered — it's a synchronous request to the MDS, but
>> it's not a synchronous disk commit. They get batched up normally in
>> the metadata log. :)
>> Still, a sync MDS request can take a little bit of time. Someday we
>> will make the client able to respond to these more quickly locally and
>> batch up MDS requests or something, but it'll be tricky. Faster file
>> creates will probably come first. (If we're lucky they can use some of
>> the same client-side machinery.)
>> -Greg
>>
>> >
>> >
>> > On Monday, April 4, 2016, Kenneth Waegeman <kenneth.waegeman@xxxxxxxx>
>> > wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I want to remove a large directory containing +- 40M files /160TB of
>> >> data
>> >> in CephFS by running rm -rf on the directory via the ceph kernel
>> >> client.
>> >> After 7h , the rm command is still running. I checked the rados df
>> >> output,
>> >> and saw that only about 2TB and 2M files are gone.
>> >> I know this output of rados df can be confusing because ceph should
>> >> delete
>> >> objects asyncroniously, but then I don't know why the rm command still
>> >> hangs.
>> >> Is there some way to speed this up? And is there a way to check how far
>> >> the marked for deletion has progressed ?
>> >>
>> >> Thank you very much!
>> >>
>> >> Kenneth
>> >>
>> >> _______________________________________________
>> >> ceph-users mailing list
>> >> ceph-users@xxxxxxxxxxxxxx
>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
