I have been running some speed tests in POSIX file operations and I noticed even just listing files can take a while compared to an attached HDD. I am wondering is there a reason it takes so long to even just list files.
Here is the test I ran
time for i in {1..100000}; do touch $i; done
Internal HDD:
real 4m37.492s
user 0m18.125s
sys 1m5.040s
Ceph Dir
real 12m30.059s
user 0m16.749s
sys 0m53.451s
~300% faster on HDD
*I am actually ok with this but nice to be quicker.
When I am listing the directory it is taking a lot longer compared to an attached HDD
time ls -1
Internal HDD
real 0m2.112s
user 0m0.560s
sys 0m0.440s
Ceph Dir
real 3m35.982s
user 0m2.788s
sys 0m4.580s
~1000% faster on HDD
*I understand there is some time in the display so what is really making it odd is the following test.
time ls -1 > /dev/null
Internal HDD
real 0m0.367s
user 0m0.324s
sys 0m0.040s
Ceph Dir
real 0m2.807s
user 0m0.128s
sys 0m0.052s
~700% faster on HDD
My guess the performance issue is with the batch requests as you stated. So I am wondering if the file deletion of the 40M files is not just deleting the files but even just traversing that many files takes a while.
I am running this on 0.94.6 with Ceph Fuse Client
And config
fuse multithreaded = false
Since multithreaded crashes in hammer.
It would be interesting to see the performance on newer versions.
Any thoughts or comments would be good.
On Tue, Apr 5, 2016 at 9:22 AM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
On Mon, Apr 4, 2016 at 9:55 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
> Deletes are just slow right now. You can look at the ops in flight on you
> client or MDS admin socket to see how far along it is and watch them to see
> how long stuff is taking -- I think it's a sync disk commit for each unlink
> though so at 40M it's going to be a good looong while. :/
> -Greg
Oh good, I misremembered — it's a synchronous request to the MDS, but
it's not a synchronous disk commit. They get batched up normally in
the metadata log. :)
Still, a sync MDS request can take a little bit of time. Someday we
will make the client able to respond to these more quickly locally and
batch up MDS requests or something, but it'll be tricky. Faster file
creates will probably come first. (If we're lucky they can use some of
the same client-side machinery.)
-Greg
>
>
> On Monday, April 4, 2016, Kenneth Waegeman <kenneth.waegeman@xxxxxxxx>
> wrote:
>>
>> Hi all,
>>
>> I want to remove a large directory containing +- 40M files /160TB of data
>> in CephFS by running rm -rf on the directory via the ceph kernel client.
>> After 7h , the rm command is still running. I checked the rados df output,
>> and saw that only about 2TB and 2M files are gone.
>> I know this output of rados df can be confusing because ceph should delete
>> objects asyncroniously, but then I don't know why the rm command still
>> hangs.
>> Is there some way to speed this up? And is there a way to check how far
>> the marked for deletion has progressed ?
>>
>> Thank you very much!
>>
>> Kenneth
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
