Re: CephFS and caching

[Les <les@xxxxxxxxxx>]

Cephfs can use fscache. I am testing it at the moment.

Some lines from my deployment process:

sudo apt-get install linux-generic-lts-utopic cachefilesd

sudo reboot

sudo mkdir /mnt/cephfs

sudo mkdir /mnt/ceph_cache

sudo mkfs -t xfs /dev/md3 # A 100gb local raid partition


sudo bash -c "echo /dev/md/3 /mnt/ceph_cache xfs defaults,noatime 0 0 >> /etc/fstab"

sudo bash -c "echo M1:6789,M2:6789:/ /mnt/cephfs ceph name=fsuser,secretfile=/etc/ceph/fsuser.secret,noatime,fsc,_netdev 0 0 >> /etc/fstab"

sudo bash -c "echo REDACTED > /etc/ceph/fsuser.secret"

sudo chmod 400 /etc/ceph/fsuser.secret

sudo mount /mnt/ceph_cache/

sudo sed -i 's/#RUN=yes/RUN=yes/g' /etc/default/cachefilesd

sudo vim /etc/cachefilesd.conf #Change dir to /mnt/ceph_cache, tag to ceph_cache and comment everything else

sudo service cachefilesd start

sudo mount /mnt/cephfs

Cheers, Les

On 04.09.2015 00:58, Kyle Hutson wrote:

I was wondering if anybody could give me some insight as to how CephFS does its caching - read-caching in particular.

We are using CephFS with an EC pool on the backend with a replicated cache pool in front of it. We're seeing some very slow read times. Trying to compute an md5sum on a 15GB file twice in a row (so it should be in cache) takes the time from 23 minutes down to 17 minutes, but this is over a 10Gbps network and with a crap-ton of OSDs (over 300), so I would expect it to be down in the 2-3 minute range.

I'm just trying to figure out what we can do to increase the performance. I have over 300 TB of live data that I have to be careful with, though, so I have to have some level of caution.

Is there some other caching we can do (client-side or server-side) that might give us a decent performance boost?

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com