On 10/20/2015 09:00 AM, Wido den Hollander wrote:
Hi,
In the "newstore direction" thread on ceph-devel I wrote that I'm using
bcache in production and Mark Nelson asked me to share some details.
Bcache is running in two clusters now that I manage, but I'll keep this
information to one of them (the one at PCextreme behind CloudStack).
In this cluster has been running for over 2 years now:
epoch 284353
fsid 0d56dd8f-7ae0-4447-b51b-f8b818749307
created 2013-09-23 11:06:11.819520
modified 2015-10-20 15:27:48.734213
The system consists out of 39 hosts:
2U SuperMicro chassis:
* 80GB Intel SSD for OS
* 240GB Intel S3700 SSD for Journaling + Bcache
* 6x 3TB disk
This isn't the newest hardware. The next batch of hardware will be more
disks per chassis, but this is it for now.
All systems were installed with Ubuntu 12.04, but they are all running
14.04 now with bcache.
The Intel S3700 SSD is partitioned with a GPT label:
- 5GB Journal for each OSD
- 200GB Partition for bcache
root@ceph11:~# df -h|grep osd
/dev/bcache0 2.8T 1.1T 1.8T 38% /var/lib/ceph/osd/ceph-60
/dev/bcache1 2.8T 1.2T 1.7T 41% /var/lib/ceph/osd/ceph-61
/dev/bcache2 2.8T 930G 1.9T 34% /var/lib/ceph/osd/ceph-62
/dev/bcache3 2.8T 970G 1.8T 35% /var/lib/ceph/osd/ceph-63
/dev/bcache4 2.8T 814G 2.0T 30% /var/lib/ceph/osd/ceph-64
/dev/bcache5 2.8T 915G 1.9T 33% /var/lib/ceph/osd/ceph-65
root@ceph11:~#
root@ceph11:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty
root@ceph11:~# uname -r
3.19.0-30-generic
root@ceph11:~#
"apply_latency": {
"avgcount": 2985023,
"sum": 226219.891559000
}
What did we notice?
- Less spikes on the disk
- Lower commit latencies on the OSDs
- Almost no 'slow requests' during backfills
- Cache-hit ratio of about 60%
Max backfills and recovery active are both set to 1 on all OSDs.
For the next generation hardware we are looking into using 3U chassis
with 16 4TB SATA drives and a 1.2TB NVM-E SSD for bcache, but we haven't
tested those yet, so nothing to say about it.
The current setup is 200GB of cache for 18TB of disks. The new setup
will be 1200GB for 64TB, curious to see what that does.
Our main conclusion however is that it does smoothen the I/O-pattern
towards the disks and that gives a overall better response of the disks.
Hi Wido, thanks for the big writeup! Did you guys happen to do any
benchmarking? I think Xiaoxi looked at flashcache a while back but had
mixed results if I remember right. It would be interesting to know how
bcache is affecting performance in different scenarios.
Thanks,
Mark
Wido
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
