Re: Ceph OSDs with bcache experience

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 10/21/2015 03:30 PM, Mark Nelson wrote:
> 
> 
> On 10/21/2015 01:59 AM, Wido den Hollander wrote:
>> On 10/20/2015 07:44 PM, Mark Nelson wrote:
>>> On 10/20/2015 09:00 AM, Wido den Hollander wrote:
>>>> Hi,
>>>>
>>>> In the "newstore direction" thread on ceph-devel I wrote that I'm using
>>>> bcache in production and Mark Nelson asked me to share some details.
>>>>
>>>> Bcache is running in two clusters now that I manage, but I'll keep this
>>>> information to one of them (the one at PCextreme behind CloudStack).
>>>>
>>>> In this cluster has been running for over 2 years now:
>>>>
>>>> epoch 284353
>>>> fsid 0d56dd8f-7ae0-4447-b51b-f8b818749307
>>>> created 2013-09-23 11:06:11.819520
>>>> modified 2015-10-20 15:27:48.734213
>>>>
>>>> The system consists out of 39 hosts:
>>>>
>>>> 2U SuperMicro chassis:
>>>> * 80GB Intel SSD for OS
>>>> * 240GB Intel S3700 SSD for Journaling + Bcache
>>>> * 6x 3TB disk
>>>>
>>>> This isn't the newest hardware. The next batch of hardware will be more
>>>> disks per chassis, but this is it for now.
>>>>
>>>> All systems were installed with Ubuntu 12.04, but they are all running
>>>> 14.04 now with bcache.
>>>>
>>>> The Intel S3700 SSD is partitioned with a GPT label:
>>>> - 5GB Journal for each OSD
>>>> - 200GB Partition for bcache
>>>>
>>>> root@ceph11:~# df -h|grep osd
>>>> /dev/bcache0    2.8T  1.1T  1.8T  38% /var/lib/ceph/osd/ceph-60
>>>> /dev/bcache1    2.8T  1.2T  1.7T  41% /var/lib/ceph/osd/ceph-61
>>>> /dev/bcache2    2.8T  930G  1.9T  34% /var/lib/ceph/osd/ceph-62
>>>> /dev/bcache3    2.8T  970G  1.8T  35% /var/lib/ceph/osd/ceph-63
>>>> /dev/bcache4    2.8T  814G  2.0T  30% /var/lib/ceph/osd/ceph-64
>>>> /dev/bcache5    2.8T  915G  1.9T  33% /var/lib/ceph/osd/ceph-65
>>>> root@ceph11:~#
>>>>
>>>> root@ceph11:~# lsb_release -a
>>>> No LSB modules are available.
>>>> Distributor ID:    Ubuntu
>>>> Description:    Ubuntu 14.04.3 LTS
>>>> Release:    14.04
>>>> Codename:    trusty
>>>> root@ceph11:~# uname -r
>>>> 3.19.0-30-generic
>>>> root@ceph11:~#
>>>>
>>>> "apply_latency": {
>>>>       "avgcount": 2985023,
>>>>       "sum": 226219.891559000
>>>> }
>>>>
>>>> What did we notice?
>>>> - Less spikes on the disk
>>>> - Lower commit latencies on the OSDs
>>>> - Almost no 'slow requests' during backfills
>>>> - Cache-hit ratio of about 60%
>>>>
>>>> Max backfills and recovery active are both set to 1 on all OSDs.
>>>>
>>>> For the next generation hardware we are looking into using 3U chassis
>>>> with 16 4TB SATA drives and a 1.2TB NVM-E SSD for bcache, but we
>>>> haven't
>>>> tested those yet, so nothing to say about it.
>>>>
>>>> The current setup is 200GB of cache for 18TB of disks. The new setup
>>>> will be 1200GB for 64TB, curious to see what that does.
>>>>
>>>> Our main conclusion however is that it does smoothen the I/O-pattern
>>>> towards the disks and that gives a overall better response of the
>>>> disks.
>>>
>>> Hi Wido, thanks for the big writeup!  Did you guys happen to do any
>>> benchmarking?  I think Xiaoxi looked at flashcache a while back but had
>>> mixed results if I remember right.  It would be interesting to know how
>>> bcache is affecting performance in different scenarios.
>>>
>>
>> No, we didn't do any benchmarking. Initially this cluster was build for
>> just the RADOS Gateway, so we went for 2Gbit (2x 1Gbit) per machine. 90%
>> is still Gbit networking and we are in the process of upgrading it all
>> to 10Gbit.
>>
>> Since the 1Gbit network latency is about 4 times higher then 10Gbit we
>> aren't really benchmarking the cluster.
>>
>> What counts for us most is that we can do recovery operations without
>> any slow requests.
>>
>> Before bcache we saw disks spike to 100% busy while a backfill was busy.
>> Now bcache smoothens this and we see peaks of maybe 70%, but that's it.
> 
> In the testing I was doing to figure out our new lab hardware, I was
> seeing SSDs handle recovery dramatically better than spinning disks as
> well during cephtestrados runs.  It might be worth digging in to see
> what the IO patterns look like.  In the mean time though, it's very
> interesting that bcache helps in this case so much.  Good to know!
> 

Keep in mind that CentOS 7.1 doesn't support bcache natively in the kernel.

Would be nice if RHEL/CentOS would also support bcache.

Wido

>>
>>> Thanks,
>>> Mark
>>>
>>>>
>>>> Wido
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@xxxxxxxxxxxxxx
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux