Primary workload - rbd, kvm
пятница, 14 августа 2015 г. пользователь Ben Hines написал:
пятница, 14 августа 2015 г. пользователь Ben Hines написал:
Nice to hear that you have no SSD failures yet in 10months.
How many OSDs are you running, and what is your primary ceph workload?
(RBD, rgw, etc?)
-Ben
On Fri, Aug 14, 2015 at 2:23 AM, Межов Игорь Александрович
<megov@xxxxxxxxxx> wrote:
> Hi!
>
>
> Of course, it isn't cheap at all, but we use Intel DC S3700 200Gb for ceph
> journals
> and DC S3700 400Gb in the SSD pool: same hosts, separate root in crushmap.
>
> SSD pool are not yet in production, journаlling SSDs works under production
> load
> for 10 months. They're in good condition - no faults, no degradation.
>
> We specially take 200Gb SSD for journals to reduce costs, and also have a
> higher
> than recommended OSD/SSD ratio: 1 SSD per 10-12 ODS, whille recommended
> 1/3 to 1/6.
>
> So, as a conclusion - I'll recommend you to get a bigger budget and buy
> durable
> and fast SSDs for Ceph.
>
> Megov Igor
> CIO, Yuterra
>
> ________________________________
> От: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> от имени Voloshanenko
> Igor <igor.voloshanenko@xxxxxxxxx>
> Отправлено: 13 августа 2015 г. 15:54
> Кому: Jan Schermer
> Копия: ceph-users@xxxxxxxxxxxxxx
> Тема: Re: CEPH cache layer. Very slow
>
> So, good, but price for 845 DC PRO 400 GB higher in about 2x times than
> intel S3500 240G (((
>
> Any other models? (((
>
> 2015-08-13 15:45 GMT+03:00 Jan Schermer <jan@xxxxxxxxxxx>:
>>
>> I tested and can recommend the Samsung 845 DC PRO (make sure it is DC PRO
>> and not just "PRO" or "DC EVO"!).
>> Those were very cheap but are out of stock at the moment (here).
>> Faster than Intels, cheaper, and slightly different technology (3D V-NAND)
>> which IMO makes them superior without needing many tricks to do its job.
>>
>> Jan
>>
>> On 13 Aug 2015, at 14:40, Voloshanenko Igor <igor.voloshanenko@xxxxxxxxx>
>> wrote:
>>
>> Tnx, Irek! Will try!
>>
>> but another question to all, which SSD good enough for CEPH now?
>>
>> I'm looking into S3500 240G (I have some S3500 120G which show great
>> results. Around 8x times better than Samsung)
>>
>> Possible you can give advice about other vendors/models with same or below
>> price level as S3500 240G?
>>
>> 2015-08-13 12:11 GMT+03:00 Irek Fasikhov <malmyzh@xxxxxxxxx>:
>>>
>>> Hi, Igor.
>>> Try to roll the patch here:
>>>
>>> http://www.theirek.com/blog/2014/02/16/patch-dlia-raboty-s-enierghoniezavisimym-keshiem-ssd-diskov
>>>
>>> P.S. I am no longer tracks changes in this direction(kernel), because we
>>> use already recommended SSD
>>>
>>> С уважением, Фасихов Ирек Нургаязович
>>> Моб.: +79229045757
>>>
>>> 2015-08-13 11:56 GMT+03:00 Voloshanenko Igor
>>> <igor.voloshanenko@xxxxxxxxx>:
>>>>
>>>> So, after testing SSD (i wipe 1 SSD, and used it for tests)
>>>>
>>>> root@ix-s2:~# sudo fio --filename=/dev/sda --direct=1 --sync=1
>>>> --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based
>>>> --gr[53/1800]
>>>> ting --name=journal-test
>>>> journal-test: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync,
>>>> iodepth=1
>>>> fio-2.1.3
>>>> Starting 1 process
>>>> Jobs: 1 (f=1): [W] [100.0% done] [0KB/1152KB/0KB /s] [0/288/0 iops] [eta
>>>> 00m:00s]
>>>> journal-test: (groupid=0, jobs=1): err= 0: pid=2849460: Thu Aug 13
>>>> 10:46:42 2015
>>>> write: io=68972KB, bw=1149.6KB/s, iops=287, runt= 60001msec
>>>> clat (msec): min=2, max=15, avg= 3.48, stdev= 1.08
>>>> lat (msec): min=2, max=15, avg= 3.48, stdev= 1.08
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 2704], 5.00th=[ 2800], 10.00th=[ 2864], 20.00th=[
>>>> 2928],
>>>> | 30.00th=[ 3024], 40.00th=[ 3088], 50.00th=[ 3280], 60.00th=[
>>>> 3408],
>>>> | 70.00th=[ 3504], 80.00th=[ 3728], 90.00th=[ 3856], 95.00th=[
>>>> 4016],
>>>> | 99.00th=[ 9024], 99.50th=[ 9280], 99.90th=[ 9792],
>>>> 99.95th=[10048],
>>>> | 99.99th=[14912]
>>>> bw (KB /s): min= 1064, max= 1213, per=100.00%, avg=1150.07,
>>>> stdev=34.31
>>>> lat (msec) : 4=94.99%, 10=4.96%, 20=0.05%
>>>> cpu : usr=0.13%, sys=0.57%, ctx=17248, majf=0, minf=7
>>>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>>> >=64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> >=64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> >=64=0.0%
>>>> issued : total=r=0/w=17243/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=68972KB, aggrb=1149KB/s, minb=1149KB/s, maxb=1149KB/s,
>>>> mint=60001msec, maxt=60001msec
>>>>
>>>> Disk stats (read/write):
>>>> sda: ios=0/17224, merge=0/0, ticks=0/59584, in_queue=59576,
>>>> util=99.30%
>>>>
>>>> So, it's pain... SSD do only 287 iops on 4K... 1,1 MB/s
>>>>
>>>> I try to change cache mode :
>>>> echo temporary write through > /sys/class/scsi_disk/2:0:0:0/cache_type
>>>> echo temporary write through > /sys/class/scsi_disk/3:0:0:0/cache_type
>>>>
>>>> no luck, still same shit results, also i found this article:
>>>> https://lkml.org/lkml/2013/11/20/264 pointed to old very simple patch,
>>>> which disable CMD_FLUSH
>>>> https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414ba
>>>>
>>>> Has everybody better ideas, how to improve this? (or disable CMD_FLUSH
>>>> without recompile kernel, i used ubuntu and 4.0.4 for now (4.x branch
>>>> because SSD 850 Pro have issue with NCQ TRIM< and before 4.0.4 this
>>>> exception was not included into libsata.c)
>>>>
>>>> 2015-08-12 19:17 GMT+03:00 Pieter Koorts <pieter.koorts@xxxxxx>:
>>>>>
>>>>> Hi Igor
>>>>>
>>>>> I suspect you have very much the same problem as me.
>>>>>
>>>>> https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg22260.html
>>>>>
>>>>> Basically Samsung drives (like many SATA SSD's) are very much hit and
>>>>> miss so you will need to test them like described here to see if they are
>>>>> any good.
>>>>> http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
>>>>>
>>>>> To give you an idea my average performance went from 11MB/s (with
>>>>> Samsung SSD) to 30MB/s (without any SSD) on write performance. This is a
>>>>> very small cluster.
>>>>>
>>>>> Pieter
>>>>>
>>>>> On Aug 12, 2015, at 04:33 PM, Voloshanenko Igor
>>>>> <igor.voloshanenko@xxxxxxxxx> wrote:
>>>>>
>>>>> Hi all, we have setup CEPH cluster with 60 OSD (2 diff types) (5 nodes,
>>>>> 12 disks on each, 10 HDD, 2 SSD)
>>>>>
>>>>> Also we cover this with custom crushmap with 2 root leaf
>>>>>
>>>>> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
>>>>> -100 5.00000 root ssd
>>>>> -102 1.00000 host ix-s2-ssd
>>>>> 2 1.00000 osd.2 up 1.00000 1.00000
>>>>> 9 1.00000 osd.9 up 1.00000 1.00000
>>>>> -103 1.00000 host ix-s3-ssd
>>>>> 3 1.00000 osd.3 up 1.00000 1.00000
>>>>> 7 1.00000 osd.7 up 1.00000 1.00000
>>>>> -104 1.00000 host ix-s5-ssd
>>>>> 1 1.00000 osd.1 up 1.00000 1.00000
>>>>> 6 1.00000 osd.6 up 1.00000 1.00000
>>>>> -105 1.00000 host ix-s6-ssd
>>>>> 4 1.00000 osd.4 up 1.00000 1.00000
>>>>> 8 1.00000 osd.8 up 1.00000 1.00000
>>>>> -106 1.00000 host ix-s7-ssd
>>>>> 0 1.00000 osd.0 up 1.00000 1.00000
>>>>> 5 1.00000 osd.5 up 1.00000 1.00000
>>>>> -1 5.00000 root platter
>>>>> -2 1.00000 host ix-s2-platter
>>>>> 13 1.00000 osd.13 up 1.00000 1.00000
>>>>> 17 1.00000 osd.17 up 1.00000 1.00000
>>>>> 21 1.00000 osd.21 up 1.00000 1.00000
>>>>> 27 1.00000 osd.27 up 1.00000 1.00000
>>>>> 32 1.00000 osd.32 up 1.00000 1.00000
>>>>> 37 1.00000 osd.37 up 1.00000 1.00000
>>>>> 44 1.00000 osd.44 up 1.00000 1.00000
>>>>> 48 1.00000 osd.48 up 1.00000 1.00000
>>>>> 55 1.00000 osd.55 up 1.00000 1.00000
>>>>> 59 1.00000 osd.59 up 1.00000 1.00000
>>>>> -3 1.00000 host ix-s3-platter
>>>>> 14 1.00000 osd.14 up 1.00000 1.00000
>>>>> 18 1.00000 osd.18 up 1.00000 1.00000
>>>>> 23 1.00000 osd.23 up 1.00000 1.00000
>>>>> 28 1.00000 osd.28 up 1.00000 1.00000
>>>>> 33 1.00000 osd.33 up 1.00000 1.00000
>>>>> 39 1.00000 osd.39 up 1.00000 1.00000
>>>>> 43 1.00000 osd.43 up 1.00000 1.00000
>>>>> 47 1.00000 osd.47 up 1.00000 1.00000
>>>>> 54 1.00000 osd.54 up 1.00000 1.00000
>>>>> 58 1.00000 osd.58 up 1.00000 1.00000
>>>>> -4 1.00000 host ix-s5-platter
>>>>> 11 1.00000 osd.11 up 1.00000 1.00000
>>>>> 16 1.00000 osd.16 up 1.00000 1.00000
>>>>> 22 1.00000 osd.22 up 1.00000 1.00000
>>>>> 26 1.00000 osd.26 up 1.00000 1.00000
>>>>> 31 1.00000 osd.31 up 1.00000 1.00000
>>>>> 36 1.00000 osd.36 up 1.00000 1.00000
>>>>> 41 1.00000 osd.41 up 1.00000 1.00000
>>>>> 46 1.00000 osd.46 up 1.00000 1.00000
>>>>> 51 1.00000 osd.51 up 1.00000 1.00000
>>>>> 56 1.00000 osd.56 up 1.00000 1.00000
>>>>> -5 1.00000 host ix-s6-platter
>>>>> 12 1.00000 osd.12 up 1.00000 1.00000
>>>>> 19 1.00000 osd.19 up 1.00000 1.00000
>>>>> 24 1.00000 osd.24 up 1.00000 1.00000
>>>>> 29 1.00000 osd.29 up 1.00000 1.00000
>>>>> 34 1.00000 osd.34 up 1.00000 1.00000
>>>>> 38 1.00000 osd.38 up 1.00000 1.00000
>>>>> 42 1.00000 osd.42 up 1.00000 1.00000
>>>>> 50 1.00000 osd.50 up 1.00000 1.00000
>>>>> 53 1.00000 osd.53 up 1.00000 1.00000
>>>>> 57 1.00000 osd.57 up 1.00000 1.00000
>>>>> -6 1.00000 host ix-s7-platter
>>>>> 10 1.00000 osd.10 up 1.00000 1.00000
>>>>> 15 1.00000 osd.15 up 1.00000 1.00000
>>>>> 20 1.00000 osd.20 up 1.00000 1.00000
>>>>> 25 1.00000 osd.25 up 1.00000 1.00000
>>>>> 30 1.00000 osd.30 up 1.00000 1.00000
>>>>> 35 1.00000 osd.35 up 1.00000 1.00000
>>>>> 40 1.00000 osd.40 up 1.00000 1.00000
>>>>> 45 1.00000 osd.45 up 1.00000 1.00000
>>>>> 49 1.00000 osd.49 up 1.00000 1.00000
>>>>> 52 1.00000 osd.52 up 1.00000 1.00000
>>>>>
>>>>>
>>>>> Then create 2 pools, 1 on HDD (platters), 1 on SSD/
>>>>> and put SSD pul in from of HDD pool (cache tier)
>>>>>
>>>>> now we receive very bad performance results from cluster.
>>>>> Even with rados bench we received very unstable performance with even
>>>>> zero speed. So it's create very big issues for our clients.
>>>>>
>>>>> I try to tune all possible values, including OSD, but still no luck.
>>>>>
>>>>> Also very unbelievble situation, when i do
>>>>> ceph tell... bench on SSD OSD - i receive about 20MB/s
>>>>> If for HDD - 67 MB/s...
>>>>>
>>>>> I don;t understand why cache pools which consist of SSD works so bad...
>>>>> We used Samsung 850 Pro 256 Gb as SSDs
>>>>>
>>>>> Can you guys give me advice please...
>>>>>
>>>>> Also very idiotic thing, when i set cache-mode to forward and try to
>>>>> flush-evict all object (not all object evicted, some busy (locked on KVM
>>>>> sides). but now i receive quite stable results for rados bench
>>>>>
>>>>> Total time run: 30.275871
>>>>> Total writes made: 2076
>>>>> Write size: 4194304
>>>>> Bandwidth (MB/sec): 274.278
>>>>>
>>>>> Stddev Bandwidth: 75.1445
>>>>> Max bandwidth (MB/sec): 368
>>>>> Min bandwidth (MB/sec): 0
>>>>> Average Latency: 0.232892
>>>>> Stddev Latency: 0.240356
>>>>> Max latency: 2.01436
>>>>> Min latency: 0.0716344
>>>>>
>>>>> Without zeros, etc... So i don't understand how it's possible.
>>>>>
>>>>> Also interesting thing, when i disable overlay for pool, rados bench
>>>>> become around 70MB/s as for ordinary HDD, but in same time rados bench for
>>>>> SSD pool, which not used anymore show same bad results...
>>>>>
>>>>> So please, give me some direction to deeg...
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@xxxxxxxxxxxxxx
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
