Hi Nick, How large NVMe drives are you running per 12 disks? In my current setup I have 4xP3700 per 36 disks but I feel like I could get by with 2… Just looking for community experience :-) Cheers, Zoltan > On 07 Jul 2016, at 10:45, Nick Fisk <nick@xxxxxxxxxx> wrote: > > Just to add if you really want to go with lots of HDD's to Journals then go > NVME. They are not a lot more expensive than the equivalent SATA based > 3700's, but the latency is low low low. Here is an example of a node I have > just commissioned with 12 HDD's to one P3700 > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > sdb 0.00 0.00 68.00 0.00 8210.00 0.00 241.47 > 0.26 3.85 3.85 0.00 2.09 14.20 > sdd 2.50 0.00 198.50 22.00 24938.00 9422.00 311.66 > 4.34 27.80 6.21 222.64 2.45 54.00 > sdc 0.00 0.00 63.00 0.00 7760.00 0.00 246.35 > 0.15 2.16 2.16 0.00 1.56 9.80 > sda 0.00 0.00 61.50 47.00 7600.00 22424.00 553.44 > 2.77 25.57 2.63 55.57 3.82 41.40 > nvme0n1 0.00 22.50 2.00 2605.00 8.00 139638.00 107.13 > 0.14 0.05 0.00 0.05 0.03 6.60 > sdg 0.00 0.00 61.00 28.00 6230.00 12696.00 425.30 > 3.66 74.79 5.84 225.00 3.87 34.40 > sdf 0.00 0.00 34.50 47.00 4108.00 21702.00 633.37 > 3.56 43.75 1.51 74.77 2.85 23.20 > sdh 0.00 0.00 75.00 15.50 9180.00 4984.00 313.02 > 0.45 12.55 3.28 57.42 3.51 31.80 > sdi 1.50 0.50 142.00 48.50 18102.00 21924.00 420.22 > 3.60 18.92 4.99 59.71 2.70 51.40 > sdj 0.50 0.00 74.50 5.00 9362.00 1832.00 281.61 > 0.33 4.10 3.33 15.60 2.44 19.40 > sdk 0.00 0.00 54.00 0.00 6420.00 0.00 237.78 > 0.12 2.30 2.30 0.00 1.70 9.20 > sdl 0.00 0.00 21.00 1.50 2286.00 16.00 204.62 > 0.32 18.13 13.81 78.67 6.67 15.00 > sde 0.00 0.00 98.00 0.00 12304.00 0.00 251.10 > 0.30 3.10 3.10 0.00 2.08 20.40 > > 50us latency at 2605 iops!!! > > Compared to one of the other nodes with 2 100GB S3700's, 6 disks each > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > sda 0.00 30.50 0.00 894.50 0.00 50082.00 111.98 > 0.36 0.41 0.00 0.41 0.20 17.80 > sdb 0.00 9.00 0.00 551.00 0.00 32044.00 116.31 > 0.23 0.42 0.00 0.42 0.19 10.40 > sdc 0.00 2.00 6.50 17.50 278.00 8422.00 725.00 > 1.08 44.92 18.46 54.74 8.08 19.40 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sde 0.00 2.50 27.50 21.50 2112.00 9866.00 488.90 > 0.59 12.04 6.91 18.60 6.53 32.00 > sdf 0.50 0.00 50.50 0.00 6170.00 0.00 244.36 > 0.18 4.63 4.63 0.00 2.10 10.60 > md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdg 0.00 1.50 32.00 386.50 3970.00 12188.00 77.22 > 0.15 0.35 0.50 0.34 0.15 6.40 > sdh 0.00 0.00 6.00 0.00 34.00 0.00 11.33 > 0.07 12.67 12.67 0.00 11.00 6.60 > sdi 0.00 0.50 1.50 19.50 6.00 8862.00 844.57 > 0.96 45.71 33.33 46.67 6.57 13.80 > sdj 0.00 0.00 67.00 0.00 8214.00 0.00 245.19 > 0.17 2.51 2.51 0.00 1.88 12.60 > sdk 1.50 2.50 61.00 48.00 6216.00 21020.00 499.74 > 2.01 18.46 11.41 27.42 5.05 55.00 > sdm 0.00 0.00 30.50 0.00 3576.00 0.00 234.49 > 0.07 2.43 2.43 0.00 1.90 5.80 > sdl 0.00 4.50 25.00 23.50 2092.00 12648.00 607.84 > 1.36 19.42 5.60 34.13 4.04 19.60 > sdn 0.50 0.00 23.00 0.00 2670.00 0.00 232.17 > 0.07 2.96 2.96 0.00 2.43 5.60 > > Pretty much 10x the latency. I'm seriously impressed with these NVME things. > > >> -----Original Message----- >> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of >> Christian Balzer >> Sent: 07 July 2016 03:23 >> To: ceph-users@xxxxxxxxxxxxxx >> Subject: Re: multiple journals on SSD >> >> >> Hello, >> >> I have a multitude of of problems with the benchmarks and conclusions > here, >> more below. >> >> But firstly to address the question of the OP, definitely not filesystem > based >> journals. >> Another layer of overhead and delays, something I'd be willing to ignore > if >> we're talking about a full SSD as OSD with an inline journal, but not with >> journal SSDs. >> Similar with LVM, though with a lower impact. >> >> Partitions really are your best bet. >> >> On Wed, 6 Jul 2016 18:20:43 +0300 George Shuklin wrote: >> >>> Yes. >>> >>> On my lab (not production yet) with 9 7200 SATA (OSD) and one INTEL >>> SSDSC2BB800G4 (800G, 9 journals) >> >> First and foremost, a DC 3510 with 1 DWPD endurance is not my idea of good >> journal device, even if it had the performance. >> If you search in the ML archives there is at least one case where somebody >> lost a full storage node precisely because their DC S3500s were worn out: >> https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg28083.html >> >> Unless you have a read-mostly cluster, a 400GB DC S3610 (same or lower >> price) would be a better deal, at 50% more endurance and only slightly > lower >> sequential write speed. >> >> And depending on your expected write volume (which you should >> know/estimate as close as possible before buying HW), a 400GB DC S3710 >> might be the best deal when it comes to TBW/$. >> It's 30% more expensive than your 3510, but has the same speed and an >> endurance that's 5 times greater. >> >>> during random write I got ~90% >>> utilization of 9 HDD with ~5% utilization of SSD (2.4k IOPS). With >>> linear writing it somehow worse: I got 250Mb/s on SSD, which >>> translated to 240Mb of all OSD combined. >>> >> This test shows us a lot of things, mostly the failings of filestore. >> But only partially if a SSD is a good fit for journals or not. >> >> How are you measuring these things on the storage node, iostat, atop? >> At 250MB/s (Mb would be mega-bit) your 800 GB DC S3500 should register >> about/over 50% utilization, given that its top speed is 460MB/s. >> >> With Intel DC SSDs you can pretty much take the sequential write speed > from >> their specifications page and roughly expect that to be the speed of your >> journal. >> >> For example a 100GB DC S3700 (200MB/s) doing journaling for 2 plain SATA >> HDDs will give us this when running "ceph tell osd.nn bench" in parallel >> against 2 OSDs that share a journal SSD: >> --- >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz > avgqu-sz >> await r_await w_await svctm %util >> sdd 0.00 2.00 0.00 409.50 0.00 191370.75 > 934.66 146.52 356.46 >> 0.00 356.46 2.44 100.00 >> sdl 0.00 85.50 0.50 120.50 2.00 49614.00 820.10 > 2.25 18.51 >> 0.00 18.59 8.20 99.20 >> sdk 0.00 89.50 1.50 119.00 6.00 49348.00 819.15 > 2.04 16.91 >> 0.00 17.13 8.23 99.20 >> --- >> >> Where sdd is the journal SSD and sdl/sdk are the OSD HDDs. >> And the SSD is nearly at 200MB/s (and 100%). >> For the record, that bench command is good for testing, but the result: >> --- >> # ceph tell osd.30 bench >> { >> "bytes_written": 1073741824, >> "blocksize": 4194304, >> "bytes_per_sec": 100960114.000000 >> } >> --- >> should be taken with a grain of salt, realistically those OSDs can do > about >> 50MB/s sustained. >> >> On another cluster I have 200GB DC S3700s (360MB/s), holding 3 journals > for >> 4 disk RAID10 (4GB HW cache Areca controller) OSDs. >> Thus the results are more impressive: >> --- >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz > avgqu-sz >> await r_await w_await svctm %util >> sda 0.00 381.00 0.00 485.00 0.00 200374.00 > 826.28 3.16 6.49 >> 0.00 6.49 1.53 74.20 >> sdb 0.00 350.50 1.00 429.00 4.00 177692.00 > 826.49 2.78 6.46 >> 4.00 6.46 1.53 65.60 >> sdg 0.00 1.00 0.00 795.00 0.00 375514.50 > 944.69 143.68 180.43 >> 0.00 180.43 1.26 100.00 >> --- >> >> Where sda/sdb are the OSD RAIDs and sdg is the journal SSD. >> Again, a near perfect match to the Intel specifications and also an > example >> where the journal is the bottleneck (never mind that his cluster is all > about >> IOPS, not throughput). >> >> As for the endurance mentioned above, these 200GB DC 3700s are/were >> overkill: >> --- >> 233 Media_Wearout_Indicator 0x0032 098 098 000 Old_age Always > - >> 0 >> 241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always > - >> 4818100 >> 242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always > - >> 84403 >> --- >> >> Again, this cluster is all about (small) IOPS, it only sees about 5MB/s > sustained >> I/O. >> So a 3610 might been a better fit, but not only didn't they exist back > then, it >> would have to be the 400GB model to match the speed, which is more >> expensive. >> A DC S3510 would be down 20% in terms of wearout (assuming same size) >> and of course significantly slower. >> With a 480GB 3510 (similar speed) it would still be about 10% worn out and >> thus still no match for the expected life time of this cluster. >> >> The numbers above do correlate nicely with dd or fio tests (with 4MB >> blocks) from VMs against the same clusters. >> >>> Obviously, it sucked with cold randread too (as expected). >>> >> Reads never touch the journal SSDs. >> >>> Just for comparacment, my baseline benchmark (fio/librbd, 4k, >>> iodepth=32, randwrite) for single OSD in the pool with size=1: >>> >>> Intel 53x and Pro 2500 Series SSDs - 600 IOPS Intel 730 >> Consumer models, avoid. >> >>> and DC S35x0/3610/3700 Series SSDs - 6605 IOPS >> Again, what you're comparing here is only part of the picture. >> With tests as shown above you'd see significant differences. >> >>> Samsung SSD 840 Series - 739 IOPS >> Also consumer model, with impressive and unpredicted deaths reported. >> >> Christian >> >>> EDGE Boost Pro Plus 7mm - 1000 IOPS >>> >>> (so 3500 is clear winner) >>> >>> On 07/06/2016 03:22 PM, Alwin Antreich wrote: >>>> Hi George, >>>> >>>> interesting result for your benchmark. May you please supply some >>>> more numbers? As we didn't get that good of a result on our tests. >>>> >>>> Thanks. >>>> >>>> Cheers, >>>> Alwin >>>> >>>> >>>> On 07/06/2016 02:03 PM, George Shuklin wrote: >>>>> Hello. >>>>> >>>>> I've been testing Intel 3500 as journal store for few HDD-based OSD. >>>>> I stumble on issues with multiple partitions (>4) and UDEV (sda5, >>>>> sda6,etc sometime do not appear after partition creation). And I'm >>>>> thinking that partition is not that useful for OSD management, >>>>> because linux do no allow partition rereading with it contains used >>>>> volumes. >>>>> >>>>> So my question: How you store many journals on SSD? My initial >>>>> thoughts: >>>>> >>>>> 1) filesystem with filebased journals >>>>> 2) LVM with volumes >>>>> >>>>> Anything else? Best practice? >>>>> >>>>> P.S. I've done benchmarking: 3500 can support up to 16 10k-RPM HDD. >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@xxxxxxxxxxxxxx >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> -- >> Christian Balzer Network/Systems Engineer >> chibi@xxxxxxx Global OnLine Japan/Rakuten Communications >> http://www.gol.com/ >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |