i think i asked it before with brown, a raid card with memory is a nice tool here but maybe... you could use a ssd and a bcache, flashcache, or md cache solution to increase performace of hdds (ok, ram/raid card are faster than ssd, but just another idea) 2014-06-03 4:58 GMT-03:00 David Brown <david.brown@xxxxxxxxxxxx>: > Hi, > > It's not that SSD's are /bad/ for VM images - it is simply that they are > not so much better than HD's that they are worth the money. VM files > are big - it costs a lot to get that much SSD space, so you have to be > sure that the faster IOPs and faster random access is actually worth > that cost. SSD's are not significantly faster in bandwidth than HD's - > for the price of one high throughput SSD, you can buy four HD's in > raid10,f2 with a higher throughput. > > The OP needs HD's for the space. So the question is whether he should > spend additional money on two good quality SSDs - or should he spend it > on an extra HD, more ram, and perhaps a little UPS? (I'm assuming he > has a limited budget.) > > I don't think the IOPs rate of SSDs will make such a difference over the > layers of indirection - Windows on the VM's, the VM's disk caching > system, the VM image file format, the caches on the host ram, the raid > layers, etc. These all conspire to add latency and reduce the peak IOPs > - within the VM, you are never going to see anything like the SSD's > theoretical IOPs rate. You will get a little higher IOPs than with HD's > at the back end, but not much more. The VM's will see high IOP's if and > only if the data is in ram cache somewhere, regardless of the disk type > - so more ram will always help. > > Of course, there are other reasons you might prefer SSD's - size, space, > power, noise, reliability, etc. > > mvh., > > David > > > > On 03/06/14 01:13, Craig Curtin wrote: >> Dave, >> >> What part of a VM is not ideally suited to running from SSDs. The >> right SSDs support a high level of IOPS (much higher sustained that >> any SATA based RAID array is going to get to) and has he has >> predefined (preallocated/thick) disks already defined for the VMs >> they are ideal candidates to move onto SSDs. >> >> As a real world example - I have 4 HP N40L microservers running in a >> VMware Cluster at home - they all source their VMs from another N40L >> that has a HP P410 RAID controller in it and dual gigabit Ethernet >> ports. >> >> The box running as the disk store is running Centos 6.3. >> >> It has two RAID sets defined on the P410 - a pair of Samsung EVO >> 240GB SSDs in RAID 1 and 4 x WD (Enterprise Series) 500GB SATA drives >> in RAID0+1 >> >> I can categorically state that the throughput from the SSD VMs is >> approx. 4 times more than I can sustain to the SATA drives - the SATA >> drives come out at around 1/2 the throughput of a single Gigabit card >> whilst the SSDs flood both channels of the card. You can also see the >> point where the cache on the controller is flooded when writing to >> the SATA drives as everything slows down - whereas with the SSDs this >> never happens. This is doing disk intensive operations like live >> state migrating VMs etc. >> >> Craig >> >> -----Original Message----- From: linux-raid-owner@xxxxxxxxxxxxxxx >> [mailto:linux-raid-owner@xxxxxxxxxxxxxxx] On Behalf Of David Brown >> Sent: Tuesday, 3 June 2014 9:05 AM To: Mark Knecht Cc: Craig Curtin; >> L.M.J; Linux-RAID Subject: Re: Home desktop/server RAID upgrade >> >> Hi Mark, >> >> I would say forget the SSD's - they are not ideal for VM files, and I >> don't think they would be worth the cost. Raid 10 (any arrangement) >> is likely to give the best speed for such files, and would do a lot >> better than raid 6. Raid 10,f2 is probably a good choice - but you >> might want to test things out a bit if that is possible. >> >> I don't know how much ram you've got in the machine, but if you can >> afford more, it will always help (especially if you make sure the >> VM's use the host's cache rather than direct writes). >> >> mvh., >> >> David >> >> >> On 01/06/14 17:59, Mark Knecht wrote: >>> David, You are correct and I'm sorry I didn't do that. I started >>> this question on a Gentoo list where I put a lot more information >>> about the machine/ When I came here I should have included more. >>> >>> The machine is used 7 days a week. I'm self employed writing >>> software analyzing the stock & futures markets. Most of it is >>> written in R in Linux, some of it in proprietary languages in >>> Windows. Some of it is quite computational but mostly it's just >>> looking at a _lot_ of locally stored financial data. Almost all >>> financial data is currently stored on the machine in Linux in ext4. >>> Over the past year this data has been growing at around 30GB/month. >>> With 100GB left on my current RAID6 I don't have much time before >>> I'm full. >>> >>> When I'm actually trading in the market I have a few Virtualbox VMs >>> running Windows 7. They aren't overly large in terms of disk >>> space. (Currently about 150GB total.) The VMs are each stored in >>> massive single files which I suspect basically represent a hard >>> drive to Virtualbox. I have no idea what size any IO might be >>> coming from the VM. The financial data in the previous paragraph is >>> available to these Windows VMs as a network mount from the Windows >>> perspective. Read & write speeds of this data in Windows is not >>> overly high. >>> >>> These VMs are the area where my current RAID6 (5 drive, 16k chunk >>> size) seems to have been a bad decision. The machine is powered >>> off every night. Loading these VMs takes at least 10-15 minutes >>> each morning where I see disk activity lights just grinding away >>> the whole time. If I had a single _performance_ goal in upgrading >>> the disks it would be to improve this significantly. Craig's SSD >>> RAID1 suggestion would certainly help here but at 240GB there >>> wouldn't be a lot of room left. That may be OK though. >>> >>> The last area is video storage. Write speed is unimportant, read >>> speeds are quite low. Over time I hope to migrate it off to a NAS >>> box but for now this is where it's stored. This is currently using >>> about 1/2 the storage my RAID6 provides. >>> >>> Most important to me is data safety. I currently do weekly >>> rotational backups to a couple of USB drives. I have no real-time >>> issues at all if the machine goes down. I have 2 other machines I >>> can do day-to-day work on while I fix this machine. What I am most >>> concerned about is not losing anything more than a couple of >>> previous days work. If I took a week to rebuild the machine after >>> a failure it's pretty much a non-issue to me. >>> >>> Thanks, Mark >>> >>> On Sun, Jun 1, 2014 at 8:06 AM, David Brown >>> <david.brown@xxxxxxxxxxxx> wrote: >>>> Hi Mark, >>>> >>>> What would be really useful here is a description of what you >>>> actually /want/. What do you want to do with these drives? >>>> What sort of files are they - big or small? Do you need fast >>>> access for large files? Do you need fast access for many files >>>> in parallel? How important is the data? How important is uptime? >>>> What sort of backups do you have? What will the future be like - >>>> are you making one big system to last for the foreseeable future, >>>> or do you need something that can easily be expanded? Are you >>>> looking for "fun, interesting and modern" or "boring but >>>> well-tested" solutions? >>>> >>>> Then you need to make a list of the hardware you have, or the >>>> budget for new hardware. >>>> >>>> Without know at least roughly what you are looking for, it's easy >>>> to end up with expensive SSDs because they are "cool", even >>>> though you might get more speed for your money with a couple of >>>> slow rust disks and a bit more ram in your system. It may be >>>> that there is no need for any sort of raid at all - perhaps one >>>> big main disk is fine, and the rest of the money spent on a >>>> backup disk (possibly external) with rsync'd copies of your data. >>>> This would mean longer downtime if your main disk failed - but it >>>> also gives some protection against user error. >>>> >>>> And perhaps btrfs with raid1 would be the best choice. >>>> >>>> A raid10,f2 is often the best choice for desktops or >>>> workstations with 2 or 3 hard disks, but it is not necessarily >>>> /the/ best choice. >>>> >>>> mvh., >>>> >>>> David >>>> >>>> >>>> >>>> On 01/06/14 16:25, Mark Knecht wrote: >>>>> >>>>> Hi Craig, Responding to both you and David Brown. Thanks for >>>>> your ideas. >>>>> >>>>> - Mark >>>>> >>>>> On Sat, May 31, 2014 at 9:40 AM, Craig Curtin >>>>> <craigc@xxxxxxxxxxxxx> wrote: >>>>>> >>>>>> It sounds like the op has additional data ports on his MOBO >>>>>> - wouldn't he be better off looking at a couple of SSDs in >>>>>> raid 1 for his OS, swap etc and his VMs and then leave the >>>>>> rest for data as raid5 - By moving the things from the >>>>>> existing drives he gets back space and only purchases a >>>>>> couple of good sized fast SSDs now >>>>>> >>>>> >>>>> It's a possibility. I can get 240GB SSDs in the $120 range so >>>>> that's $240 for RAID1. If I take the five existing 500GB drives >>>>> and reconfigure for RAID5 that's 2TB. Overall it's not bad >>>>> going from 1.4TB to about 2.2TB but being it's not all one big >>>>> disk I'll likely never use it all as efficiently. Still, it's >>>>> an option. >>>>> >>>>> I do in fact have extra ports: >>>>> >>>>> c2RAID6 ~ # lspci | grep SATA 00:1f.2 IDE interface: Intel >>>>> Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller >>>>> #1 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 >>>>> Family) 2 port SATA IDE Controller #2 03:00.0 SATA controller: >>>>> Marvell Technology Group Ltd. 88SE9123 PCIe SATA 6.0 Gb/s >>>>> controller (rev 11) 06:00.0 SATA controller: JMicron Technology >>>>> Corp. JMB363 SATA/IDE Controller (rev 03) 06:00.1 IDE >>>>> interface: JMicron Technology Corp. JMB363 SATA/IDE Controller >>>>> (rev 03) c2RAID6 ~ # >>>>> >>>>> Currently my 5-drive RAID6 uses 5 of the Intel ports. The 6th >>>>> port goes to the CD/DVD drive. Some time ago I bought the SATA3 >>>>> Marvell card and a smaller (120GB) SSD. I put Gentoo on it and >>>>> played around a bit but I've never really used it day-to-day. >>>>> Part of my 2-drive RAID1 thinking was that I could build the >>>>> new RAID1 on the SATA3 controller not even touch the existing >>>>> RAID6. If it works reliably on that controller I'd be done and >>>>> have 3TB. >>>>> >>>>> I think David's RAID10 3-drive solution could possibly work if >>>>> I buy 3 of the lower cost new WD drives. I'll need to think >>>>> about that. Not sure. >>>>> >>>>> Thanks, Mark >>>>> >>>>> >>>>> On Sat, May 31, 2014 at 9:40 AM, Craig Curtin >>>>> <craigc@xxxxxxxxxxxxx> wrote: >>>>>> >>>>>> It sounds like the op has additional data ports on his MOBO >>>>>> - wouldn't he be better off looking at a couple of SSDs in >>>>>> raid 1 for his OS, swap etc and his VMs and then leave the >>>>>> rest for data as raid5 - By moving the things from the >>>>>> existing drives he gets back space and only purchases a >>>>>> couple of good sized fast SSDs now >>>>>> >>>>>> >>>>>> Sent from my Samsung tablet >>>>>> >>>>>> . >>>>>> >>>>>> >>>>>> -------- Original message -------- From: David Brown >>>>>> Date:31/05/2014 21:01 (GMT+10:00) To: Mark Knecht ,"L.M.J" >>>>>> Cc: Linux-RAID Subject: Re: Home desktop/server RAID upgrade >>>>>> >>>>>> On 30/05/14 22:14, Mark Knecht wrote: >>>>>>> >>>>>>> On Fri, May 30, 2014 at 12:29 PM, L.M.J >>>>>>> <linuxmasterjedi@xxxxxxx> wrote: >>>>>>>> >>>>>>>> Le Fri, 30 May 2014 12:04:07 -0700, Mark Knecht >>>>>>>> <markknecht@xxxxxxxxx> a écrit : >>>>>>>> >>>>>>>>> In a RAID1 would a 3-drive Red RAID1 possibly be faster >>>>>>>>> than the 2-drive Se RAID1 and at the same time give me >>>>>>>>> more safety? >>>>>>>> >>>>>>>> >>>>>>>> Just a question inside the question : how do you manager >>>>>>>> a RAID1 with 3 drives ? Maybe you're talking about RAID5 >>>>>>>> then ? >>>>>>> >>>>>>> >>>>>>> OK, I'm no RAID expert but RAID1 is just drives in parallel >>>>>>> right. 2 drives, 3 drives, 4 drives, all holding exactly >>>>>>> the same data. In the case of a 3-drive RAID1 - if there is >>>>>>> such a beast - I could safely lose 2 drives. You ask a >>>>>>> reasonable question though as maybe the way this is >>>>>>> actually done is 2 drives + a hot spare in the box that >>>>>>> gets sync'ed if and only if one drive fails. Not sure and >>>>>>> maybe I'm totally wrong about that. >>>>>>> >>>>>>> A 3-drive RAID5 would be 2 drives in series - in this case >>>>>>> making 6TB - and then the 3rd drive being the redundancy. >>>>>>> In the case of a 3-drive RAID5 I could safely lose 1 >>>>>>> drive. >>>>>>> >>>>>>> In my case I don't need more than 3TB, so an option would >>>>>>> be a 3-drive RAID5 made out of 2TB drives which would give >>>>>>> me 4TB but I don't need the space as much as I want the >>>>>>> redundancy and I think RAID5 is slower than RAID1. >>>>>>> Additionally some more mdadm RAID knowledgeable people on >>>>>>> other lists say Linux mdadm RAID1 would be faster as it >>>>>>> will get data from more than one drive at a time. (Or >>>>>>> possibly get data from which ever drive returns it the >>>>>>> fastest. Not sure.) >>>>>>> >>>>>>> I believe one good option if I wanted 4 physical drives >>>>>>> would be RAID10 but that's getting more complicated again >>>>>>> which I didn't really want to do. >>>>>>> >>>>>>> So maybe it is just 2 drives and the 3 drive version isn't >>>>>>> even a possibility? Could be. >>>>>> >>>>>> >>>>>> With 3 drives, you have several possibilities. >>>>>> >>>>>> Raid5 makes "stripes" across the three drives, with 2 parts >>>>>> holding data and one part holding parity to provide >>>>>> redundancy. >>>>>> >>>>>> Raid1 is commonly called "mirroring", because you get the >>>>>> same data on each disk. md raid has no problem making a >>>>>> 3-way mirror, so that each disk is identical. This gives you >>>>>> excellent redundancy, and you can make three different reads >>>>>> in parallel - but writes have to go to each disk, which can >>>>>> be a little slower than using 2 disks. It's not often that >>>>>> people need that level of redundancy. >>>>>> >>>>>> Another option with md raid is the raid10 setups. For many >>>>>> uses, the fastest arrangement is raid10,f2. This means there >>>>>> is two copies of all your data (f3 would be three copies), >>>>>> with a "far" layout. >>>>>> >>>>>> <http://en.wikipedia.org/wiki/Linux_MD_RAID_10#LINUX-MD-RAID-10> >>>>>> >>>>>> >>>>>> > With this arrangement, reads are striped across all three disks, >>>>>> which is fast for large reads. Small reads can be handled >>>>>> in parallel. Most reads while be handled from the outer half >>>>>> of the disk, which is faster and needs less head movement - >>>>>> so reading is on average faster than a raid0 on the same >>>>>> disks. Small writes are fast, but large writes require quite >>>>>> a bit of head movement to get everything written twice to >>>>>> different parts of the disks. >>>>>> >>>>>> The "best" option always depends on your needs - how you want >>>>>> to access your files. A layout geared to fast striped reads >>>>>> of large files will be poorer for parallel small writes, and >>>>>> vice versa. raid10,f2 is often the best choice for a desktop >>>>>> or small system - but it is not very flexible if you later >>>>>> want to add new disks or replace the disks with bigger ones. >>>>>> >>>>>> md raid is flexible enough that it will even let you make a 3 >>>>>> disk raid6 array if you want - but a 3-way raid1 mirror will >>>>>> give you the same disk space and much better performance. >>>>>> >>>> >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Roberto Spadim -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
