The other option is to scale out rather than scale up. I'm currently building nodes based on a fast Xeon E3 with 12 Drives in 1U. The MB/CPU is very attractively priced and the higher clock gives you much lower write latency if that is important. The density is slightly lower, but I guess you gain an advantage in more granularity of the cluster. > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Jack Makenz > Sent: 30 May 2016 08:40 > To: Christian Balzer <chibi@xxxxxxx> > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Fwd: [Ceph-community] Wasting the Storage > capacity when using Ceph based On high-end storage systems > > Thanks Christian, and all of ceph users > > Your guidance was very helpful, appreciate ! > > Regards > Jack Makenz > > On Mon, May 30, 2016 at 11:08 AM, Christian Balzer <chibi@xxxxxxx> wrote: > > Hello, > > you may want to read up on the various high-density node threads and > conversations here. > > You most certainly do NOT need high end-storage systems to create > multi-petabyte storage systems with Ceph. > > If you were to use these chassis as a basis: > > https://www.supermicro.com.tw/products/system/4U/6048/SSG-6048R- > E1CR60N.cfm > [We (and surely others) urged Supermicro to provide a design like this] > > And fill them with 6TB HDDs, configure them as 5x 12HDD RAID6s, set your > replication to 2 in Ceph, you will wind up with VERY reliable, resilient > 1.2PB per rack (32U, leaving space for other bits and not melting the > PDUs). > Add fast SSDs or NVMes to this case for journals and you have decently > performing mass storage. > > Need more IOPS for really hot data? > Add a cache tier or dedicated SSD pools for special needs/customers. > > Alternatively, do "classic" Ceph with 3x replication or EC coding, but in > either case (even more so with EC) you will need the most firebreathing > CPUs available, so compared to the above design it may be a zero sum game > cost wise, if not performance wise as well. > This leaves you with 960TB in the same space when doing 3x replication. > > A middle of the road approach would be to use RAID1 or 10 based OSDs to > bring down the computational needs in exchange for higher storage costs > (effective 4x replication). > This only gives you 720TB, alas it will be easier (and cheaper CPU cost > wise) to achieve peak performance with this approach compared to the one > above with 60 OSDs per node. > > Lastly, I give you this (and not being a fan of Fujitsu, mind): > http://www.fujitsu.com/global/products/computing/storage/eternus-cd/ > > Christian > > On Mon, 30 May 2016 10:25:35 +0430 Jack Makenz wrote: > > > Forwarded conversation > > Subject: Wasting the Storage capacity when using Ceph based On high-end > > storage systems > > ------------------------ > > > > From: *Jack Makenz* <jack.makenz@xxxxxxxxx> > > Date: Sun, May 29, 2016 at 6:52 PM > > To: ceph-community@xxxxxxxxxxxxxx > > > > > > Hello All, > > There are some serious problem about ceph that may waste storage > capacity > > when using high-end storage system(Hitachi, IBM, EMC, HP ,...) as > > back-end for OSD hosts. > > > > Imagine in the real cloud we need *n Petabytes* of storage capacity that > > commodity hardware's hard disks or OSD server's hard disks can't provide > > this amount of storage capacity. thus we have to use storage systems as > > back-end for OSD hosts(to implement OSD daemons ). > > > > But because almost all of these storage systems ( Regardless of their > > brand) use Raid technology and also ceph replicate at least two copy of > > each Object, lot's amount of storage capacity waste. > > > > So is there any solution to solve this problem/misunderstand ? > > > > Regards > > Jack Makenz > > > > ---------- > > From: *Nate Curry* <curry@xxxxxxxxxxxxx> > > Date: Mon, May 30, 2016 at 5:50 AM > > To: Jack Makenz <jack.makenz@xxxxxxxxx> > > Cc: Unknown <ceph-community@xxxxxxxxxxxxxx> > > > > > > I think that purpose of ceph is to get away from having to rely on high > > end storage systems and to be provide the capacity to utilize multiple > > less expensive servers as the storage system. > > > > That being said you should still be able to use the high end storage > > systems with or without RAID enabled. You could do away with RAID > > altogether and let Ceph handle the redundancy or you can have LUNs > > assigned to hosts be put into use as OSDs. You could make it work > > however but to get the most out of your storage with Ceph I think a > > non-RAID configuration would be best. > > > > Nate Curry > > > > > _______________________________________________ > > > Ceph-community mailing list > > > Ceph-community@xxxxxxxxxxxxxx > > > http://lists.ceph.com/listinfo.cgi/ceph-community-ceph.com > > > > > > > > ---------- > > From: *Doug Dressler* <darbymorrison@xxxxxxxxx> > > Date: Mon, May 30, 2016 at 6:02 AM > > To: Nate Curry <curry@xxxxxxxxxxxxx> > > Cc: Jack Makenz <jack.makenz@xxxxxxxxx>, Unknown < > > ceph-community@xxxxxxxxxxxxxx> > > > > > > For non-technical reasons I had to run ceph initially using SAN disks. > > > > Lesson learned: > > > > Make sure deduplication is disabled on the SAN :-) > > > > > > > > ---------- > > From: *Jack Makenz* <jack.makenz@xxxxxxxxx> > > Date: Mon, May 30, 2016 at 9:05 AM > > To: Nate Curry <curry@xxxxxxxxxxxxx>, ceph-community@xxxxxxxxxxxxxx > > > > > > Thanks Nate, > > But as i mentioned before , providing petabytes of storage capacity on > > commodity hardware or enterprise servers is almost impossible, of course > > that it's possible by installing hundreds of servers with 3 terabytes > > hard disks, but this solution waste data center raise floor, power > > consumption and also *money* :) > > > -- > Christian Balzer Network/Systems Engineer > chibi@xxxxxxx Global OnLine Japan/Rakuten Communications > http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |