Re: Fwd: [Ceph-community] Wasting the Storage capacity when using Ceph based On high-end storage systems

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

E3 CPUs have 4 Cores, with HT Unit. So 8 logical Cores. And they are not
multi CPU.

That means you will be naturally ( fastly ) limited in the number of
OSD's you can run with that.

Because no matter how much Ghz it has, the OSD process occupy a cpu core
for ever. Not for 100%, but still enough, to ruin ur day, if you have 8
logical cores and 12 disks ( in scrubbing/backfilling/high load ).

So all single Core CPU's are just good for a very limited amount of OSD's.

-- 
Mit freundlichen Gruessen / Best regards

Oliver Dzombic
IP-Interactive

mailto:info@xxxxxxxxxxxxxxxxx

Anschrift:

IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen

HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic

Steuer Nr.: 35 236 3622 1
UST ID: DE274086107


Am 30.05.2016 um 17:13 schrieb Christian Balzer:
> 
> Hello,
> 
> On Mon, 30 May 2016 09:40:11 +0100 Nick Fisk wrote:
> 
>> The other option is to scale out rather than scale up. I'm currently
>> building nodes based on a fast Xeon E3 with 12 Drives in 1U. The MB/CPU
>> is very attractively priced and the higher clock gives you much lower
>> write latency if that is important. The density is slightly lower, but I
>> guess you gain an advantage in more granularity of the cluster.
>>
> Most definitely, granularity and number of OSDs (up to a point, mind ya)
> is a good thing [TM].
> 
> I was citing the designs I did to basically counter the "not dense enough"
> argument. 
> 
> Ultimately with Ceph (unless you throw lots of money and brain cells at
> it), the less dense, the better it will perform.
> 
> Christian
>>> -----Original Message-----
>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf
>>> Of Jack Makenz
>>> Sent: 30 May 2016 08:40
>>> To: Christian Balzer <chibi@xxxxxxx>
>>> Cc: ceph-users@xxxxxxxxxxxxxx
>>> Subject: Re:  Fwd: [Ceph-community] Wasting the Storage
>>> capacity when using Ceph based On high-end storage systems
>>>
>>> Thanks Christian, and all of ceph users
>>>
>>> Your guidance was very helpful, appreciate !
>>>
>>> Regards
>>> Jack Makenz
>>>
>>> On Mon, May 30, 2016 at 11:08 AM, Christian Balzer <chibi@xxxxxxx>
>>> wrote:
>>>
>>> Hello,
>>>
>>> you may want to read up on the various high-density node threads and
>>> conversations here.
>>>
>>> You most certainly do NOT need high end-storage systems to create
>>> multi-petabyte storage systems with Ceph.
>>>
>>> If you were to use these chassis as a basis:
>>>
>>> https://www.supermicro.com.tw/products/system/4U/6048/SSG-6048R-
>>> E1CR60N.cfm
>>> [We (and surely others) urged Supermicro to provide a design like this]
>>>
>>> And fill them with 6TB HDDs, configure them as 5x 12HDD RAID6s, set
>>> your replication to 2 in Ceph, you will wind up with VERY reliable,
>>> resilient 1.2PB per rack (32U, leaving space for other bits and not
>>> melting the PDUs).
>>> Add fast SSDs or NVMes to this case for journals and you have decently
>>> performing mass storage.
>>>
>>> Need more IOPS for really hot data?
>>> Add a cache tier or dedicated SSD pools for special needs/customers.
>>>
>>> Alternatively, do "classic" Ceph with 3x replication or EC coding, but
>>> in either case (even more so with EC) you will need the most
>>> firebreathing CPUs available, so compared to the above design it may
>>> be a zero sum game cost wise, if not performance wise as well.
>>> This leaves you with 960TB in the same space when doing 3x replication.
>>>
>>> A middle of the road approach would be to use RAID1 or 10 based OSDs to
>>> bring down the computational needs in exchange for higher storage costs
>>> (effective 4x replication).
>>> This only gives you 720TB, alas it will be easier (and cheaper CPU cost
>>> wise) to achieve peak performance with this approach compared to the
>>> one above with 60 OSDs per node.
>>>
>>> Lastly, I give you this (and not being a fan of Fujitsu, mind):
>>> http://www.fujitsu.com/global/products/computing/storage/eternus-cd/
>>>
>>> Christian
>>>
>>> On Mon, 30 May 2016 10:25:35 +0430 Jack Makenz wrote:
>>>
>>>> Forwarded conversation
>>>> Subject: Wasting the Storage capacity when using Ceph based On
>>>> high-end storage systems
>>>> ------------------------
>>>>
>>>> From: *Jack Makenz* <jack.makenz@xxxxxxxxx>
>>>> Date: Sun, May 29, 2016 at 6:52 PM
>>>> To: ceph-community@xxxxxxxxxxxxxx
>>>>
>>>>
>>>> Hello All,
>>>> There are some serious problem about ceph that may waste storage
>>> capacity
>>>> when using high-end storage system(Hitachi, IBM, EMC, HP ,...) as
>>>> back-end for OSD hosts.
>>>>
>>>> Imagine in the real cloud we need  *n Petabytes* of storage capacity
>>>> that commodity hardware's hard disks or OSD server's hard disks
>>>> can't provide this amount of storage capacity. thus we have to use
>>>> storage systems as back-end for OSD hosts(to implement OSD daemons ).
>>>>
>>>> But because almost all of these storage systems ( Regardless of their
>>>> brand) use Raid technology and also ceph replicate at least two copy
>>>> of each Object, lot's amount of storage capacity waste.
>>>>
>>>> So is there any solution to solve this problem/misunderstand ?
>>>>
>>>> Regards
>>>> Jack Makenz
>>>>
>>>> ----------
>>>> From: *Nate Curry* <curry@xxxxxxxxxxxxx>
>>>> Date: Mon, May 30, 2016 at 5:50 AM
>>>> To: Jack Makenz <jack.makenz@xxxxxxxxx>
>>>> Cc: Unknown <ceph-community@xxxxxxxxxxxxxx>
>>>>
>>>>
>>>> I think that purpose of ceph is to get away from having to rely on
>>>> high end storage systems and to be provide the capacity to utilize
>>>> multiple less expensive servers as the storage system.
>>>>
>>>> That being said you should still be able to use the high end storage
>>>> systems with or without RAID enabled.  You could do away with RAID
>>>> altogether and let Ceph handle the redundancy or you can have LUNs
>>>> assigned to hosts be put into use as OSDs.  You could make it work
>>>> however but to get the most out of your storage with Ceph I think a
>>>> non-RAID configuration would be best.
>>>>
>>>> Nate Curry
>>>>
>>>>> _______________________________________________
>>>>> Ceph-community mailing list
>>>>> Ceph-community@xxxxxxxxxxxxxx
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-community-ceph.com
>>>>>
>>>>>
>>>> ----------
>>>> From: *Doug Dressler* <darbymorrison@xxxxxxxxx>
>>>> Date: Mon, May 30, 2016 at 6:02 AM
>>>> To: Nate Curry <curry@xxxxxxxxxxxxx>
>>>> Cc: Jack Makenz <jack.makenz@xxxxxxxxx>, Unknown <
>>>> ceph-community@xxxxxxxxxxxxxx>
>>>>
>>>>
>>>> For non-technical reasons I had to run ceph initially using SAN
>>>> disks.
>>>>
>>>> Lesson learned:
>>>>
>>>> Make sure deduplication is disabled on the SAN :-)
>>>>
>>>>
>>>>
>>>> ----------
>>>> From: *Jack Makenz* <jack.makenz@xxxxxxxxx>
>>>> Date: Mon, May 30, 2016 at 9:05 AM
>>>> To: Nate Curry <curry@xxxxxxxxxxxxx>, ceph-community@xxxxxxxxxxxxxx
>>>>
>>>>
>>>> Thanks Nate,
>>>> But as i mentioned before , providing petabytes of storage capacity
>>>> on commodity hardware or enterprise servers is almost impossible, of
>>>> course that it's possible by installing hundreds of servers with 3
>>>> terabytes hard disks, but this solution waste data center raise
>>>> floor, power consumption and also *money* :)
>>>
>>>
>>> --
>>> Christian Balzer        Network/Systems Engineer
>>> chibi@xxxxxxx           Global OnLine Japan/Rakuten Communications
>>> http://www.gol.com/
>>
>>
>>
> 
> 
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux