The cost of the chassis component[1] is likely to influence totals a fair bit. I notice that in their reference design there are only two 10Gb ports for 60 drives -- this would be the cheap bulk storage option, if you had a bandwidth-conscious application you'd be looking at more expensive 10Gb ports per chassis.
The demands on the top of rack network would presumably be high, as all the replication has to be driven from the client side, rather than happening p2p between storage servers. Compared with a Ceph cluster doing replication on a separate backend network, a Kinetic-based app with N way replication would require a factor of N more bandwidth on the (probably expensive) network between the storage racks and the clients.
I'll certainly be following with interest, but I'm very sceptical about cost benefits until I see an overall system including the application-level redundancy, the chassis and the networking. The drive cost might vanish in the noise once we see how heavily an application would hit the ToR network on a system like this (e.g. imagine recovering from a drive failure, clients are going to have to eat ToR bandwidth to do recovery too). Could be a lucky break for switch vendors :-)
John
1. https://developers.seagate.com/display/KV/Kinetic+Deployment+Chassis
On Tue, Oct 29, 2013 at 11:23 AM, <james@xxxxxxxxxxxx> wrote:
That's unfortunate; hopefully 2nd-gens will improve and open things up.
Some numbers:
- Commercial grid-style SAN is maybe £1.70 per usable GB
- Ceph cluster of about 1PB built on Dell hardware is maybe £1.25 per usable GB
- Bare drives like WD RE4 3TB are about £0.21/GB (assuming 1/3rd capacity ends up usable)
So if Ethernet hybrid drives could be 2x or 3x the price of standard block, so cluster cost could be halved :)
It'd be interesting to know what £/GB (or $/GB) others have achieved with their Ceph implementations.
On 2013-10-28 15:50, Gregory Farnum wrote:
On Monday, October 28, 2013, wrote:
Kinetic is interesting, but I think it's going to find more uptakeSoftware Engineer #42 @ http://inktank.com [3] | http://ceph.com [4]
among big Open Compute users like Facebook than in general distributed
storage systems. In particular, these drives don't appear to have the
CPU power required to run OSDs, and their native interfaces don't have
the strength to be useful underneath.
-Greg
--
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
