We have the single-socket version of this chassis with 4 nodes in our
test lab. E3-1240v2 CPU with 10 spinners for OSDs, 2 DC S3700s, a 250GB
spinner for OS, 10GbE, and a SAS2308 HBA + on-board SATA. They work
well but were oddly a little slow for sequential reads from what I
remember. Overall not bad though and I think a very reasonable
solution, especially if you want smaller clusters while maintaining
similar (actually slightly better) drive density vs the 36 drive
chassis. They weren't quite able to saturate a 10GbE link for writes
(about 700MB/s including OSD->OSD replica writes if I recall). Close
enough that you won't feel like you are wasting the 10GbE. Gives them a
bit of room to grow too as Ceph performance improves.
Mark
On 04/13/2015 11:34 AM, Nick Fisk wrote:
I went for something similar to the Quantas boxes but 4 stacked in 1x 4U box
http://www.supermicro.nl/products/system/4U/F617/SYS-F617H6-FTPT_.cfm
When you do the maths, even something like a banana pi + disk starts costing
a similar amount and you get so much more for your money in temrs of
processing power, NIC bandwidth...etc
-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of
Robert LeBlanc
Sent: 13 April 2015 17:27
To: Jerker Nyberg
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: low power single disk nodes
We are getting ready to put the Quantas into production. We looked at the
Supermico Atoms (we have 6 of them), the rails were crap (they exploded
the first time you pull the server out, and they stick out of the back of
the
cabinet about 8 inches, these boxes are already very deep), we also ran
out
of CPU on these boxes and had limited PCI I/O).
They may work fine for really cold data. It may also work fine with XIO
and
Infiniband. The Atoms still had pretty decent performance given these
limitations.
The Quantas removed some of the issues with NUMA, had much better PCI
I/O bandwidth, comes with a 10 Gb NIC on board. The biggest drawback is
that 8 drives is on a SAS controller and 4 drives are on a SATA
controller, plus
SATADOM and a free port. So you have to manage two different controller
types and speeds (6Gb SAS and 3Gb SATA).
I'd say neither is perfect, but we decided on Quanta in the end.
On Mon, Apr 13, 2015 at 5:17 AM, Jerker Nyberg <jerker@xxxxxxxxxxxx>
wrote:
Hello,
Thanks for all replies! The Banana Pi could work. The built in
SATA-power in Banana Pi can power a 2.5" SATA disk. Cool. (Not 3.5"
SATA since that seem to require 12 V too.)
I found this post from Vess Bakalov about the same subject:
http://millibit.blogspot.se/2015/01/ceph-pi-adding-osd-and-more-perfor
mance.html
For PoE I have only found Intel Galileo Gen 2 or RouterBOARD RB450G
which are too slow and/or miss IO-expansion. (But good for
signage/Xibo maybe!)
I found two boxes from Quanta and SuperMicro with single socket Xeon
or with Intel Atom (Avaton) that might be quite ok. I was only aware
of the dual-Xeons before.
http://www.quantaqct.com/Product/Servers/Rackmount-
Servers/STRATOS-S10
0-L11SL-p151c77c70c83
http://www.supermicro.nl/products/system/1U/5018/SSG-5018A-
AR12L.cfm
Kind regards,
Jerker Nyberg
On Thu, 9 Apr 2015, Quentin Hartman wrote:
I'm skeptical about how well this would work, but a Banana Pi might
be a place to start. Like a raspberry pi, but it has a SATA connector:
http://www.bananapi.org/
On Thu, Apr 9, 2015 at 3:18 AM, Jerker Nyberg <jerker@xxxxxxxxxxxx>
wrote:
Hello ceph users,
Is anyone running any low powered single disk nodes with Ceph now?
Calxeda
seems to be no more according to Wikipedia. I do not think HP
moonshot is what I am looking for - I want stand-alone nodes, not
server cartridges integrated into server chassis. And I do not want
to be locked to a single vendor.
I was playing with Raspberry Pi 2 for signage when I thought of my
old experiments with Ceph.
I am thinking of for example Odroid-C1 or Odroid-XU3 Lite or maybe
something with a low-power Intel x64/x86 processor. Together with
one SSD or one low power HDD the node could get all power via PoE
(via splitter or integrated into board if such boards exist). PoE
provide remote power-on power-off even for consumer grade nodes.
The cost for a single low power node should be able to compete with
traditional PC-servers price per disk. Ceph take care of redundancy.
I think simple custom casing should be good enough - maybe just
strap or velcro everything on trays in the rack, at least for the
nodes with
SSD.
Kind regards,
--
Jerker Nyberg, Uppsala, Sweden.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
