Re: arm cluster install

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Le 15/02/2015 16:48, hp cre a écrit :

Hello all,  I'm currently studying the possibility of creating a small ceph cluster on arm nodes.

The reasonably priced boards I found (like the banana pi/pro, Orange pi/pro/h3, etc..) most have either dual core or quad core Allwinner chips and 1GB RAM. They also use a micro sd card for os and a sata drive connection.

Has anyone ever tried a deployment like this? I want to know if that hardware is enough to create a usable cluster.


I have, at home :)

I have 3 Banana pi (each is an OSD), and 1 Odroid C1 (which is acting as MON, and in the same time, hosts other LXC servers)

Each banana pi has a Sata 240 GB  SSD + 1 USB 1,5 TB drive.

In a few words : It works. But it has lots of limitations, and it's absolutely not advisable to put production on it.

I you want to experiment on this, some advices :

-> You need recent kernel on Banana pi & Odroid to have good Ethernet bandwidth (over 600 Mbit/s)
-> Another reason to put a recent vanilla kernel (3.6.18 or more on Bananapi) is stability, I had lots of glitches with 3.10 customs Bananapi kernels ( with data corruption :) )
-> If you plan to use XFS, BEWARE !!! there is miscompilation with XFS on arm if you use standard (-O2) optimization. Use (-Os)  to have a reliable XFS module.
-> banana pi has Sata, performance is  OK for read (over 150 MB/s) but it's definitively SLOW for writes (not more than 40 MB/s), so putting an SSD on Sata for journal don't have much interest. In fact you have better bandwidth using 2 separate USB 2.0 disks !!! (Bananapi has 2 independents USB channels)
-> Bananapi (and Odroid) are limited to 1 GB of RAM, it's adequate for OSD in the 1/2 TB range, not more, or you'll end up swapping like mad (on slow device)
-> Bananapi or Odroid are not servers anyway so you don't have any kind of error protection !!!

And even if you avoid all those traps, you'll find that performance is not that great.

Using RBD, I can barely reach 20 MB/s (which is not THAT bad). Even with ONLY SSD pool (the root cause is : you need journal anyway and limitation is Bananapi (in fact Allwinner A20) Sata chip or driver .

Odroid C1 has an eMMC channel, but it seems you won't have better throughput using this....

I can give further details if you wish,

Cheers







I dont want loads of storage,  just want to end up with a responsive os and a 10 gb storage pool.

Beyond the basic minimum hardware listed on the ceph documents,  i haven't found much info on this.  The pilot implementation with startup company calxeda was on 64-bit arm boards,  not 32-bit,  and they were a custom spec.

Any ideas welcome.  Thanks.



_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux