Hi Brian and Martin, Physical space isn't a constraint at this point, the only requirement I've in mind is to maintain a *low level of noise* (since the equipment will be in my office) and *if possible low energy consumption*. Based on my limited experience, the only downside with used hardware is the level of noise and power consumption. I think that building a new one with a non-fancy big box <https://www.newegg.com/black-fractal-design-node-micro-atx-cube-case/p/N82E16811352047> and a typical CPU fan <https://www.amazon.com/gp/product/B01KBXKP8W/> can help. But I'm not quite convinced this is a good idea, I would rather prefer to purchase old hardware that works and add new HDD, SSD, NVMe just I don't know which one to choice. *If you guys have any suggestions about used hardware that can be a good fit considering mainly low noise, please let me know.* Your feedback has been extremely helpful, I will actually modify the plan to utilize *SFP+ instead of ethernet*, the MikroTik Router Switch is wonderful, I want it. For the spindles (Are you referring to HDD right?) I selected a 6TB HDD (without any prior experience, I've read that it's like a good balance between a big disk and a small disk, that will impact the recovery time), my plan is to have 5 HDD per Node, with a total capacity of 30 TB per node. Regarding the kind of workloads I've in mind, in order of priority are: *Block Storage for OpenStack *(it will be a mix of apps: databases, web servers, etc), *Object Storage* (to develop apps compatible with S3), I don't think I will need File Storage at this point. I know HDDs are slow and provide a low IOPS, but since I will have 15 HDD spread across the 3-nodes Ceph cluster, I thought it will help a little bit with the performance, and if needed, I will slowly increase the number of OSD nodes to continue linearly scaling the performance and capacity of the cluster. Should I consider enterprise-grade SSD instead of HDD from day 1? Thanks for your support! On Sun, Oct 4, 2020 at 10:31 AM Brian Topping <brian.topping@xxxxxxxxx> wrote: > Hi Ignacio, apologies I missed your responses here. > > I would agree with Martin about buying used hardware for as cheap as > possible, but also understand the desire to have hardware you can promote > into future OpenStack usage. > > Regarding networking, I started to use SFP+ cables like > https://amzn.to/36sHZo1. These run with less energy consumption, so are > far cooler, and can be easily swapped out with Fiber or 10GBase-T modules > if / when I need them. https://amzn.to/34s6Ohj is the switch that I am > currently using and will be moving to a SONiC based EdgeCore when I get > some time to fix the crashes i’m having in the software. If the > motherboards you found are built with 10GBase-T, it may be cheaper now to > just stick with that in your switch, but long term, you’ll probably save > money going with SFP+, so it’s worth considering. > > Regarding the MikroTik switch, I bought it on price since I thought I was > just getting a switch. I turns out the thing is a full-fledged router on > the level of a Cisco Catalyst. It has a very complete and mature CLI > configuration language, as well as a web UI that is a bit overwhelming at > first but very useful once one gets the hang of it. I thought it was > something I’d be putting back on eBay pretty rapidly and I ended up growing > quite fond of it. > > Regarding bottlenecks, there’s always going to be a bottleneck. Parts are > never completely balanced. Spindles still have good characteristics for > certain workloads, and I’d say it’s doubtful you’d exceed them with a three > node experimentation cluster. The question I’d ask is whether you can use > the spindles in a long term solution for things like log storage and > backups. If the answer is yes, then having them now will allow you to get a > better feel for the differences. It’s a lot easier to discover that way > than it is to describe it. (In the old days, this was like people comparing > statistics about stereo equipment - today people don’t care about all that, > they care about whether they can find the song they want and headphones are > more than good enough…) > > Brian > > On Oct 2, 2020, at 2:03 AM, Ignacio Ocampo <nafiux@xxxxxxxxx> wrote: > > What about the network cards? The motherboard I’m looking for has 2 x > 10Gbe, with that and the CPU frequency, I think the bottleneck will be the > HDD. Is that overkill? Thanks! > > *Ignacio Ocampo* > > On 2 Oct 2020, at 0:38, Martin Verges <martin.verges@xxxxxxxx> wrote: > > > For private projects, you can search small 1U servers with up to 4 3.5" > disk slots and some e3-1230 v3/4/5 cpu. They can be bought for 250-350€ > (used) and then you just plug in a disk. > They are also good for SATA SSDs and work quite well. You can mix both > drives in the same system as well. > > -- > Martin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.verges@xxxxxxxx > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > > Web: https://croit.io > YouTube: https://goo.gl/PGE1Bx > > > Am Fr., 2. Okt. 2020 um 08:32 Uhr schrieb Ignacio Ocampo <nafiux@xxxxxxxxx > >: > >> Hi Brian, >> >> Here more context about what I want to accomplish: I've migrated a bunch >> of >> services from AWS to a local server, but having everything in a single >> server is not safe, and instead of investing in RAID, I would like to >> start >> setting up a small Ceph Cluster to have redundancy and a robust mechanism >> in case any component fails. >> >> Also, in the mid-term, I do have plans to deploy a small OpenStack >> Cluster. >> >> Because of that, I would like to set up the first small Ceph Cluster that >> can scale as my needs grow, the idea is to have 3 OSD nodes with the same >> characteristics and add additional HDDs as needed, up to 5 HDD per OSD >> node, starting with 1 HDD per node. >> >> Thanks! >> >> On Thu, Oct 1, 2020 at 11:35 AM Brian Topping <brian.topping@xxxxxxxxx> >> wrote: >> >> > Welcome to Ceph! >> > >> > I think better questions to start with are “what are your objectives in >> > your study?” Is it just seeing Ceph run with many disks, or are you >> trying >> > to see how much performance you can get out of it with distributed disk? >> > What is your budget? Do you want to try different combinations of >> storage >> > devices to learn how they differ in performance or do you just want to >> jump >> > to the fastest things out there? >> > >> > One often doesn’t need a bunch of machines to determine that Ceph is a >> > really versatile and robust solution. I pretty regularly deploy Ceph on >> a >> > single node using Kubernetes and Rook. Some would ask “why would one >> ever >> > do that, just use direct storage!”. The answer is when I want to expand >> a >> > cluster, I am willing to have traded initial performance overhead for >> > letting Ceph distribute data at a later date. And the overhead is far >> lower >> > than one might think when there’s not a network bottleneck to deal >> with. I >> > do use direct storage on LVM when I have distributed workloads such as >> > Kafka that abstract storage that a service instance depends on. It >> doesn’t >> > make much sense in my mind for Kafka or Cassandra to use Ceph because I >> can >> > afford to lose nodes using those services. >> > >> > In other words, Ceph is virtualized storage. You have likely come to it >> > because your workloads need to be able to come up anywhere on your >> network >> > and reach that storage. How do you see those workloads exercising the >> > capabilities of Ceph? That’s where your interesting use cases come from, >> > and can help you better decide what the best lab platform is to get >> started. >> > >> > Hope that helps, Brian >> > >> > On Sep 29, 2020, at 12:44 AM, Ignacio Ocampo <nafiux@xxxxxxxxx> wrote: >> > >> > Hi All :), >> > >> > I would like to get your feedback about the components below to build a >> > PoC OSD Node (I will build 3 of these). >> > >> > SSD for OS. >> > NVMe for cache. >> > HDD for storage. >> > >> > The Supermicro motherboard has 2 10Gb cards, and I will use ECC >> memories. >> > >> > <image.png> >> > >> > Thanks for your feedback! >> > >> > -- >> > Ignacio Ocampo >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > >> > >> > >> >> -- >> Ignacio Ocampo >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > > -- Ignacio Ocampo _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |