On Sat, 26 Jun 2021 08:01:46 -0500 Mark Nelson <mnelson@xxxxxxxxxx> wrote: > FWIW, you can lower both the osd_memory_target and tweak a couple of > other settings that will lower bluestore memory usage. A 2GB target is > about the lowest you can reasonably set it to (and you'll likely hurt > performance due to cache misses), but saying you need a host with 8+GB > of RAM is probably a little excessive. There's also a good chance that > filestore memory usage isn't as consistently low as you think it is. > Yes you can avoid the in-memory caches that bluestore has since > filestore relies more heavily on page cache, but things like osdmap, > pglog, and various other buffers are still going to use memory in > filestore just like bluestore. You might find yourself working fine 99% > of the time and then going OOM during recovery or something if you try > to deploy filestore on a low memory SBC. To be honest, the smallest of my nodes has 8GB RAM presently, but I'd like to scale out, and most of my cost-effective options for scale-out are 4GB or less. Raspberry Pi4 is the only I've seen available to mere mortals like myself that exceeds this limit, and even then it's far from an ideal system. PC Engines APU3s look good as nodes, as they have SATA on-board, multiple Ethernet interfaces that can be bonded for quick networking, and they use CoreBoot managed over a serial port, but needing 8GB is a killer. I presently use filestore on HDD-based OSDs because I've found it gives me the best performance. The SSD-based OSDs are running Bluestore, since they seem to be able to keep up better. Never tried FileStore on less than 8GB, so you could well be right, but my experience with BlueStore on these OSDs was abysmal. > Having said all of that, you can get 16GB of ECC DDR4 RAM in the US new > for around $70-100USD. A quick search on google makes it look like > you'll pay about twice that new in AU, but there's plenty of stuff on > the used market for ~$60-100AUD (like $50-70USD). I don't think that's > super unreasonable and frankly would be far more reliable than running > on SBCs with non-ECC memory. I would love to see SBCs become more > prolific, but memory has always been a big constraint (especially before > the 8GB devices came out), and not only for Ceph. Yep, but remember when you buy a SBC, the RAM comes soldered-to-the-board in most cases. If you want removable RAM, you're looking at a small-form-factor server board of some kind like the Supermicro A1SAi boards that have been my storage nodes since 2016. Regards, -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |