On 6/26/21 12:53 AM, Stuart Longland wrote:
On Tue, 1 Jun 2021 12:24:12 -0700
Neha Ojha <nojha@xxxxxxxxxx> wrote:
Given that BlueStore has been the default and more widely used
objectstore since quite some time, we would like to understand whether
we can consider deprecating FileStore in our next release, Quincy and
remove it in the R release. There is also a proposal [0] to add a
health warning to report FileStore OSDs.
I'd consder this:
- Bluestore requires OSD hosts with 8GB+ of RAM
- In my experience, it performs poorly on HDD-based clusters with a
small number of disks
- There are very few single-board computers that have 8GB+ of RAM:
- Raspberry Pi 4 has 8GB, at about AU$100 a pop -- such a machine
could service *one* OSD. Also, only one Ethernet port.
- PC Engines APU[234] stop at 4GB RAM at present, a shame since
they'd otherwise make great small-form-factor OSD nodes.
- Lots of other ARM-based SBCs stop at 2GB RAM. There are a few out
there that have SATA and multiple Ethernet ports, but RAM rules them
out for BlueStore.
- Intel NUCs and similar machines can do Ceph work, but only one
Ethernet port is a limitation. (Plus the need to use a console
to manage them instead of using a BMC with a server board or
a multiplexed serial console is a nuisance.)
Not all of us using Ceph are big corporates with deep pockets.
FWIW, you can lower both the osd_memory_target and tweak a couple of
other settings that will lower bluestore memory usage. A 2GB target is
about the lowest you can reasonably set it to (and you'll likely hurt
performance due to cache misses), but saying you need a host with 8+GB
of RAM is probably a little excessive. There's also a good chance that
filestore memory usage isn't as consistently low as you think it is.
Yes you can avoid the in-memory caches that bluestore has since
filestore relies more heavily on page cache, but things like osdmap,
pglog, and various other buffers are still going to use memory in
filestore just like bluestore. You might find yourself working fine 99%
of the time and then going OOM during recovery or something if you try
to deploy filestore on a low memory SBC.
Having said all of that, you can get 16GB of ECC DDR4 RAM in the US new
for around $70-100USD. A quick search on google makes it look like
you'll pay about twice that new in AU, but there's plenty of stuff on
the used market for ~$60-100AUD (like $50-70USD). I don't think that's
super unreasonable and frankly would be far more reliable than running
on SBCs with non-ECC memory. I would love to see SBCs become more
prolific, but memory has always been a big constraint (especially before
the 8GB devices came out), and not only for Ceph.
Mark
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
