Good evening,
as an operator running Ceph clusters based on Debian and later Devuan
for years and recently testing ceph in rook, I would like to chime in to
some of the topics mentioned here with short review:
Devuan/OS package:
- Over all the years changing from Debian to Devuan, changing the Devuan
versions, dist-upgrading - we did not encounter a single issues on the
OS basis. The only real problems where, when ceph version
incompatibilities between the major versions happened. However this
will not change with containers.
I do see the lack of proper packages for Alpine Linux, which would be
an amazing lean target for running ceph.
The biggest problem I see is that ceph/cephadm is the longer the more
relying on systemd and that actually locks out folks.
Cephadm/Containers:
- I am sceptical about this approach to be honest. On one hand yes,
putting it into a controlled environment, great. But there are two
things that make me really wonder if this is a good approach:
- ceph-deploy and friends have been quite a pain over the years and we
had to reverse engineer scripts and data to find out how to create
OSDs, because in many situations ceph-deploy would fail doing the
right thing [tm]. So what is different now with cephadm that the
story does not repeat again?
- Re-inventing the deployment/distribution system which is basically
given by kubernetes. I see many overlaps here and I guess the
motivation was "do not force people going into kubernetes" (... but
bind them to systemd only operating systems)
rook/kubernetes:
- If we have to go towards containers, rook seems to be the most
plausible solution out there, as it is k8s native, it has solved
things like autodetection of disks, restart of daemons by k8s is
given, etc. - So if going full blown containers, you might as well go
down the road and settle on rook only.
Both rook and cephadm come with the obvious disadvantage that the
underlying complexity grows quite a lot. And that is a problem, the
entrance barrier for ceph just goes up. It's not about understanding
concepts of mon, mgr, mds, osd anymore, it's about "how to debug a
containerisation deployment framework" and on top of that you begin to
understand ceph, effectively adding resources on top of running a ceph
cluster.
Don't get me wrong, there are good use cases for k8s and we are the
longer the more going towards that, but I really don't see the advantage
of the in-between approach.
Thus my suggestion for the ceph team is to focus on 2 out of the three
variants:
- Keep providing a native, even manual deployment mode. Let people get
an understanding of ceph, develop even their own tooling around it.
This enables distros, SMEs, Open Source communities, hackers,
developers. Low entrance barrier, easy access, low degree of
automation.
- For those who are into containers, advise them how to embrace k8s. How
to use k8s on bare metal. Is it potentially even smarter to run ceph
on IPv6 only clusters? What does the architecture look like with k8s?
How does rook do autodetection, what metrics can the kube-prometheus
grafana help with? etc. etc. The whole shebang that you'll need to
develop and create over time anyway.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
