So, turns out ceph-volume has two modes: raw and lvm. Rook uses
the raw mode,
while typical non-Rook clusters are using the lvm mode of
ceph-volume. And
as the lvm mode doesn't use lsblk to list devices, we're only
seeing this in Rook
deployed clusters.
Is there a way we can fix the raw mode here?
Best,
Sebastian
From what I have been reading, it's actually quite easy because AHDI partitions are so easily/loosely defined. I suspect Ubuntu has the kernel built with this enabled to allow for users emulating Atari systems.See here for reference: https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1531404/comments/12
I suspect that Ceph's Bluestore layout has a high probability of matching these loose AHDI partitions at least in some cases. It seems very good that Centos 7 isn't built with AHDI support.
Blaine
On Tue, 22 Jun 2021 at 16:53, Steven Ellis <sellis@xxxxxxxxxx> wrote:
I'm rather puzzled as to why a disk is showing AHDI partition layout. It isn't something that should happen on a modern disk
On Wed, 23 Jun 2021 at 10:07, Blaine Gardner <brgardne@xxxxxxxxxx> wrote:
Hi All,
We have had a strange issue crop up in Rook, and I wanted to bring it
up for discussion more widely since I think it has the potential to
affect cephadm deployments as well as Rook.
Here if you want to look through it:
https://github.com/rook/rook/issues/7940#issuecomment-865005220
In Rook, we list disks with `lsblk` to be provisioned. I'm not sure
how ceph-volume does this internally or how cephadm does this. In Rook
(especially with 500+GB SSDs for some reason), many users are starting
to see unexpected partitions appear. One user finally found out that
this is because the Ubuntu kernel they're using has support for ATARI
(AHDI) partitions. The kernel detects these as partitions, but lsblk
does not understand the ATARI type and thus sees them as empty
partitions. (It's unclear to me yet if `blkid` is able to see the
ATARI type.)
In Rook, this is really only a problem for OSDs that are provisioned
on nodes (not a downstream-supported use-case). On PVCs (our
downstream use-case), we only run the detection/provisioning step once
at first setup, and so I believe Rook's downstream isn't affected.
When provisioning OSDs on nodes, disk detection and OSD provisioning
both run each time we reconcile. I'm not sure if that is true for
cephadm or not.
Additionally, the upstream user reports that ATARI/AHDI disk support
is disabled for Centos 7. I hope this means we don't have to worry
about this with Centos 8 or any RHEL we ship, but it might be good to
confirm this for the RHEL versions we ship to customers.
I'll leave the discussion of Rook-specific ideas for me, Seb, and
Travis, but here are the questions I have for what we might need to
think about for Ceph more broadly:
Do we need to check Centos/RHEL versions for ATARI/AHDI disk support?
Do we need to communicate to Ceph users that their bluestore OSD disks
might appear as ATARI partitions on their host systems?
Do we need to modify ceph-volume to do anything special if it finds
ATARI/AHDI partitions?
Are there other considerations I'm missing?
Thanks again! And feel free to chime in with questions if I haven't
explained things very well.
Blaine
--
Steven Ellis
TECHNICAL PORTFOLIO EVANGELIST - APAC
APAC SENIOR PRINCIPAL PRODUCT MANAGER - OPENSHIFT INFRA & DATA SERVICES
Red Hat Asia Pacific - Auckland NZ
E: sellis@xxxxxxxxxx M: +64 21 321 673
![]()
Attachment:
OpenPGP_signature
Description: OpenPGP digital signature
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
