Hello,
I got something strange on a Pacific (16.2.6) cluster.
I have added 8 new empty spinning disk on this running cluster that is configured with:
# ceph orch ls osd --export
service_type: osd
service_id: ar_osd_hdd_spec
service_name: osd.ar_osd_hdd_spec
placement:
host_pattern: '*'
spec:
data_devices:
rotational: 1
filter_logic: AND
objectstore: bluestore
---
service_type: osd
service_id: ar_osd_ssd_spec
service_name: osd.ar_osd_ssd_spec
placement:
host_pattern: '*'
spec:
data_devices:
rotational: 0
filter_logic: AND
objectstore: bluestore
Before adding them I had:
# ceph orch ls osd
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
osd.ar_osd_hdd_spec 16/24 8m ago 4M *
osd.ar_osd_ssd_spec 8/16 8m ago 4M *
After adding the disk I have:
# ceph orch ls osd
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
osd.ar_osd_hdd_spec 16/24 8m ago 4M *
osd.ar_osd_ssd_spec 16/24 8m ago 4M *
I do not understand why the disk have been detected as osd.ar_osd_ssd_spec.
New disk are on /dev/sdf.
# ceph orch device ls —wide
Hostname Path Type Transport RPM Vendor Model Size Health Ident Fault Avail Reject Reasons
host10 /dev/sdc ssd ATA/SATA Unknown ATA Micron_5300_MTFD 960G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host10 /dev/sdd hdd ATA/SATA 7200 ATA HGST HUH721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host10 /dev/sde hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host10 /dev/sdf hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host11 /dev/sdc ssd ATA/SATA Unknown ATA Micron_5300_MTFD 960G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host11 /dev/sdd hdd ATA/SATA 7200 ATA HGST HUH721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host11 /dev/sde hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host11 /dev/sdf hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host12 /dev/sdc ssd ATA/SATA Unknown ATA Micron_5300_MTFD 960G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host12 /dev/sdd hdd ATA/SATA 7200 ATA HGST HUH721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host12 /dev/sde hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host12 /dev/sdf hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host13 /dev/sdc ssd ATA/SATA Unknown ATA Micron_5300_MTFD 960G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host13 /dev/sdd hdd ATA/SATA 7200 ATA HGST HUH721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host13 /dev/sde hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host13 /dev/sdf hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host14 /dev/sdc ssd ATA/SATA Unknown ATA Micron_5300_MTFD 960G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host14 /dev/sdd hdd ATA/SATA 7200 ATA HGST HUH721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host14 /dev/sde hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host14 /dev/sdf hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host15 /dev/sdc ssd ATA/SATA Unknown ATA Micron_5300_MTFD 960G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host15 /dev/sdd hdd ATA/SATA 7200 ATA HGST HUH721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host15 /dev/sde hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host15 /dev/sdf hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host16 /dev/sdc ssd ATA/SATA Unknown ATA Micron_5300_MTFD 960G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host16 /dev/sdd hdd ATA/SATA 7200 ATA HGST HUH721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host16 /dev/sde hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host16 /dev/sdf hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host17 /dev/sdc ssd ATA/SATA Unknown ATA Micron_5300_MTFD 960G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host17 /dev/sdd hdd ATA/SATA 7200 ATA HGST HUH721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host17 /dev/sde hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
host17 /dev/sdf hdd ATA/SATA 7200 ATA WDC WUS721010AL 10.0T Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked
# for f in /sys/block/sd[cdef]/queue/rotational; do printf "$f is "; cat $f; done
/sys/block/sdc/queue/rotational is 0
/sys/block/sdd/queue/rotational is 1
/sys/block/sde/queue/rotational is 1
/sys/block/sdf/queue/rotational is 1
Is there a way to :
1/ see which disk are in each OSD service_id?
2/ move a disk from one service_id to another one?
Another question is why “ ceph orch ls osd” reports in the RUNNING column the value x/24, why 24?
Each server has (8 servers in the cluster):
# ceph-volume inventory
Device Path Size rotates available Model name
/dev/sda 59.00 GB False False SuperMicro SSD
/dev/sdb 59.00 GB False False SuperMicro SSD
/dev/sdc 894.25 GB False False Micron_5300_MTFD
/dev/sdd 9.10 TB True False HGST HUH721010AL
/dev/sde 9.10 TB True False WDC WUS721010AL
/dev/sdf 9.10 TB True False WDC WUS721010AL
PS: of course this is not a big problem as the 2 specs are equal, but I did not understand why it did that
PS2: on another ceph 16.2.6 cluster that have the same service_spec, we did not get the same strange thing: the disk have been linked to the right service_spec.
Thank you,
--
Guillaume de Lafond
Aqua Ray
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
