Hello,
This is an interesting topic and would like to know a solution to this problem. Does that mean we should never use Dell storage as ceph storage device? I have similar setup with Dell 4 iscsi LUNs attached to openstack controller and compute node in active-active situation.
As they were in active active 1 selected first 2 luns as osd on node1 and last 2 as osd on node 2.
Is it ok to have this configuration specially when and node will be down or considering live migration.
Regards
Gaurav Goyal
On 10-Jul-2016 9:02 pm, "Christian Balzer" <chibi@xxxxxxx> wrote:
Hello,
On Sun, 10 Jul 2016 12:46:39 +0000 (UTC) William Josefsson wrote:
> Hi everyone,
>
> I have problem with swapping drive and partition names on reboot. My
> Ceph is Hammer on CentOS7, Dell R730 6xSSD (2xSSD OS RAID1 PERC,
> 4xSSD=Journal drives), 18x1.8T SAS for OSDs.
>
> Whenever I reboot, drives randomly seem to change names. This is
> extremely dangerous and frustrating when I've initially setup CEPH with
> ceph-deploy, zap, prepare and activate. It has happened that I've
> accidentally erased wrong disk too when e.g. /dev/sdX had
> become /dev/sdY.
>
This isn't a Ceph specific question per se and you could probably keep
things from moving around by enforcing module loads in a particular order.
But that of course still wouldn't help if something else changed or a
drive totally failed.
So in the context of Ceph, it doesn't (shouldn't) care if the OSD (HDD)
changes names, especially since you did set it up with ceph-deploy.
And to avoid the journals getting jumbled up, do what everybody does
(outside of Ceph as well), use /dev/disk/by-id or uuid.
Like:
---
# ls -la /var/lib/ceph/osd/ceph-28/
journal -> /dev/disk/by-id/wwn-0x55cd2e404b73d569-part3
---
Christian
> Please see an output below of how this drive swapping below appears SDC
> is shifted, indexes and drive names got shuffled. Ceph OSDs didn't come
> up properly.
>
> Please advice on how to get this corrected, with no more drive name
> shuffling. Can this be due to the PERC HW raid? thx will
>
>
>
> POST REBOOT 2 (expected outcome.. with sda,sdb,sdc,sdd as journal. sdw
> is a perc raid1)
>
>
> [cephnode3][INFO ] Running command: sudo /usr/sbin/ceph-disk list
> [cephnode3][DEBUG ] /dev/sda :
> [cephnode3][DEBUG ] /dev/sda1 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda2
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sda3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
> [cephnode3][DEBUG ] /dev/sda4 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda5
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdb : [cephnode3][DEBUG ] /dev/sdb1 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb2
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdb3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
> [cephnode3][DEBUG ] /dev/sdb4 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb5
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdc : [cephnode3][DEBUG ] /dev/sdc1 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdc2
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdc3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
> [cephnode3][DEBUG ] /dev/sdc4 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdc5
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdd : [cephnode3][DEBUG ] /dev/sdd1 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdd2
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdd3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
> [cephnode3][DEBUG ] /dev/sdd4 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdd5
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sde : [cephnode3][DEBUG ] /dev/sde1 ceph data, active, cluster
> ceph, osd.0 [cephnode3][DEBUG ] /dev/sdf : [cephnode3][DEBUG
> ] /dev/sdf1 ceph data, active, cluster ceph, osd.1 [cephnode3][DEBUG
> ] /dev/sdg : [cephnode3][DEBUG ] /dev/sdg1 ceph data, active, cluster
> ceph, osd.2 [cephnode3][DEBUG ] /dev/sdh : [cephnode3][DEBUG
> ] /dev/sdh1 ceph data, active, cluster ceph, osd.3 [cephnode3][DEBUG
> ] /dev/sdi : [cephnode3][DEBUG ] /dev/sdi1 ceph data, active, cluster
> ceph, osd.4 [cephnode3][DEBUG ] /dev/sdj : [cephnode3][DEBUG
> ] /dev/sdj1 ceph data, active, cluster ceph, osd.5 [cephnode3][DEBUG
> ] /dev/sdk : [cephnode3][DEBUG ] /dev/sdk1 ceph data, active, cluster
> ceph, osd.6 [cephnode3][DEBUG ] /dev/sdl : [cephnode3][DEBUG
> ] /dev/sdl1 ceph data, active, cluster ceph, osd.7 [cephnode3][DEBUG
> ] /dev/sdm : [cephnode3][DEBUG ] /dev/sdm1 other, xfs
> [cephnode3][DEBUG ] /dev/sdn :
> [cephnode3][DEBUG ] /dev/sdn1 ceph data, active, cluster ceph, osd.9
> [cephnode3][DEBUG ] /dev/sdo :
> [cephnode3][DEBUG ] /dev/sdo1 ceph data, active, cluster ceph, osd.10
> [cephnode3][DEBUG ] /dev/sdp :
> [cephnode3][DEBUG ] /dev/sdp1 ceph data, active, cluster ceph, osd.11
> [cephnode3][DEBUG ] /dev/sdq :
> [cephnode3][DEBUG ] /dev/sdq1 ceph data, active, cluster ceph, osd.12
> [cephnode3][DEBUG ] /dev/sdr :
> [cephnode3][DEBUG ] /dev/sdr1 ceph data, active, cluster ceph, osd.13
> [cephnode3][DEBUG ] /dev/sds :
> [cephnode3][DEBUG ] /dev/sds1 ceph data, active, cluster ceph, osd.14
> [cephnode3][DEBUG ] /dev/sdt :
> [cephnode3][DEBUG ] /dev/sdt1 ceph data, active, cluster ceph, osd.15
> [cephnode3][DEBUG ] /dev/sdu :
> [cephnode3][DEBUG ] /dev/sdu1 ceph data, active, cluster ceph, osd.16
> [cephnode3][DEBUG ] /dev/sdv :
> [cephnode3][DEBUG ] /dev/sdv1 ceph data, active, cluster ceph, osd.17
> [cephnode3][DEBUG ] /dev/sdw :
> [cephnode3][DEBUG ] /dev/sdw1 other, xfs, mounted on /
> [cephnode3][DEBUG ] /dev/sdw2 swap, swap
>
>
> POST REBOOT 1:
>
>
> [cephnode3][DEBUG ] /dev/sda :
> [cephnode3][DEBUG ] /dev/sda1 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda2
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sda3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
> [cephnode3][DEBUG ] /dev/sda4 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda5
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdb : [cephnode3][DEBUG ] /dev/sdb1 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb2
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdb3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
> [cephnode3][DEBUG ] /dev/sdb4 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb5
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdc : [cephnode3][DEBUG ] /dev/sdc1 ceph data, active, cluster
> ceph, osd.13 [cephnode3][DEBUG ] /dev/sdd : [cephnode3][DEBUG
> ] /dev/sdd1 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
> [cephnode3][DEBUG ] /dev/sdd2 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdd3
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdd4 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
> [cephnode3][DEBUG ] /dev/sdd5 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sde :
> [cephnode3][DEBUG ] /dev/sde1 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sde2
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sde3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
> [cephnode3][DEBUG ] /dev/sde4 other,
> ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sde5
> other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG
> ] /dev/sdf : [cephnode3][DEBUG ] /dev/sdf1 ceph data, active, cluster
> ceph, osd.0 [cephnode3][DEBUG ] /dev/sdg : [cephnode3][DEBUG
> ] /dev/sdg1 ceph data, active, cluster ceph, osd.14 [cephnode3][DEBUG
> ] /dev/sdh : [cephnode3][DEBUG ] /dev/sdh1 ceph data, active, cluster
> ceph, osd.1 [cephnode3][DEBUG ] /dev/sdi : [cephnode3][DEBUG
> ] /dev/sdi1 ceph data, active, cluster ceph, osd.2 [cephnode3][DEBUG
> ] /dev/sdj : [cephnode3][DEBUG ] /dev/sdj1 ceph data, active, cluster
> ceph, osd.3 [cephnode3][DEBUG ] /dev/sdk : [cephnode3][DEBUG
> ] /dev/sdk1 ceph data, active, cluster ceph, osd.4 [cephnode3][DEBUG
> ] /dev/sdl : [cephnode3][DEBUG ] /dev/sdl1 ceph data, active, cluster
> ceph, osd.15 [cephnode3][DEBUG ] /dev/sdm : [cephnode3][DEBUG
> ] /dev/sdm1 ceph data, active, cluster ceph, osd.5 [cephnode3][DEBUG
> ] /dev/sdn : [cephnode3][DEBUG ] /dev/sdn1 ceph data, active, cluster
> ceph, osd.6 [cephnode3][DEBUG ] /dev/sdo :
> [cephnode3][DEBUG ] /dev/sdo1 ceph data, active, cluster ceph, osd.7
> [cephnode3][DEBUG ] /dev/sdp :
> [cephnode3][DEBUG ] /dev/sdp1 other, xfs
> [cephnode3][DEBUG ] /dev/sdq :
> [cephnode3][DEBUG ] /dev/sdq1 ceph data, active, cluster ceph, osd.9
> [cephnode3][DEBUG ] /dev/sdr :
> [cephnode3][DEBUG ] /dev/sdr1 ceph data, active, cluster ceph, osd.16
> [cephnode3][DEBUG ] /dev/sds :
> [cephnode3][DEBUG ] /dev/sds1 ceph data, active, cluster ceph, osd.10
> [cephnode3][DEBUG ] /dev/sdt :
> [cephnode3][DEBUG ] /dev/sdt1 ceph data, active, cluster ceph, osd.11
> [cephnode3][DEBUG ] /dev/sdu :
> [cephnode3][DEBUG ] /dev/sdu1 ceph data, active, cluster ceph, osd.12
> [cephnode3][DEBUG ] /dev/sdv :
> [cephnode3][DEBUG ] /dev/sdv1 ceph data, active, cluster ceph, osd.17
> [cephnode3][DEBUG ] /dev/sdw :
> [cephnode3][DEBUG ] /dev/sdw1 other, xfs, mounted on /
> [cephnode3][DEBUG ] /dev/sdw2 swap, swap
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Christian Balzer Network/Systems Engineer
chibi@xxxxxxx Global OnLine Japan/Rakuten Communications
http://www.gol.com/
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
