Re: Ceph-deploy won't write journal if partition exists and using -- dmcrypt

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Sorry for the broken post previously. I have looked into this more and it looks like ceph-deploy is not seeing that it is a partition and attempting to create an additional partition in the journals place. I read in the documentation that if I set osd journal size = 0, that it will assume that the target is a block device and use the entire block. I tried this and it still doesn't work. I have since zapped the journals and specified a 20G journal size. Now in my ceph-deploy line I just specify :

ceph-deploy osd --dmcrypt --fs-type ${fs} create ${host}:${disk}:/dev/${journal_disk}

IE::
ceph-deploy osd --dmcrypt --fs-type btrfs create kh28-1:sde:/dev/sdab
ceph-deploy osd --dmcrypt --fs-type btrfs create kh28-1:sdf:/dev/sdab

and ceph-deploy seems to try to create a new partition every time.

I have now run into a new issue though. After ceph-deploy creates the partitions and seems to bootstrap the disks successfully it doest not mount them properly to create the journal.



[ceph_deploy.osd][DEBUG ] Calling partprobe on zapped device /dev/sdr                                                                                                                                                                                                             [1565/1920]
[kh28-3.osdc.io][INFO  ] Running command: sudo partprobe /dev/sdr
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/lacadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /usr/local/bin/ceph-deploy osd --dmcrypt --fs-type btrfs create kh28-3.osdc.io:sdr:/dev/sdp2
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks kh28-3.osdc.io:/dev/sdr:/dev/sdp2
[kh28-3.osdc.io][DEBUG ] connection detected need for sudo
[kh28-3.osdc.io][DEBUG ] connected to host: kh28-3.osdc.io
[kh28-3.osdc.io][DEBUG ] detect platform information from remote host
[kh28-3.osdc.io][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] Deploying osd to kh28-3.osdc.io
[kh28-3.osdc.io][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[kh28-3.osdc.io][INFO  ] Running command: sudo udevadm trigger --subsystem-match=block --action=""> [ceph_deploy.osd][DEBUG ] Preparing host kh28-3.osdc.io disk /dev/sdr journal /dev/sdp2 activate True
[kh28-3.osdc.io][INFO  ] Running command: sudo ceph-disk -v prepare --fs-type btrfs --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --cluster ceph -- /dev/sdr /dev/sdp2
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_btrfs
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_btrfs
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_btrfs
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating journal file /dev/sdp2 with size 0 (ceph-osd will resize and allocate)
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Journal is file /dev/sdp2
[kh28-3.osdc.io][WARNIN] WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating osd partition on /dev/sdr
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:c1879421-bcd0-4419-bc96-63d2d51176db --typecode=1:89c57f98-2fe5-4dc0-89c1-5ec00ceff2be -- /dev/sdr
[kh28-3.osdc.io][DEBUG ] The operation has completed successfully.
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Calling partprobe on created device /dev/sdr
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sdr
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --batch-mode --key-file /etc/ceph/dmcrypt-keys/c1879421-bcd0-4419-bc96-63d2d51176db.luks.key luksFormat /dev/sdr1
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --key-file /etc/ceph/dmcrypt-keys/c1879421-bcd0-4419-bc96-63d2d51176db.luks.key luksOpen /dev/sdr1 c1879421-bcd0-4419-bc96-63d2d51176db
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating btrfs fs on /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/mkfs -t btrfs -m single -l 32768 -n 32768 -- /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db
[kh28-3.osdc.io][WARNIN] Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
[kh28-3.osdc.io][DEBUG ]
[kh28-3.osdc.io][DEBUG ] WARNING! - Btrfs v3.12 IS EXPERIMENTAL
[kh28-3.osdc.io][DEBUG ] WARNING! - see http://btrfs.wiki.kernel.org before using
[kh28-3.osdc.io][DEBUG ]


[kh28-3.osdc.io][DEBUG ] fs created label (null) on /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db


[kh28-3.osdc.io][DEBUG ]        nodesize 32768 leafsize 32768 sectorsize 4096 size 3.64TiB
[kh28-3.osdc.io][DEBUG ] Btrfs v3.12
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Mounting /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db on /var/lib/ceph/tmp/mnt.lYDwEH with options noatime,autodefrag,space_cache,recovery
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t btrfs -o noatime,autodefrag,space_cache,recovery -- /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db /var/lib/ceph/tmp/mnt.lYDwEH
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.lYDwEH
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.lYDwEH/journal -> /dev/sdp2
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.lYDwEH
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.lYDwEH
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup remove c1879421-bcd0-4419-bc96-63d2d51176db
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-35865ceff05d -- /dev/sdr
[kh28-3.osdc.io][DEBUG ] The operation has completed successfully.
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/sdr
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sdr
[kh28-3.osdc.io][INFO  ] Running command: sudo udevadm trigger --subsystem-match=block --action=""> [kh28-3.osdc.io][INFO  ] checking OSD status...
[kh28-3.osdc.io][INFO  ] Running command: sudo ceph --cluster=ceph osd stat --format=json
[kh28-3.osdc.io][WARNIN] there are 36 OSDs down
[kh28-3.osdc.io][WARNIN] there are 36 OSDs out
[ceph_deploy.osd][DEBUG ] Host kh28-3.osdc.io is now ready for osd use.





I've tried this with xfs as well and it doesn't seem to make a difference. Checking the log for any of the osds on this host in /var/log/ceph/ seems to show that the journal is not created properly::

trying XFS::
2015-07-23 15:18:47.236679 7efd5aeab900  0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 973519
2015-07-23 15:18:47.239134 7efd5aeab900  1 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) mkfs in /var/lib/ceph/tmp/mnt.TZy7HL
2015-07-23 15:18:47.239156 7efd5aeab900  1 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) mkfs fsid is already set to a473d735-672f-4835-9217-55b084ad5a35
2015-07-23 15:18:47.239554 7efd5aeab900  0 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) backend xfs (magic 0x58465342)
2015-07-23 15:18:47.363235 7efd5aeab900  1 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) leveldb db exists/created
2015-07-23 15:18:47.363330 7efd5aeab900 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2015-07-23 15:18:47.363337 7efd5aeab900  1 journal _open /var/lib/ceph/tmp/mnt.TZy7HL/journal fd 10: 0 bytes, block size 4096 bytes, directio = 1, aio = 0
2015-07-23 15:18:47.363357 7efd5aeab900 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected a473d735-672f-4835-9217-55b084ad5a35, invalid (someone else's?) journal
2015-07-23 15:18:47.363385 7efd5aeab900 -1 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.TZy7HL/journal: (22) Invalid argument
2015-07-23 15:18:47.363406 7efd5aeab900 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22
2015-07-23 15:18:47.363439 7efd5aeab900 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.TZy7HL: (22) Invalid argument


trying btrfs::
2015-07-23 16:40:12.429496 7fef6fbb2900 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.cbhhm7: (22) Invalid argument
2015-07-23 16:40:31.380336 7fb07eff1900  0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1153140
2015-07-23 16:40:31.382097 7fb07eff1900  1 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) mkfs in /var/lib/ceph/tmp/mnt.Aa9khA
2015-07-23 16:40:31.382118 7fb07eff1900  1 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) mkfs fsid is already set to f8b57033-dd8d-41ea-ace1-593bd6048789
2015-07-23 16:40:31.382193 7fb07eff1900  0 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) backend btrfs (magic 0x9123683e)
2015-07-23 16:40:31.411131 7fb07eff1900  1 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) leveldb db exists/created
2015-07-23 16:40:31.411203 7fb07eff1900 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2015-07-23 16:40:31.411211 7fb07eff1900  1 journal _open /var/lib/ceph/tmp/mnt.Aa9khA/journal fd 10: 0 bytes, block size 4096 bytes, directio = 1, aio = 0
2015-07-23 16:40:31.411225 7fb07eff1900 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected f8b57033-dd8d-41ea-ace1-593bd6048789, invalid (someone else's?) journal
2015-07-23 16:40:31.411251 7fb07eff1900 -1 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.Aa9khA/journal: (22) Invalid argument
2015-07-23 16:40:31.411263 7fb07eff1900 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22
2015-07-23 16:40:31.411291 7fb07eff1900 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.Aa9khA: (22) Invalid argument


What is strange is that this all seemed to work without issue in Giant so I am not sure what has changed. I am about to try a manual deploy next but i've grown lazy with ceph-deploy.

Has anyone experienced anything similar or know a work around (outside of bootstrapping and deploying each osd manually?)



_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux