|
Sorry for the broken post previously. I have looked into this more
and it looks like ceph-deploy is not seeing that it is a partition
and attempting to create an additional partition in the journals
place. I read in the documentation that if I set osd journal size =
0, that it will assume that the target is a block device and use the
entire block. I tried this and it still doesn't work. I have since
zapped the journals and specified a 20G journal size. Now in my
ceph-deploy line I just specify : ceph-deploy osd --dmcrypt --fs-type ${fs} create ${host}:${disk}:/dev/${journal_disk} IE:: ceph-deploy osd --dmcrypt --fs-type btrfs create kh28-1:sde:/dev/sdab ceph-deploy osd --dmcrypt --fs-type btrfs create kh28-1:sdf:/dev/sdab and ceph-deploy seems to try to create a new partition every time. I have now run into a new issue though. After ceph-deploy creates the partitions and seems to bootstrap the disks successfully it doest not mount them properly to create the journal. [ceph_deploy.osd][DEBUG ] Calling partprobe on zapped device /dev/sdr [1565/1920] [kh28-3.osdc.io][INFO ] Running command: sudo partprobe /dev/sdr [ceph_deploy.conf][DEBUG ] found configuration file at: /home/lacadmin/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.25): /usr/local/bin/ceph-deploy osd --dmcrypt --fs-type btrfs create kh28-3.osdc.io:sdr:/dev/sdp2 [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks kh28-3.osdc.io:/dev/sdr:/dev/sdp2 [kh28-3.osdc.io][DEBUG ] connection detected need for sudo [kh28-3.osdc.io][DEBUG ] connected to host: kh28-3.osdc.io [kh28-3.osdc.io][DEBUG ] detect platform information from remote host [kh28-3.osdc.io][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty [ceph_deploy.osd][DEBUG ] Deploying osd to kh28-3.osdc.io [kh28-3.osdc.io][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [kh28-3.osdc.io][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=""> [ceph_deploy.osd][DEBUG ] Preparing host kh28-3.osdc.io disk /dev/sdr journal /dev/sdp2 activate True [kh28-3.osdc.io][INFO ] Running command: sudo ceph-disk -v prepare --fs-type btrfs --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --cluster ceph -- /dev/sdr /dev/sdp2 [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_btrfs [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_btrfs [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_btrfs [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating journal file /dev/sdp2 with size 0 (ceph-osd will resize and allocate) [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Journal is file /dev/sdp2 [kh28-3.osdc.io][WARNIN] WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating osd partition on /dev/sdr [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:c1879421-bcd0-4419-bc96-63d2d51176db --typecode=1:89c57f98-2fe5-4dc0-89c1-5ec00ceff2be -- /dev/sdr [kh28-3.osdc.io][DEBUG ] The operation has completed successfully. [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Calling partprobe on created device /dev/sdr [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sdr [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --batch-mode --key-file /etc/ceph/dmcrypt-keys/c1879421-bcd0-4419-bc96-63d2d51176db.luks.key luksFormat /dev/sdr1 [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --key-file /etc/ceph/dmcrypt-keys/c1879421-bcd0-4419-bc96-63d2d51176db.luks.key luksOpen /dev/sdr1 c1879421-bcd0-4419-bc96-63d2d51176db [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating btrfs fs on /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/mkfs -t btrfs -m single -l 32768 -n 32768 -- /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db [kh28-3.osdc.io][WARNIN] Turning ON incompat feature 'extref': increased hardlink limit per file to 65536 [kh28-3.osdc.io][DEBUG ] [kh28-3.osdc.io][DEBUG ] WARNING! - Btrfs v3.12 IS EXPERIMENTAL [kh28-3.osdc.io][DEBUG ] WARNING! - see http://btrfs.wiki.kernel.org before using [kh28-3.osdc.io][DEBUG ] [kh28-3.osdc.io][DEBUG ] fs created label (null) on /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db [kh28-3.osdc.io][DEBUG ] nodesize 32768 leafsize 32768 sectorsize 4096 size 3.64TiB [kh28-3.osdc.io][DEBUG ] Btrfs v3.12 [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Mounting /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db on /var/lib/ceph/tmp/mnt.lYDwEH with options noatime,autodefrag,space_cache,recovery [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t btrfs -o noatime,autodefrag,space_cache,recovery -- /dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db /var/lib/ceph/tmp/mnt.lYDwEH [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.lYDwEH [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.lYDwEH/journal -> /dev/sdp2 [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.lYDwEH [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.lYDwEH [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup remove c1879421-bcd0-4419-bc96-63d2d51176db [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-35865ceff05d -- /dev/sdr [kh28-3.osdc.io][DEBUG ] The operation has completed successfully. [kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/sdr [kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sdr [kh28-3.osdc.io][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=""> [kh28-3.osdc.io][INFO ] checking OSD status... [kh28-3.osdc.io][INFO ] Running command: sudo ceph --cluster=ceph osd stat --format=json [kh28-3.osdc.io][WARNIN] there are 36 OSDs down [kh28-3.osdc.io][WARNIN] there are 36 OSDs out [ceph_deploy.osd][DEBUG ] Host kh28-3.osdc.io is now ready for osd use. I've tried this with xfs as well and it doesn't seem to make a difference. Checking the log for any of the osds on this host in /var/log/ceph/ seems to show that the journal is not created properly:: trying XFS:: 2015-07-23 15:18:47.236679 7efd5aeab900 0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 973519 2015-07-23 15:18:47.239134 7efd5aeab900 1 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) mkfs in /var/lib/ceph/tmp/mnt.TZy7HL 2015-07-23 15:18:47.239156 7efd5aeab900 1 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) mkfs fsid is already set to a473d735-672f-4835-9217-55b084ad5a35 2015-07-23 15:18:47.239554 7efd5aeab900 0 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) backend xfs (magic 0x58465342) 2015-07-23 15:18:47.363235 7efd5aeab900 1 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) leveldb db exists/created 2015-07-23 15:18:47.363330 7efd5aeab900 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2015-07-23 15:18:47.363337 7efd5aeab900 1 journal _open /var/lib/ceph/tmp/mnt.TZy7HL/journal fd 10: 0 bytes, block size 4096 bytes, directio = 1, aio = 0 2015-07-23 15:18:47.363357 7efd5aeab900 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected a473d735-672f-4835-9217-55b084ad5a35, invalid (someone else's?) journal 2015-07-23 15:18:47.363385 7efd5aeab900 -1 filestore(/var/lib/ceph/tmp/mnt.TZy7HL) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.TZy7HL/journal: (22) Invalid argument 2015-07-23 15:18:47.363406 7efd5aeab900 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22 2015-07-23 15:18:47.363439 7efd5aeab900 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.TZy7HL: (22) Invalid argument trying btrfs:: 2015-07-23 16:40:12.429496 7fef6fbb2900 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.cbhhm7: (22) Invalid argument 2015-07-23 16:40:31.380336 7fb07eff1900 0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1153140 2015-07-23 16:40:31.382097 7fb07eff1900 1 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) mkfs in /var/lib/ceph/tmp/mnt.Aa9khA 2015-07-23 16:40:31.382118 7fb07eff1900 1 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) mkfs fsid is already set to f8b57033-dd8d-41ea-ace1-593bd6048789 2015-07-23 16:40:31.382193 7fb07eff1900 0 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) backend btrfs (magic 0x9123683e) 2015-07-23 16:40:31.411131 7fb07eff1900 1 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) leveldb db exists/created 2015-07-23 16:40:31.411203 7fb07eff1900 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2015-07-23 16:40:31.411211 7fb07eff1900 1 journal _open /var/lib/ceph/tmp/mnt.Aa9khA/journal fd 10: 0 bytes, block size 4096 bytes, directio = 1, aio = 0 2015-07-23 16:40:31.411225 7fb07eff1900 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected f8b57033-dd8d-41ea-ace1-593bd6048789, invalid (someone else's?) journal 2015-07-23 16:40:31.411251 7fb07eff1900 -1 filestore(/var/lib/ceph/tmp/mnt.Aa9khA) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.Aa9khA/journal: (22) Invalid argument 2015-07-23 16:40:31.411263 7fb07eff1900 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22 2015-07-23 16:40:31.411291 7fb07eff1900 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.Aa9khA: (22) Invalid argument What is strange is that this all seemed to work without issue in Giant so I am not sure what has changed. I am about to try a manual deploy next but i've grown lazy with ceph-deploy. Has anyone experienced anything similar or know a work around (outside of bootstrapping and deploying each osd manually?) |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
