Hi,
News. I tried activate disk without --dmcrypt and there is no problem. After
activate on sdb are two partitions (sdb2 for jounral and sdb1 for data).
In my opinion there is a bug with switch --dmcrypt and activating
journal on disk (partitions are created, but mounting done by ceph-disk fail).
Here are logs without --dmcrypt
root@ceph-deploy:~/ceph# ceph-deploy osd prepare ceph-node0:/dev/sdb
[ceph_deploy.cli][INFO ] Invoked (1.3.1): /usr/bin/ceph-deploy osd prepare ceph-node0:/dev/sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph-node0:/dev/sdb:
[ceph-node0][DEBUG ] connected to host: ceph-node0
[ceph-node0][DEBUG ] detect platform information from remote host
[ceph-node0][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 13.04 raring
[ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node0
[ceph-node0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node0][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host ceph-node0 disk /dev/sdb journal None activate False
[ceph-node0][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdb
[ceph-node0][ERROR ] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
[ceph-node0][DEBUG ] Information: Moved requested sector from 34 to 2048 in
[ceph-node0][DEBUG ] order to align on 2048-sector boundaries.
[ceph-node0][DEBUG ] The operation has completed successfully.
[ceph-node0][DEBUG ] Information: Moved requested sector from 2097153 to 2099200 in
[ceph-node0][DEBUG ] order to align on 2048-sector boundaries.
[ceph-node0][DEBUG ] The operation has completed successfully.
[ceph-node0][DEBUG ] meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=917439 blks
[ceph-node0][DEBUG ] = sectsz=512 attr=2, projid32bit=0
[ceph-node0][DEBUG ] data = bsize=4096 blocks=3669755, imaxpct=25
[ceph-node0][DEBUG ] = sunit=0 swidth=0 blks
[ceph-node0][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
[ceph-node0][DEBUG ] log =internal log bsize=4096 blocks=2560, version=2
[ceph-node0][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1
[ceph-node0][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0
[ceph-node0][DEBUG ] The operation has completed successfully.
[ceph_deploy.osd][DEBUG ] Host ceph-node0 is now ready for osd use.
Disk are properly activated. With --dmcrypt journal partition are not
propery mounted and ceph-disk cannot use it.
Best Regards,
Michael
>
> Hi!
>
> I have a question about activating OSD on whole disk. I can't bypass this issue.
> Conf spec: 8 VMs - ceph-deploy; ceph-admin; ceph-mon0-2 and ceph-node0-2;
>
> I started from creating MON - all good .
> After that I want to prepare and activate 3x OSD with dm-crypt.
>
> So I put on ceph.conf this
>
> [osd.0]
> host = ceph-node0
> cluster addr = 10.0.0.75:6800
> public addr = 10.0.0.75:6801
> devs = /dev/sdb
>
> Next I use ceph-deploy to activate a OSD and this shows
>
> root@ceph-deploy:~/ceph# ceph-deploy osd prepare ceph-node0:/dev/sdb --dmcrypt
> [ceph_deploy.cli][INFO ] Invoked (1.3.1): /usr/bin/ceph-deploy
> osd prepare ceph-node0:/dev/sdb --dmcrypt
> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph-node0:/dev/sdb:
> [ceph-node0][DEBUG ] connected to host: ceph-node0
> [ceph-node0][DEBUG ] detect platform information from remote host
> [ceph-node0][DEBUG ] detect machine type
> [ceph_deploy.osd][INFO ] Distro info: Ubuntu 13.04 raring
> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node0
> [ceph-node0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> [ceph-node0][INFO ] Running command: udevadm trigger
> --subsystem-match=block --action=add
> [ceph_deploy.osd][DEBUG ] Preparing host ceph-node0 disk /dev/sdb journal None activate False
> [ceph-node0][INFO ] Running command: ceph-disk-prepare --fs-type
> xfs --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --cluster ceph -- /dev/sdb
> [ceph-node0][ERROR ] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
> [ceph-node0][ERROR ] ceph-disk: Error: partition 1 for /dev/sdb does not appear to exist
> [ceph-node0][DEBUG ] Information: Moved requested sector from 34 to 2048 in
> [ceph-node0][DEBUG ] order to align on 2048-sector boundaries.
> [ceph-node0][DEBUG ] The operation has completed successfully.
> [ceph-node0][DEBUG ] Information: Moved requested sector from 2097153 to 2099200 in
> [ceph-node0][DEBUG ] order to align on 2048-sector boundaries.
> [ceph-node0][DEBUG ] Warning: The kernel is still using the old partition table.
> [ceph-node0][DEBUG ] The new table will be used at the next reboot.
> [ceph-node0][DEBUG ] The operation has completed successfully.
> [ceph-node0][ERROR ] Traceback (most recent call last):
> [ceph-node0][ERROR ] File
> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/process.py", line 68, in run
> [ceph-node0][ERROR ] reporting(conn, result, timeout)
> [ceph-node0][ERROR ] File
> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/log.py", line 13, in reporting
> [ceph-node0][ERROR ] received = result.receive(timeout)
> [ceph-node0][ERROR ] File
> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py", line 455, in receive
> [ceph-node0][ERROR ] raise self._getremoteerror() or EOFError()
> [ceph-node0][ERROR ] RemoteError: Traceback (most recent call last):
> [ceph-node0][ERROR ] File "<string>", line 806, in executetask
> [ceph-node0][ERROR ] File "", line 35, in _remote_run
> [ceph-node0][ERROR ] RuntimeError: command returned non-zero exit status: 1
> [ceph-node0][ERROR ]
> [ceph-node0][ERROR ]
> [ceph_deploy.osd][ERROR ] Failed to execute command:
> ceph-disk-prepare --fs-type xfs --dmcrypt --dmcrypt-key-dir
> /etc/ceph/dmcrypt-keys --cluster ceph -- /dev/sdb
> [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
>
>
> It's looks like ceph-disk-prepare can't mount (activate?) the one of disk.
> So I go to ceph-node0 and listed disk, this shows:
>
> root@ceph-node0:~# ls /dev/sd
> sda sda1 sda2 sda5 sdb sdb2
>
> Ups - there are no sdb1.
>
> So I printed all partitions on /dev/sdb and there is two:
>
> Number Beg End Size Filesystem Name Flags
> 2 1049kB 1074MB 1073MB ceph journal
> 1 1075MB 16,1GB 15,0GB ceph data
>
> Where sdb1 should be for data and sdb2 for journal.
>
> When I restart the VM /dev/sdb1 start showing.
> root@ceph-node0:~# ls /dev/sd
> sda sda1 sda2 sda5 sdb sdb1 sdb2
> But I cant mount
>
> When I put journal to separate file/disk, there is no problem with activating
> (journal are on separate disk, and all partition data are on sdb1).
> There is log from this acction (I put journal to file in /mnt/sdb2)
>
> root@ceph-deploy:~/ceph# ceph-deploy osd prepare
> ceph-node0:/dev/sdb:/mnt/sdb2 --dmcrypt
> [ceph_deploy.cli][INFO ] Invoked (1.3.1): /usr/bin/ceph-deploy
> osd prepare ceph-node0:/dev/sdb:/mnt/sdb2 --dmcrypt
> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph-node0:/dev/sdb:/mnt/sdb2
> [ceph-node0][DEBUG ] connected to host: ceph-node0
> [ceph-node0][DEBUG ] detect platform information from remote host
> [ceph-node0][DEBUG ] detect machine type
> [ceph_deploy.osd][INFO ] Distro info: Ubuntu 13.04 raring
> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node0
> [ceph-node0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> [ceph-node0][INFO ] Running command: udevadm trigger
> --subsystem-match=block --action=add
> [ceph_deploy.osd][DEBUG ] Preparing host ceph-node0 disk /dev/sdb journal /mnt/sdb2 activate False
> [ceph-node0][INFO ] Running command: ceph-disk-prepare --fs-type
> xfs --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --cluster ceph -- /dev/sdb /mnt/sdb2
> [ceph-node0][ERROR ] WARNING:ceph-disk:OSD will not be
> hot-swappable if journal is not the same device as the osd data
> [ceph-node0][DEBUG ] Information: Moved requested sector from 34 to 2048 in
> [ceph-node0][DEBUG ] order to align on 2048-sector boundaries.
> [ceph-node0][DEBUG ] The operation has completed successfully.
> [ceph-node0][DEBUG ]
> meta-data=/dev/mapper/299e07ff-31bf-49c4-a8de-62a8e4203c04
> isize=2048 agcount=4, agsize=982975 blks
> [ceph-node0][DEBUG ] = sectsz=512 attr=2, projid32bit=0
> [ceph-node0][DEBUG ] data = bsize=4096 blocks=3931899, imaxpct=25
> [ceph-node0][DEBUG ] = sunit=0 swidth=0 blks
> [ceph-node0][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
> [ceph-node0][DEBUG ] log =internal log bsize=4096 blocks=2560, version=2
> [ceph-node0][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1
> [ceph-node0][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0
> [ceph-node0][DEBUG ] The operation has completed successfully.
> [ceph_deploy.osd][DEBUG ] Host ceph-node0 is now ready for osd use.
>
> Listed partitions (there are only one)
> Number Beg End Size Filesystem Name Flags
> 1 1049kB 16,1GB 16,1GB ceph data
>
> Keys are properly stored in /etc/ceph/dmcrypt-keys/ and sdb1 are
> mounted to /var/lib/ceph/osd/ceph-1
> Ceph start showing that this OSD are in cluster. Yupi. But this is no solution to me ;)
>
> So the final question is - Where I'm wrong? Why I can't activate journal on the same disk.
>
> --
> Best Regards,
> Michael Lukzak
--
Pozdrowienia,
Michael Lukzak
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
