i have meet the same issue , when i want to use prepare .
when i use --zap-disk , it is ok. but if use prepare to define journal
device, failed
ceph-disk-prepare --zap-disk --fs-type btrfs --cluster ceph -- /dev/sdb
/dev/sdc
2014-07-01 1:00 GMT+07:00 Iban Cabrillo <cabrillo at ifca.unican.es>:
> Hi Alfredo,
> During this morning, I have purged all the deployment.
> I just prepared 4 SAN Servers with 4 FC-Atacched disk (2.7 TB per disk)
> each one of them.
>
> Tomorrow I will try to deploy anew installation leaving the VMs machines
> as mons and the OSDs with this physical servers.
>
> The local disk Is SAS RAID1, Should I put the journal over the local
> disk (SAS RAID1), or should be a better solution used the RAID5 nlSAS disk
> FC-attached (journal + data on the same disk)?
>
> Which would be the recommended size for the journal? 10GB per disk, for
> example?
>
> Regards, I
>
>
> 2014-06-30 18:50 GMT+02:00 Alfredo Deza <alfredo.deza at inktank.com>:
>
> On Mon, Jun 30, 2014 at 11:22 AM, Iban Cabrillo <cabrillo at ifca.unican.es>
>> wrote:
>> > Hi Alfredo and folk,
>> > Could you have a look at this?
>> > Someone else has any idea why i am getting this error?
>> >
>> > Thanks in advance, I
>> >
>> >
>> >
>> > 2014-06-27 16:37 GMT+02:00 Iban Cabrillo <cabrillo at ifca.unican.es>:
>> >
>> >> Hi Alfredo,
>> >> This is the complete procedure:
>> >>
>> >>
>> >> On OSD node:
>> >>
>> >> [ceph at ceph02 ~]$ sudo parted /dev/xvdb
>> >>
>> >> GNU Parted 2.1
>> >> Using /dev/xvdb
>> >> Welcome to GNU Parted! Type 'help' to view a list of commands.
>> >> (parted) p
>> >> Model: Xen Virtual Block Device (xvd)
>> >> Disk /dev/xvdb: 107GB
>> >> Sector size (logical/physical): 512B/512B
>> >> Partition Table: gpt
>> >>
>> >> Number Start End Size File system Name Flags
>> >>
>> >> [ceph at ceph02 ~]$ sudo ls -la /var/lib/ceph/tmp/
>> >> total 8
>> >> drwxr-xr-x 2 root root 4096 Jun 27 16:30 .
>> >> drwxr-xr-x 7 root root 4096 Jun 26 22:30 ..
>> >> [ceph at ceph02 ~]$ sudo ls -la /var/lib/ceph/osd/
>> >> total 8
>> >> drwxr-xr-x 2 root root 4096 Jun 27 12:14 .
>> >> drwxr-xr-x 7 root root 4096 Jun 26 22:30 ..
>> >>
>> >> On ceph admin node:
>> >>
>> >> [ceph at cephadm ~]$ sudo ceph osd tree
>> >> # id weight type name up/down reweight
>> >> -1 0.14 root default
>> >> -2 0.009995 host ceph02
>> >> 1 0.009995 osd.1 DNE
>> >> -3 0.03999 host ceph04
>> >> 4 0.03999 osd.4 up 1
>> >> -4 0.09 host ceph03
>> >> 6 0.09 osd.6 up 1
>> >>
>> >>
>> >> [ceph at cephadm ceph-cloud]$ ceph-deploy osd prepare ceph02:xvdb
>> >> [ceph_deploy.conf][DEBUG ] found configuration file at:
>> >> /home/ceph/.cephdeploy.conf
>> >> [ceph_deploy.cli][INFO ] Invoked (1.5.5): /usr/bin/ceph-deploy osd
>> >> prepare ceph02:xvdb
>> >> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
>> ceph02:/dev/xvdb:
>> >> [ceph02][DEBUG ] connected to host: ceph02
>> >> [ceph02][DEBUG ] detect platform information from remote host
>> >> [ceph02][DEBUG ] detect machine type
>> >> [ceph_deploy.osd][INFO ] Distro info: Scientific Linux 6.2 Carbon
>> >> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph02
>> >> [ceph02][DEBUG ] write cluster configuration to
>> /etc/ceph/{cluster}.conf
>> >> [ceph02][INFO ] Running command: sudo udevadm trigger
>> >> --subsystem-match=block --action=add
>> >> [ceph_deploy.osd][DEBUG ] Preparing host ceph02 disk /dev/xvdb journal
>> >> None activate False
>> >> [ceph02][INFO ] Running command: sudo ceph-disk-prepare --fs-type xfs
>> >> --cluster ceph -- /dev/xvdb
>> >> [ceph02][DEBUG ] Setting name!
>> >> [ceph02][DEBUG ] partNum is 1
>> >> [ceph02][DEBUG ] REALLY setting name!
>> >> [ceph02][DEBUG ] The operation has completed successfully.
>> >> [ceph02][DEBUG ] Setting name!
>> >> [ceph02][DEBUG ] partNum is 0
>> >> [ceph02][DEBUG ] REALLY setting name!
>> >> [ceph02][DEBUG ] The operation has completed successfully.
>> >> [ceph02][DEBUG ] meta-data=/dev/xvdb1 isize=2048
>> agcount=4,
>> >> agsize=5897919 blks
>> >> [ceph02][DEBUG ] = sectsz=512 attr=2
>> >> [ceph02][DEBUG ] data = bsize=4096
>> >> blocks=23591675, imaxpct=25
>> >> [ceph02][DEBUG ] = sunit=0 swidth=0
>> >> blks
>> >> [ceph02][DEBUG ] naming =version 2 bsize=4096
>> ascii-ci=0
>> >> [ceph02][DEBUG ] log =internal log bsize=4096
>> >> blocks=11519, version=2
>> >> [ceph02][DEBUG ] = sectsz=512 sunit=0
>> >> blks, lazy-count=1
>> >> [ceph02][DEBUG ] realtime =none extsz=4096
>> blocks=0,
>> >> rtextents=0
>> >> [ceph02][DEBUG ] The operation has completed successfully.
>> >> [ceph02][WARNIN] INFO:ceph-disk:Will colocate journal with data on
>> >> /dev/xvdb
>> >> [ceph02][INFO ] checking OSD status...
>> >> [ceph02][INFO ] Running command: sudo ceph --cluster=ceph osd stat
>> >> --format=json
>> >> [ceph_deploy.osd][DEBUG ] Host ceph02 is now ready for osd use.
>> >>
>> >>
>> >>
>> >> If i make create instead of prepare do the same (create do not make the
>> >> trick prepare+activate )
>> >>
>> >>
>> >> In the OSD:
>> >>
>> >> [ceph at ceph02 ~]$ sudo parted /dev/xvdb
>> >>
>> >> GNU Parted 2.1
>> >> Using /dev/xvdb
>> >> Welcome to GNU Parted! Type 'help' to view a list of commands.
>> >> (parted) p
>> >> Model: Xen Virtual Block Device (xvd)
>> >> Disk /dev/xvdb: 107GB
>> >> Sector size (logical/physical): 512B/512B
>> >> Partition Table: gpt
>> >>
>> >> Number Start End Size File system Name Flags
>> >> 2 1049kB 10.7GB 10.7GB xfs ceph journal
>> >> 1 10.7GB 107GB 96.6GB xfs ceph data
>> >>
>> >> (parted) q
>> >>
>> >> [ceph at ceph02 ~]$ sudo ls -la /var/lib/ceph/osd/
>> >> total 8
>> >> drwxr-xr-x 2 root root 4096 Jun 27 12:14 .
>> >> drwxr-xr-x 7 root root 4096 Jun 26 22:30 ..
>> >> [ceph at ceph02 ~]$ sudo ls -la /var/lib/ceph/tmp/
>> >> total 8
>> >> drwxr-xr-x 2 root root 4096 Jun 27 16:32 .
>> >> drwxr-xr-x 7 root root 4096 Jun 26 22:30 ..
>> >> -rw-r--r-- 1 root root 0 Jun 27 16:32 ceph-disk.prepare.lock
>> >>
>> >> [ceph at ceph02 ~]$ sudo ceph-disk list
>> >> /dev/xvda1 other, ext4, mounted on /
>> >> /dev/xvdb :
>> >> /dev/xvdb1 ceph data, prepared, cluster ceph, journal /dev/xvdb2
>> >> /dev/xvdb2 ceph journal, for /dev/xvdb1
>> >>
>> >>
>> >> In the cephadm:
>> >>
>> >> [ceph at cephadm ceph-cloud]$ ceph-deploy osd activate ceph02:xvdb1
>> >> [ceph_deploy.conf][DEBUG ] found configuration file at:
>> >> /home/ceph/.cephdeploy.conf
>> >> [ceph_deploy.cli][INFO ] Invoked (1.5.5): /usr/bin/ceph-deploy osd
>> >> activate ceph02:xvdb1
>> >> [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
>> ceph02:/dev/xvdb1:
>> >> [ceph02][DEBUG ] connected to host: ceph02
>> >> [ceph02][DEBUG ] detect platform information from remote host
>> >> [ceph02][DEBUG ] detect machine type
>> >> [ceph_deploy.osd][INFO ] Distro info: Scientific Linux 6.2 Carbon
>> >> [ceph_deploy.osd][DEBUG ] activating host ceph02 disk /dev/xvdb1
>> >> [ceph_deploy.osd][DEBUG ] will use init type: sysvinit
>> >> [ceph02][INFO ] Running command: sudo ceph-disk-activate --mark-init
>> >> sysvinit --mount /dev/xvdb1
>> >> [ceph02][WARNIN] got monmap epoch 2
>> >> [ceph02][WARNIN] 2014-06-27 16:35:49.948865 7f143254e7a0 -1
>> >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) mkjournal error creating
>> journal on
>> >> /var/lib/ceph/tmp/mnt.9VZHpR/journal: (2) No such file or directory
>> >> [ceph02][WARNIN] 2014-06-27 16:35:49.948893 7f143254e7a0 -1 OSD::mkfs:
>> >> ObjectStore::mkfs failed with error -2
>> >> [ceph02][WARNIN] 2014-06-27 16:35:49.948957 7f143254e7a0 -1 ** ERROR:
>> >> error creating empty object store in /var/lib/ceph/tmp/mnt.9VZHpR: (2)
>> No
>> >> such file or directory
>> >>
>> >> [ceph02][WARNIN] ERROR:ceph-disk:Failed to activate
>> >> [ceph02][WARNIN] Traceback (most recent call last):
>> >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 2579, in <module>
>> >> [ceph02][WARNIN] main()
>> >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 2557, in main
>> >> [ceph02][WARNIN] args.func(args)
>> >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 1910, in
>> main_activate
>> >> [ceph02][WARNIN] init=args.mark_init,
>> >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 1686, in
>> >> mount_activate
>> >> [ceph02][WARNIN] (osd_id, cluster) = activate(path,
>> >> activate_key_template, init)
>> >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 1849, in activate
>> >> [ceph02][WARNIN] keyring=keyring,
>> >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 1484, in mkfs
>> >> [ceph02][WARNIN] '--keyring', os.path.join(path, 'keyring'),
>> >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 303, in
>> >> command_check_call
>> >> [ceph02][WARNIN] return subprocess.check_call(arguments)
>> >> [ceph02][WARNIN] File "/usr/lib64/python2.6/subprocess.py", line
>> 505, in
>> >> check_call
>> >> [ceph02][WARNIN] raise CalledProcessError(retcode, cmd)
>> >> [ceph02][WARNIN] subprocess.CalledProcessError: Command
>> >> '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i',
>> '0',
>> >> '--monmap', '/var/lib/ceph/tmp/mnt.9VZHpR/activate.monmap',
>> '--osd-data',
>> >> '/var/lib/ceph/tmp/mnt.9VZHpR', '--osd-journal',
>> >> '/var/lib/ceph/tmp/mnt.9VZHpR/journal', '--osd-uuid',
>> >> '5e93fa7c-b6f7-4684-981b-bf73254bd87a', '--keyring',
>> >> '/var/lib/ceph/tmp/mnt.9VZHpR/keyring']' returned non-zero exit status
>> 1
>> >> [ceph02][ERROR ] RuntimeError: command returned non-zero exit status: 1
>> >> [ceph_deploy][ERROR ] RuntimeError: Failed to execute command:
>> >> ceph-disk-activate --mark-init sysvinit --mount /dev/xvdb1
>> >>
>>
>> Is it possible you have tried to install Ceph a few times to get into
>> this state? It is odd that /var/lib/ceph/tmp doesn't exist
>> (as presented by the logs).
>>
>> Are you able to replicate this problem from scratch? I would try to
>> redo everything from a fresh host. If that is not possible, you
>> could tell ceph-deploy to wipe everything. Just as long as you do not
>> care about the data (the following will wipe all data stored in
>> ceph) you could run this:
>>
>> ceph-deploy purge ceph02 && ceph-deploy purgedata ceph02 &&
>> ceph-deploy install ceph02
>>
>>
>> >>
>> >> At OSD log:
>> >>
>> >> [ceph at ceph02 ~]$ tail -100 /var/log/ceph/ceph-osd.0.log
>> >> 2014-06-27 16:35:49.859984 7f143254e7a0 0 ceph version 0.80.1
>> >> (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-osd, pid 6590
>> >> 2014-06-27 16:35:49.861265 7f143254e7a0 1
>> >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) mkfs in
>> /var/lib/ceph/tmp/mnt.9VZHpR
>> >> 2014-06-27 16:35:49.861319 7f143254e7a0 1
>> >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) mkfs fsid is already set to
>> >> 5e93fa7c-b6f7-4684-981b-bf73254bd87a
>> >> 2014-06-27 16:35:49.948589 7f143254e7a0 1
>> >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) leveldb db exists/created
>> >> 2014-06-27 16:35:49.948865 7f143254e7a0 -1
>> >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) mkjournal error creating
>> journal on
>> >> /var/lib/ceph/tmp/mnt.9VZHpR/journal: (2) No such file or directory
>> >> 2014-06-27 16:35:49.948893 7f143254e7a0 -1 OSD::mkfs: ObjectStore::mkfs
>> >> failed with error -2
>> >> 2014-06-27 16:35:49.948957 7f143254e7a0 -1 ** ERROR: error creating
>> empty
>> >> object store in /var/lib/ceph/tmp/mnt.9VZHpR: (2) No such file or
>> directory
>> >> [ceph at ceph02 ~]$
>> >>
>> >> [ceph at ceph02 ~]$ sudo ceph-disk list
>> >> /dev/xvda1 other, ext4, mounted on /
>> >> /dev/xvdb :
>> >> /dev/xvdb1 ceph data, prepared, cluster ceph, osd.0, journal
>> /dev/xvdb2
>> >> /dev/xvdb2 ceph journal, for /dev/xvdb1
>> >>
>> >>
>> >> Thanks in advance, I
>> >>
>> >>
>> >>
>> >> 2014-06-27 15:30 GMT+02:00 Alfredo Deza <alfredo.deza at inktank.com>:
>> >>
>> >>> Can you paste the full ceph-deploy logs? there are a few reasons why
>> >>> this might be happening.
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Jun 27, 2014 at 6:42 AM, Iban Cabrillo <
>> cabrillo at ifca.unican.es>
>> >>> wrote:
>> >>> > Hi,
>> >>> >
>> >>> > I am a little frustrated. After 6 times trying to deploy a test
>> ceph
>> >>> > always
>> >>> > get the same error in the osd activation stage.
>> >>> > The version is firefly (for el6 repo), 3 mons, 3 osds all of then
>> Xen
>> >>> > VMs.
>> >>> >
>> >>> > The mons wake up correctly and I do not know why two osd servers
>> too
>> >>> > after
>> >>> > a lot of errors, always the same
>> >>> >
>> >>> > #ceph-deploy --verbose osd prepare ceph02:xvdb (works fine)
>> >>> >
>> >>> > [root at ceph02 ~]# parted /dev/xvdb
>> >>> > GNU Parted 2.1
>> >>> > Using /dev/xvdb
>> >>> > Welcome to GNU Parted! Type 'help' to view a list of commands.
>> >>> > (parted) p
>> >>> > Model: Xen Virtual Block Device (xvd)
>> >>> > Disk /dev/xvdb: 107GB
>> >>> > Sector size (logical/physical): 512B/512B
>> >>> > Partition Table: gpt
>> >>> >
>> >>> > Number Start End Size File system Name Flags
>> >>> > 2 1049kB 10,7GB 10,7GB xfs ceph journal
>> >>> > 1 10,7GB 107GB 96,6GB xfs ceph data
>> >>> >
>> >>> >
>> >>> > But the activate give us this error:
>> >>> >
>> >>> > ceph-deploy --verbose osd activate ceph02:xvdb1:/dev/xvdb2
>> >>> >
>> >>> > [ceph02][WARNIN] 2014-06-27 12:27:34.750160 7f123b33d7a0 -1
>> >>> > filestore(/var/lib/ceph/tmp/mnt.HacFAP) mkjournal error creating
>> >>> > journal on
>> >>> > /var/lib/ceph/tmp/mnt.HacFAP/journal: (2) No such file or directory
>> >>> > [ceph02][WARNIN] 2014-06-27 12:27:34.750281 7f123b33d7a0 -1
>> OSD::mkfs:
>> >>> > ObjectStore::mkfs failed with error -2
>> >>> > [ceph02][WARNIN] 2014-06-27 12:27:34.750416 7f123b33d7a0 -1 **
>> ERROR:
>> >>> > error
>> >>> > creating empty object store in /var/lib/ceph/tmp/mnt.HacFAP: (2) No
>> >>> > such
>> >>> > file or directory
>> >>> > [ceph02][WARNIN] ERROR:ceph-disk:Failed to activate
>> >>> >
>> >>> > Two of then following the same procedure after the same error
>> several
>> >>> > time,
>> >>> > raise up.
>> >>> > [ceph at ceph03 ~]$ df -h
>> >>> > Filesystem Size Used Avail Use% Mounted on
>> >>> > /dev/xvda1 5.0G 2.2G 2.6G 46% /
>> >>> > tmpfs 935M 0 935M 0% /dev/shm
>> >>> > /dev/xvdb1 90G 37M 90G 1% /var/lib/ceph/osd/ceph-6
>> >>> >
>> >>> >
>> >>> > Any idea please?
>> >>> >
>> >>> >
>> >>> > Bertrand Russell:
>> >>> > "El problema con el mundo es que los est?pidos est?n seguros de
>> todo y
>> >>> > los
>> >>> > inteligentes est?n llenos de dudas"
>> >>> >
>> >>> > _______________________________________________
>> >>> > ceph-users mailing list
>> >>> > ceph-users at lists.ceph.com
>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>> >
>> >>
>> >>
>> >
>>
>
>
>
> --
>
> ############################################################################
> Iban Cabrillo Bartolome
> Instituto de Fisica de Cantabria (IFCA)
> Santander, Spain
> Tel: +34942200969
> PGP PUBLIC KEY:
> http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC
>
> ############################################################################
> Bertrand Russell:
> *"El problema con el mundo es que los est?pidos est?n seguros de todo y
> los inteligentes est?n llenos de dudas*"
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140801/8ae13dfd/attachment.htm>
