OSD created only if I use single disk for data and journal.
Situation with separate disks:
1.
ceph-deploy disk zap ceph001:sdaa ceph001:sda1 [ceph_deploy.osd][DEBUG ] zapping /dev/sdaa on ceph001 [ceph_deploy.osd][DEBUG ] zapping /dev/sda1 on ceph001
2.
Wiped file system on ceph001
wipefs /dev/sdaa
wipefs: WARNING: /dev/sdaa: appears to contain 'gpt' partition table wipefs /dev/sdaa1
wipefs: error: /dev/sdaa1: probing initialization failed
3.
ceph-deploy osd create ceph001:sdaa:/dev/sda1 [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph001:/dev/sdaa:/dev/sda1
[ceph_deploy.osd][DEBUG ] Deploying osd to ceph001 [ceph_deploy.osd][DEBUG ] Host ceph001 is now ready for osd use.
[ceph_deploy.osd][DEBUG ] Preparing host ceph001 disk /dev/sdaa journal /dev/sda1 activate True
4.
ceph -k ceph.client.admin.keyring -s
cluster d4d39e90-9610-41f3-be73-db361908b433
health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds
monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election epoch 2, quorum 0 ceph001
osdmap e1: 0 osds: 0 up, 0 in
pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
mdsmap e1: 0/0/1 up
With single disk:
1.
ceph-deploy disk zap ceph001:sdaa
[ceph_deploy.osd][DEBUG ] zapping /dev/sdaa on ceph001
2.
ceph-deploy osd create ceph001:sdaa
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph001:/dev/sdaa:
[ceph_deploy.osd][DEBUG ] Deploying osd to ceph001 [ceph_deploy.osd][DEBUG ] Host ceph001 is now ready for osd use.
[ceph_deploy.osd][DEBUG ] Preparing host ceph001 disk /dev/sdaa journal None activate True
3.
ceph@ceph-admin:~$ ceph -k ceph.client.admin.keyring -s
cluster d4d39e90-9610-41f3-be73-db361908b433
health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean
monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election epoch 2, quorum 0 ceph001
osdmap e2: 1 osds: 0 up, 0 in
pgmap v3: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
mdsmap e1: 0/0/1 up
-----Original Message-----
From: Sage Weil [mailto:sage@xxxxxxxxxxx]
Sent: Monday, September 09, 2013 7:09 PM
To: Pavel Timoschenkov
Cc: Alfredo Deza; ceph-users@xxxxxxxxxxxxxx
Subject: RE: trouble with ceph-deploy
If you manually use wipefs to clear out the fs signatures after you zap, does it work then?
I've opened http://tracker.ceph.com/issues/6258 as I think that is the answer here, but if you could confirm that wipefs does in fact solve the problem, that would be helpful!
Thanks-
sage
On Mon, 9 Sep 2013, Pavel Timoschenkov wrote:
> for the experiment:
>
> - blank disk sdae for data
>
> blkid -p /dev/sdaf
> /dev/sdaf: PTTYPE="gpt"
>
> - and sda4 partition for journal
>
> blkid -p /dev/sda4
> /dev/sda4: PTTYPE="gpt" PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="Linux filesystem" PART_ENTRY_UUID="cdc46436-b6ed-40bb-adb4-63cf1c41cbe3" PART_ENTRY_TYPE="0fc63daf-8483-4772-8e79-3d69d8477de4" PART_ENTRY_NUMBER="4" PART_ENTRY_OFFSET="62916608" PART_ENTRY_SIZE="20971520" PART_ENTRY_DISK="8:0"
>
> - zapped disk
>
> ceph-deploy disk zap ceph001:sdaf ceph001:sda4 [ceph_deploy.osd][DEBUG
> ] zapping /dev/sdaf on ceph001 [ceph_deploy.osd][DEBUG ] zapping
> /dev/sda4 on ceph001
>
> - after this:
>
> ceph-deploy osd create ceph001:sdae:/dev/sda4 [ceph_deploy.osd][DEBUG
> ] Preparing cluster ceph disks ceph001:/dev/sdaf:/dev/sda4
> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph001
> [ceph_deploy.osd][DEBUG ] Host ceph001 is now ready for osd use.
> [ceph_deploy.osd][DEBUG ] Preparing host ceph001 disk /dev/sdaf
> journal /dev/sda4 activate True
>
>
> - after this:
>
> blkid -p /dev/sdaf1
> /dev/sdaf1: ambivalent result (probably more filesystems on the
> device, use wipefs(8) to see more details)
>
> wipefs /dev/sdaf1
> offset type
> ----------------------------------------------------------------
> 0x30000 zfs_member [raid]
>
> 0x0 xfs [filesystem]
> UUID: aba50262-0427-4f8b-8eb9-513814af6b81
>
> - and OSD not created
>
> but if I'm using sungle disk for data and journal:
>
> ceph-deploy disk zap ceph001:sdaf
> [ceph_deploy.osd][DEBUG ] zapping /dev/sdaf on ceph001
>
> ceph-deploy osd create ceph001:sdaf
> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph001:/dev/sdaf:
> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph001
> [ceph_deploy.osd][DEBUG ] Host ceph001 is now ready for osd use.
> [ceph_deploy.osd][DEBUG ] Preparing host ceph001 disk /dev/sdaf
> journal None activate True
>
> OSD created!
>
> -----Original Message-----
> From: Sage Weil [mailto:sage@xxxxxxxxxxx]
> Sent: Friday, September 06, 2013 6:41 PM
> To: Pavel Timoschenkov
> Cc: Alfredo Deza; ceph-users@xxxxxxxxxxxxxx
> Subject: RE: trouble with ceph-deploy
>
> On Fri, 6 Sep 2013, Pavel Timoschenkov wrote:
> > >>>Try
> > >>>ceph-disk -v activate /dev/sdaa1
> >
> > ceph-disk -v activate /dev/sdaa1
> > /dev/sdaa1: ambivalent result (probably more filesystems on the
> > device, use wipefs(8) to see more details)
>
> Looks like thre are multiple fs signatures on that partition. See
>
> http://ozancaglayan.com/2013/01/29/multiple-filesystem-signatures-on-a
> -partition/
>
> for how to clean that up. And please share the wipefs output that you see; it may be that we need to make the --zap-disk behavior also explicitly clear any signatures on the device.
>
> Thanks!
> sage
>
>
> > >>>as there is probably a partition there. And/or tell us what
> > >>>/proc/partitions contains,
> >
> > cat /proc/partitions
> > major minor #blocks name
> > ....
> > 65 160 2930266584 sdaa
> > 65 161 2930265543 sdaa1
> > ....
> > >>>and/or what you get from
> > >>>ceph-disk list
> >
> > ceph-disk list
> > Traceback (most recent call last):
> > File "/usr/sbin/ceph-disk", line 2328, in <module>
> > main()
> > File "/usr/sbin/ceph-disk", line 2317, in main
> > args.func(args)
> > File "/usr/sbin/ceph-disk", line 2001, in main_list
> > tpath = mount(dev=dev, fstype=fs_type, options='')
> > File "/usr/sbin/ceph-disk", line 678, in mount
> > path,
> > File "/usr/lib/python2.7/subprocess.py", line 506, in check_call
> > retcode = call(*popenargs, **kwargs)
> > File "/usr/lib/python2.7/subprocess.py", line 493, in call
> > return Popen(*popenargs, **kwargs).wait()
> > File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
> > errread, errwrite)
> > File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
> > raise child_exception
> > TypeError: execv() arg 2 must contain only strings
> >
> > ==================================================================
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@xxxxxxxxxxx]
> > Sent: Thursday, September 05, 2013 6:37 PM
> > To: Pavel Timoschenkov
> > Cc: Alfredo Deza; ceph-users@xxxxxxxxxxxxxx
> > Subject: RE: trouble with ceph-deploy
> >
> > On Thu, 5 Sep 2013, Pavel Timoschenkov wrote:
> > > >>>What happens if you do
> > > >>>ceph-disk -v activate /dev/sdaa1 on ceph001?
> > >
> > > Hi. My issue has not been solved. When i execute ceph-disk -v activate /dev/sdaa - all is ok:
> > > ceph-disk -v activate /dev/sdaa
> >
> > Try
> >
> > ceph-disk -v activate /dev/sdaa1
> >
> > as there is probably a partition there. And/or tell us what
> > /proc/partitions contains, and/or what you get from
> >
> > ceph-disk list
> >
> > Thanks!
> > sage
> >
> >
> > > DEBUG:ceph-disk:Mounting /dev/sdaa on /var/lib/ceph/tmp/mnt.yQuXIa
> > > with options noatime
> > > mount: Structure needs cleaning
> > > but OSD not created all the same:
> > > ceph -k ceph.client.admin.keyring -s
> > > cluster 0a2e18d2-fd53-4f01-b63a-84851576c076
> > > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds
> > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election epoch 2, quorum 0 ceph001
> > > osdmap e1: 0 osds: 0 up, 0 in
> > > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
> > > mdsmap e1: 0/0/1 up
> > >
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sage@xxxxxxxxxxx]
> > > Sent: Friday, August 30, 2013 6:14 PM
> > > To: Pavel Timoschenkov
> > > Cc: Alfredo Deza; ceph-users@xxxxxxxxxxxxxx
> > > Subject: Re: trouble with ceph-deploy
> > >
> > > On Fri, 30 Aug 2013, Pavel Timoschenkov wrote:
> > >
> > > >
> > > > <<<<Can you share the output of the commands that do not work for you?
> > > > How <<<<did `create` not work ? what did you see in the logs?
> > > >
> > > >
> > > >
> > > > In logs everything looks good. After
> > > >
> > > > ceph-deploy disk zap ceph001:sdaa ceph001:sda1
> > > >
> > > > and
> > > >
> > > > ceph-deploy osd create ceph001:sdaa:/dev/sda1
> > > >
> > > > where:
> > > >
> > > > HOST: ceph001
> > > >
> > > > DISK: sdaa
> > > >
> > > > JOURNAL: /dev/sda1
> > > >
> > > > in log:
> > > >
> > > > ==============================================
> > > >
> > > > cat ceph.log
> > > >
> > > > 2013-08-30 13:06:42,030 [ceph_deploy.osd][DEBUG ] Preparing
> > > > cluster ceph disks ceph001:/dev/sdaa:/dev/sda1
> > > >
> > > > 2013-08-30 13:06:42,590 [ceph_deploy.osd][DEBUG ] Deploying osd
> > > > to
> > > > ceph001
> > > >
> > > > 2013-08-30 13:06:42,627 [ceph_deploy.osd][DEBUG ] Host ceph001
> > > > is now ready for osd use.
> > > >
> > > > 2013-08-30 13:06:42,627 [ceph_deploy.osd][DEBUG ] Preparing host
> > > > ceph001 disk /dev/sdaa journal /dev/sda1 activate True
> > > >
> > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > >
> > > > But:
> > > >
> > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > >
> > > > ceph -k ceph.client.admin.keyring -s
> > > >
> > > > cluster 0a2e18d2-fd53-4f01-b63a-84851576c076
> > > >
> > > > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck
> > > > unclean; no osds
> > > >
> > > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election
> > > > epoch 2, quorum 0 ceph001
> > > >
> > > > osdmap e1: 0 osds: 0 up, 0 in
> > > >
> > > > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0
> > > > KB /
> > > > 0 KB avail
> > > >
> > > > mdsmap e1: 0/0/1 up
> > > >
> > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > >
> > > > And
> > > >
> > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > >
> > > > ceph -k ceph.client.admin.keyring osd tree
> > > >
> > > > # id weight type name up/down reweight
> > > >
> > > > -1 0 root default
> > > >
> > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > >
> > > > OSD not created (
> > >
> > > What happens if you do
> > >
> > > ceph-disk -v activate /dev/sdaa1
> > >
> > > on ceph001?
> > >
> > > sage
> > >
> > >
> > > >
> > > >
> > > >
> > > > From: Alfredo Deza [mailto:alfredo.deza@xxxxxxxxxxx]
> > > > Sent: Thursday, August 29, 2013 5:41 PM
> > > > To: Pavel Timoschenkov
> > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > Subject: Re: trouble with ceph-deploy
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 29, 2013 at 10:23 AM, Pavel Timoschenkov
> > > > <Pavel@xxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
> > > >
> > > > Hi.
> > > >
> > > > If I use the example of the doc:
> > > >
> > > > http://ceph.com/docs/master/rados/deployment/ceph-deploy-osd/#cr
> > > > ea
> > > > te
> > > > -o
> > > > sds
> > > >
> > > > ceph-deploy osd prepare ceph001:sdaa:/dev/sda1
> > > > ceph-deploy osd activate ceph001:sdaa:/dev/sda1
> > > > or
> > > > ceph-deploy osd prepare ceph001:/dev/sdaa1:/dev/sda1
> > > > ceph-deploy osd activate ceph001:/dev/sdaa:/dev/sda1
> > > >
> > > > or
> > > >
> > > > ceph-deploy osd create ceph001:sdaa:/dev/sda1
> > > >
> > > > OSD is not created. No errors, but when I execute
> > > >
> > > > ceph -k ceph.client.admin.keyring ?s
> > > >
> > > > I see the following:
> > > >
> > > > cluster 4b91a9e9-0e6c-4570-98c6-1398c6900a9e
> > > > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck
> > > > unclean; no osds
> > > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election
> > > > epoch 2, quorum 0 ceph001
> > > > osdmap e1: 0 osds: 0 up, 0 in
> > > > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0
> > > > KB /
> > > > 0 KB avail
> > > > mdsmap e1: 0/0/1 up
> > > >
> > > >
> > > >
> > > > 0 OSD.
> > > >
> > > >
> > > >
> > > > But if I use as an DISK argument to a local folder
> > > > (/var/lib/ceph/osd/osd001) - it works, but only if used prepare
> > > > + activate construction:
> > > >
> > > > ceph-deploy osd prepare
> > > > ceph001:/var/lib/ceph/osd/osd001:/dev/sda1
> > > > ceph-deploy osd activate
> > > > ceph001:/var/lib/ceph/osd/osd001:/dev/sda1
> > > >
> > > > If I use CREATE, OSD is not created also.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Alfredo Deza [mailto:alfredo.deza@xxxxxxxxxxx]
> > > > Sent: Thursday, August 29, 2013 4:36 PM
> > > > To: Pavel Timoschenkov
> > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > Subject: Re: trouble with ceph-deploy
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 29, 2013 at 8:00 AM, Pavel Timoschenkov
> > > > <Pavel@xxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
> > > >
> > > > Hi.
> > > > New trouble with ceph-deploy. When i'm executing:
> > > >
> > > > ceph-deploy osd prepare ceph001:sdaa:/dev/sda1
> > > > ceph-deploy osd activate ceph001:sdaa:/dev/sda1
> > > > or
> > > > ceph-deploy osd prepare ceph001:/dev/sdaa1:/dev/sda1
> > > > ceph-deploy osd activate ceph001:/dev/sdaa:/dev/sda1
> > > >
> > > >
> > > >
> > > > Have you tried with
> > > >
> > > > ceph-deploy osd create ceph001:sdaa:/dev/sda1
> > > >
> > > > ?
> > > >
> > > > `create` should do `prepare` and `activate` for you. Also be
> > > > mindful that the requirements for the arguments are that you
> > > > need to pass something like:
> > > >
> > > > HOST:DISK[:JOURNAL]
> > > >
> > > > Where JOURNAL is completely optional, this is also detailed here:
> > > > http://ceph.com/docs/master/rados/deployment/ceph-deploy-osd/#cr
> > > > ea
> > > > te
> > > > -o
> > > > sds
> > > >
> > > > Have you followed those instructions to deploy your OSDs ?
> > > >
> > > >
> > > >
> > > >
> > > > OSD not created:
> > > >
> > > > ceph -k ceph.client.admin.keyring -s
> > > > cluster 4b91a9e9-0e6c-4570-98c6-1398c6900a9e
> > > > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck
> > > > unclean; no osds
> > > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0},
> > > > election epoch 2, quorum 0 ceph001
> > > > osdmap e1: 0 osds: 0 up, 0 in
> > > > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB
> > > > used, 0 KB / 0 KB avail
> > > > mdsmap e1: 0/0/1 up
> > > >
> > > > ceph -k ceph.client.admin.keyring osd tree
> > > > # id weight type name up/down reweight
> > > > -1 0 root default
> > > >
> > > > but if i'm creating folder for ceph data and executing:
> > > >
> > > > ceph-deploy osd prepare
> > > > ceph001:/var/lib/ceph/osd/osd001:/dev/sda1
> > > > ceph-deploy osd activate
> > > > ceph001:/var/lib/ceph/osd/osd001:/dev/sda1
> > > >
> > > > Those do not look right to me.
> > > >
> > > >
> > > >
> > > > OSD created:
> > > >
> > > > ceph -k ceph.client.admin.keyring -s
> > > > cluster 4b91a9e9-0e6c-4570-98c6-1398c6900a9e
> > > > health HEALTH_WARN 192 pgs stuck inactive; 192 pgs
> > > > stuck unclean
> > > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0},
> > > > election epoch 2, quorum 0 ceph001
> > > > osdmap e5: 1 osds: 1 up, 1 in
> > > > pgmap v6: 192 pgs: 192 creating; 0 bytes data, 0 KB
> > > > used, 0 KB / 0 KB avail
> > > > mdsmap e1: 0/0/1 up
> > > >
> > > > ceph -k ceph.client.admin.keyring osd
> > > > tree
> > > > # id weight type name up/down reweight
> > > > -1 0.03999 root default
> > > > -2 0.03999 host ceph001
> > > > 0 0.03999 osd.0 up 1
> > > >
> > > > This is a bug or should I mount disks for data to some
> > > > catalog?
> > > >
> > > >
> > > > and more:
> > > > The 'ceph-deploy osd create' construction don't work from
> > > > me. Only 'prepare&activate'.
> > > >
> > > >
> > > >
> > > > When you say `create` didn't work for you, how so? What output
> > > > did you see? Can you share some logs/output?
> > > >
> > > >
> > > >
> > > > Can you share the output of the commands that do not work for you?
> > > > How did `create` not work ? what did you see in the logs?
> > > >
> > > >
> > > > dpkg -s ceph-deploy
> > > > Version: 1.2.1-1precise
> > > >
> > > >
> > > > _______________________________________________
> > > > ceph-users mailing list
> > > > ceph-users@xxxxxxxxxxxxxx
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
