On Tue, 10 Sep 2013, Pavel Timoschenkov wrote:
> OSD created only if I use single disk for data and journal.
>
> Situation with separate disks:
> 1.
> ceph-deploy disk zap ceph001:sdaa ceph001:sda1
> [ceph_deploy.osd][DEBUG ] zapping /dev/sdaa on ceph001
> [ceph_deploy.osd][DEBUG ] zapping /dev/sda1 on ceph001
> 2.
> Wiped file system on ceph001
> wipefs /dev/sdaa
> wipefs: WARNING: /dev/sdaa: appears to contain 'gpt' partition table
> wipefs /dev/sdaa1
> wipefs: error: /dev/sdaa1: probing initialization failed
I think this is still the problem. What happens if you do wipefs *before*
the zap? I wonder if the signature offsets are relative to sdaa1 and it
doesn't see them after the partition table is zeroed out by zap?
Thanks-
sage
> 3.
> ceph-deploy osd create ceph001:sdaa:/dev/sda1 [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph001:/dev/sdaa:/dev/sda1
> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph001
> [ceph_deploy.osd][DEBUG ] Host ceph001 is now ready for osd use.
> [ceph_deploy.osd][DEBUG ] Preparing host ceph001 disk /dev/sdaa journal /dev/sda1 activate True
> 4.
> ceph -k ceph.client.admin.keyring -s
> cluster d4d39e90-9610-41f3-be73-db361908b433
> health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds
> monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election epoch 2, quorum 0 ceph001
> osdmap e1: 0 osds: 0 up, 0 in
> pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
> mdsmap e1: 0/0/1 up
>
> With single disk:
> 1.
> ceph-deploy disk zap ceph001:sdaa
> [ceph_deploy.osd][DEBUG ] zapping /dev/sdaa on ceph001
> 2.
> ceph-deploy osd create ceph001:sdaa
> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph001:/dev/sdaa:
> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph001
> [ceph_deploy.osd][DEBUG ] Host ceph001 is now ready for osd use.
> [ceph_deploy.osd][DEBUG ] Preparing host ceph001 disk /dev/sdaa journal None activate True
> 3.
> ceph@ceph-admin:~$ ceph -k ceph.client.admin.keyring -s
> cluster d4d39e90-9610-41f3-be73-db361908b433
> health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean
> monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election epoch 2, quorum 0 ceph001
> osdmap e2: 1 osds: 0 up, 0 in
> pgmap v3: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
> mdsmap e1: 0/0/1 up
>
>
> -----Original Message-----
> From: Sage Weil [mailto:sage@xxxxxxxxxxx]
> Sent: Monday, September 09, 2013 7:09 PM
> To: Pavel Timoschenkov
> Cc: Alfredo Deza; ceph-users@xxxxxxxxxxxxxx
> Subject: RE: trouble with ceph-deploy
>
> If you manually use wipefs to clear out the fs signatures after you zap, does it work then?
>
> I've opened http://tracker.ceph.com/issues/6258 as I think that is the answer here, but if you could confirm that wipefs does in fact solve the problem, that would be helpful!
>
> Thanks-
> sage
>
>
> On Mon, 9 Sep 2013, Pavel Timoschenkov wrote:
>
> > for the experiment:
> >
> > - blank disk sdae for data
> >
> > blkid -p /dev/sdaf
> > /dev/sdaf: PTTYPE="gpt"
> >
> > - and sda4 partition for journal
> >
> > blkid -p /dev/sda4
> > /dev/sda4: PTTYPE="gpt" PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="Linux filesystem" PART_ENTRY_UUID="cdc46436-b6ed-40bb-adb4-63cf1c41cbe3" PART_ENTRY_TYPE="0fc63daf-8483-4772-8e79-3d69d8477de4" PART_ENTRY_NUMBER="4" PART_ENTRY_OFFSET="62916608" PART_ENTRY_SIZE="20971520" PART_ENTRY_DISK="8:0"
> >
> > - zapped disk
> >
> > ceph-deploy disk zap ceph001:sdaf ceph001:sda4 [ceph_deploy.osd][DEBUG
> > ] zapping /dev/sdaf on ceph001 [ceph_deploy.osd][DEBUG ] zapping
> > /dev/sda4 on ceph001
> >
> > - after this:
> >
> > ceph-deploy osd create ceph001:sdae:/dev/sda4 [ceph_deploy.osd][DEBUG
> > ] Preparing cluster ceph disks ceph001:/dev/sdaf:/dev/sda4
> > [ceph_deploy.osd][DEBUG ] Deploying osd to ceph001
> > [ceph_deploy.osd][DEBUG ] Host ceph001 is now ready for osd use.
> > [ceph_deploy.osd][DEBUG ] Preparing host ceph001 disk /dev/sdaf
> > journal /dev/sda4 activate True
> >
> >
> > - after this:
> >
> > blkid -p /dev/sdaf1
> > /dev/sdaf1: ambivalent result (probably more filesystems on the
> > device, use wipefs(8) to see more details)
> >
> > wipefs /dev/sdaf1
> > offset type
> > ----------------------------------------------------------------
> > 0x30000 zfs_member [raid]
> >
> > 0x0 xfs [filesystem]
> > UUID: aba50262-0427-4f8b-8eb9-513814af6b81
> >
> > - and OSD not created
> >
> > but if I'm using sungle disk for data and journal:
> >
> > ceph-deploy disk zap ceph001:sdaf
> > [ceph_deploy.osd][DEBUG ] zapping /dev/sdaf on ceph001
> >
> > ceph-deploy osd create ceph001:sdaf
> > [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph001:/dev/sdaf:
> > [ceph_deploy.osd][DEBUG ] Deploying osd to ceph001
> > [ceph_deploy.osd][DEBUG ] Host ceph001 is now ready for osd use.
> > [ceph_deploy.osd][DEBUG ] Preparing host ceph001 disk /dev/sdaf
> > journal None activate True
> >
> > OSD created!
> >
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@xxxxxxxxxxx]
> > Sent: Friday, September 06, 2013 6:41 PM
> > To: Pavel Timoschenkov
> > Cc: Alfredo Deza; ceph-users@xxxxxxxxxxxxxx
> > Subject: RE: trouble with ceph-deploy
> >
> > On Fri, 6 Sep 2013, Pavel Timoschenkov wrote:
> > > >>>Try
> > > >>>ceph-disk -v activate /dev/sdaa1
> > >
> > > ceph-disk -v activate /dev/sdaa1
> > > /dev/sdaa1: ambivalent result (probably more filesystems on the
> > > device, use wipefs(8) to see more details)
> >
> > Looks like thre are multiple fs signatures on that partition. See
> >
> > http://ozancaglayan.com/2013/01/29/multiple-filesystem-signatures-on-a
> > -partition/
> >
> > for how to clean that up. And please share the wipefs output that you see; it may be that we need to make the --zap-disk behavior also explicitly clear any signatures on the device.
> >
> > Thanks!
> > sage
> >
> >
> > > >>>as there is probably a partition there. And/or tell us what
> > > >>>/proc/partitions contains,
> > >
> > > cat /proc/partitions
> > > major minor #blocks name
> > > ....
> > > 65 160 2930266584 sdaa
> > > 65 161 2930265543 sdaa1
> > > ....
> > > >>>and/or what you get from
> > > >>>ceph-disk list
> > >
> > > ceph-disk list
> > > Traceback (most recent call last):
> > > File "/usr/sbin/ceph-disk", line 2328, in <module>
> > > main()
> > > File "/usr/sbin/ceph-disk", line 2317, in main
> > > args.func(args)
> > > File "/usr/sbin/ceph-disk", line 2001, in main_list
> > > tpath = mount(dev=dev, fstype=fs_type, options='')
> > > File "/usr/sbin/ceph-disk", line 678, in mount
> > > path,
> > > File "/usr/lib/python2.7/subprocess.py", line 506, in check_call
> > > retcode = call(*popenargs, **kwargs)
> > > File "/usr/lib/python2.7/subprocess.py", line 493, in call
> > > return Popen(*popenargs, **kwargs).wait()
> > > File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
> > > errread, errwrite)
> > > File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
> > > raise child_exception
> > > TypeError: execv() arg 2 must contain only strings
> > >
> > > ==================================================================
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sage@xxxxxxxxxxx]
> > > Sent: Thursday, September 05, 2013 6:37 PM
> > > To: Pavel Timoschenkov
> > > Cc: Alfredo Deza; ceph-users@xxxxxxxxxxxxxx
> > > Subject: RE: trouble with ceph-deploy
> > >
> > > On Thu, 5 Sep 2013, Pavel Timoschenkov wrote:
> > > > >>>What happens if you do
> > > > >>>ceph-disk -v activate /dev/sdaa1 on ceph001?
> > > >
> > > > Hi. My issue has not been solved. When i execute ceph-disk -v activate /dev/sdaa - all is ok:
> > > > ceph-disk -v activate /dev/sdaa
> > >
> > > Try
> > >
> > > ceph-disk -v activate /dev/sdaa1
> > >
> > > as there is probably a partition there. And/or tell us what
> > > /proc/partitions contains, and/or what you get from
> > >
> > > ceph-disk list
> > >
> > > Thanks!
> > > sage
> > >
> > >
> > > > DEBUG:ceph-disk:Mounting /dev/sdaa on /var/lib/ceph/tmp/mnt.yQuXIa
> > > > with options noatime
> > > > mount: Structure needs cleaning
> > > > but OSD not created all the same:
> > > > ceph -k ceph.client.admin.keyring -s
> > > > cluster 0a2e18d2-fd53-4f01-b63a-84851576c076
> > > > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds
> > > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election epoch 2, quorum 0 ceph001
> > > > osdmap e1: 0 osds: 0 up, 0 in
> > > > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
> > > > mdsmap e1: 0/0/1 up
> > > >
> > > > -----Original Message-----
> > > > From: Sage Weil [mailto:sage@xxxxxxxxxxx]
> > > > Sent: Friday, August 30, 2013 6:14 PM
> > > > To: Pavel Timoschenkov
> > > > Cc: Alfredo Deza; ceph-users@xxxxxxxxxxxxxx
> > > > Subject: Re: trouble with ceph-deploy
> > > >
> > > > On Fri, 30 Aug 2013, Pavel Timoschenkov wrote:
> > > >
> > > > >
> > > > > <<<<Can you share the output of the commands that do not work for you?
> > > > > How <<<<did `create` not work ? what did you see in the logs?
> > > > >
> > > > >
> > > > >
> > > > > In logs everything looks good. After
> > > > >
> > > > > ceph-deploy disk zap ceph001:sdaa ceph001:sda1
> > > > >
> > > > > and
> > > > >
> > > > > ceph-deploy osd create ceph001:sdaa:/dev/sda1
> > > > >
> > > > > where:
> > > > >
> > > > > HOST: ceph001
> > > > >
> > > > > DISK: sdaa
> > > > >
> > > > > JOURNAL: /dev/sda1
> > > > >
> > > > > in log:
> > > > >
> > > > > ==============================================
> > > > >
> > > > > cat ceph.log
> > > > >
> > > > > 2013-08-30 13:06:42,030 [ceph_deploy.osd][DEBUG ] Preparing
> > > > > cluster ceph disks ceph001:/dev/sdaa:/dev/sda1
> > > > >
> > > > > 2013-08-30 13:06:42,590 [ceph_deploy.osd][DEBUG ] Deploying osd
> > > > > to
> > > > > ceph001
> > > > >
> > > > > 2013-08-30 13:06:42,627 [ceph_deploy.osd][DEBUG ] Host ceph001
> > > > > is now ready for osd use.
> > > > >
> > > > > 2013-08-30 13:06:42,627 [ceph_deploy.osd][DEBUG ] Preparing host
> > > > > ceph001 disk /dev/sdaa journal /dev/sda1 activate True
> > > > >
> > > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >
> > > > > But:
> > > > >
> > > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >
> > > > > ceph -k ceph.client.admin.keyring -s
> > > > >
> > > > > cluster 0a2e18d2-fd53-4f01-b63a-84851576c076
> > > > >
> > > > > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck
> > > > > unclean; no osds
> > > > >
> > > > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election
> > > > > epoch 2, quorum 0 ceph001
> > > > >
> > > > > osdmap e1: 0 osds: 0 up, 0 in
> > > > >
> > > > > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0
> > > > > KB /
> > > > > 0 KB avail
> > > > >
> > > > > mdsmap e1: 0/0/1 up
> > > > >
> > > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >
> > > > > And
> > > > >
> > > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >
> > > > > ceph -k ceph.client.admin.keyring osd tree
> > > > >
> > > > > # id weight type name up/down reweight
> > > > >
> > > > > -1 0 root default
> > > > >
> > > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >
> > > > > OSD not created (
> > > >
> > > > What happens if you do
> > > >
> > > > ceph-disk -v activate /dev/sdaa1
> > > >
> > > > on ceph001?
> > > >
> > > > sage
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Alfredo Deza [mailto:alfredo.deza@xxxxxxxxxxx]
> > > > > Sent: Thursday, August 29, 2013 5:41 PM
> > > > > To: Pavel Timoschenkov
> > > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > > Subject: Re: trouble with ceph-deploy
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 29, 2013 at 10:23 AM, Pavel Timoschenkov
> > > > > <Pavel@xxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
> > > > >
> > > > > Hi.
> > > > >
> > > > > If I use the example of the doc:
> > > > >
> > > > > http://ceph.com/docs/master/rados/deployment/ceph-deploy-osd/#cr
> > > > > ea
> > > > > te
> > > > > -o
> > > > > sds
> > > > >
> > > > > ceph-deploy osd prepare ceph001:sdaa:/dev/sda1
> > > > > ceph-deploy osd activate ceph001:sdaa:/dev/sda1
> > > > > or
> > > > > ceph-deploy osd prepare ceph001:/dev/sdaa1:/dev/sda1
> > > > > ceph-deploy osd activate ceph001:/dev/sdaa:/dev/sda1
> > > > >
> > > > > or
> > > > >
> > > > > ceph-deploy osd create ceph001:sdaa:/dev/sda1
> > > > >
> > > > > OSD is not created. No errors, but when I execute
> > > > >
> > > > > ceph -k ceph.client.admin.keyring ?s
> > > > >
> > > > > I see the following:
> > > > >
> > > > > cluster 4b91a9e9-0e6c-4570-98c6-1398c6900a9e
> > > > > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck
> > > > > unclean; no osds
> > > > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0}, election
> > > > > epoch 2, quorum 0 ceph001
> > > > > osdmap e1: 0 osds: 0 up, 0 in
> > > > > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0
> > > > > KB /
> > > > > 0 KB avail
> > > > > mdsmap e1: 0/0/1 up
> > > > >
> > > > >
> > > > >
> > > > > 0 OSD.
> > > > >
> > > > >
> > > > >
> > > > > But if I use as an DISK argument to a local folder
> > > > > (/var/lib/ceph/osd/osd001) - it works, but only if used prepare
> > > > > + activate construction:
> > > > >
> > > > > ceph-deploy osd prepare
> > > > > ceph001:/var/lib/ceph/osd/osd001:/dev/sda1
> > > > > ceph-deploy osd activate
> > > > > ceph001:/var/lib/ceph/osd/osd001:/dev/sda1
> > > > >
> > > > > If I use CREATE, OSD is not created also.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Alfredo Deza [mailto:alfredo.deza@xxxxxxxxxxx]
> > > > > Sent: Thursday, August 29, 2013 4:36 PM
> > > > > To: Pavel Timoschenkov
> > > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > > Subject: Re: trouble with ceph-deploy
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 29, 2013 at 8:00 AM, Pavel Timoschenkov
> > > > > <Pavel@xxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
> > > > >
> > > > > Hi.
> > > > > New trouble with ceph-deploy. When i'm executing:
> > > > >
> > > > > ceph-deploy osd prepare ceph001:sdaa:/dev/sda1
> > > > > ceph-deploy osd activate ceph001:sdaa:/dev/sda1
> > > > > or
> > > > > ceph-deploy osd prepare ceph001:/dev/sdaa1:/dev/sda1
> > > > > ceph-deploy osd activate ceph001:/dev/sdaa:/dev/sda1
> > > > >
> > > > >
> > > > >
> > > > > Have you tried with
> > > > >
> > > > > ceph-deploy osd create ceph001:sdaa:/dev/sda1
> > > > >
> > > > > ?
> > > > >
> > > > > `create` should do `prepare` and `activate` for you. Also be
> > > > > mindful that the requirements for the arguments are that you
> > > > > need to pass something like:
> > > > >
> > > > > HOST:DISK[:JOURNAL]
> > > > >
> > > > > Where JOURNAL is completely optional, this is also detailed here:
> > > > > http://ceph.com/docs/master/rados/deployment/ceph-deploy-osd/#cr
> > > > > ea
> > > > > te
> > > > > -o
> > > > > sds
> > > > >
> > > > > Have you followed those instructions to deploy your OSDs ?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > OSD not created:
> > > > >
> > > > > ceph -k ceph.client.admin.keyring -s
> > > > > cluster 4b91a9e9-0e6c-4570-98c6-1398c6900a9e
> > > > > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck
> > > > > unclean; no osds
> > > > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0},
> > > > > election epoch 2, quorum 0 ceph001
> > > > > osdmap e1: 0 osds: 0 up, 0 in
> > > > > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB
> > > > > used, 0 KB / 0 KB avail
> > > > > mdsmap e1: 0/0/1 up
> > > > >
> > > > > ceph -k ceph.client.admin.keyring osd tree
> > > > > # id weight type name up/down reweight
> > > > > -1 0 root default
> > > > >
> > > > > but if i'm creating folder for ceph data and executing:
> > > > >
> > > > > ceph-deploy osd prepare
> > > > > ceph001:/var/lib/ceph/osd/osd001:/dev/sda1
> > > > > ceph-deploy osd activate
> > > > > ceph001:/var/lib/ceph/osd/osd001:/dev/sda1
> > > > >
> > > > > Those do not look right to me.
> > > > >
> > > > >
> > > > >
> > > > > OSD created:
> > > > >
> > > > > ceph -k ceph.client.admin.keyring -s
> > > > > cluster 4b91a9e9-0e6c-4570-98c6-1398c6900a9e
> > > > > health HEALTH_WARN 192 pgs stuck inactive; 192 pgs
> > > > > stuck unclean
> > > > > monmap e1: 1 mons at {ceph001=172.16.4.32:6789/0},
> > > > > election epoch 2, quorum 0 ceph001
> > > > > osdmap e5: 1 osds: 1 up, 1 in
> > > > > pgmap v6: 192 pgs: 192 creating; 0 bytes data, 0 KB
> > > > > used, 0 KB / 0 KB avail
> > > > > mdsmap e1: 0/0/1 up
> > > > >
> > > > > ceph -k ceph.client.admin.keyring osd
> > > > > tree
> > > > > # id weight type name up/down reweight
> > > > > -1 0.03999 root default
> > > > > -2 0.03999 host ceph001
> > > > > 0 0.03999 osd.0 up 1
> > > > >
> > > > > This is a bug or should I mount disks for data to some
> > > > > catalog?
> > > > >
> > > > >
> > > > > and more:
> > > > > The 'ceph-deploy osd create' construction don't work from
> > > > > me. Only 'prepare&activate'.
> > > > >
> > > > >
> > > > >
> > > > > When you say `create` didn't work for you, how so? What output
> > > > > did you see? Can you share some logs/output?
> > > > >
> > > > >
> > > > >
> > > > > Can you share the output of the commands that do not work for you?
> > > > > How did `create` not work ? what did you see in the logs?
> > > > >
> > > > >
> > > > > dpkg -s ceph-deploy
> > > > > Version: 1.2.1-1precise
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
> _______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
