Re: storage pools ceph (bobtail) auth failure in xenserver SR creation

[Sébastien RICCIO <sr@xxxxxxxxxxxxxxx>]

Hi,

thanks a lot for your answer. I was successfully able to create the 
storage pool with virsh.

However it is not listed when I issue a virsh pool-list:

Name                 State      Autostart
-----------------------------------------

If I try to add it again:
[root@xen-blade05 ~]# virsh pool-create ceph.xml
error: Failed to create pool from ceph.xml
error: operation failed: pool 'rbd' already exists with uuid 
34609207-3ea0-28c5-fee9-92e6b493e6cb

When i try to get info about it:

virsh # pool-info rbd
Name:           rbd
UUID:           34609207-3ea0-28c5-fee9-92e6b493e6cb
State:          inactive
Persistent:     yes
Autostart:      no

When I try to start it:
virsh# pool-start rbd
error: Failed to start pool rbd
error: An error occurred, but the cause is unknown

dumping the config

virsh # pool-dumpxml rbd
<pool type='rbd'>
  <name>rbd</name>
  <uuid>34609207-3ea0-28c5-fee9-92e6b493e6cb</uuid>
  <capacity unit='bytes'>0</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>0</available>
  <source>
    <host name='10.111.80.1' port='6789'/>
    <name>rbd</name>
    <auth username='admin' type='ceph'>
      <secret uuid='4cae707d-e049-4ec1-ac51-2cf999969dd3'/>
    </auth>
  </source>
</pool>

virsh # secret-dumpxml 4cae707d-e049-4ec1-ac51-2cf999969dd3
<secret ephemeral='no' private='no'>
  <uuid>4cae707d-e049-4ec1-ac51-2cf999969dd3</uuid>
  <usage type='ceph'>
    <name>client.admin</name>
  </usage>
</secret>

Ceph is working on the box

[root@xen-blade05 ~]# ceph status
  cluster f58c1405-8891-46cb-adf0-c6ab2da4050a
   health HEALTH_OK
   monmap e1: 3 mons at 
{ceph01=10.111.80.1:6789/0,ceph02=10.111.80.2:6789/0,ceph03=10.111.80.3:6789/0}, 
election epoch 18, quorum 0,1,2 ceph01,ceph02,ceph03
   osdmap e96: 6 osds: 6 up, 6 in
    pgmap v1317: 192 pgs: 192 active+clean; 17400 MB data, 41192 MB 
used, 5367 GB / 5407 GB avail
   mdsmap e1: 0/0/1 up

[root@xen-blade05 ~]# ceph osd lspools
0 data,1 metadata,2 rbd,

Any idea where to look now ? :)

Thanks for your help.

Cheers,
Sébastien

On 21.07.2013 11:42, Wido den Hollander wrote:
> Hi,
>
> On 07/21/2013 08:14 AM, Sébastien RICCIO wrote:
> > Hi !
> >
> > I'm currently trying to get the xenserver on centos 6.4 tech preview
> > working against a test ceph cluster and having the same issue.
> >
> > Some infos: the cluster is  named "ceph", the pool is named "rbd".
> >
> > ceph.xml:
> > <pool type='rbd'>
> >    <name>rbd</name>
> >    <source>
> >      <name>ceph</name>
> >      <host name='10.111.80.1' port='6789'/>
> >    </source>
> > </pool>
>
> You need a secret section inside the <source> like this:
>
> <pool type='rbd'>
>   <name>MyCephPool</name>
>   <source>
>     <host name='10.111.80.1' port='6789'/>
>     <name>rbd</name>
>     <auth username='admin' type='ceph'>
>       <secret uuid='4cae707d-e049-4ec1-ac51-2cf999969dd3'/>
>     </auth>
>   </source>
> </pool>
>
> The secret "uuid" should reference back to the UUID of the secret 
> you'll define.
>
> > secret.xml:
> > <secret ephemeral='no' private='no'>
> >    <usage type='ceph'>
> >      <name>client.admin <key_from_ceph_auth_list></name>
> >    </usage>
> > </secret>
>
>
> Don't put all that information in the <name> section, the XML should 
> be like this:
>
> <secret ephemeral='no' private='no'>
>   <uuid>4cae707d-e049-4ec1-ac51-2cf999969dd3</uuid>
>   <usage type='ceph'>
>     <name>client.admin</name>
>   </usage>
> </secret>
>
> > [root@xen-blade05 ~]# virsh pool-create ceph.xml
>
> This should be the procedure to follow:
>
> $ virsh secret-define secret.xml
> $ virsh secret-set-value <uuid> <cephx key>
> $ virsh pool-define ceph.xml
>
>
> > error: Failed to create pool from ceph.xml
> > error: Invalid secret: virSecretFree
>
> You get this error due to cephx being disabled since you didn't define 
> a <auth> section. This error is a bug in libvirt, but that will go 
> away as soon as you use cephx.
>
> Wido
>
> > same error :/
> >
> > Any ideas ?
> >
> > Don't hesitate to ask if you need more infos.
> >
> > Cheers,
> > Sébastien
> >
> > > Hi John,
> > >
> > > Could you try without the cat'ing and such?
> > >
> > > Could you also try this:
> > >
> > > $ virsh secret define secret.xml
> > > $ virsh secret-set-value <uuid> <secret>
> > > $ virsh pool-create ceph.xml
> > >
> > > Could you post both XML files and not use any Xen commands like 'xe'?
> > >
> > > I want to verify where this problem is.
> > >
> > > Wido
> > >
> > > On 07/11/2013 10:34 PM, John Shen wrote:
> > > >/  Wido, Thanks! I tried again with your command syntax but the 
> > > result is
> > > />/  the same.
> > > />/
> > > />/  [root at xen02 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> ~]# virsh 
> > > secret-set-value $(cat uuid) $(cat client.admin.key)
> > > />/  Secret value set
> > > />/
> > > />/  [root at xen02 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> ~]# xe 
> > > sr-create type=libvirt name-label=ceph
> > > />/  device-config:xml-filename=ceph.xml
> > > />/  Error code: libvirt
> > > />/  Error parameters: libvirt: VIR_ERR_65: VIR_FROM_30: Invalid 
> > > secret:
> > > />/  virSecretFree
> > > />/  [root at xen02 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> ~]#  virsh 
> > > pool-create ceph.xml
> > > />/  error: Failed to create pool from ceph.xml
> > > />/  error: Invalid secret: virSecretFree
> > > />/
> > > />/  [root at xen02 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> ~]#
> > > />/
> > > />/
> > > />/
> > > />/  On Thu, Jul 11, 2013 at 1:14 PM, Wido den Hollander <wido at 
> > > 42on.com <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> > > />/  <mailto:wido at 42on.com 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>> wrote:
> > > />/
> > > />/      Hi.
> > > />/
> > > />/      So, the problem here is a couple of things.
> > > />/
> > > />/      First: libvirt doesn't handle RBD storage pools without 
> > > auth. That's
> > > />/      my bad, but I never resolved that bug:
> > > />/      http://tracker.ceph.com/__issues/3493
> > > />/      <http://tracker.ceph.com/issues/3493>
> > > />/
> > > />/      For now, make sure cephx is enabled.
> > > />/
> > > />/      Also, the commands you are using don't seem to be right.
> > > />/
> > > />/      It should be:
> > > />/
> > > />/      $ virsh secret-set-value $(cat uuid) <base64 secret key>
> > > />/
> > > />/      Could you try again with cephx enabled and setting the 
> > > secret value
> > > />/      like mentioned above?
> > > />/
> > > />/      Wido
> > > />/
> > > />/
> > > />/      On 07/11/2013 06:00 PM, John Shen wrote:
> > > />/
> > > />/          Hi Dave, Thank you so much for getting back to me.
> > > />/
> > > />/          the command returns the same errors:
> > > />/
> > > />/          [root at xen02 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> ~]# virsh 
> > > pool-create ceph.xml
> > > />/          error: Failed to create pool from ceph.xml
> > > />/          error: Invalid secret: virSecretFree
> > > />/
> > > />/          [root at xen02 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> ~]#
> > > />/
> > > />/          the secret was precreated for the user admin that I use
> > > />/          elsewhere with
> > > />/          no issues (rbd mount, cephfs etc.), and per the ceph
> > > />/          documentation, i
> > > />/          just set the secret value with this command
> > > />/
> > > />/             virsh secret-set-value $(cat uuid) --base64 $(cat
> > > />/          client.admin.key)
> > > />/
> > > />/          where the key is obtained from
> > > />/
> > > />/             ceph auth list
> > > />/
> > > />/          and uuid is generated by
> > > />/
> > > />/          virsh secret-define --file secret.xml
> > > />/
> > > />/          # cat secret.xml
> > > />/          <secret ephemeral='no' private='no'>
> > > />/                    <usage type='ceph'>
> > > />/                            <name>client.admin $(cat 
> > > client.admin.key)</name>
> > > />/                    </usage>
> > > />/          </secret>
> > > />/
> > > />/
> > > />/
> > > />/          On Thu, Jul 11, 2013 at 7:22 AM, Dave Scott
> > > />/          <Dave.Scott at eu.citrix.com 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
> > > <mailto:Dave.Scott at eu.citrix.com 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>
> > > />/          <mailto:Dave.Scott at eu.citrix.__com 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> > > />/          <mailto:Dave.Scott at eu.citrix.com 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>>> wrote:
> > > />/
> > > />/               [sorry I didn't manage to reply to the original 
> > > message; I
> > > />/          only just
> > > />/               joined this list.
> > > />/               Sorry if this breaks your threading!]
> > > />/
> > > />/               On 10 Jul 2013 at 16:01 John Shen wrote:
> > > />/
> > > />/                > I was following the tech preview of libvirt/ceph
> > > />/          integration in
> > > />/               xenserver, but ran
> > > />/                > into an issue with ceph auth in setting up the 
> > > SR. any
> > > />/          help would
> > > />/               be greatly
> > > />/                > appreciated.
> > > />/
> > > />/               I must confess that I've disabled auth in my test 
> > > environment.
> > > />/               Clearly I should
> > > />/               go back and enable it again :-)
> > > />/
> > > />/                > uuid was generated per:
> > > />/ http://eu.ceph.com/docs/wip-__dump/rbd/libvirt/
> > > />/ <http://eu.ceph.com/docs/wip-dump/rbd/libvirt/>
> > > />/                >
> > > />/                > according to inktank, storage pool auth syntax 
> > > differs
> > > />/          slightly
> > > />/               from block device
> > > />/                > attachment. I tried both format but got the 
> > > same error.
> > > />/                >
> > > />/                > Ref:
> > > />/                >
> > > />/                >
> > > />/ 
> > > http://xenserver.org/blog/__entry/tech-preview-of-__xenserver-libvirt-ceph.html
> > > />/ 
> > > <http://xenserver.org/blog/entry/tech-preview-of-xenserver-libvirt-ceph.html>
> > > />/                >
> > > />/                > [root <at> xen01 ~]# xe sr-create type=libvirt
> > > />/          name-label=ceph
> > > />/               device-config:xml-filename=__ceph.xml
> > > />/                > Error code: libvirt
> > > />/                > Error parameters: libvirt: VIR_ERR_65: 
> > > VIR_FROM_30: Invalid
> > > />/               secret: virSecretFree
> > > />/
> > > />/               The "xe sr-create" call is handled by "xapi" which 
> > > calls
> > > />/               "xapi-libvirt-storage" which
> > > />/               uses the libvirt API directly to create the pool. It
> > > />/          _should_ do the
> > > />/               same as running
> > > />/
> > > />/               virsh pool-create ceph.xml
> > > />/
> > > />/               Could you try the "virsh pool-create" and see if that
> > > />/          works? If it
> > > />/               does, then we need
> > > />/               to figure out what the "virsh" CLI is doing that 
> > > my Pool.create
> > > />/               function call isn't. If
> > > />/               it doesn't then there might be some other missing 
> > > step. Did
> > > />/          you have
> > > />/               to pre-create
> > > />/               a secret (is that "virsh secret-create"?)
> > > />/
> > > />/               Cheers,
> > > />/               Dave Scott
> > > />/
> > > />/
> > > />/
> > > />/
> > > />/          --
> > > />/          --John Shen
> > > />/
> > > />/
> > > />/ _________________________________________________
> > > />/          ceph-users mailing list
> > > />/          ceph-users at lists.ceph.com 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
> > > <mailto:ceph-users at lists.ceph.com 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>
> > > />/ http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
> > > />/ <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> > > />/
> > > />/
> > > />/
> > > />/      --
> > > />/      Wido den Hollander
> > > />/      42on B.V.
> > > />/
> > > />/      Phone: +31 (0)20 700 9902<tel:%2B31%20%280%2920%20700%209902>
> > > />/      Skype: contact42on
> > > />/      _________________________________________________
> > > />/      ceph-users mailing list
> > > />/      ceph-users at lists.ceph.com 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
> > > <mailto:ceph-users at lists.ceph.com 
> > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>
> > > />/ http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
> > > />/ <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> > > />/
> > > />/
> > > />/
> > > />/
> > > />/  --
> > > />/  --John Shen
> > > /
> > >
> > > --
> > > Wido den Hollander
> > > 42on B.V.
> > >
> > > Phone: +31 (0)20 700 9902
> > > Skype: contact42on
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com