I believe I found my problem, naturally it's user error. I used 'hostname' rather than 'host' in the daemon sections. I'll retry later.
-Steve
----- Original Message -----
> From: "Steve Carter" <svenphx@xxxxxxxxx>
> To: ceph-users@xxxxxxxxxxxxxx
> Sent: Monday, March 25, 2013 10:38:13 PM
> Subject: Re: Weird problem with mkcephfs
>
> Sage,
>
> Sure, here you go:
>
> [global]
> auth cluster required = cephx
> auth service required = cephx
> auth client required = cephx
> max open files = 4096
>
> [mon]
> mon data = /data/${name}
> keyring = /data/${name}/keyring
>
> [osd]
> osd data = /data/${name}
> keyring = /data/${name}/keyring
> btrfs devs = /dev/disk/by-label/${name}-data
> osd journal = /dev/sda_vg/${name}-journal
>
> [mon.a]
> hostname = smon
> mon addr = 192.168.0.253:6789
>
> [osd.0]
> hostname = s1
>
> [osd.1]
> hostname = s1
>
> [osd.2]
> hostname = s1
>
> [osd.3]
> hostname = s1
>
> [osd.4]
> hostname = s1
>
> [osd.5]
> hostname = s1
>
> [osd.6]
> hostname = s2
>
> [osd.7]
> hostname = s2
>
> [osd.8]
> hostname = s2
>
> [osd.9]
> hostname = s2
>
> [osd.10]
> hostname = s2
>
> [osd.11]
> hostname = s2
>
>
> ----- Original Message -----
> > From: "Sage Weil" <sage@xxxxxxxxxxx>
> > To: "Steve Carter" <svenphx@xxxxxxxxx>
> > Cc: ceph-users@xxxxxxxxxxxxxx
> > Sent: Monday, March 25, 2013 8:26:54 AM
> > Subject: Re: Weird problem with mkcephfs
> >
> > They keyring.* vs key.* distinction in mkcephfs appears correct. Can you
> > attach your ceph.conf? It looks a bit like no daemons are defined.
> >
> > sage
> >
> >
> > On Mon, 25 Mar 2013, Steve Carter wrote:
> >
> > > Although it doesn't attempt to login to my other machines as I
> thought it
> > > was
> > > designed to do, as I know it did the last time I built a cluster.
> Not sure
> > > what I'm doing wrong.
> > >
> > > -Steve
> > >
> > > On 03/23/2013 10:35 PM, Steve Carter wrote:
> > > > I changed:
> > > >
> > > > for k in $dir/key.*
> > > >
> > > > to:
> > > >
> > > > for k in $dir/key*
> > > >
> > > > and it appeared to run correctly:
> > > >
> > > > root@smon:/etc/ceph# mkcephfs -a -c /etc/ceph/ceph.conf -d /tmp -k
> > > > /etc/ceph/keyring
> > > > preparing monmap in /tmp/monmap
> > > > /usr/bin/monmaptool --create --clobber --add a 192.168.0.253:6789
> --print
> > > > /tmp/monmap
> > > > /usr/bin/monmaptool: monmap file /tmp/monmap
> > > > /usr/bin/monmaptool: generated fsid
> 46e4ae99-3df6-41ae-8d45-474c95b98852
> > > > epoch 0
> > > > fsid 46e4ae99-3df6-41ae-8d45-474c95b98852
> > > > last_changed 2013-03-23 22:33:26.254974
> > > > created 2013-03-23 22:33:26.254974
> > > > 0: 192.168.0.253:6789/0 mon.a
> > > > /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
> > > > Building generic osdmap from /tmp/conf
> > > > /usr/bin/osdmaptool: osdmap file '/tmp/osdmap'
> > > > /usr/bin/osdmaptool: writing epoch 1 to /tmp/osdmap
> > > > Generating admin key at /tmp/keyring.admin
> > > > creating /tmp/keyring.admin
> > > > Building initial monitor keyring
> > > > placing client.admin keyring in /etc/ceph/keyring
> > > >
> > > > On 03/23/2013 10:29 PM, Steve Carter wrote:
> > > > > The below part of the mkcephfs code seems responsible for this.
> > > > > specifically the 'for' loop below. I wonder if I installed
> from the
> > > > > wrong
> > > > > place? I installed from the ubuntu source rather than the ceph
> source.
> > > > >
> > > > > # admin keyring
> > > > > echo Generating admin key at $dir/keyring.admin
> > > > > $BINDIR/ceph-authtool --create-keyring --gen-key -n
> client.admin
> > > > > $dir/keyring.admin
> > > > >
> > > > > # mon keyring
> > > > > echo Building initial monitor keyring
> > > > > cp $dir/keyring.admin $dir/keyring.mon
> > > > > $BINDIR/ceph-authtool -n client.admin --set-uid=0 \
> > > > > --cap mon 'allow *' \
> > > > > --cap osd 'allow *' \
> > > > > --cap mds 'allow' \
> > > > > $dir/keyring.mon
> > > > >
> > > > > $BINDIR/ceph-authtool --gen-key -n mon. $dir/keyring.mon
> > > > >
> > > > > for k in $dir/key.*
> > > > > do
> > > > > kname=`echo $k | sed 's/.*key\.//'`
> > > > > ktype=`echo $kname | cut -c 1-3`
> > > > > kid=`echo $kname | cut -c 4- | sed 's/^\\.//'`
> > > > > kname="$ktype.$kid"
> > > > > secret=`cat $k`
> > > > > if [ "$ktype" = "osd" ]; then
> > > > > $BINDIR/ceph-authtool -n $kname --add-key $secret
> > > > > $dir/keyring.mon \
> > > > > --cap mon 'allow rwx' \
> > > > > --cap osd 'allow *'
> > > > > fi
> > > > > if [ "$ktype" = "mds" ]; then
> > > > > $BINDIR/ceph-authtool -n $kname --add-key $secret
> > > > > $dir/keyring.mon \
> > > > > --cap mon "allow rwx" \
> > > > > --cap osd 'allow *' \
> > > > > --cap mds 'allow'
> > > > > fi
> > > > > done
> > > > >
> > > > > exit 0
> > > > > fi
> > > > >
> > > > >
> > > > > On 03/23/2013 01:50 PM, Steve Carter wrote:
> > > > > > This is consistently repeatable on my system. This is the
> latest of
> > > > > > two
> > > > > > cluster builds I have done. This is a brand new deployment on
> > > > > > hardware I
> > > > > > haven't deployed on previously.
> > > > > >
> > > > > > You see the error below is referencing /tmp/key.* and the keyring
> > > > > > files
> > > > > > are actually keyring.*.
> > > > > >
> > > > > > Any help is much appreciated.
> > > > > >
> > > > > > root@mon:~# uname -a
> > > > > > Linux mon.XXXXX.com 3.2.0-39-generic #62-Ubuntu SMP Thu Feb 28
> > > > > > 00:28:53
> > > > > > UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> > > > > > root@mon:~# ceph -v
> > > > > > ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
> > > > > > root@mon:~# ls -al /tmp/
> > > > > > total 8
> > > > > > drwxrwxrwx 2 root root 4096 Mar 23 12:15 .
> > > > > > drwxr-xr-x 25 root root 4096 Mar 22 23:18 ..
> > > > > > root@mon:~# ls -al / | grep tmp
> > > > > > drwxrwxrwx 2 root root 4096 Mar 23 12:15 tmp
> > > > > > root@mon:~# mkcephfs -d /tmp -a -c /etc/ceph/ceph.conf -k
> > > > > > /etc/ceph/keyring
> > > > > > preparing monmap in /tmp/monmap
> > > > > > /usr/bin/monmaptool --create --clobber --add a 192.168.0.253:6789
> > > > > > --print /tmp/monmap
> > > > > > /usr/bin/monmaptool: monmap file /tmp/monmap
> > > > > > /usr/bin/monmaptool: generated fsid
> > > > > > 68b9c724-21c0-4d54-8237-674ced7adbfe
> > > > > > epoch 0
> > > > > > fsid 68b9c724-21c0-4d54-8237-674ced7adbfe
> > > > > > last_changed 2013-03-23 12:17:03.087018
> > > > > > created 2013-03-23 12:17:03.087018
> > > > > > 0: 192.168.0.253:6789/0 mon.a
> > > > > > /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
> > > > > > Building generic osdmap from /tmp/conf
> > > > > > /usr/bin/osdmaptool: osdmap file '/tmp/osdmap'
> > > > > > /usr/bin/osdmaptool: writing epoch 1 to /tmp/osdmap
> > > > > > Generating admin key at /tmp/keyring.admin
> > > > > > creating /tmp/keyring.admin
> > > > > > Building initial monitor keyring
> > > > > > cat: /tmp/key.*: No such file or directory
> > > > > > root@mon:~# ls -al /tmp/
> > > > > > total 32
> > > > > > drwxrwxrwx 2 root root 4096 Mar 23 12:17 .
> > > > > > drwxr-xr-x 25 root root 4096 Mar 22 23:18 ..
> > > > > > -rw-r--r-- 1 root root 695 Mar 23 12:17 conf
> > > > > > -rw------- 1 root root 63 Mar 23 12:17 keyring.admin
> > > > > > -rw------- 1 root root 192 Mar 23 12:17 keyring.mon
> > > > > > -rw-r--r-- 1 root root 187 Mar 23 12:17 monmap
> > > > > > -rw-r--r-- 1 root root 6886 Mar 23 12:17 osdmap
> > > > > >
> > > > >
> > > >
> > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@xxxxxxxxxxxxxx
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
