Re: keyring generation

["Kei.masumoto" <kei.masumoto@xxxxxxxxx>]

(2014/02/10 23:33), Alfredo Deza wrote:
> On Sat, Feb 8, 2014 at 7:56 AM, Kei.masumoto <kei.masumoto@xxxxxxxxx> wrote:
> > (2014/02/05 23:49), Alfredo Deza wrote:
> > > On Mon, Feb 3, 2014 at 11:28 AM, Kei.masumoto <kei.masumoto@xxxxxxxxx>
> > > wrote:
> > > > Hi Alfredo,
> > > >
> > > > Thanks for your reply!
> > > >
> > > > I think I pasted all logs from ceph.log,  but anyway, I re-executed
> > > > "ceph-deploy mon create-initial again"
> > > > Does that make sense? It seems like stack strace are added...
> > > Those seem bad enough. There is a ticket open for these type of
> > > tracebacks that should be gone with the
> > > up coming release of ceph-deploy.
> > >
> > > Your monitor does seem like in a good state. Have you checked the
> > > monitor logs to see if they are complaining
> > > about something?
> > >
> > > I would also raise the log level in ceph.conf for the monitors
> > > specifically to:
> > >
> > >     debug mon = 10
> > Thanks for your reply.
> > I did "debug mon = 10", but I could not find any error logs in
> > /var/log/ceph/ceph-mon.ceph1.log.
> > So I tried to let ceph-create-keys generate logs to files, and inserted log
> > by myself for debugging purpose.
> > Then I found get_key() in ceph-create-keys complains like below. ( first
> > line is inserted by me)
> >
> > > INFO:ceph-create-keys: ceph --cluster=ceph --name=mon.
> > > --keyring=/var/lib/ceph/mon/ceph-ceph1/keyring auth get-or-create
> > > client.admin mon allow * osd allow * mds allow
> > > INFO:ceph-create-keys:Talking to monitor...
> > > INFO:ceph-create-keys:Cannot get or create admin key, permission denied
> How did you started ceph-create-keys? with root? or with the ceph user?
>
> That process is usually fired up by the init script that is usually
> called by root.
I just use  "ceph-deploy --overwrite-conf mon create-initial" by user ceph.
Before doing that, "start ceph-all && stop ceph-all" by root.
BTW, I inserted debugging logs to /usr/bin/ceph, and found the below log.

> INFO:debug:Exception error calling connect
> INFO:debug:Exception error calling connect TimedOut

Those logs are generated by cluster_handle.connect().  e.g. 
rados.Rados.connect().

>    try:
>        if childargs and childargs[0] == 'ping':
>            return ping_monitor(cluster_handle, childargs[1])
>        cluster_handle.connect(timeout=timeout)

Any hints where to check? 6789 is listened by mon.


> > So I tried,
> > > root@ceph1:~/my-cluster# chown -R ceph:ceph /var/lib/ceph/mon/
> > > root@ceph1:~/my-cluster# start ceph-all && stop ceph-all
> > > ceph-all start/running
> > > ceph-all stop/waiting
> > Then I re-tried:
> > > ceph@ceph1:~/my-cluster$ ceph-deploy --overwrite-conf mon create-initial
> > After that, I found some files are still owned b root. Is this a correct
> > behavior?
> > > root@ceph1:~/my-cluster# ls -l /var/lib/ceph/mon/ceph-ceph1/store.db
> > > total 1184
> > > -rw-r--r-- 1 ceph ceph 1081168 Feb  8 02:25 000133.sst
> > > -rw-r--r-- 1 ceph ceph   25530 Feb  8 02:38 000135.sst
> > > -rw-r--r-- 1 ceph ceph   25530 Feb  8 02:38 000138.sst
> > > -rw-r--r-- 1 root root   25530 Feb  8 02:44 000141.sst
> > > -rw-r--r-- 1 root root   65536 Feb  8 02:44 000142.log
> > > -rw-r--r-- 1 root root      16 Feb  8 02:44 CURRENT
> > > -rw-r--r-- 1 ceph ceph       0 Jan 26 05:50 LOCK
> > > -rw-r--r-- 1 ceph ceph     315 Jan 26 06:28 LOG
> > > -rw-r--r-- 1 ceph ceph      57 Jan 26 05:50 LOG.old
> > > -rw-r--r-- 1 root root   65536 Feb  8 02:44 MANIFEST-000140
> > Also, looks like ceph-create-keys and ceph-mon are launched by root. Is this
> > correct?
> Yes these processes are launched by root.
>
> > > root@ceph1:~/my-cluster# ps -ef | grep ceph
> > > root     18943     1  0 03:11 ?        00:00:00 /bin/sh -e -c
> > > /usr/bin/ceph-mon --cluster="${cluster:-ceph}" -i "$id" -f /bin/sh
> > > root     18944 18943  0 03:11 ?        00:00:00 /usr/bin/ceph-mon
> > > --cluster=ceph -i ceph1 -f
> > > root     18945     1  0 03:11 ?        00:00:00 /bin/sh -e -c
> > > /usr/sbin/ceph-create-keys --cluster="${cluster:-ceph}" -i
> > > "${id:-$(hostname)}" /bin/sh
> > > root     18946 18945  0 03:11 ?        00:00:00 /usr/bin/python
> > > /usr/sbin/ceph-create-keys --cluster=ceph -i ceph1
> > > root     18960 18946  1 03:11 ?        00:00:00 /usr/bin/python
> > > /usr/bin/ceph --cluster=ceph --name=mon.
> > > --keyring=/var/lib/ceph/mon/ceph-ceph1/keyring auth get-or-create
> > > client.admin mon allow * osd allow * mds allow
> > > root     19036 16175  0 03:11 pts/7    00:00:00 tail -f
> > > /var/log/ceph/ceph-mon.ceph1.log
> >
> >
> >
> >
> > > > --------------------------------------------------------------------------------------------------------------------------
> > > >
> > > > [ceph_deploy.cli][INFO  ] Invoked (1.3.4): /usr/bin/ceph-deploy mon
> > > > create-initial
> > > > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph1
> > > > [ceph_deploy.mon][DEBUG ] detecting platform for host ceph1 ...
> > > > [ceph1][DEBUG ] connected to host: ceph1
> > > > [ceph1][DEBUG ] detect platform information from remote host
> > > > [ceph1][DEBUG ] detect machine type
> > > > [ceph_deploy.mon][INFO  ] distro info: Ubuntu 13.04 raring
> > > > [ceph1][DEBUG ] determining if provided host has same hostname in remote
> > > > [ceph1][DEBUG ] get remote short hostname
> > > > [ceph1][DEBUG ] deploying mon to ceph1
> > > > [ceph1][DEBUG ] get remote short hostname
> > > > [ceph1][DEBUG ] remote hostname: ceph1
> > > > [ceph1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> > > > [ceph1][DEBUG ] create the mon path if it does not exist
> > > > [ceph1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph1/done
> > > > [ceph1][DEBUG ] create a done file to avoid re-doing the mon deployment
> > > > [ceph1][DEBUG ] create the init path if it does not exist
> > > > [ceph1][DEBUG ] locating the `service` executable...
> > > > [ceph1][INFO  ] Running command: sudo initctl emit ceph-mon cluster=ceph
> > > > id=ceph1
> > > > [ceph1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon
> > > > /var/run/ceph/ceph-mon.ceph1.asok mon_status
> > > > [ceph1][DEBUG ]
> > > >
> > > > ********************************************************************************
> > > >
> > > > [ceph1][DEBUG ] status for monitor: mon.ceph1
> > > > [ceph1][DEBUG ] {
> > > > [ceph1][DEBUG ]   "election_epoch": 1,
> > > > [ceph1][DEBUG ]   "extra_probe_peers": [],
> > > > [ceph1][DEBUG ]   "monmap": {
> > > >
> > > > [ceph1][DEBUG ]     "created": "0.000000",
> > > > [ceph1][DEBUG ]     "epoch": 1,
> > > >
> > > > [ceph1][DEBUG ]     "fsid": "26835656-6b29-455d-9d1f-545cad8f1e23",
> > > > [ceph1][DEBUG ]     "modified": "0.000000",
> > > > [ceph1][DEBUG ]     "mons": [
> > > > [ceph1][DEBUG ]       {
> > > >
> > > > [ceph1][DEBUG ]         "addr": "192.168.111.11:6789/0",
> > > > [ceph1][DEBUG ]         "name": "ceph1",
> > > > [ceph1][DEBUG ]         "rank": 0
> > > > [ceph1][DEBUG ]       }
> > > > [ceph1][DEBUG ]     ]
> > > > [ceph1][DEBUG ]   },
> > > >
> > > > [ceph1][DEBUG ]   "name": "ceph1",
> > > > [ceph1][DEBUG ]   "outside_quorum": [],
> > > > [ceph1][DEBUG ]   "quorum": [
> > > > [ceph1][DEBUG ]     0
> > > > [ceph1][DEBUG ]   ],
> > > > [ceph1][DEBUG ]   "rank": 0,
> > > >
> > > > [ceph1][DEBUG ]   "state": "leader",
> > > > [ceph1][DEBUG ]   "sync_provider": []
> > > > [ceph1][DEBUG ] }
> > > > [ceph1][DEBUG ]
> > > >
> > > > ********************************************************************************
> > > >
> > > > [ceph1][INFO  ] monitor: mon.ceph1 is running
> > > > [ceph1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon
> > > > /var/run/ceph/ceph-mon.ceph1.asok mon_status
> > > > [ceph_deploy.mon][INFO  ] processing monitor mon.ceph1
> > > > [ceph1][DEBUG ] connected to host: ceph1
> > > > [ceph1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon
> > > > /var/run/ceph/ceph-mon.ceph1.asok mon_status
> > > > [ceph_deploy.mon][INFO  ] mon.ceph1 monitor has reached quorum!
> > > > [ceph_deploy.mon][INFO  ] all initial monitors are running and have
> > > > formed
> > > > quorum
> > > > [ceph_deploy.mon][INFO  ] Running gatherkeys...
> > > > gatherkeys.fetch_file    Namespace(cluster='ceph', dry_run=False,
> > > > func=<function mon at 0xe14e60>, mon=['ceph1'], overwrite_conf=False,
> > > > prog='ceph-deploy', quiet=False, subcommand='create-initial',
> > > > username=None,
> > > > verbose=False) :: /etc/ceph/ceph.client.admin.keyring ::
> > > > ceph.client.admin.keyring :: ['ceph1']
> > > >
> > > > [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1 for
> > > > /etc/ceph/ceph.client.admin.keyring
> > > > [ceph1][DEBUG ] connected to host: ceph1
> > > > [ceph1][DEBUG ] detect platform information from remote host
> > > > [ceph1][DEBUG ] detect machine type
> > > > [ceph1][DEBUG ] fetch remote file
> > > > [ceph_deploy.gatherkeys][WARNIN] Unable to find
> > > > /etc/ceph/ceph.client.admin.keyring on ['ceph1']
> > > > Traceback (most recent call last):
> > > >     File "<string>", line 1, in <module>
> > > >     File "<string>", line 6, in <module>
> > > >     File
> > > >
> > > > "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> > > > line 1220, in serve
> > > >       gatherkeys.fetch_file    Namespace(cluster='ceph', dry_run=False,
> > > > func=<function mon at 0xe14e60>, mon=['ceph1'], overwrite_conf=False,
> > > > prog='ceph-deploy', quiet=False, subcommand='create-initial',
> > > > username=None,
> > > > verbose=False) :: /var/lib/ceph/mon/ceph-{hostname}/keyring ::
> > > > ceph.mon.keyring :: ['ceph1']
> > > > [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
> > > > gatherkeys.fetch_file    Namespace(cluster='ceph', dry_run=False,
> > > > func=<function mon at 0xe14e60>, mon=['ceph1'], overwrite_conf=False,
> > > > prog='ceph-deploy', quiet=False, subcommand='create-initial',
> > > > username=None,
> > > > verbose=False) :: /var/lib/ceph/bootstrap-osd/ceph.keyring ::
> > > > ceph.bootstrap-osd.keyring :: ['ceph1']
> > > > SlaveGateway(io=io, id=id, _startcount=2).serve()
> > > >     File
> > > >
> > > > "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> > > > line 764, in serve
> > > >
> > > >       [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1 for
> > > > /var/lib/ceph/bootstrap-osd/ceph.keyring
> > > > self._io.close_write()
> > > >     File
> > > >
> > > > "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> > > > line 105, in close_write
> > > >       self.outfile.close()
> > > > IOError: close() called during concurrent operation on the same file
> > > > object.
> > > >
> > > > [ceph1][DEBUG ] connected to host: ceph1
> > > > [ceph1][DEBUG ] detect platform information from remote host
> > > > [ceph1][DEBUG ] detect machine type
> > > > [ceph1][DEBUG ] fetch remote file
> > > > [ceph_deploy.gatherkeys][WARNIN] Unable to find
> > > > /var/lib/ceph/bootstrap-osd/ceph.keyring on ['ceph1']
> > > > Traceback (most recent call last):
> > > >     File "<string>", line 1, in <module>
> > > >     File "<string>", line 6, in <module>
> > > >     File
> > > >
> > > > "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> > > > line 1220, in serve
> > > >       SlaveGateway(io=io, id=id, _startcount=2).serve()
> > > >     File
> > > >
> > > > "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> > > > line 764, in serve
> > > >       self._io.close_write()
> > > >     File
> > > >
> > > > "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> > > > line 105, in close_write
> > > >       gatherkeys.fetch_file    Namespace(cluster='ceph', dry_run=False,
> > > > func=<function mon at 0xe14e60>, mon=['ceph1'], overwrite_conf=False,
> > > > prog='ceph-deploy', quiet=False, subcommand='create-initial',
> > > > username=None,
> > > > verbose=False) :: /var/lib/ceph/bootstrap-mds/ceph.keyring ::
> > > > ceph.bootstrap-mds.keyring :: ['ceph1']
> > > > self.outfile.close()
> > > > IOError: close() called during concurrent operation on the same file
> > > > object.
> > > >
> > > > [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1 for
> > > > /var/lib/ceph/bootstrap-mds/ceph.keyring
> > > > [ceph1][DEBUG ] connected to host: ceph1
> > > > [ceph1][DEBUG ] detect platform information from remote host
> > > > [ceph1][DEBUG ] detect machine type
> > > > [ceph1][DEBUG ] fetch remote file
> > > > [ceph_deploy.gatherkeys][WARNIN] Unable to find
> > > > /var/lib/ceph/bootstrap-mds/ceph.keyring on ['ceph1']
> > > > Traceback (most recent call last):
> > > >     File "<string>", line 1, in <module>
> > > >     File "<string>", line 6, in <module>
> > > >     File
> > > >
> > > > "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> > > > line 1220, in serve
> > > >       SlaveGateway(io=io, id=id, _startcount=2).serve()
> > > >     File
> > > >
> > > > "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> > > > line 764, in serve
> > > >       self._io.close_write()
> > > >     File
> > > >
> > > > "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> > > > line 105, in close_write
> > > >       self.outfile.close()
> > > > IOError: close() called during concurrent operation on the same file
> > > > object.
> > > >
> > > >
> > > > ----------------------------------------------------------------------------------------------------------
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > (2014/02/04 1:12), Alfredo Deza wrote:
> > > > > On Mon, Feb 3, 2014 at 10:07 AM, Kei.masumoto <kei.masumoto@xxxxxxxxx>
> > > > > wrote:
> > > > > > Hi Alfredo,
> > > > > >
> > > > > > Thanks for reply!  I pasted the logs below.
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------------------------------
> > > > > > 2014-02-01 14:06:33,350 [ceph_deploy.cli][INFO  ] Invoked (1.3.4):
> > > > > > /usr/bin/ceph-deploy mon create-initial
> > > > > > 2014-02-01 14:06:33,353 [ceph_deploy.mon][DEBUG ] Deploying mon,
> > > > > > cluster
> > > > > > ceph hosts ceph1
> > > > > > 2014-02-01 14:06:33,354 [ceph_deploy.mon][DEBUG ] detecting platform
> > > > > > for
> > > > > > host ceph1 ...
> > > > > > 2014-02-01 14:06:33,770 [ceph1][DEBUG ] connected to host: ceph1
> > > > > > 2014-02-01 14:06:33,775 [ceph1][DEBUG ] detect platform information
> > > > > > from
> > > > > > remote host
> > > > > > 2014-02-01 14:06:33,874 [ceph1][DEBUG ] detect machine type
> > > > > > 2014-02-01 14:06:33,909 [ceph_deploy.mon][INFO  ] distro info: Ubuntu
> > > > > > 13.04
> > > > > > raring
> > > > > > 2014-02-01 14:06:33,910 [ceph1][DEBUG ] determining if provided host
> > > > > > has
> > > > > > same hostname in remote
> > > > > > 2014-02-01 14:06:33,911 [ceph1][DEBUG ] get remote short hostname
> > > > > > 2014-02-01 14:06:33,914 [ceph1][DEBUG ] deploying mon to ceph1
> > > > > > 2014-02-01 14:06:33,915 [ceph1][DEBUG ] get remote short hostname
> > > > > > 2014-02-01 14:06:33,917 [ceph1][DEBUG ] remote hostname: ceph1
> > > > > > 2014-02-01 14:06:33,919 [ceph1][DEBUG ] write cluster configuration to
> > > > > > /etc/ceph/{cluster}.conf
> > > > > > 2014-02-01 14:06:33,933 [ceph1][DEBUG ] create the mon path if it does
> > > > > > not
> > > > > > exist
> > > > > > 2014-02-01 14:06:33,939 [ceph1][DEBUG ] checking for done path:
> > > > > > /var/lib/ceph/mon/ceph-ceph1/done
> > > > > > 2014-02-01 14:06:33,941 [ceph1][DEBUG ] create a done file to avoid
> > > > > > re-doing
> > > > > > the mon deployment
> > > > > > 2014-02-01 14:06:33,944 [ceph1][DEBUG ] create the init path if it does
> > > > > > not
> > > > > > exist
> > > > > > 2014-02-01 14:06:33,946 [ceph1][DEBUG ] locating the `service`
> > > > > > executable...
> > > > > > 2014-02-01 14:06:33,949 [ceph1][INFO  ] Running command: sudo initctl
> > > > > > emit
> > > > > > ceph-mon cluster=ceph id=ceph1
> > > > > > 2014-02-01 14:06:36,119 [ceph1][INFO  ] Running command: sudo ceph
> > > > > > --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph1.asok
> > > > > > mon_status
> > > > > > 2014-02-01 14:06:36,805 [ceph1][DEBUG ]
> > > > > >
> > > > > >
> > > > > > ********************************************************************************
> > > > > > 2014-02-01 14:06:36,807 [ceph1][DEBUG ] status for monitor: mon.ceph1
> > > > > > 2014-02-01 14:06:36,809 [ceph1][DEBUG ] {
> > > > > > 2014-02-01 14:06:36,810 [ceph1][DEBUG ]   "election_epoch": 1,
> > > > > > 2014-02-01 14:06:36,812 [ceph1][DEBUG ]   "extra_probe_peers": [],
> > > > > > 2014-02-01 14:06:36,813 [ceph1][DEBUG ]   "monmap": {
> > > > > > 2014-02-01 14:06:36,814 [ceph1][DEBUG ]     "created": "0.000000",
> > > > > > 2014-02-01 14:06:36,815 [ceph1][DEBUG ]     "epoch": 1,
> > > > > > 2014-02-01 14:06:36,815 [ceph1][DEBUG ]     "fsid":
> > > > > > "26835656-6b29-455d-9d1f-545cad8f1e23",
> > > > > > 2014-02-01 14:06:36,816 [ceph1][DEBUG ]     "modified": "0.000000",
> > > > > > 2014-02-01 14:06:36,816 [ceph1][DEBUG ]     "mons": [
> > > > > > 2014-02-01 14:06:36,817 [ceph1][DEBUG ]       {
> > > > > > 2014-02-01 14:06:36,818 [ceph1][DEBUG ]         "addr":
> > > > > > "192.168.111.11:6789/0",
> > > > > > 2014-02-01 14:06:36,818 [ceph1][DEBUG ]         "name": "ceph1",
> > > > > > 2014-02-01 14:06:36,819 [ceph1][DEBUG ]         "rank": 0
> > > > > > 2014-02-01 14:06:36,820 [ceph1][DEBUG ]       }
> > > > > > 2014-02-01 14:06:36,821 [ceph1][DEBUG ]     ]
> > > > > > 2014-02-01 14:06:36,822 [ceph1][DEBUG ]   },
> > > > > > 2014-02-01 14:06:36,826 [ceph1][DEBUG ]   "name": "ceph1",
> > > > > > 2014-02-01 14:06:36,826 [ceph1][DEBUG ]   "outside_quorum": [],
> > > > > > 2014-02-01 14:06:36,826 [ceph1][DEBUG ]   "quorum": [
> > > > > > 2014-02-01 14:06:36,827 [ceph1][DEBUG ]     0
> > > > > > 2014-02-01 14:06:36,827 [ceph1][DEBUG ]   ],
> > > > > > 2014-02-01 14:06:36,827 [ceph1][DEBUG ]   "rank": 0,
> > > > > > 2014-02-01 14:06:36,827 [ceph1][DEBUG ]   "state": "leader",
> > > > > > 2014-02-01 14:06:36,828 [ceph1][DEBUG ]   "sync_provider": []
> > > > > > 2014-02-01 14:06:36,828 [ceph1][DEBUG ] }
> > > > > > 2014-02-01 14:06:36,828 [ceph1][DEBUG ]
> > > > > >
> > > > > >
> > > > > > ********************************************************************************
> > > > > > 2014-02-01 14:06:36,829 [ceph1][INFO  ] monitor: mon.ceph1 is running
> > > > > > 2014-02-01 14:06:36,830 [ceph1][INFO  ] Running command: sudo ceph
> > > > > > --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph1.asok
> > > > > > mon_status
> > > > > > 2014-02-01 14:06:37,005 [ceph_deploy.mon][INFO  ] processing monitor
> > > > > > mon.ceph1
> > > > > > 2014-02-01 14:06:37,079 [ceph1][DEBUG ] connected to host: ceph1
> > > > > > 2014-02-01 14:06:37,081 [ceph1][INFO  ] Running command: sudo ceph
> > > > > > --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph1.asok
> > > > > > mon_status
> > > > > > 2014-02-01 14:06:37,258 [ceph_deploy.mon][INFO  ] mon.ceph1 monitor has
> > > > > > reached quorum!
> > > > > > 2014-02-01 14:06:37,259 [ceph_deploy.mon][INFO  ] all initial monitors
> > > > > > are
> > > > > > running and have formed quorum
> > > > > > 2014-02-01 14:06:37,266 [ceph_deploy.mon][INFO  ] Running gatherkeys...
> > > > > > 2014-02-01 14:06:37,268 [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1
> > > > > > for
> > > > > > /etc/ceph/ceph.client.admin.keyring
> > > > > > 2014-02-01 14:06:37,336 [ceph1][DEBUG ] connected to host: ceph1
> > > > > > 2014-02-01 14:06:37,340 [ceph1][DEBUG ] detect platform information
> > > > > > from
> > > > > > remote host
> > > > > > 2014-02-01 14:06:37,373 [ceph1][DEBUG ] detect machine type
> > > > > > 2014-02-01 14:06:37,383 [ceph1][DEBUG ] fetch remote file
> > > > > >
> > > > > > 2014-02-01 14:06:37,385 [ceph_deploy.gatherkeys][WARNING] Unable to
> > > > > > find
> > > > > > /etc/ceph/ceph.client.admin.keyring on ['ceph1']
> > > > > > 2014-02-01 14:06:37,391 [ceph_deploy.gatherkeys][DEBUG ] Have
> > > > > > ceph.mon.keyring
> > > > > > 2014-02-01 14:06:37,398 [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1
> > > > > > for
> > > > > > /var/lib/ceph/bootstrap-osd/ceph.keyring
> > > > > > 2014-02-01 14:06:37,468 [ceph1][DEBUG ] connected to host: ceph1
> > > > > > 2014-02-01 14:06:37,471 [ceph1][DEBUG ] detect platform information
> > > > > > from
> > > > > > remote host
> > > > > > 2014-02-01 14:06:37,506 [ceph1][DEBUG ] detect machine type
> > > > > > 2014-02-01 14:06:37,514 [ceph1][DEBUG ] fetch remote file
> > > > > >
> > > > > > 2014-02-01 14:06:37,516 [ceph_deploy.gatherkeys][WARNING] Unable to
> > > > > > find
> > > > > > /var/lib/ceph/bootstrap-osd/ceph.keyring on ['ceph1']
> > > > > > 2014-02-01 14:06:37,523 [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1
> > > > > > for
> > > > > > /var/lib/ceph/bootstrap-mds/ceph.keyring
> > > > > > 2014-02-01 14:06:37,591 [ceph1][DEBUG ] connected to host: ceph1
> > > > > > 2014-02-01 14:06:37,594 [ceph1][DEBUG ] detect platform information
> > > > > > from
> > > > > > remote host
> > > > > > 2014-02-01 14:06:37,627 [ceph1][DEBUG ] detect machine type
> > > > > > 2014-02-01 14:06:37,636 [ceph1][DEBUG ] fetch remote file
> > > > > >
> > > > > > 2014-02-01 14:06:37,639 [ceph_deploy.gatherkeys][WARNING] Unable to
> > > > > > find
> > > > > > /var/lib/ceph/bootstrap-mds/ceph.keyring on ['ceph1']
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------------------------------
> > > > > Does it end here? seems like the output was trimmed...
> > > > >
> > > > > > (2014/02/03 22:26), Alfredo Deza wrote:
> > > > > > > On Sun, Feb 2, 2014 at 12:18 AM, Kei.masumoto <kei.masumoto@xxxxxxxxx>
> > > > > > > wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I am newbie of ceph, now I am trying to deploy following
> > > > > > > > "http://ceph.com/docs/master/start/quick-ceph-deploy/";
> > > > > > > > ceph1, ceph2 and ceph3 exists according to the above tutorial. I got
> > > > > > > > a
> > > > > > > > WARNING message when I exec ceph-deploy "mon create-initial".
> > > > > > > >
> > > > > > > > 2014-02-01 14:06:37,385 [ceph_deploy.gatherkeys][WARNING] Unable to
> > > > > > > > find
> > > > > > > > /etc/ceph/ceph.client.admin.keyring on ['ceph1']
> > > > > > > > 2014-02-01 14:06:37,516 [ceph_deploy.gatherkeys][WARNING] Unable to
> > > > > > > > find
> > > > > > > > /var/lib/ceph/bootstrap-osd/ceph.keyring on ['ceph1']
> > > > > > > > 2014-02-01 14:06:37,639 [ceph_deploy.gatherkeys][WARNING] Unable to
> > > > > > > > find
> > > > > > > > /var/lib/ceph/bootstrap-mds/ceph.keyring on ['ceph1']
> > > > > > > >
> > > > > > > > Thinking about when those 3 keyrings should be created, I thins
> > > > > > > > "ceph-deploy mon create " is a right timing for keyring creation. I
> > > > > > > > checked my environment, and found
> > > > > > > > /etc/ceph/ceph.client.admin.keyring.14081.tmp. It looks like this
> > > > > > > > file
> > > > > > > > is created by ceph-create-keys on executing stop ceph-all && start
> > > > > > > > ceph-all. but ceph-create-keys never finishes.
> > > > > > > ceph-deploy tries to help here a lot with create-initial, and although
> > > > > > > the warnings are useful,
> > > > > > > they are only good depending on the context of the rest of the output.
> > > > > > >
> > > > > > > When the whole process completes, does ceph-deploy say all mons are up
> > > > > > > and running?
> > > > > > >
> > > > > > > It would be better to paste the complete output of the call so we can
> > > > > > > see the details.
> > > > > > > > When I execute ceph-create-keys manually, it continues to generate
> > > > > > > > below
> > > > > > > > log, looks like waiting reply...
> > > > > > > >
> > > > > > > > 2014-02-01 20:13:02.847737 7f55e81a4700  0 -- :/1001774 >>
> > > > > > > > 192.168.11.8:6789/0 pipe(0x7f55e4024400 sd=3 :0 s=1 pgs=0 cs=0 l=1
> > > > > > > > c=0x7f55e4024660).fault
> > > > > > > >
> > > > > > > > Since I found that mon listens 6789, so I strace mon, then mon also
> > > > > > > > waiting something...
> > > > > > > >
> > > > > > > > root@ceph1:~/src/ceph-0.56.7# strace -p 1047
> > > > > > > > Process 1047 attached - interrupt to quit
> > > > > > > > futex(0x7f37c14839d0, FUTEX_WAIT, 1102, NULL
> > > > > > > >
> > > > > > > > I have no idea what situation should be, any hints?
> > > > > > > >
> > > > > > > > P.S. somebody give me an adivce to check below, but I dont see any
> > > > > > > > from
> > > > > > > > here.
> > > > > > > > root@ceph1:~/my-cluster# ceph daemon mon.`hostname` mon_status
> > > > > > > > { "name": "ceph1",
> > > > > > > >       "rank": 0,
> > > > > > > >       "state": "leader",
> > > > > > > >       "election_epoch": 1,
> > > > > > > >       "quorum": [
> > > > > > > >             0],
> > > > > > > >       "outside_quorum": [],
> > > > > > > >       "extra_probe_peers": [],
> > > > > > > >       "sync_provider": [],
> > > > > > > >       "monmap": { "epoch": 1,
> > > > > > > >           "fsid": "26835656-6b29-455d-9d1f-545cad8f1e23",
> > > > > > > >           "modified": "0.000000",
> > > > > > > >           "created": "0.000000",
> > > > > > > >           "mons": [
> > > > > > > >                 { "rank": 0,
> > > > > > > >                   "name": "ceph1",
> > > > > > > >                   "addr": "192.168.111.11:6789\/0"}]}}
> > > > > > > >
> > > > > > > >
> > > > > > > > Kei
> > > > > > > > _______________________________________________
> > > > > > > > ceph-users mailing list
> > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com