On Sat, Feb 8, 2014 at 7:56 AM, Kei.masumoto <kei.masumoto@xxxxxxxxx> wrote:
>
> (2014/02/05 23:49), Alfredo Deza wrote:
>>
>> On Mon, Feb 3, 2014 at 11:28 AM, Kei.masumoto <kei.masumoto@xxxxxxxxx>
>> wrote:
>>>
>>> Hi Alfredo,
>>>
>>> Thanks for your reply!
>>>
>>> I think I pasted all logs from ceph.log, but anyway, I re-executed
>>> "ceph-deploy mon create-initial again"
>>> Does that make sense? It seems like stack strace are added...
>>
>> Those seem bad enough. There is a ticket open for these type of
>> tracebacks that should be gone with the
>> up coming release of ceph-deploy.
>>
>> Your monitor does seem like in a good state. Have you checked the
>> monitor logs to see if they are complaining
>> about something?
>>
>> I would also raise the log level in ceph.conf for the monitors
>> specifically to:
>>
>> debug mon = 10
>>
>
> Thanks for your reply.
> I did "debug mon = 10", but I could not find any error logs in
> /var/log/ceph/ceph-mon.ceph1.log.
> So I tried to let ceph-create-keys generate logs to files, and inserted log
> by myself for debugging purpose.
> Then I found get_key() in ceph-create-keys complains like below. ( first
> line is inserted by me)
>
>> INFO:ceph-create-keys: ceph --cluster=ceph --name=mon.
>> --keyring=/var/lib/ceph/mon/ceph-ceph1/keyring auth get-or-create
>> client.admin mon allow * osd allow * mds allow
>> INFO:ceph-create-keys:Talking to monitor...
>> INFO:ceph-create-keys:Cannot get or create admin key, permission denied
>
How did you started ceph-create-keys? with root? or with the ceph user?
That process is usually fired up by the init script that is usually
called by root.
> So I tried,
>> root@ceph1:~/my-cluster# chown -R ceph:ceph /var/lib/ceph/mon/
>> root@ceph1:~/my-cluster# start ceph-all && stop ceph-all
>> ceph-all start/running
>> ceph-all stop/waiting
>
> Then I re-tried:
>> ceph@ceph1:~/my-cluster$ ceph-deploy --overwrite-conf mon create-initial
>
> After that, I found some files are still owned b root. Is this a correct
> behavior?
>> root@ceph1:~/my-cluster# ls -l /var/lib/ceph/mon/ceph-ceph1/store.db
>> total 1184
>> -rw-r--r-- 1 ceph ceph 1081168 Feb 8 02:25 000133.sst
>> -rw-r--r-- 1 ceph ceph 25530 Feb 8 02:38 000135.sst
>> -rw-r--r-- 1 ceph ceph 25530 Feb 8 02:38 000138.sst
>> -rw-r--r-- 1 root root 25530 Feb 8 02:44 000141.sst
>> -rw-r--r-- 1 root root 65536 Feb 8 02:44 000142.log
>> -rw-r--r-- 1 root root 16 Feb 8 02:44 CURRENT
>> -rw-r--r-- 1 ceph ceph 0 Jan 26 05:50 LOCK
>> -rw-r--r-- 1 ceph ceph 315 Jan 26 06:28 LOG
>> -rw-r--r-- 1 ceph ceph 57 Jan 26 05:50 LOG.old
>> -rw-r--r-- 1 root root 65536 Feb 8 02:44 MANIFEST-000140
>
> Also, looks like ceph-create-keys and ceph-mon are launched by root. Is this
> correct?
Yes these processes are launched by root.
>> root@ceph1:~/my-cluster# ps -ef | grep ceph
>> root 18943 1 0 03:11 ? 00:00:00 /bin/sh -e -c
>> /usr/bin/ceph-mon --cluster="${cluster:-ceph}" -i "$id" -f /bin/sh
>> root 18944 18943 0 03:11 ? 00:00:00 /usr/bin/ceph-mon
>> --cluster=ceph -i ceph1 -f
>> root 18945 1 0 03:11 ? 00:00:00 /bin/sh -e -c
>> /usr/sbin/ceph-create-keys --cluster="${cluster:-ceph}" -i
>> "${id:-$(hostname)}" /bin/sh
>> root 18946 18945 0 03:11 ? 00:00:00 /usr/bin/python
>> /usr/sbin/ceph-create-keys --cluster=ceph -i ceph1
>> root 18960 18946 1 03:11 ? 00:00:00 /usr/bin/python
>> /usr/bin/ceph --cluster=ceph --name=mon.
>> --keyring=/var/lib/ceph/mon/ceph-ceph1/keyring auth get-or-create
>> client.admin mon allow * osd allow * mds allow
>> root 19036 16175 0 03:11 pts/7 00:00:00 tail -f
>> /var/log/ceph/ceph-mon.ceph1.log
>
>
>
>
>
>>>
>>> --------------------------------------------------------------------------------------------------------------------------
>>>
>>> [ceph_deploy.cli][INFO ] Invoked (1.3.4): /usr/bin/ceph-deploy mon
>>> create-initial
>>> [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph1
>>> [ceph_deploy.mon][DEBUG ] detecting platform for host ceph1 ...
>>> [ceph1][DEBUG ] connected to host: ceph1
>>> [ceph1][DEBUG ] detect platform information from remote host
>>> [ceph1][DEBUG ] detect machine type
>>> [ceph_deploy.mon][INFO ] distro info: Ubuntu 13.04 raring
>>> [ceph1][DEBUG ] determining if provided host has same hostname in remote
>>> [ceph1][DEBUG ] get remote short hostname
>>> [ceph1][DEBUG ] deploying mon to ceph1
>>> [ceph1][DEBUG ] get remote short hostname
>>> [ceph1][DEBUG ] remote hostname: ceph1
>>> [ceph1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
>>> [ceph1][DEBUG ] create the mon path if it does not exist
>>> [ceph1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph1/done
>>> [ceph1][DEBUG ] create a done file to avoid re-doing the mon deployment
>>> [ceph1][DEBUG ] create the init path if it does not exist
>>> [ceph1][DEBUG ] locating the `service` executable...
>>> [ceph1][INFO ] Running command: sudo initctl emit ceph-mon cluster=ceph
>>> id=ceph1
>>> [ceph1][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
>>> /var/run/ceph/ceph-mon.ceph1.asok mon_status
>>> [ceph1][DEBUG ]
>>>
>>> ********************************************************************************
>>>
>>> [ceph1][DEBUG ] status for monitor: mon.ceph1
>>> [ceph1][DEBUG ] {
>>> [ceph1][DEBUG ] "election_epoch": 1,
>>> [ceph1][DEBUG ] "extra_probe_peers": [],
>>> [ceph1][DEBUG ] "monmap": {
>>>
>>> [ceph1][DEBUG ] "created": "0.000000",
>>> [ceph1][DEBUG ] "epoch": 1,
>>>
>>> [ceph1][DEBUG ] "fsid": "26835656-6b29-455d-9d1f-545cad8f1e23",
>>> [ceph1][DEBUG ] "modified": "0.000000",
>>> [ceph1][DEBUG ] "mons": [
>>> [ceph1][DEBUG ] {
>>>
>>> [ceph1][DEBUG ] "addr": "192.168.111.11:6789/0",
>>> [ceph1][DEBUG ] "name": "ceph1",
>>> [ceph1][DEBUG ] "rank": 0
>>> [ceph1][DEBUG ] }
>>> [ceph1][DEBUG ] ]
>>> [ceph1][DEBUG ] },
>>>
>>> [ceph1][DEBUG ] "name": "ceph1",
>>> [ceph1][DEBUG ] "outside_quorum": [],
>>> [ceph1][DEBUG ] "quorum": [
>>> [ceph1][DEBUG ] 0
>>> [ceph1][DEBUG ] ],
>>> [ceph1][DEBUG ] "rank": 0,
>>>
>>> [ceph1][DEBUG ] "state": "leader",
>>> [ceph1][DEBUG ] "sync_provider": []
>>> [ceph1][DEBUG ] }
>>> [ceph1][DEBUG ]
>>>
>>> ********************************************************************************
>>>
>>> [ceph1][INFO ] monitor: mon.ceph1 is running
>>> [ceph1][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
>>> /var/run/ceph/ceph-mon.ceph1.asok mon_status
>>> [ceph_deploy.mon][INFO ] processing monitor mon.ceph1
>>> [ceph1][DEBUG ] connected to host: ceph1
>>> [ceph1][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
>>> /var/run/ceph/ceph-mon.ceph1.asok mon_status
>>> [ceph_deploy.mon][INFO ] mon.ceph1 monitor has reached quorum!
>>> [ceph_deploy.mon][INFO ] all initial monitors are running and have
>>> formed
>>> quorum
>>> [ceph_deploy.mon][INFO ] Running gatherkeys...
>>> gatherkeys.fetch_file Namespace(cluster='ceph', dry_run=False,
>>> func=<function mon at 0xe14e60>, mon=['ceph1'], overwrite_conf=False,
>>> prog='ceph-deploy', quiet=False, subcommand='create-initial',
>>> username=None,
>>> verbose=False) :: /etc/ceph/ceph.client.admin.keyring ::
>>> ceph.client.admin.keyring :: ['ceph1']
>>>
>>> [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1 for
>>> /etc/ceph/ceph.client.admin.keyring
>>> [ceph1][DEBUG ] connected to host: ceph1
>>> [ceph1][DEBUG ] detect platform information from remote host
>>> [ceph1][DEBUG ] detect machine type
>>> [ceph1][DEBUG ] fetch remote file
>>> [ceph_deploy.gatherkeys][WARNIN] Unable to find
>>> /etc/ceph/ceph.client.admin.keyring on ['ceph1']
>>> Traceback (most recent call last):
>>> File "<string>", line 1, in <module>
>>> File "<string>", line 6, in <module>
>>> File
>>>
>>> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
>>> line 1220, in serve
>>> gatherkeys.fetch_file Namespace(cluster='ceph', dry_run=False,
>>> func=<function mon at 0xe14e60>, mon=['ceph1'], overwrite_conf=False,
>>> prog='ceph-deploy', quiet=False, subcommand='create-initial',
>>> username=None,
>>> verbose=False) :: /var/lib/ceph/mon/ceph-{hostname}/keyring ::
>>> ceph.mon.keyring :: ['ceph1']
>>> [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
>>> gatherkeys.fetch_file Namespace(cluster='ceph', dry_run=False,
>>> func=<function mon at 0xe14e60>, mon=['ceph1'], overwrite_conf=False,
>>> prog='ceph-deploy', quiet=False, subcommand='create-initial',
>>> username=None,
>>> verbose=False) :: /var/lib/ceph/bootstrap-osd/ceph.keyring ::
>>> ceph.bootstrap-osd.keyring :: ['ceph1']
>>> SlaveGateway(io=io, id=id, _startcount=2).serve()
>>> File
>>>
>>> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
>>> line 764, in serve
>>>
>>> [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1 for
>>> /var/lib/ceph/bootstrap-osd/ceph.keyring
>>> self._io.close_write()
>>> File
>>>
>>> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
>>> line 105, in close_write
>>> self.outfile.close()
>>> IOError: close() called during concurrent operation on the same file
>>> object.
>>>
>>> [ceph1][DEBUG ] connected to host: ceph1
>>> [ceph1][DEBUG ] detect platform information from remote host
>>> [ceph1][DEBUG ] detect machine type
>>> [ceph1][DEBUG ] fetch remote file
>>> [ceph_deploy.gatherkeys][WARNIN] Unable to find
>>> /var/lib/ceph/bootstrap-osd/ceph.keyring on ['ceph1']
>>> Traceback (most recent call last):
>>> File "<string>", line 1, in <module>
>>> File "<string>", line 6, in <module>
>>> File
>>>
>>> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
>>> line 1220, in serve
>>> SlaveGateway(io=io, id=id, _startcount=2).serve()
>>> File
>>>
>>> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
>>> line 764, in serve
>>> self._io.close_write()
>>> File
>>>
>>> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
>>> line 105, in close_write
>>> gatherkeys.fetch_file Namespace(cluster='ceph', dry_run=False,
>>> func=<function mon at 0xe14e60>, mon=['ceph1'], overwrite_conf=False,
>>> prog='ceph-deploy', quiet=False, subcommand='create-initial',
>>> username=None,
>>> verbose=False) :: /var/lib/ceph/bootstrap-mds/ceph.keyring ::
>>> ceph.bootstrap-mds.keyring :: ['ceph1']
>>> self.outfile.close()
>>> IOError: close() called during concurrent operation on the same file
>>> object.
>>>
>>> [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1 for
>>> /var/lib/ceph/bootstrap-mds/ceph.keyring
>>> [ceph1][DEBUG ] connected to host: ceph1
>>> [ceph1][DEBUG ] detect platform information from remote host
>>> [ceph1][DEBUG ] detect machine type
>>> [ceph1][DEBUG ] fetch remote file
>>> [ceph_deploy.gatherkeys][WARNIN] Unable to find
>>> /var/lib/ceph/bootstrap-mds/ceph.keyring on ['ceph1']
>>> Traceback (most recent call last):
>>> File "<string>", line 1, in <module>
>>> File "<string>", line 6, in <module>
>>> File
>>>
>>> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
>>> line 1220, in serve
>>> SlaveGateway(io=io, id=id, _startcount=2).serve()
>>> File
>>>
>>> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
>>> line 764, in serve
>>> self._io.close_write()
>>> File
>>>
>>> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
>>> line 105, in close_write
>>> self.outfile.close()
>>> IOError: close() called during concurrent operation on the same file
>>> object.
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------------
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> (2014/02/04 1:12), Alfredo Deza wrote:
>>>>
>>>> On Mon, Feb 3, 2014 at 10:07 AM, Kei.masumoto <kei.masumoto@xxxxxxxxx>
>>>> wrote:
>>>>>
>>>>> Hi Alfredo,
>>>>>
>>>>> Thanks for reply! I pasted the logs below.
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------------
>>>>> 2014-02-01 14:06:33,350 [ceph_deploy.cli][INFO ] Invoked (1.3.4):
>>>>> /usr/bin/ceph-deploy mon create-initial
>>>>> 2014-02-01 14:06:33,353 [ceph_deploy.mon][DEBUG ] Deploying mon,
>>>>> cluster
>>>>> ceph hosts ceph1
>>>>> 2014-02-01 14:06:33,354 [ceph_deploy.mon][DEBUG ] detecting platform
>>>>> for
>>>>> host ceph1 ...
>>>>> 2014-02-01 14:06:33,770 [ceph1][DEBUG ] connected to host: ceph1
>>>>> 2014-02-01 14:06:33,775 [ceph1][DEBUG ] detect platform information
>>>>> from
>>>>> remote host
>>>>> 2014-02-01 14:06:33,874 [ceph1][DEBUG ] detect machine type
>>>>> 2014-02-01 14:06:33,909 [ceph_deploy.mon][INFO ] distro info: Ubuntu
>>>>> 13.04
>>>>> raring
>>>>> 2014-02-01 14:06:33,910 [ceph1][DEBUG ] determining if provided host
>>>>> has
>>>>> same hostname in remote
>>>>> 2014-02-01 14:06:33,911 [ceph1][DEBUG ] get remote short hostname
>>>>> 2014-02-01 14:06:33,914 [ceph1][DEBUG ] deploying mon to ceph1
>>>>> 2014-02-01 14:06:33,915 [ceph1][DEBUG ] get remote short hostname
>>>>> 2014-02-01 14:06:33,917 [ceph1][DEBUG ] remote hostname: ceph1
>>>>> 2014-02-01 14:06:33,919 [ceph1][DEBUG ] write cluster configuration to
>>>>> /etc/ceph/{cluster}.conf
>>>>> 2014-02-01 14:06:33,933 [ceph1][DEBUG ] create the mon path if it does
>>>>> not
>>>>> exist
>>>>> 2014-02-01 14:06:33,939 [ceph1][DEBUG ] checking for done path:
>>>>> /var/lib/ceph/mon/ceph-ceph1/done
>>>>> 2014-02-01 14:06:33,941 [ceph1][DEBUG ] create a done file to avoid
>>>>> re-doing
>>>>> the mon deployment
>>>>> 2014-02-01 14:06:33,944 [ceph1][DEBUG ] create the init path if it does
>>>>> not
>>>>> exist
>>>>> 2014-02-01 14:06:33,946 [ceph1][DEBUG ] locating the `service`
>>>>> executable...
>>>>> 2014-02-01 14:06:33,949 [ceph1][INFO ] Running command: sudo initctl
>>>>> emit
>>>>> ceph-mon cluster=ceph id=ceph1
>>>>> 2014-02-01 14:06:36,119 [ceph1][INFO ] Running command: sudo ceph
>>>>> --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph1.asok
>>>>> mon_status
>>>>> 2014-02-01 14:06:36,805 [ceph1][DEBUG ]
>>>>>
>>>>>
>>>>> ********************************************************************************
>>>>> 2014-02-01 14:06:36,807 [ceph1][DEBUG ] status for monitor: mon.ceph1
>>>>> 2014-02-01 14:06:36,809 [ceph1][DEBUG ] {
>>>>> 2014-02-01 14:06:36,810 [ceph1][DEBUG ] "election_epoch": 1,
>>>>> 2014-02-01 14:06:36,812 [ceph1][DEBUG ] "extra_probe_peers": [],
>>>>> 2014-02-01 14:06:36,813 [ceph1][DEBUG ] "monmap": {
>>>>> 2014-02-01 14:06:36,814 [ceph1][DEBUG ] "created": "0.000000",
>>>>> 2014-02-01 14:06:36,815 [ceph1][DEBUG ] "epoch": 1,
>>>>> 2014-02-01 14:06:36,815 [ceph1][DEBUG ] "fsid":
>>>>> "26835656-6b29-455d-9d1f-545cad8f1e23",
>>>>> 2014-02-01 14:06:36,816 [ceph1][DEBUG ] "modified": "0.000000",
>>>>> 2014-02-01 14:06:36,816 [ceph1][DEBUG ] "mons": [
>>>>> 2014-02-01 14:06:36,817 [ceph1][DEBUG ] {
>>>>> 2014-02-01 14:06:36,818 [ceph1][DEBUG ] "addr":
>>>>> "192.168.111.11:6789/0",
>>>>> 2014-02-01 14:06:36,818 [ceph1][DEBUG ] "name": "ceph1",
>>>>> 2014-02-01 14:06:36,819 [ceph1][DEBUG ] "rank": 0
>>>>> 2014-02-01 14:06:36,820 [ceph1][DEBUG ] }
>>>>> 2014-02-01 14:06:36,821 [ceph1][DEBUG ] ]
>>>>> 2014-02-01 14:06:36,822 [ceph1][DEBUG ] },
>>>>> 2014-02-01 14:06:36,826 [ceph1][DEBUG ] "name": "ceph1",
>>>>> 2014-02-01 14:06:36,826 [ceph1][DEBUG ] "outside_quorum": [],
>>>>> 2014-02-01 14:06:36,826 [ceph1][DEBUG ] "quorum": [
>>>>> 2014-02-01 14:06:36,827 [ceph1][DEBUG ] 0
>>>>> 2014-02-01 14:06:36,827 [ceph1][DEBUG ] ],
>>>>> 2014-02-01 14:06:36,827 [ceph1][DEBUG ] "rank": 0,
>>>>> 2014-02-01 14:06:36,827 [ceph1][DEBUG ] "state": "leader",
>>>>> 2014-02-01 14:06:36,828 [ceph1][DEBUG ] "sync_provider": []
>>>>> 2014-02-01 14:06:36,828 [ceph1][DEBUG ] }
>>>>> 2014-02-01 14:06:36,828 [ceph1][DEBUG ]
>>>>>
>>>>>
>>>>> ********************************************************************************
>>>>> 2014-02-01 14:06:36,829 [ceph1][INFO ] monitor: mon.ceph1 is running
>>>>> 2014-02-01 14:06:36,830 [ceph1][INFO ] Running command: sudo ceph
>>>>> --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph1.asok
>>>>> mon_status
>>>>> 2014-02-01 14:06:37,005 [ceph_deploy.mon][INFO ] processing monitor
>>>>> mon.ceph1
>>>>> 2014-02-01 14:06:37,079 [ceph1][DEBUG ] connected to host: ceph1
>>>>> 2014-02-01 14:06:37,081 [ceph1][INFO ] Running command: sudo ceph
>>>>> --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph1.asok
>>>>> mon_status
>>>>> 2014-02-01 14:06:37,258 [ceph_deploy.mon][INFO ] mon.ceph1 monitor has
>>>>> reached quorum!
>>>>> 2014-02-01 14:06:37,259 [ceph_deploy.mon][INFO ] all initial monitors
>>>>> are
>>>>> running and have formed quorum
>>>>> 2014-02-01 14:06:37,266 [ceph_deploy.mon][INFO ] Running gatherkeys...
>>>>> 2014-02-01 14:06:37,268 [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1
>>>>> for
>>>>> /etc/ceph/ceph.client.admin.keyring
>>>>> 2014-02-01 14:06:37,336 [ceph1][DEBUG ] connected to host: ceph1
>>>>> 2014-02-01 14:06:37,340 [ceph1][DEBUG ] detect platform information
>>>>> from
>>>>> remote host
>>>>> 2014-02-01 14:06:37,373 [ceph1][DEBUG ] detect machine type
>>>>> 2014-02-01 14:06:37,383 [ceph1][DEBUG ] fetch remote file
>>>>>
>>>>> 2014-02-01 14:06:37,385 [ceph_deploy.gatherkeys][WARNING] Unable to
>>>>> find
>>>>> /etc/ceph/ceph.client.admin.keyring on ['ceph1']
>>>>> 2014-02-01 14:06:37,391 [ceph_deploy.gatherkeys][DEBUG ] Have
>>>>> ceph.mon.keyring
>>>>> 2014-02-01 14:06:37,398 [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1
>>>>> for
>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring
>>>>> 2014-02-01 14:06:37,468 [ceph1][DEBUG ] connected to host: ceph1
>>>>> 2014-02-01 14:06:37,471 [ceph1][DEBUG ] detect platform information
>>>>> from
>>>>> remote host
>>>>> 2014-02-01 14:06:37,506 [ceph1][DEBUG ] detect machine type
>>>>> 2014-02-01 14:06:37,514 [ceph1][DEBUG ] fetch remote file
>>>>>
>>>>> 2014-02-01 14:06:37,516 [ceph_deploy.gatherkeys][WARNING] Unable to
>>>>> find
>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring on ['ceph1']
>>>>> 2014-02-01 14:06:37,523 [ceph_deploy.gatherkeys][DEBUG ] Checking ceph1
>>>>> for
>>>>> /var/lib/ceph/bootstrap-mds/ceph.keyring
>>>>> 2014-02-01 14:06:37,591 [ceph1][DEBUG ] connected to host: ceph1
>>>>> 2014-02-01 14:06:37,594 [ceph1][DEBUG ] detect platform information
>>>>> from
>>>>> remote host
>>>>> 2014-02-01 14:06:37,627 [ceph1][DEBUG ] detect machine type
>>>>> 2014-02-01 14:06:37,636 [ceph1][DEBUG ] fetch remote file
>>>>>
>>>>> 2014-02-01 14:06:37,639 [ceph_deploy.gatherkeys][WARNING] Unable to
>>>>> find
>>>>> /var/lib/ceph/bootstrap-mds/ceph.keyring on ['ceph1']
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------------
>>>>
>>>> Does it end here? seems like the output was trimmed...
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> (2014/02/03 22:26), Alfredo Deza wrote:
>>>>>>
>>>>>> On Sun, Feb 2, 2014 at 12:18 AM, Kei.masumoto <kei.masumoto@xxxxxxxxx>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am newbie of ceph, now I am trying to deploy following
>>>>>>> "http://ceph.com/docs/master/start/quick-ceph-deploy/";
>>>>>>> ceph1, ceph2 and ceph3 exists according to the above tutorial. I got
>>>>>>> a
>>>>>>> WARNING message when I exec ceph-deploy "mon create-initial".
>>>>>>>
>>>>>>> 2014-02-01 14:06:37,385 [ceph_deploy.gatherkeys][WARNING] Unable to
>>>>>>> find
>>>>>>> /etc/ceph/ceph.client.admin.keyring on ['ceph1']
>>>>>>> 2014-02-01 14:06:37,516 [ceph_deploy.gatherkeys][WARNING] Unable to
>>>>>>> find
>>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring on ['ceph1']
>>>>>>> 2014-02-01 14:06:37,639 [ceph_deploy.gatherkeys][WARNING] Unable to
>>>>>>> find
>>>>>>> /var/lib/ceph/bootstrap-mds/ceph.keyring on ['ceph1']
>>>>>>>
>>>>>>> Thinking about when those 3 keyrings should be created, I thins
>>>>>>> "ceph-deploy mon create " is a right timing for keyring creation. I
>>>>>>> checked my environment, and found
>>>>>>> /etc/ceph/ceph.client.admin.keyring.14081.tmp. It looks like this
>>>>>>> file
>>>>>>> is created by ceph-create-keys on executing stop ceph-all && start
>>>>>>> ceph-all. but ceph-create-keys never finishes.
>>>>>>
>>>>>> ceph-deploy tries to help here a lot with create-initial, and although
>>>>>> the warnings are useful,
>>>>>> they are only good depending on the context of the rest of the output.
>>>>>>
>>>>>> When the whole process completes, does ceph-deploy say all mons are up
>>>>>> and running?
>>>>>>
>>>>>> It would be better to paste the complete output of the call so we can
>>>>>> see the details.
>>>>>>>
>>>>>>> When I execute ceph-create-keys manually, it continues to generate
>>>>>>> below
>>>>>>> log, looks like waiting reply...
>>>>>>>
>>>>>>> 2014-02-01 20:13:02.847737 7f55e81a4700 0 -- :/1001774 >>
>>>>>>> 192.168.11.8:6789/0 pipe(0x7f55e4024400 sd=3 :0 s=1 pgs=0 cs=0 l=1
>>>>>>> c=0x7f55e4024660).fault
>>>>>>>
>>>>>>> Since I found that mon listens 6789, so I strace mon, then mon also
>>>>>>> waiting something...
>>>>>>>
>>>>>>> root@ceph1:~/src/ceph-0.56.7# strace -p 1047
>>>>>>> Process 1047 attached - interrupt to quit
>>>>>>> futex(0x7f37c14839d0, FUTEX_WAIT, 1102, NULL
>>>>>>>
>>>>>>> I have no idea what situation should be, any hints?
>>>>>>>
>>>>>>> P.S. somebody give me an adivce to check below, but I dont see any
>>>>>>> from
>>>>>>> here.
>>>>>>> root@ceph1:~/my-cluster# ceph daemon mon.`hostname` mon_status
>>>>>>> { "name": "ceph1",
>>>>>>> "rank": 0,
>>>>>>> "state": "leader",
>>>>>>> "election_epoch": 1,
>>>>>>> "quorum": [
>>>>>>> 0],
>>>>>>> "outside_quorum": [],
>>>>>>> "extra_probe_peers": [],
>>>>>>> "sync_provider": [],
>>>>>>> "monmap": { "epoch": 1,
>>>>>>> "fsid": "26835656-6b29-455d-9d1f-545cad8f1e23",
>>>>>>> "modified": "0.000000",
>>>>>>> "created": "0.000000",
>>>>>>> "mons": [
>>>>>>> { "rank": 0,
>>>>>>> "name": "ceph1",
>>>>>>> "addr": "192.168.111.11:6789\/0"}]}}
>>>>>>>
>>>>>>>
>>>>>>> Kei
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>
>>>>>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
