Hi Dulux-oz,
Are you able to share the output of "gwcli -d" from your iSCSI node?
Just a few things I can think to check off the top of my head, is
port 5000 accessible/opened on the node running iSCSI?
I think by default the API tries to listen/use a pool called rbd, so
does your cluster have a pool named that? It looks like it does
based on your logs but something to check anyways, otherwise I
believe you can change the pool it uses within the iscsi-gateway.cfg
file though.
If there's any blocklisted OSDs on the node you're running iSCSI on
it will also prevent rbd-target-api from starting I have found from
experience, but again per your logs it looks like there isn't any.
Just in case it might help I've also attached an iscsi-gateway-cfg
file from a cluster we've got with it working here:
# This is seed configuration used by the ceph_iscsi_config modules #
when handling configuration tasks for iscsi gateway(s) # # Please do
not change this file directly since it is managed by Ansible and will
be overwritten [config] api_password = admin api_port = 5000 # API
settings.
# The API supports a number of options that allow you to tailor it to
your # local environment. If you want to run the API under https, you
will need to # create cert/key files that are compatible for each
iSCSI gateway node, that is # not locked to a specific node. SSL cert
and key files *must* be called # 'iscsi-gateway.crt' and
'iscsi-gateway.key' and placed in the '/etc/ceph/' directory # on
*each* gateway node. With the SSL files in place, you can use
'api_secure = true'
# to switch to https mode.
# To support the API, the bear minimum settings are:
api_secure = False
# Optional settings related to the CLI/API service api_user = admin
cluster_name = ceph loop_delay = 1 pool = rbd trusted_ip_list =
X.X.X.X,X.X.X.X,X.X.X.X,X.X.X.X
-----Original Message-----
From: duluxoz <duluxoz@xxxxxxxxx>
Sent: September 7, 2022 6:38 AM
To: ceph-users@xxxxxxx
Subject: Ceph iSCSI rbd-target.api Failed to Load
Hi All,
I've followed the instructions on the CEPH Doco website on
Configuring the iSCSI Target. Everything went AOK up to the point
where I try to start the rbd-target-api service, which fails (the
rbd-target-gw service started OK).
A `systemctl status rbd-target-api` gives:
~~~
rbd-target-api.service - Ceph iscsi target configuration API
Loaded: loaded (/usr/lib/systemd/system/rbd-target-api.service;
enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2022-09-07
18:07:26 AEST; 1h 5min ago
Process: 32547 ExecStart=/usr/bin/rbd-target-api (code=exited,
status=16)
Main PID: 32547 (code=exited, status=16)
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]:
rbd-target-api.service: Start request repeated too quickly.
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]:
rbd-target-api.service: Failed with result 'exit-code'.
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]: Failed to
start Ceph iscsi target configuration API.
~~~
A `journalctl -xe` gives:
~~~
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]:
rbd-target-api.service: Start request repeated too quickly.
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]:
rbd-target-api.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support:
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit rbd-target-api.service has entered the 'failed' state
with result 'exit-code'.
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]: Failed to
start Ceph iscsi target configuration API.
-- Subject: Unit rbd-target-api.service has failed
-- Defined-By: systemd
-- Support:
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rbd-target-api.service has failed.
--
-- The result is failed.
~~~
The `rbd-target-api.log` gives:
~~~
2022-09-07 19:19:01,084DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:01,086DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:01,105DEBUG [common.py:438:init_config()] -
(init_config) using pre existing config object
2022-09-07 19:19:01,105DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:01,105DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:01,106DEBUG [common.py:120:_read_config_object()] -
_read_config_object reading the config object
2022-09-07 19:19:01,107DEBUG [common.py:170:_get_ceph_config()] -
(_get_rbd_config) config object contains 'b'{\n"created": "2022/09/07
07:25:58",\n"discovery_auth": {\n"mutual_password":
"",\n"mutual_password_encryption_enabled": false,\n"mutual_username":
"",\n"password": "",\n"password_encryption_enabled":
false,\n"username":
""\n},\n"disks": {},\n"epoch": 0,\n"gateways": {},\n"targets":
{},\n"updated": "",\n"version": 11\n}''
2022-09-07 19:19:01,107 INFO [rbd-target-api:2810:run()] - Started the
configuration object watcher
2022-09-07 19:19:01,107 INFO [rbd-target-api:2812:run()] - Checking
for config object changes every 1s
2022-09-07 19:19:01,109 INFO [gateway.py:82:osd_blocklist_cleanup()] -
Processing osd blocklist entries for this node
2022-09-07 19:19:01,497 INFO [gateway.py:140:osd_blocklist_cleanup()]
- No OSD blocklist entries found
2022-09-07 19:19:01,497 INFO [gateway.py:250:define()] - Reading the
configuration object to update local LIO configuration
2022-09-07 19:19:01,497 INFO [gateway.py:261:define()] - Configuration
does not have an entry for this host(ceph-host1.mydomain.local) -
nothing to define to LIO
2022-09-07 19:19:01,507 CRITICAL [rbd-target-api:2942:main()] -
Secure API requested but the crt/key files missing/incompatible?
2022-09-07 19:19:01,508 CRITICAL [rbd-target-api:2944:main()] - Unable
to start
2022-09-07 19:19:01,956DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:01,958DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:01,976DEBUG [common.py:438:init_config()] -
(init_config) using pre existing config object
2022-09-07 19:19:01,976DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:01,976DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:01,977DEBUG [common.py:120:_read_config_object()] -
_read_config_object reading the config object
2022-09-07 19:19:01,978DEBUG [common.py:170:_get_ceph_config()] -
(_get_rbd_config) config object contains 'b'{\n"created": "2022/09/07
07:25:58",\n"discovery_auth": {\n"mutual_password":
"",\n"mutual_password_encryption_enabled": false,\n"mutual_username":
"",\n"password": "",\n"password_encryption_enabled":
false,\n"username":
""\n},\n"disks": {},\n"epoch": 0,\n"gateways": {},\n"targets":
{},\n"updated": "",\n"version": 11\n}''
2022-09-07 19:19:01,979 INFO [rbd-target-api:2810:run()] - Started the
configuration object watcher
2022-09-07 19:19:01,979 INFO [rbd-target-api:2812:run()] - Checking
for config object changes every 1s
2022-09-07 19:19:01,980 INFO [gateway.py:82:osd_blocklist_cleanup()] -
Processing osd blocklist entries for this node
2022-09-07 19:19:02,367 INFO [gateway.py:140:osd_blocklist_cleanup()]
- No OSD blocklist entries found
2022-09-07 19:19:02,368 INFO [gateway.py:250:define()] - Reading the
configuration object to update local LIO configuration
2022-09-07 19:19:02,368 INFO [gateway.py:261:define()] - Configuration
does not have an entry for this host(ceph-host1.mydomain.local) -
nothing to define to LIO
2022-09-07 19:19:02,378 CRITICAL [rbd-target-api:2942:main()] -
Secure API requested but the crt/key files missing/incompatible?
2022-09-07 19:19:02,379 CRITICAL [rbd-target-api:2944:main()] - Unable
to start
2022-09-07 19:19:02,960DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:02,962DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:02,980DEBUG [common.py:438:init_config()] -
(init_config) using pre existing config object
2022-09-07 19:19:02,980DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:02,980DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:02,981DEBUG [common.py:120:_read_config_object()] -
_read_config_object reading the config object
2022-09-07 19:19:02,982DEBUG [common.py:170:_get_ceph_config()] -
(_get_rbd_config) config object contains 'b'{\n"created": "2022/09/07
07:25:58",\n"discovery_auth": {\n"mutual_password":
"",\n"mutual_password_encryption_enabled": false,\n"mutual_username":
"",\n"password": "",\n"password_encryption_enabled":
false,\n"username":
""\n},\n"disks": {},\n"epoch": 0,\n"gateways": {},\n"targets":
{},\n"updated": "",\n"version": 11\n}''
2022-09-07 19:19:02,983 INFO [rbd-target-api:2810:run()] - Started the
configuration object watcher
2022-09-07 19:19:02,983 INFO [rbd-target-api:2812:run()] - Checking
for config object changes every 1s
2022-09-07 19:19:02,985 INFO [gateway.py:82:osd_blocklist_cleanup()] -
Processing osd blocklist entries for this node
2022-09-07 19:19:03,370 INFO [gateway.py:140:osd_blocklist_cleanup()]
- No OSD blocklist entries found
2022-09-07 19:19:03,371 INFO [gateway.py:250:define()] - Reading the
configuration object to update local LIO configuration
2022-09-07 19:19:03,371 INFO [gateway.py:261:define()] - Configuration
does not have an entry for this host(ceph-host1.mydomain.local) -
nothing to define to LIO
2022-09-07 19:19:03,381 CRITICAL [rbd-target-api:2942:main()] -
Secure API requested but the crt/key files missing/incompatible?
2022-09-07 19:19:03,381 CRITICAL [rbd-target-api:2944:main()] - Unable
to start ~~~
This `iscsi-gateway.cfg` file reads:
~~~
[config]
cluster_name = ceph
gateway_keyring = ceph.client.admin.keyring
# API settings.
# The api supports a number of options that allow you to tailor it
to your # local environment. If you want to run the api under https,
you will need to # create crt/key files that are compatible for each
gateway node (i.e. not # locked to a specific node). SSL crt and key
files *must* be called # iscsi-gateway.crt and iscsi-gateway.key and
placed in /etc/ceph on *each* # gateway node. With the SSL files in
place, you can use api_secure = true # to switch to https mode.
api_secure = false
# Additional API configuration options are as follows (defaults
shown); # api_user = admin # api_password = admin # api_port = 5000
trusted_ip_list = 192.168.1.101,192.168.1.101,192.168.1.101
# Refer to the ceph-iscsi-config/settings module for more options ~~~
192.168.1.101 is the ip address for the ceph-host1.mydomain.local
(and similarly for the other ip addresses / hostnames of the other 2
nodes, which are yet to be installed).
The iSCSI Node is co-located on an OSD Node.
The Cluster is working (apart form the iSCSI part, of course).
So, could someone be kind enough to point out what I'm missing (ie
what's wrong)? - Thanks in advance
Dulux-Oz
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
email to ceph-users-leave@xxxxxxx
