Hi Dulux-oz,
Are you able to share the output of "gwcli -d" from your iSCSI node?
Just a few things I can think to check off the top of my head, is port 5000 accessible/opened on the node running iSCSI?
I think by default the API tries to listen/use a pool called rbd, so does your cluster have a pool named that? It looks like it does based on your logs but something to check anyways, otherwise I believe you can change the pool it uses within the iscsi-gateway.cfg file though.
If there's any blocklisted OSDs on the node you're running iSCSI on it will also prevent rbd-target-api from starting I have found from experience, but again per your logs it looks like there isn't any.
Just in case it might help I've also attached an iscsi-gateway-cfg file from a cluster we've got with it working here:
# This is seed configuration used by the ceph_iscsi_config modules
# when handling configuration tasks for iscsi gateway(s)
#
# Please do not change this file directly since it is managed by Ansible and will be overwritten
[config]
api_password = admin
api_port = 5000
# API settings.
# The API supports a number of options that allow you to tailor it to your
# local environment. If you want to run the API under https, you will need to
# create cert/key files that are compatible for each iSCSI gateway node, that is
# not locked to a specific node. SSL cert and key files *must* be called
# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory
# on *each* gateway node. With the SSL files in place, you can use 'api_secure = true'
# to switch to https mode.
# To support the API, the bear minimum settings are:
api_secure = False
# Optional settings related to the CLI/API service
api_user = admin
cluster_name = ceph
loop_delay = 1
pool = rbd
trusted_ip_list = X.X.X.X,X.X.X.X,X.X.X.X,X.X.X.X
-----Original Message-----
From: duluxoz <duluxoz@xxxxxxxxx>
Sent: September 7, 2022 6:38 AM
To: ceph-users@xxxxxxx
Subject: Ceph iSCSI rbd-target.api Failed to Load
Hi All,
I've followed the instructions on the CEPH Doco website on Configuring the iSCSI Target. Everything went AOK up to the point where I try to start the rbd-target-api service, which fails (the rbd-target-gw service started OK).
A `systemctl status rbd-target-api` gives:
~~~
rbd-target-api.service - Ceph iscsi target configuration API
Loaded: loaded (/usr/lib/systemd/system/rbd-target-api.service;
enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2022-09-07 18:07:26 AEST; 1h 5min ago
Process: 32547 ExecStart=/usr/bin/rbd-target-api (code=exited, status=16)
Main PID: 32547 (code=exited, status=16)
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]:
rbd-target-api.service: Start request repeated too quickly.
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]:
rbd-target-api.service: Failed with result 'exit-code'.
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]: Failed to start Ceph iscsi target configuration API.
~~~
A `journalctl -xe` gives:
~~~
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]:
rbd-target-api.service: Start request repeated too quickly.
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]:
rbd-target-api.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit rbd-target-api.service has entered the 'failed' state with result 'exit-code'.
Sep 07 19:19:03 ceph-host1.mydomain.local systemd[1]: Failed to start Ceph iscsi target configuration API.
-- Subject: Unit rbd-target-api.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rbd-target-api.service has failed.
--
-- The result is failed.
~~~
The `rbd-target-api.log` gives:
~~~
2022-09-07 19:19:01,084DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:01,086DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:01,105DEBUG [common.py:438:init_config()] -
(init_config) using pre existing config object
2022-09-07 19:19:01,105DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:01,105DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:01,106DEBUG [common.py:120:_read_config_object()] - _read_config_object reading the config object
2022-09-07 19:19:01,107DEBUG [common.py:170:_get_ceph_config()] -
(_get_rbd_config) config object contains 'b'{\n"created": "2022/09/07
07:25:58",\n"discovery_auth": {\n"mutual_password":
"",\n"mutual_password_encryption_enabled": false,\n"mutual_username":
"",\n"password": "",\n"password_encryption_enabled": false,\n"username":
""\n},\n"disks": {},\n"epoch": 0,\n"gateways": {},\n"targets":
{},\n"updated": "",\n"version": 11\n}''
2022-09-07 19:19:01,107 INFO [rbd-target-api:2810:run()] - Started the configuration object watcher
2022-09-07 19:19:01,107 INFO [rbd-target-api:2812:run()] - Checking for config object changes every 1s
2022-09-07 19:19:01,109 INFO [gateway.py:82:osd_blocklist_cleanup()] - Processing osd blocklist entries for this node
2022-09-07 19:19:01,497 INFO [gateway.py:140:osd_blocklist_cleanup()] - No OSD blocklist entries found
2022-09-07 19:19:01,497 INFO [gateway.py:250:define()] - Reading the configuration object to update local LIO configuration
2022-09-07 19:19:01,497 INFO [gateway.py:261:define()] - Configuration does not have an entry for this host(ceph-host1.mydomain.local) - nothing to define to LIO
2022-09-07 19:19:01,507 CRITICAL [rbd-target-api:2942:main()] - Secure API requested but the crt/key files missing/incompatible?
2022-09-07 19:19:01,508 CRITICAL [rbd-target-api:2944:main()] - Unable to start
2022-09-07 19:19:01,956DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:01,958DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:01,976DEBUG [common.py:438:init_config()] -
(init_config) using pre existing config object
2022-09-07 19:19:01,976DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:01,976DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:01,977DEBUG [common.py:120:_read_config_object()] - _read_config_object reading the config object
2022-09-07 19:19:01,978DEBUG [common.py:170:_get_ceph_config()] -
(_get_rbd_config) config object contains 'b'{\n"created": "2022/09/07
07:25:58",\n"discovery_auth": {\n"mutual_password":
"",\n"mutual_password_encryption_enabled": false,\n"mutual_username":
"",\n"password": "",\n"password_encryption_enabled": false,\n"username":
""\n},\n"disks": {},\n"epoch": 0,\n"gateways": {},\n"targets":
{},\n"updated": "",\n"version": 11\n}''
2022-09-07 19:19:01,979 INFO [rbd-target-api:2810:run()] - Started the configuration object watcher
2022-09-07 19:19:01,979 INFO [rbd-target-api:2812:run()] - Checking for config object changes every 1s
2022-09-07 19:19:01,980 INFO [gateway.py:82:osd_blocklist_cleanup()] - Processing osd blocklist entries for this node
2022-09-07 19:19:02,367 INFO [gateway.py:140:osd_blocklist_cleanup()] - No OSD blocklist entries found
2022-09-07 19:19:02,368 INFO [gateway.py:250:define()] - Reading the configuration object to update local LIO configuration
2022-09-07 19:19:02,368 INFO [gateway.py:261:define()] - Configuration does not have an entry for this host(ceph-host1.mydomain.local) - nothing to define to LIO
2022-09-07 19:19:02,378 CRITICAL [rbd-target-api:2942:main()] - Secure API requested but the crt/key files missing/incompatible?
2022-09-07 19:19:02,379 CRITICAL [rbd-target-api:2944:main()] - Unable to start
2022-09-07 19:19:02,960DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:02,962DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:02,980DEBUG [common.py:438:init_config()] -
(init_config) using pre existing config object
2022-09-07 19:19:02,980DEBUG [common.py:141:_open_ioctx()] -
(_open_ioctx) Opening connection to rbd pool
2022-09-07 19:19:02,980DEBUG [common.py:148:_open_ioctx()] -
(_open_ioctx) connection opened
2022-09-07 19:19:02,981DEBUG [common.py:120:_read_config_object()] - _read_config_object reading the config object
2022-09-07 19:19:02,982DEBUG [common.py:170:_get_ceph_config()] -
(_get_rbd_config) config object contains 'b'{\n"created": "2022/09/07
07:25:58",\n"discovery_auth": {\n"mutual_password":
"",\n"mutual_password_encryption_enabled": false,\n"mutual_username":
"",\n"password": "",\n"password_encryption_enabled": false,\n"username":
""\n},\n"disks": {},\n"epoch": 0,\n"gateways": {},\n"targets":
{},\n"updated": "",\n"version": 11\n}''
2022-09-07 19:19:02,983 INFO [rbd-target-api:2810:run()] - Started the configuration object watcher
2022-09-07 19:19:02,983 INFO [rbd-target-api:2812:run()] - Checking for config object changes every 1s
2022-09-07 19:19:02,985 INFO [gateway.py:82:osd_blocklist_cleanup()] - Processing osd blocklist entries for this node
2022-09-07 19:19:03,370 INFO [gateway.py:140:osd_blocklist_cleanup()] - No OSD blocklist entries found
2022-09-07 19:19:03,371 INFO [gateway.py:250:define()] - Reading the configuration object to update local LIO configuration
2022-09-07 19:19:03,371 INFO [gateway.py:261:define()] - Configuration does not have an entry for this host(ceph-host1.mydomain.local) - nothing to define to LIO
2022-09-07 19:19:03,381 CRITICAL [rbd-target-api:2942:main()] - Secure API requested but the crt/key files missing/incompatible?
2022-09-07 19:19:03,381 CRITICAL [rbd-target-api:2944:main()] - Unable to start ~~~
This `iscsi-gateway.cfg` file reads:
~~~
[config]
cluster_name = ceph
gateway_keyring = ceph.client.admin.keyring
# API settings.
# The api supports a number of options that allow you to tailor it to your # local environment. If you want to run the api under https, you will need to # create crt/key files that are compatible for each gateway node (i.e. not # locked to a specific node). SSL crt and key files *must* be called # iscsi-gateway.crt and iscsi-gateway.key and placed in /etc/ceph on *each* # gateway node. With the SSL files in place, you can use api_secure = true # to switch to https mode.
api_secure = false
# Additional API configuration options are as follows (defaults shown); # api_user = admin # api_password = admin # api_port = 5000 trusted_ip_list = 192.168.1.101,192.168.1.101,192.168.1.101
# Refer to the ceph-iscsi-config/settings module for more options ~~~
192.168.1.101 is the ip address for the ceph-host1.mydomain.local (and similarly for the other ip addresses / hostnames of the other 2 nodes, which are yet to be installed).
The iSCSI Node is co-located on an OSD Node.
The Cluster is working (apart form the iSCSI part, of course).
So, could someone be kind enough to point out what I'm missing (ie what's wrong)? - Thanks in advance
Dulux-Oz
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
