Well I dont use either the dashboard or the cephadm/containerized deployment but do use ceph-iscsi. The fact that your two gateways are not "up" might indicate that they havent been added to the target IQN yet. Once you can get into gwcli and create an iqn and associate your gateways with it, I guess they will report as "up" On the filesystem ceph-iscsi uses /etc/ceph/iscsi-gateway.cfg to store things lik the api password / trusted_ip_list etc. Maybe check this in the container and see if it is being configured correctly. Are you running gwcli on the server where the iscsi gateway is running? It wants to connect to the rbd-target-api on port 5000 via localhost. The error you reported "Unable to access the configuration object" also sounds like a cephx inability to access the configuration rados object at rbd/gateway.conf By default it would want to use the admin keyring unless otherwise specified in the .cfg with the gateway_keyring option. Hope that helps in some way. Respectfully, *Wes Dillingham* wes@xxxxxxxxxxxxxxxxx LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, May 17, 2022 at 1:45 PM Erik Andersen <eandersen@xxxxxxxx> wrote: > I am attempting to set up a 3 node Ceph cluster using Ubuntu server > 22.04LTS, and the Cehpadm deployment tool. > > 3 times I've succeeded in setting up ceph itself, getting the cluster > healthy, and OSDs all set up. The nodes (all monitors) are at > 192.168.122.3, 192.168.122.4, and 192.168.122.5. All nodes have a second > "backend" network on separate interface in the 10.0.0.3-10.0.0.5 range. > > I then create a RBD pool called "rbd". > > All is healthy with the cluster per the dashboard up to this point. > > I then try to set up iSCSI gateways on 192.168.122.3 and 192.168.122.5, > following these directions: > https://docs.ceph.com/en/pacific/cephadm/services/iscsi/ > > That means doing `cephadm shell`, getting the `iscsi.yaml` file into the > docker container (with echo since there seems to be no text editors > available) and then running their recommended deployment command of `ceph > orch apply -i iscsi.yaml`. The yaml file has in it: > > service_type: iscsi > service_id: iscsi > placement: > hosts: > - ceph1 > - ceph3 > spec: > pool: rbd # RADOS pool where ceph-iscsi config data is stored. > trusted_ip_list: > "192.168.122.3,192.168.122.5,10.0.0.3,10.0.0.5,192.168.122.4,10.0.0.4" > > I then get in the dashboard status page there there are 2 iSCSI gateways > configured, but down. > https://i.stack.imgur.com/wr619.png > > In services, it shows that the services are running: > > https://i.stack.imgur.com/PwSik.png > > > On the iSCSI gateways page it shows this: > > (Ceph dashboard iSCSI gateway page showing both gateways down) > https://i.stack.imgur.com/Se2Mv.png > > > Looking on one of the node's containers, it does look like cephadm > started/deployed containers for this (apologies in advance for the horrible > email formatting of a console table - look at the first two): > > root@ceph1:~# docker ps > CONTAINER ID IMAGE COMMAND > CREATED STATUS PORTS NAMES > cefaf78b98ee quay.ceph.io/ceph-ci/ceph > "/usr/bin/rbd-target…" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-iscsi-iscsi-ceph1-alnale > b405b321bd6a quay.ceph.io/ceph-ci/ceph > "/usr/bin/tcmu-runner" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-iscsi-iscsi-ceph1-alnale-tcmu > a05af7ac9609 quay.io/prometheus/prometheus:v2.33.4 > "/bin/prometheus --c…" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-prometheus-ceph1 > 4699606a7878 quay.io/prometheus/alertmanager:v0.23.0 > "/bin/alertmanager -…" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-alertmanager-ceph1 > 103abafd0c19 quay.ceph.io/ceph-ci/ceph > "/usr/bin/ceph-osd -…" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-osd-2 > adcad13a1dcb quay.ceph.io/ceph-ci/ceph > "/usr/bin/ceph-osd -…" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-osd-0 > 9626b0794794 quay.io/ceph/ceph-grafana:8.3.5 "/bin/sh -c > 'grafana…" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-grafana-ceph1 > 9a717edbf83f quay.io/prometheus/node-exporter:v1.3.1 > "/bin/node_exporter …" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-node-exporter-ceph1 > c1c52d37baf1 quay.ceph.io/ceph-ci/ceph > "/usr/bin/ceph-crash…" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-crash-ceph1 > f6b2c9fef7e9 quay.ceph.io/ceph-ci/ceph:master > "/usr/bin/ceph-mgr -…" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-mgr-ceph1-mpqrst > 8889082c55b1 quay.ceph.io/ceph-ci/ceph:master > "/usr/bin/ceph-mon -…" About an hour ago Up About an hour > ceph-9f724dc4-d2de-11ec-b7be-8f11f39bf88a-mon-ceph1 > > > I have tried a fresh redeployment with setting the API username and > password in the iscsi.yaml file to the same as the main Ceph dashboard > login, but that just gave 500 errors when trying to got to the iSCSI > gateway page. > > I have also tried setting the dashboard not to verify SSL certs as I don't > have any signed ones: > > root@ceph1:/# ceph dashboard set-iscsi-api-ssl-verification false > Option ISCSI_API_SSL_VERIFICATION updated > > I have also tried looking at the URLs it is using, and going to them in a > browser. I get 404 errors when trying to go to those in a browser, but that > may be normal if it's an API base URL: > > root@ceph1:/# ceph dashboard iscsi-gateway-list > {"gateways": {"ceph1": {"service_url":" > http://admin:admin@192.168.122.3:5000"}, "ceph3": {"service_url":" > http://admin:admin@192.168.122.5:5000"}}} > > (The above had the admin username and password for the main dashboard > encoded into the URL on a previous deployment where I set the api_user and > api_password is the iscsi.yaml file. > > Doing `cephadm shell` and then trying `gwcli` gives: > > REST API failure, code : 500 > Unable to access the configuration object > Unable to contact the local API endpoint (https://localhost:5000/api) > > Trying the API with wget from one of the nodes (so we are coming from a > trusted IP) gives: > > wgethttp://192.168.122.3:5000/api/targets --user "admin" --password > "admin" > --2022-05-13 19:32:54--http://192.168.122.3:5000/api/targets > Connecting to 192.168.122.3:5000... connected. > HTTP request sent, awaiting response... 401 UNAUTHORIZED > Unknown authentication scheme. > > Username/Password Authentication Failed. > The same occurrs whether I use empty username and password, the main ceph > username and password, or the (above) admin/admin. > > How can I get these iSCSI gateways to come up? How do I troubleshoot this > from here? What can I check to get more information on what is going off > the rails here? Is it just that the dashboard is having trouble, but it > would work from CLI (if so, what commands can I use (and where do I run > them with all these docker containers) to see status and set up ISCSI > targets?) > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |