iSCSI - failed, gateway(s) unavailable UNKNOWN

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I’m new to Ceph and have just deployed my first cluster using ceph-ansible and running into some issues I was hoping someone could point me in the right direction. 

I have 5 servers to start with. 5 OSD, 3 Monitors, 2 Manager, 4 iSCSI gateway. My intention is to use this environment as iSCSI storage for ESXi. 

When I went to add an iSCSI Target, I got an error that popped up in the browser (went away to fast to read it) but after this, now if I click on the “Targets” tab in Dashboard for iSCSI, I just continuously get errors popping up: 

500 - Internal Server Error
The server encountered an unexpected condition which prevented it from fulfilling the request.

I’m unable to add any targets now. The log files show this: 

2021-05-19T12:58:05.294-0400 7f2db6115700  0 [dashboard ERROR exception] Internal Server Error
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 694, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 907, in wrapper
    return func(*vpath, **params)
  File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 266, in list
    IscsiTarget._set_info(target)
  File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 990, in _set_info
    raise e
  File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 980, in _set_info
    target_iqn)
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 531, in func_wrapper
    **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_client.py", line 254, in get_targetinfo
    return request()
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 326, in __call__
    data, raw_content, headers)
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 449, in do_request
    resp.content)
dashboard.rest_client.RequestException: iscsi REST API failed request with status code 503
(b'{\n  "message": "failed, gateway(s) unavailable:cxcto-c240-j27-01(UNKNOWN'
 b' state)"\n}\n')
2021-05-19T12:58:05.294-0400 7f2db6115700  0 [dashboard ERROR request] [10.117.244.166:58270] [GET] [500] [0.398s] [admin] [513.0B] /api/iscsi/target
2021-05-19T12:58:05.294-0400 7f2db6115700  0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "7bc691e8-e814-48c1-a69c-57ea1f3f3dbd"}                                                                                                                                                                                                                                                                                                                        ‘]

Clearly it’s showing an error indicating that the gateway is “unavailable”. 

The output from gwcli shows this: 

[root@cxcto-c240-j27-01 ~]# ./gwcli

1 gateway is inaccessible - updates will be disabled
/> ls
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph ............................................................................................................ [HEALTH_OK]
  |   o- pools .......................................................................................................... [Pools: 4]
  |   | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/15911599M (0%), Used: 0.00Y]
  |   | o- iscsi ............................................................... [(x3), Commit: 0.00Y/15911599M (0%), Used: 262511b]
  |   | o- rbd ................................................................... [(x3), Commit: 0.00Y/15911599M (0%), Used: 1650b]
  |   | o- test .................................................................. [(x3), Commit: 0.00Y/15911599M (0%), Used: 0.00Y]
  |   o- topology .............................................................................................. [OSDs: 110,MONs: 3]
  o- disks ....................................................................................................... [0.00Y, Disks: 0]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: CHAP, Targets: 1]
    o- iqn.2001-07.com.ceph:1621437640904 ................................................................ [Auth: None, Gateways: 1]
      o- disks .......................................................................................................... [Disks: 0]
      o- gateways ............................................................................................ [Up: 0/1, Portals: 1]
      | o- cxcto-c240-j27-01 ....................................................................... [10.122.242.196 (UNAUTHORIZED)]
      o- host-groups .................................................................................................. [Groups : 0]
      o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 0]


In this case it says “UNAUTHORIZED” 

When I created the target, I had selected all 4 iSCSI gateways, but it looks like something happened during the addition process that has left it in a weird state. 

The dashboard seems to think that all the gateways are up and running :



Notice that it only shows the target on the one node which it is reporting an error. 

I’ve tried deleting the target from gwcli but that fails and I’m not really sure where to look next. The “UNAUTHORIZED” in the gwcli output makes me wonder if there is some kind of authorization issue, but I’m not sure what that would be. 

This is what I see if I navigate to the Targets tab on the iSCSI page: 



Any thoughts or guidance are greatly appreciated. 

-Paul





_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux