Re: lack of RGW_API_HOST in ceph dashboard, 17.2.6, causes ceph mgr dashboard problems

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


 I solved this.
It has multiple layers.
1. RGW_API_HOST is no longer available in 17.2.6 as a configuration option for the ceph mgr. (I was wrong below when I said it could be queried on an *upgraded* host with:
# ceph dashboard get-rgw-api-host
You *can* query it with:
# ceph config dump | grep mgr | grep dashboard
But because the mgr doesn't use it, it's existence in the config doesn't matter.
My setup uses a DNS record that points to multiple IP addresses for the rgw servers. This record is, and these IP numbers are VIPs that are controlled by keepalived, to share them amongst the radosgw servers in the case one of the radosgw servers dies, one of the others will answer for it.
But each radosgw server itself has a name similar to:,,, in DNS each of these points to the IP of the actual server, (not the VIPs above).
In 17.2.6, I *can* do this:

$ aws --endpoint s3 ls s3://mybucket

But, in 17.2.6, I can no longer do:
$ aws --endpoint s3 ls s3://mybucket

It returns a "NoSuchBucket" error with a 403 error.
This looks to be the same error that the ceph mgr is returning in the GUI, as it tries to query the individual radosgw server name, NOT the which it was before the upgrade to 17.2.6 (It was using the RGW_API_HOST which I have set to
In the radosgw zonegroup, (radosgw-admin zonegroup get) I have:
"endpoints": [
"hostnames": [   ""]

and in the individual zone definition I have:
"endpoints": [   "";]
The code in has this section:
    if (subdomain.empty()
        && (domain.empty() || domain !=
        && !looks_like_ip_address(
        && RGWHandler_REST::validate_bucket_name( == 0
        && !(hostnames_set.empty() && hostnames_s3website_set.empty())) {
      in_hosted_domain = 1;

and later this:
    if (in_hosted_domain && !subdomain.empty()) {
      string encoded_bucket = "/";
      if (s->info.request_uri[0] != '/')
      s->info.request_uri = encoded_bucket;

In my situation, this ends up with the bucket name being changed, resulting in the NoSuchBucket error.
The way to fix this is to have "rgw dns name" set separately for each rgw host in ceph.conf and restarting the radosgw servers. 
That way, the rgw hostname is set and the bucket name does not get changed. It appears that in my setup, without "rgw dns name" set,along with the other settings I have, radosgw ASSUMES? we are using the newer domain style bucket names (we are not), and thatthe radosgw servers no longer recognize (with my settings) their own server name as valid. Adding "rgw dns name" for each server to ceph.confallows the aws cli to work with --endpoint set to the individual server name and for the ceph-mgr to work when querying radosgw information.

In pacific 16.x.y this was not a problem.

    On Wednesday, August 30, 2023 at 05:08:36 AM MDT, Eugen Block <eblock@xxxxxx> wrote:  

there have been multiple discussions on this list without any  
satisfying solution for all possible configurations. One of the  
changes [1] made in Pacific was to use hostname instead of IP, but it  
only uses the shortname (you can check the "hostname" in 'ceph service  
dump' output. But this seems to only impact the dashboard access if  
you have ssl-verify set to true. I'm still waiting for a solution as  
well for a customer cluster which uses wildcard certificates only,  
until then we let ssl-verify disabled. But I didn't check the tracker  
for any pending tickets, so someone might be working on it.



Zitat von Christopher Durham <caduceus42@xxxxxxx>:

> Hi,
> I am using 17.2.6 on Rocky Linux 8
> The ceph mgr dashboard, in my situation, (bare metal install,  
> upgraded from 15->16-> 17.2.6), can no longer hit the  
> ObjectStore->(Daemons,Users,Buckets) pages.
> When I try to hit those pages, it gives an error:
> RGW REST API failed request with status code 403 {"Code":  
> "AccessDenied", RequestId: "xxxxxxx", HostId: "yyyy-<my zone>"}
> The log of the rgw server it hit has:
> "GET /admin/metadata/user?myself HTTP/1.1" 403 125
> It appears that the mgr dashboard setting RGW_API_HOST is no longer  
> an option that can be set, nor does that name exist anywhere under  
> /usr/share/ceph/mgr/dashboard, and:
> # ceph dashboard set-rgw-api-host <host>
> is no longer in existence in 17.2.6
> However, since my situation is an upgrade, the config value still  
> exists in my config, and I can retrieve it with:
> # ceph dashboard get-rgw-api-host
> To get the  to work in my situation, I have modified  
> /usr/share/ceph/mgr/dashboard/ and re-added RGW_API_HOST  
> to the Options class using
> RGW_API_HOST = Settings('', [dict,str])
> I then modified  
> /usr/share/ceph/mgr/dashboard/services/ such that each  
> rgw daemon retrieved has its 'host' member set to  
> Settings.RGW_API_HOST.
> Then after restarting the mgr, I was able to access the  
> Objectstore->(Daemons,Users,Buckets) pages in the dashboard.
> HOWEVER, I know this is NOT the right way to fix this, it is a hack.  
> It seems like the dashboard is trying to contact an rgw server  
> individually. For us, the RGW_API_HOST is
> a name in DNS:, that has multiple A records, one for each  
> of our rgw servers, each of which have the *same* SSL cert with CN  
> and SubjectAltNames that allow
> the cert to present itself as both as well as the  
> individual host name (SubjectAltName has ALL the rgw servers in it).  
> This works well for us and has
> done so since 15.x.y, The endpoint for the zone is set to  
> Thus my users only have a single endpoint to care about, unless  
> there is a failure situation onan rgw server. (We have other ways of  
> handling that).
> Any thoughts on the CORRECT way to handle this so I can have the  
> ceph dashboard work with the ObjectStore->(Daemons,Users,Buckets)  
> pages? Thanks.
> -Chris
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx

ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux