SOLVED! OK, there was some last-minute flailing around so I can't quite report a cookbook recipe, but it goes something like this: 1. ceph config set client.mousetech rgw_admin_entry admin Note: the standard example is for client.rgw, but I named my RGW "mousetech" to make it distinguishable from possible internal magic names. Tried both "client.mousetech" and "client.rgw" just to be sure. Peeve: the "ceph config" command is silent. You cannot tell if it successfully updated (or added?) a value and when it simply threw away a bad config request. 2. Re-issued "ceph dashboard set-rgw-credentials" just to jam in anything that might not have been set right previously. 3. Restarted the RGW container on the RGW host. I got to the point where I got the "503" error, but still no dashboard even after logging in and out. Noted that the RGW logs were reporting my manual requests, but the dashboard requests weren't showing up. Huh? Head scratching. Realized that I can't get a "404" error unless there's a host on the other end, so SOMETHING was talking and it didn't make any sense that the RGW only logged some requests and not others. Finally got a suspicion and did a "ceph orch ps". Yup, I have TWO RGW servers running now (it was intentional, partly to flush out some of the older failures). So I dialed into the alternate RGW, restarted it and Dashboard Joy! So basically, everything I wanted Ceph to do is working and working clean, from the Ceph filesystem to NFS to VM backing stores to RGW and I'm delirious with joy. Thanks, guys! On Tue, 2023-10-17 at 10:23 -0400, Casey Bodley wrote: > you're right that many docs still mention ceph.conf, after the mimic > release added a centralized config database to ceph-mon. you can read > about the mon-based 'ceph config' commands in > https://docs.ceph.com/en/reef/rados/configuration/ceph-conf/#commands > > to modify rgw_admin_entry for all radosgw instances, you'd use a > command like: > > $ ceph config set client.rgw rgw_admin_entry admin > > then restart radosgws because they only read that value on startup > > On Tue, Oct 17, 2023 at 9:54 AM Tim Holloway <timh@xxxxxxxxxxxxx> > wrote: > > > > Thanks, Casey! > > > > I'm not really certain where to set this option. While Ceph is very > > well-behaved once you know what to do, the nature of Internet-based > > documentation (and occasionally incompletely-updated manuals) is > > that > > stale information is often given equal weight to the obsolete > > information. It's a problem I had as support for JavaServer Faces, > > in > > fact. I spent literally years correcting people who'd got their > > examples from obsoleted sources. > > > > If I was to concoct a "Really, Really Newbies Intro to Ceph" I > > think > > that the two most fundamental items explained would be "Ceph as > > traditional services" versus "Ceph as Containerized services" (As > > far > > as I can tell, both are still viable but containerization - at > > least > > for me - is a preferable approach). And the ceph.conf file versus > > storing operational parameters within Ceph entities (e.g. buckets > > or > > pseudo-buckets like RGW is doing). While lots of stuff still > > reference > > ceph.conf for configuration, I'm feeling like it's actually no > > longer > > authoritative for some options, may be an alternative source for > > others > > (with which source has priority being unclear) and stuff that Ceph > > no > > longer even looks at because it has moved on. > > > > Such is my plight. > > > > I have no problem with making the administrative interface look > > "bucket-like". Or for that matter, having the RGW report it as a > > (missing) bucket if it isn't configured. But knowing where to > > inject > > the magic that activates that interface eludes me and whether to do > > it > > directly on the RGW container hos (and how) or on my master host is > > totally unclear to me. It doesn't help that this is an item that > > has > > multiple values, not just on/off or that by default the docs seem > > to > > imply it should be already preset to standard values out of the > > box. > > > > Thanks, > > Tim > > > > On Tue, 2023-10-17 at 09:11 -0400, Casey Bodley wrote: > > > hey Tim, > > > > > > your changes to rgw_admin_entry probably aren't taking effect on > > > the > > > running radosgws. you'd need to restart them in order to set up > > > the > > > new route > > > > > > there also seems to be some confusion about the need for a bucket > > > named 'default'. radosgw just routes requests with paths starting > > > with > > > '/{rgw_admin_entry}' to a separate set of admin-related rest > > > apis. > > > otherwise they fall back to the s3 api, which treats '/foo' as a > > > request for bucket foo - that's why you see NoSuchBucket errors > > > when > > > it's misconfigured > > > > > > also note that, because of how these apis are nested, > > > rgw_admin_entry='default' would prevent users from creating and > > > operating on a bucket named 'default' > > > > > > On Tue, Oct 17, 2023 at 7:03 AM Tim Holloway <timh@xxxxxxxxxxxxx> > > > wrote: > > > > > > > > Thank you, Ondřej! > > > > > > > > Yes, I set the admin entry set to "default". It's just the > > > > latest > > > > result of failed attempts ("admin" didn't work for me either). > > > > I > > > > did > > > > say there were some horrors in there! > > > > > > > > If I got your sample URL pattern right, the results of a GET on > > > > "http://x.y.z/default"; return 404, NoSuchBucket. If that means > > > > that > > > > I > > > > didn't properly set rgw_enable_apis, then I probably don't know > > > > how > > > > to > > > > set it right. > > > > > > > > Best Regards, > > > > Tim > > > > > > > > On Tue, 2023-10-17 at 08:35 +0200, Ondřej Kukla wrote: > > > > > Hello Tim, > > > > > > > > > > I was also struggling with this when I was configuring the > > > > > object > > > > > gateway for the first time. > > > > > > > > > > There is a few things that you should check to make sure the > > > > > dashboard would work. > > > > > > > > > > 1. You need to have the admin api enabled on all rgws with > > > > > the > > > > > rgw_enable_apis option. (As far as I know you are not able to > > > > > force > > > > > the dashboard to use one rgw instance) > > > > > 2. It seems that you have the rgw_admin_entry set to a non > > > > > default > > > > > value - the default is admin but it seems that you have > > > > > “default" > > > > > (by > > > > > the name of the bucket) make sure that you have this also set > > > > > on > > > > > all > > > > > rgws. > > > > > > > > > > You can confirm that both of these settings are set properly > > > > > by > > > > > sending GET request to ${rgw-ip}:${port}/${rgw_admin_entry} > > > > > “default" in your case -> it should return 405 Method Not > > > > > Supported > > > > > > > > > > Btw there is actually no bucket that you would be able to see > > > > > in > > > > > the > > > > > administration. It’s just abstraction on the rgw. > > > > > > > > > > Reagards, > > > > > > > > > > Ondrej > > > > > > > > > > > On 16. 10. 2023, at 22:00, Tim Holloway > > > > > > <timh@xxxxxxxxxxxxx> > > > > > > wrote: > > > > > > > > > > > > First, an abject apology for the horrors I'm about to > > > > > > unveil. I > > > > > > made a > > > > > > cold migration from GlusterFS to Ceph a few months back, so > > > > > > it > > > > > > was > > > > > > a > > > > > > learn-/screwup/-as-you-go affair. > > > > > > > > > > > > For reasons of presumed compatibility with some of my older > > > > > > servers, I > > > > > > started with Ceph Octopus. Unfortunately, Octopus seems to > > > > > > have > > > > > > been a > > > > > > nexus of transitions from older Ceph organization and > > > > > > management to > > > > > > a > > > > > > newer (cephadm) system combined with a relocation of many > > > > > > ceph > > > > > > resources and compounded by stale bits of documentation > > > > > > (notably > > > > > > some > > > > > > references to SysV procedures and an obsolete installer > > > > > > that > > > > > > doesn't > > > > > > even come with Octopus). > > > > > > > > > > > > A far bigger problem was a known issue where actions would > > > > > > be > > > > > > scheduled > > > > > > but never executed if the system was even slightly dirty. > > > > > > And > > > > > > of > > > > > > course, since my system was hopelessly dirty, that was a > > > > > > major > > > > > > issue. > > > > > > Finally I took a risk and bumped up to Pacific, where that > > > > > > issue no > > > > > > longer exists. I won't say that I'm 100% clean even now, > > > > > > but at > > > > > > least > > > > > > the remaining crud is in areas where it cannot do any harm. > > > > > > Presumably. > > > > > > > > > > > > Given that, the only bar now remaining to total joy has > > > > > > been my > > > > > > inability to connect via the Ceph Dashboard to the Object > > > > > > Gateway. > > > > > > > > > > > > This seems to be an oft-reported problem, but generally > > > > > > referenced > > > > > > relative to higher-level administrative interfaces like > > > > > > Kubernetes > > > > > > and > > > > > > rook. I'm interfacing more directly, however. Regardless, > > > > > > the > > > > > > error > > > > > > reported is notably familiar: > > > > > > > > > > > > [quote] > > > > > > The Object Gateway Service is not configured > > > > > > Error connecting to Object Gateway: RGW REST API failed > > > > > > request > > > > > > with > > > > > > status code 404 > > > > > > (b'{"Code":"NoSuchBucket","Message":"","BucketName":"defaul > > > > > > t"," > > > > > > Requ > > > > > > estI > > > > > > d":"tx00' b'000dd0c65b8bda685b4-00652d8e0f-5e3a9b- > > > > > > default","HostId":"5e3a9b-default-defa' b'ult"}') > > > > > > Please consult the documentation on how to configure and > > > > > > enable > > > > > > the > > > > > > Object Gateway management functionality. > > > > > > [/quote] > > > > > > > > > > > > In point of fact, what this REALLY means in my case is that > > > > > > the > > > > > > bucket > > > > > > that is supposed to contain the necessary information for > > > > > > the > > > > > > dashboard > > > > > > and rgw to communicate has not been created. Presumably > > > > > > that > > > > > > SHOULDhave > > > > > > been done by the "ceph dashboard set-rgw-credentials" > > > > > > command, > > > > > > but > > > > > > apparently isn't, because the default zone has no buckets > > > > > > at > > > > > > all, > > > > > > much > > > > > > less one named "default". > > > > > > > > > > > > By way of reference, the dashboard is definitely trying to > > > > > > interact > > > > > > with the rgw container, because trying object gateway > > > > > > options > > > > > > on > > > > > > the > > > > > > dashboard result in the container logging the following. > > > > > > > > > > > > beast: 0x7efd29621620: 10.0.1.16 - dashboard > > > > > > [16/Oct/2023:19:25:03.678 > > > > > > +0000] "GET /default/metadata/user?myself HTTP/1.1" 404 > > > > > > > > > > > > To make everything happy, I'd be glad to accept > > > > > > instructions on > > > > > > how > > > > > > to > > > > > > manually brute-force construct this bucket. > > > > > > > > > > > > Of course, as a cleaner long-term solution, it would be > > > > > > nice if > > > > > > the > > > > > > failure to create could be detected and logged. > > > > > > > > > > > > And of course, the ultimate solution: something that would > > > > > > assist > > > > > > in > > > > > > making whatever processes are unhappy be happy. > > > > > > > > > > > > Thanks, > > > > > > Tim > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |