On Thu, Nov 4, 2021 at 10:52 AM Casey Bodley <cbodley@xxxxxxxxxx> wrote: > > On Wed, Nov 3, 2021 at 3:34 PM Sage Weil <sage@xxxxxxxxxxxx> wrote: > > > > On Tue, Oct 26, 2021 at 2:26 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote: > >> > >> there's an rgw bug about mon config no longer working for the > >> 'rgw_frontends' variable at https://tracker.ceph.com/issues/50249 > >> > >> the tracker identifies the regression as an rdma fix from July 2019, > >> "global/global_init: do first transport connection after setuid()" > >> from https://github.com/ceph/ceph/pull/28012. this moves the code to > >> fetch mon config after the setuid() > >> > >> rgw starts the http frontends before setuid(), so that it's able to > >> bind privileged ports like 80/443 > >> > >> so it seems to me that one of 3 things has to give here: > >> 1. ability to store frontend config in mon > >> 2. ability to bind privileged ports > >> 3. rgw support for rdma in AsyncMessenger > >> > >> > >> currently it's #1 that's broken, meaning that you either have to > >> override rgw_frontends in a config file, or live with the default > >> config (port 80, and port 443 if we can fetch the cert/key from the > >> monitor?) > >> > >> with respect to #2, the use of privileged ports isn't a big deal if > >> you have a proxy in front. but rgw has long supported privileged > >> ports, and this configuration without a proxy is the simplest > >> standalone deployment possible > >> > >> support for #3 is the one i'd least like to drop as, unlike #1 and #2, > >> i don't know of any workaround to get it back > >> > >> so is the current situation the best we can do? i'd love to hear any > >> suggestions! > > > > > > I think they key thing with #1 is that the mon config fetching is a *temporary* MonClient and messenger startup/shutdown cycle. We connect to the mon, fetch config, tear it all down, and then proceed with daemon startup. So... unless that config fetch has to use RDMA for some reason (does the environment *require* RDMA and fail with the normal TCP-based stack?), it seems like reverting the 2019 change (or similar) would get us out of this situation. > > > > Storing the rgw_frontends in the mon config seems like a strong requirement. I thought that's what cephadm was doing, which makes me wonder how it is working at all? > > thanks Sage, that's a good question. i'll need to do some more testing > and get back to you ok, mon config does appear to be working as intended here. i actually see that radosgw is fetching mon config -3 times- before the frontend is created: once in global_init(), and two other times on librados client creation radosgw uses the CINIT_FLAG_DEFER_DROP_PRIVILEGES flag to prevent global_init() from doing the setuid itself, so the rdma change from https://github.com/ceph/ceph/pull/28012 shouldn't have any effect here. it's still fetching mon config inside of global_init() long before radosgw initializes frontends and calls setuid > > > > > s > > > > > >> > >> > >> _______________________________________________ > >> Dev mailing list -- dev@xxxxxxx > >> To unsubscribe send an email to dev-leave@xxxxxxx _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
