sure. assuming this compiles, I'm fine with it; I'll try it, and make it do if needed :) Matt On Tue, Feb 9, 2021 at 11:42 AM Sebastien Han <shan@xxxxxxxxxx> wrote: > > Thank Matt, I just sent this to kick in the discussion > https://github.com/ceph/ceph/pull/39380 > If someone wants to take over it's preferable I guess, this is mainly > due to my limited C++ knowledge. > > So feel free to assign someone from your team to take over so we can > move faster with this one. > Thanks! > ––––––––– > Sébastien Han > Senior Principal Software Engineer, Storage Architect > > "Always give 100%. Unless you're giving blood." > > On Mon, Feb 8, 2021 at 3:53 PM Matt Benjamin <mbenjami@xxxxxxxxxx> wrote: > > > > HI Sebastien, > > > > That seems like a concise and reasonable solution to me. It seems > > like the metrics from a single instance should in fact be transient > > (leaving the problem of maintaining aggregate values to prometheus or > > even downstream of that? > > > > Matt > > > > On Mon, Feb 8, 2021 at 9:47 AM Sebastien Han <shan@xxxxxxxxxx> wrote: > > > > > > Hi Jiffin, > > > > > > From my perspective, one simple way to fix this (although we must be > > > careful with backward compatibility) would be for rgw to register to > > > service map differently. > > > Today it is using the daemon name like rgw.foo, then it will register > > > as foo. Essentially, if you try to run that pod twice you would still > > > see a single instance in the service map as well as the prometheus > > > metrics. > > > > > > It would be nice to register with RADOS client session ID instead , > > > just like rbd-mirror does by using instance_id. Something like: > > > > > > std::string instance_id = stringify(rados->get_instance_id()); > > > int ret = rados.service_daemon_register(daemon_type, name, metadata); > > > > > > Here https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rados.cc#L1139 > > > With that we can re-use the same cephx user and scale to any number, > > > all instances will use the same cephx to authenticate to the cluster > > > but they will show up as N in the service map. > > > > > > I guess one downside is that as soon as the daemon restart, we get a > > > new RADOS client session ID, and thus our name changes, which means we > > > are losing all the metrics... > > > Thoughts? > > > > > > Thanks! > > > ––––––––– > > > Sébastien Han > > > Senior Principal Software Engineer, Storage Architect > > > > > > "Always give 100%. Unless you're giving blood." > > > > > > On Thu, Feb 4, 2021 at 3:39 PM Jiffin Thottan <jthottan@xxxxxxxxxx> wrote: > > > > > > > > Hi all, > > > > > > > > In OCS(Rook) env workflow for RGW daemons as follows, > > > > > > > > Normally for creating ceph object-store, the first Rook creates pools for rgw daemon with the specified configuration. > > > > > > > > Then depending on the no of instances, Rook create cephxuser and then rgw spawn daemon in the container(pod) using its id > > > > with following arguments for radosgw binary > > > > Args: > > > > --fsid=91501490-4b55-47db-b226-f9d9968774c1 > > > > --keyring=/etc/ceph/keyring-store/keyring > > > > --log-to-stderr=true > > > > --err-to-stderr=true > > > > --mon-cluster-log-to-stderr=true > > > > --log-stderr-prefix=debug > > > > --default-log-to-file=false > > > > --default-mon-cluster-log-to-file=false > > > > --mon-host=$(ROOK_CEPH_MON_HOST) > > > > --mon-initial-members=$(ROOK_CEPH_MON_INITIAL_MEMBERS) > > > > --id=rgw.my.store.a > > > > --setuser=ceph > > > > --setgroup=ceph > > > > --foreground > > > > --rgw-frontends=beast port=8080 > > > > --host=$(POD_NAME) > > > > --rgw-mime-types-file=/etc/ceph/rgw/mime.types > > > > --rgw-realm=my-store > > > > --rgw-zonegroup=my-store > > > > --rgw-zone=my-store > > > > > > > > And here cephxuser will be "client.rgw.my.store.a" and all the pools for rgw will be created as my-store*. Normally if there is > > > > a request for another instance in the config file for a ceph-object-store config file[1] for rook, another user "client.rgw.mystore.b" > > > > will be created by rook and will consume the same pools. > > > > > > > > There is a feature in Kubernetes known as autoscale in which pods can be automatically scaled based on specified metrics. If we apply that > > > > feature for rgw pods, Kubernetes will automatically scale the rgw pods(like a clone of the existing pod) with the same argument for "--id" > > > > based on the metrics, but ceph cannot distinguish those as different rgw daemons even though multiple pods of rgw are running simultaneously. > > > > In "ceph status" shows only one daemon rgw as well > > > > > > > > In vstart or ceph ansible(Ali help me to figure it out), I can see for each rgw daemon a cephxuser is getting created as well > > > > > > > > Is this behaviour intended ? or am I hitting any corner case which was never tested before? > > > > > > > > There is no point of autoscaling of rgw pod if it considered to the same daemon, the s3 client will talk to only one of the pods and ceph mgr > > > > provides metrics can give incorrect data as well which can affect the autoscale feature > > > > > > > > Also opened an issue in rook for the time being [2] > > > > > > > > [1] https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/object-test.yaml > > > > [2] https://github.com/rook/rook/issues/6943 > > > > > > > > Regards, > > > > Jiffin > > > > > > > > > > > > > -- > > > > Matt Benjamin > > Red Hat, Inc. > > 315 West Huron Street, Suite 140A > > Ann Arbor, Michigan 48103 > > > > http://www.redhat.com/en/technologies/storage > > > > tel. 734-821-5101 > > fax. 734-769-8938 > > cel. 734-216-5309 > > > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
