Re: RGW blocking on large objects

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks very much, Robert.

Matt

On Thu, Oct 17, 2019 at 5:24 PM Robert LeBlanc <robert@xxxxxxxxxxxxx> wrote:
>
> On Thu, Oct 17, 2019 at 2:03 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote:
> > > This is great news. Anything we can do to help in this effort as it is
> > > very important for us?
> >
> > We would love help here. While most of the groundwork is done, so the
> > remaining work is mostly mechanical.
> >
> > To summarize the strategy, the beast frontend spawns a coroutine for
> > each client connection, and that coroutine is represented by a
> > boost::asio::yield_context. We wrap this in an 'optional_yield' struct
> > that gets passed to process_request(). The civetweb frontend always
> > passes an empty object (ie null_yield) so that everything runs
> > synchronously. When making calls into librados, we have a
> > rgw_rados_operate() function that supports this optional_yield argument.
> > If it gets a null_yield, it calls the blocking version of
> > librados::IoCtx::operate(). Otherwise it calls a special
> > librados::async_operate() function which suspends the coroutine until
> > completion instead of blocking the thread.
> >
> > So most of the remaining work is in plumbing this optional_yield
> > variable through all of the code paths under process_request() that call
> > into librados. The rgw_rados_operate() helpers will log a "WARNING:
> > blocking librados call" whenever they block inside of a beast frontend
> > thread, so we can go through the rgw log to identify all of the places
> > that still need a yield context. By iterating on this process, we can
> > eventually remove all of the blocking calls, then set up regression
> > testing to verify that no rgw logs contain that warning.
> >
> > Here's an example pr from Ali that adds the optional_yield to requests
> > for bucket instance info: https://github.com/ceph/ceph/pull/27898. It
> > extends the get_bucket_info() call to take optional_yield, and passes
> > one in where available, using null_yield to mark the synchronous cases
> > where one isn't available.
>
> I'll work to get familiar with the code base and see if I can submit
> some PRs to help out. Things are a bit crazy, but this is very
> important to us too.
>
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
> _______________________________________________
> Dev mailing list -- dev@xxxxxxx
> To unsubscribe send an email to dev-leave@xxxxxxx



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux