Hey Frank, just going through some old email and came across this. Not sure if you got help elsewhere? On Mon, Jun 28, 2021 at 8:17 AM Frank Filz <ffilz@xxxxxxxxxx> wrote: > > Recently (or not so recently, it's been almost 2 years), the nfs-ganesha > project implemented capability to utilize asynchronous non-blocking I/O > to storage backends to prevent thread starvation. The assumption is that > the backend provides non-blocking I/O with a callback mechanism to > notify nfs-ganesha when the I/O is complete so that nfs-ganesha can > subsequently asynchronously respond to the client indicating I/O completion. > > Ceph looks like it is structured to allow for such with Context objects > having finish and complete methods that allow the I/O path to notify > completion. In general libcephfs seems to use some form of condition > variable Context to block and wait for this notification. This would be > relatively easy to replace with a call back Context. > > However, libcephfs does use ObjectCacher and sets the > block_writes_upfront flag which seems to make any writes that go through > ObjectCacher to block using an internal condition variable and not > utilize the onfreespace Context object (which maybe should have been > named onfinish?). > > I'm wondering what the implication of setting block_writes_upfront to > false would be for libcephs beyond needing to assure an onfreespace > Context object is passed. Wow, I haven't looked at this code in a while. I think what's going on here is simply that the primary user of Client is our ceph-fuse, which is a synchronous interface, and the existing users of libcephfs have also been synchronous in the past. The ObjectCacher ought to work fine without doing blocking IO as RBD has been doing that for ages, but the bigger issue is needing to split apart all of the Client interfaces. It'd be pretty mechanical work to code up — functions tend to have a pretty obvious Cond.wait() sort of thing after sending data off over the wire, and the lower layers are using the same Contexts as you see in higher layers to implement it — but testing to be sure of stability would probably take some time. Hopefully somebody from the FS team can chime in if there are more details to go in to. -Greg > > Thanks > > Frank Filz > > _______________________________________________ > Dev mailing list -- dev@xxxxxxx > To unsubscribe send an email to dev-leave@xxxxxxx > _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
