On Fri, Nov 02, 2018 at 07:31:38AM -0400, Brian Foster wrote: > Fair enough, but I'm still curious if doing something like changing the > hash trylock in the shaker to a blocking lock would improve shaker > effectiveness enough to avoid the need for the time-based hackery. It's > possible it has no effect or just replaces one problem with a set of > others, but it's an even more trivial change than this patch is. > > Another approach may be to lift the cache shaker from a per-lookup-miss > execution context to something more central (like its own thread(s)) > such that lookups can block on (bounded) shakes without introducing too > much concurrency therein. That is certainly more involved than bolting a > throttle onto cache expansion and may not be worth the effort if the > long term plan is to replace the whole cache mechanism. I'm more inclined to kill the entire libxfs buffer cache implementation and MRUs and port the kernel code across with it's reference based LRU and shrinker. And with that, use AIO so that we don't need huge numbers of prefetch threads to keep IO in flight. Getting rid of the repair prefetcher threads removes the major concurrency component that is placed on the cache. Using the kernel infrastrucutre also moves from a global cache to a per-ag cache which matches how xfs_repair operates and hence further reduces lock contention. i.e. it allows threads working within an AG to work without interfence from other AGs. Basically, we are hitting on the scalability limits of the libxfs architecture right now, and we have an architecure in the kernel that scales way beyond what we have in userspace. And it fits right in with the way userspace algorithms operate. Add to that the need for AIO and delwri lists to scale mkfs to really large filesystems, and it really comes down to "we need to port the kernel code and move to AIO" rather than tinker around the edges with an architecture that we can't easily scale further that it current does... Cheers, Dave -- Dave Chinner david@xxxxxxxxxxxxx
