On Wed, Mar 8, 2023 at 10:55 PM Juri Lelli <juri.lelli@xxxxxxxxxx> wrote: > > On 08/03/23 10:01, Hao Luo wrote: > > On Wed, Mar 8, 2023 at 2:20 AM Juri Lelli <juri.lelli@xxxxxxxxxx> wrote: > > > > > > On 01/03/23 17:03, Qais Yousef wrote: > > > > On 03/01/23 15:26, Juri Lelli wrote: > > <...> > > > > > BTW, do you have a repro script of some sort handy I might play with? > > > > > > > > Sorry no. You'll just need to suspend to ram. I had a simple patch to measure > > > > the time around the call and trace_printk'ed the result. > > > > > > > > I was working on a android phone which just suspends to ram if you turn the > > > > screen off and disconnect the usb. > > > > > > Looks like I could come up with the following > > > > > > https://github.com/jlelli/linux.git deadline/rework-cpusets > > > https://github.com/jlelli/linux/tree/deadline/rework-cpusets > > > > > > which I don't think it's at a point that I feel comfortable to propose > > > as an RFC (not even sure if it actually makes sense), but it survived my > > > very light testing. > > > > > > Could you please take a look and, if it makes some sense in theory, give > > > it a try on your end? > > > > > > > Hi Juri, > > > > Thanks for coming up with the RFC. I can test your changes in the > > server environment. I observed the same issue on my side and I can > > reproduce. > > > > I sync'ed up with Qais offline earlier yesterday, and was preparing a > > couple of patches that optimize the cpuset.cpus writes. Tracking dl > > tasks in cpusets is one of them. But I am happy to take your patches > > and do the testing. Note that I won't be able to test the dl part of > > the patch, only the latency impact on rebuild_root_domains(), as we > > don't have dl tasks in our system. > > > > The other patch is fixing cpuset_rwsem. I see you switched it back to > > mutex. I did observe performance issues with cpuset_rwsem. Basically, > > using percpu_rwsem generates very very long latency tails for writers, > > but mutex doesn't. After some debugging, I found it was because > > percpu_rwsem requires every writer to call a synchronize_rcu() for > > acquiring the lock. So in my patches, I disabled the fastpath of > > readers for cpuset_rwsem. This has been done before[1]. But mutex also > > worked. > > > > Anyway, I'm happy to test your patches and ack once they are sent out. > > Do you strictly need a proper RFC or could you please test the above for > now? If you could please do the latter, and if tests look ok, I could > then put together proper changelogs etc. and propose an RFC (it would > save me some time not to do that if the above doesn't work, apologies > for not going the proper route from the start). Guess this question > applies to Qais as well. Hummm, or maybe you are actually saying that > you are indeed going to test them already, just wanted to make sure > then. :) Juri, I ported your patches to a 5.10 kernel, because my workload can only run on 5.10. But unfortunately the kernel crashed at cpuset_can_attach(). I'll put a few comments in your github branch. Hao Hao
