On Mon, Sep 04, 2023 at 12:13:38AM +0100, Al Viro wrote: > On Mon, Sep 04, 2023 at 08:27:15AM +1000, Dave Chinner wrote: > > > It already is (sysrq-t), but I'm not sure that will help - if it is > > a leaked unlock then nothing will show up at all. > > Unlikely; grep and you'll see - very few callers, and for all of them > there's an fdput_pos() downstream of any fdget_pos() that had picked > non-NULL file reference. > > In theory, it's not impossible that something had stripped FDPUT_POS_UNLOCK > from the flags, but that's basically "something might've corrupted the > local variables" scenario. Entirely possible - this is syzbot we are talking about here. Especially if reiser or ntfs has been tested back before the logs we have start, as both are known to corrupt memory and/or leak locks when trying to parse corrupt filesystem images that syzbot feeds them. That's why we largely ignore syzbot reports that involve those filesystems... Unfortunately, the logs from what was being done around when the tasks actually hung are long gone (seems like only the last 20-30s of log activity is reported) so when the hung task timer goes off at 143s, there is nothing left to tell us what might have caused it. IOWs, it's entirely possible that it is a memory corruption that has resulted in a leaked lock somewhere... > There are 12 functions total where we might > be calling fdget_pos() and all of them are pretty small (1 in alpha > osf_sys.c, 6 in read_write.c and 5 in readdir.c); none of those takes > an address of struct fd, none of them has assignments to it after fdget_pos() > and the only accesses to its members are those to fd.file - all fetches. > Control flow is also easy to check - they are all short. > > IMO it's much more likely that we'll find something like > > thread A: > grabs some fs lock > gets stuck on something > thread B: write() > finds file > grabs ->f_pos_lock > calls into filesystem > blocks on fs lock held by A > thread C: read()/write()/lseek() on the same file > blocks on ->f_pos_lock Yes, that's exactly what I said in a followup email - we need to know what happened to thread A, because that might be where we are stuck on a leaked lock. I saw quite a few reports where lookup/readdir are also stuck trying to get an inode lock - those at the "thread B"s in the above example - but there's no indication left of what happened with thread A. If thread A was blocked iall that time on something, then the hung task timer should fire on it, too. If it is running in a tight loop, the NMI would have dumped a stack trace from it. But neither of those things happened, so it's either leaked something or it's in a loop with a short term sleep so doesn't trigger the hung task timer. sysrq-w output will capture that without all the noise of sysrq-t.... -Dave. -- Dave Chinner david@xxxxxxxxxxxxx
