On Thu, Sep 29, 2016 at 12:37:22AM -0400, Theodore Ts'o wrote: > On Fri, Jun 19, 2015 at 09:34:30AM +1000, Dave Chinner wrote: > > On Thu, Jun 18, 2015 at 11:53:37AM -0400, Theodore Ts'o wrote: > > > I've been trying to figure out why generic/299 has occasionally been > > > stalling forever. After taking a closer look, it appears the problem > > > is that the fio process is stalling in userspace. Looking at the ps > > > listing, the fio process hasn't run in over six hours, and using > > > attaching strace to the fio process, it's stalled in a FUTUEX_WAIT. > > > > > > Has anyone else seen this? I'm using fio 2.2.6, and I have a feeling > > > that I started seeing this when I started using a newer version of > > > fio. So I'm going to try roll back to an older version of fio and see > > > if that causes the problem to go away. > > > > I'm running on fio 2.1.3 at the moment and I havne't seen any > > problems like this for months. Keep in mind that fio does tend to > > break in strange ways fairly regularly, so I'd suggest an > > upgrade/downgrade of fio as your first move. > > Out of curiosity, Dave, are you still using fio 2.1.3? I had upgraded No. $ fio -v fio-2.1.11 $ > to the latest fio to fix other test breaks, and I'm stil seeing the > occasional generic/299 test failure. In fact, it's been happening > often enough on one of my test platforms[1] that I decided to really > dig down and investigate it, and all of the threads were blocking on > td->verify_cond in fio's verify.c. > > It bisected down to this commit: > > commit e5437a073e658e8154b9e87bab5c7b3b06ed4255 > Author: Vasily Tarasov <tarasov@xxxxxxxxxxx> > Date: Sun Nov 9 20:22:24 2014 -0700 > > Fix for a race when fio prints I/O statistics periodically > > Below is the demonstration for the latest code in git: > ... > > So generic/299 passes reliably with this commits parent, and it fails > on this commit within a dozen tries or so. The commit first landed in > fio 2.1.14, so it's consistent with Dave's report a year ago he was > still using fio 2.1.3. But I'm still not using a fio recent enough to hit this. Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
