> Il giorno 21 nov 2019, alle ore 09:00, Paolo Valente <paolo.valente@xxxxxxxxxx> ha scritto: > > > >> Il giorno 21 nov 2019, alle ore 08:13, Oleksandr Natalenko <oleksandr@xxxxxxxxxxxxxx> ha scritto: >> >> Hi Paolo et al. >> > > Hi > >> I have a strong suspect that something is going wrong when the underlying block device responds with a large delay. What makes me thinking so is that I use a VM on some cloud provider, and they have substantial block device latency resulting in permanently high (~20%) iowait. It spikes occasionally when their cluster is overloaded, and when that happens, the I/O in my VM may stop and never recover. This is a rare occasion, but it really happens. >> >> What's worse, so far I've seen such a behaviour with BFQ only. I'm still testing other schedulers though. >> >> Important note: I have no strict evidences that this is *the* case, thus I'm asking for some suggestions. My idea is to fire up a local VM and inject delays to a block device while performing some I/O from within the VM. >> >> So the question is: how can those delays be injected? Using dm-delay? Can those delays be random? >> > > So far I have used scsi_debug [1] for this kind of tests. In my S > suite [2], it boils down to setting SCSI_DEBUG=yes in the S config > file, and then launching any of the benchmarks. Unfortunately, AFAIK > scsi_debug gives you only constant delays; but you can emulate delay > spikes very easily, by changing the delay parameter manually during > the test. > > If this option sounds reasonable to you, then I'm willing to help you > for every step. > Hi Oleksandr, Simone (in CC) and I have worked a little bit on reproducing the I/O freeze you report. Simone made a small change in SCSI_debug, which makes the latter serve I/O with a highly varying random delay (100ms - 1s), about twice a second. Then, to generate some fluctuating and heavy I/O, he ran the comm_startup_lat.sh script of my S suite with SCSI_debug a few times. Unfortunately, he didn't succeed in reproducing the problem. If you want, we can send you a patch with his change for SCSI_debug. Any news on your side? Thanks, Simone > Thanks, > Paolo > > [1] http://sg.danny.cz/sg/sdebug26.html > [2] https://github.com/Algodev-github/S > >> Thanks in advance. >> >> -- >> Oleksandr Natalenko (post-factum)
