Hi Bart, I would be willing to work on the latency logging issue in the IO threads, but don't know where to start. Do you know which parts and/or functions to start looking at? -----Original Message----- From: Bart Van Assche <bvanassche@xxxxxxx> Sent: Sunday, September 20, 2020 8:49 PM To: Siriporn Swart <Siriporn.Swart@xxxxxxx>; Jeffrey Lien <Jeff.Lien@xxxxxxx>; fio@xxxxxxxxxxxxxxx Cc: Jeff Furlong <jeff.furlong@xxxxxxx>; Kris Davis <Kris.Davis@xxxxxxx> Subject: Re: Inconsistent Status Intervals CAUTION: This email originated from outside of Western Digital. Do not click on links or open attachments unless you recognize the sender and know that the content is safe. On 2020-09-18 12:24, Siriporn Swart wrote: > I got addition test results from https://github.com/bvanassche/fio. Timestamp issue is fixed when running in single job ( --numjobs=1), however on a test with multiple jobs, each logging latency at 1s, there are sporadic deltas of 0.240 - 0.250s in iops and bw log. > Second issue, there are +1 second in lat log of the intended runtime. I was able to recreate the problem with fio command line in below including snippet of output from my test bench. > > fio --name=multijobs_noramp --filename=/dev/nvme1n1 --ioengine=libaio > --rw=randread --bs=4k --direct=1 --iodepth=64 --numjobs=4 > --runtime=20m --write_iops_log=multijobs_noramp > --write_lat_log=multijobs_noramp --group_reporting=1 > --log_avg_msec=1000 --output=multijobs_noramp Hi Siriporn, Thank you for the detailed report. I have done something this weekend that I should have done much earlier, namely analyze on the context of which thread the latency logging happens. My conclusion is that this happens on the context of the I/O threads. Or in other words, the sporadic deltas can't be a side effect of the changes I made in the helper thread. If someone else wants to work on eliminating these latency logging deltas I will be happy to assist. Bart.
