Hello Bart, I got addition test results from https://github.com/bvanassche/fio. Timestamp issue is fixed when running in single job ( --numjobs=1), however on a test with multiple jobs, each logging latency at 1s, there are sporadic deltas of 0.240 - 0.250s in iops and bw log. Second issue, there are +1 second in lat log of the intended runtime. I was able to recreate the problem with fio command line in below including snippet of output from my test bench. fio --name=multijobs_noramp --filename=/dev/nvme1n1 --ioengine=libaio --rw=randread --bs=4k --direct=1 --iodepth=64 --numjobs=4 --runtime=20m --write_iops_log=multijobs_noramp --write_lat_log=multijobs_noramp --group_reporting=1 --log_avg_msec=1000 --output=multijobs_noramp cat multijobs_noramp_iops.4.log 1120, 195213, 0, 0, 0 2120, 184542, 0, 0, 0 3120, 184310, 0, 0, 0 cat multijobs_noramp_iops.3.log 1121, 195074, 0, 0, 0 2121, 184694, 0, 0, 0 3121, 184359, 0, 0, 0 cat multijobs_noramp_iops.2.log 1000, 194010, 0, 0, 0 2000, 206103, 0, 0, 0 3000, 206836, 0, 0, 0 cat multijobs_noramp_iops.1.log 1000, 193926, 0, 0, 0 2000, 206252, 0, 0, 0 3000, 206148, 0, 0, 0 cat multijobs_noramp_lat.4.log 1198000, 309048, 0, 0, 0 1199000, 309715, 0, 0, 0 1200000, 309382, 0, 0, 0 1200000, 587410, 0, 0, 0 cat multijobs_noramp_lat.3.log 1198000, 308499, 0, 0, 0 1199000, 308733, 0, 0, 0 1200000, 308176, 0, 0, 0 1200000, 818831, 0, 0, 0 cat multijobs_noramp_lat.2.log 1198000, 310959, 0, 0, 0 1199000, 308661, 0, 0, 0 1200000, 309743, 0, 0, 0 1200000, 1036165, 0, 0, 0 cat multijobs_noramp_lat.1.log 1198000, 308318, 0, 0, 0 1199000, 309911, 0, 0, 0 1200000, 309643, 0, 0, 0 1200000, 1005811, 0, 0, 0 Best Regards ************************************************** Siriporn Swart SIT Lab SSD Performance Test Engineer Western Digital® Email: siriporn.swart@xxxxxxx Office: +1-507-322-2123 -----Original Message----- From: Siriporn Swart Sent: Monday, September 14, 2020 6:23 PM To: Bart Van Assche <bvanassche@xxxxxxx>; Jeffrey Lien <Jeff.Lien@xxxxxxx>; fio@xxxxxxxxxxxxxxx Cc: Jeff Furlong <jeff.furlong@xxxxxxx>; Kris Davis <Kris.Davis@xxxxxxx> Subject: RE: Inconsistent Status Intervals Hello Bart, Thank you very much for your support. Jeff Lien is on vacation this week. My name is Siriporn from performance test team. I ran quick test for 30sec with master branch of https://github.com/bvanassche/fio and result looks good. I will rerun with longer duration. Any update will be sent. ----- test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.17-458-g3803 Starting 1 process [root@echo-sr635-01 result]# cat test_iops.1.log 1000, 90808, 0, 0, 0 2000, 99240, 0, 0, 0 3000, 99095, 0, 0, 0 4000, 99229, 0, 0, 0 5000, 99270, 0, 0, 0 6000, 99351, 0, 0, 0 7000, 99209, 0, 0, 0 8000, 99299, 0, 0, 0 9000, 99297, 0, 0, 0 10000, 99395, 0, 0, 0 11000, 99292, 0, 0, 0 12000, 99385, 0, 0, 0 13000, 99322, 0, 0, 0 14000, 99293, 0, 0, 0 15000, 99266, 0, 0, 0 16000, 99335, 0, 0, 0 17000, 99367, 0, 0, 0 18000, 99380, 0, 0, 0 19000, 99309, 0, 0, 0 20000, 99290, 0, 0, 0 21000, 99300, 0, 0, 0 22000, 99377, 0, 0, 0 23000, 99381, 0, 0, 0 24000, 99395, 0, 0, 0 25000, 99355, 0, 0, 0 26000, 99371, 0, 0, 0 27000, 99312, 0, 0, 0 28000, 99337, 0, 0, 0 29000, 99264, 0, 0, 0 [root@echo-sr635-01 result]# Result from fio-3.20 [root@echo-sr635-01 PB319]# cat test_iops.1.log 1000, 84767, 0, 0, 0 2001, 99750, 0, 0, 0 3002, 99973, 0, 0, 0 4003, 99833, 0, 0, 0 5004, 99844, 0, 0, 0 6005, 99817, 0, 0, 0 7006, 99842, 0, 0, 0 8007, 99856, 0, 0, 0 9008, 99824, 0, 0, 0 10009, 99824, 0, 0, 0 11010, 99807, 0, 0, 0 12011, 99874, 0, 0, 0 13012, 99779, 0, 0, 0 14013, 99968, 0, 0, 0 15014, 99954, 0, 0, 0 16015, 99900, 0, 0, 0 17016, 99867, 0, 0, 0 18017, 99812, 0, 0, 0 19018, 99826, 0, 0, 0 20019, 99844, 0, 0, 0 21020, 99833, 0, 0, 0 22021, 99782, 0, 0, 0 23022, 99814, 0, 0, 0 24023, 99875, 0, 0, 0 25024, 99964, 0, 0, 0 26025, 99847, 0, 0, 0 27026, 99859, 0, 0, 0 28027, 99825, 0, 0, 0 29028, 99863, 0, 0, 0 Best Regards ************************************************** Siriporn Swart SIT Lab SSD Performance Test Engineer Western Digital® Email: siriporn.swart@xxxxxxx Office: +1-507-322-2123 -----Original Message----- From: Bart Van Assche <bvanassche@xxxxxxx> Sent: Friday, September 11, 2020 11:16 PM To: Jeffrey Lien <Jeff.Lien@xxxxxxx>; fio@xxxxxxxxxxxxxxx Cc: Siriporn Swart <Siriporn.Swart@xxxxxxx>; Jeff Furlong <jeff.furlong@xxxxxxx>; Kris Davis <Kris.Davis@xxxxxxx> Subject: Re: Inconsistent Status Intervals CAUTION: This email originated from outside of Western Digital. Do not click on links or open attachments unless you recognize the sender and know that the content is safe. On 2020-09-11 10:10, Jeffrey Lien wrote: > Do you have any other suggestions or potential fixes to try? If so, please let us know. Thanks. Hi Jeff, The master branch of https://github.com/bvanassche/fio has been updated. Please retry. Thanks, Bart.
