> -----Original Message----- > From: Nick Neumann [mailto:nick@xxxxxxxxxxxxxxxx] > Sent: Thursday, February 24, 2022 3:07 PM > To: fio@xxxxxxxxxxxxxxx > Cc: axboe@xxxxxxxxx > Subject: Not logging zero entry intervals when using --log_avg_msec seems > problematic > > I was digging into the bandwidth/iops logging code in stat.c after thinking > about how to compute an average bandwidth/iops over a period of time > given interval log entries. > > I noticed something concerning. When there is an interval where nothing > happened (e.g. no read iops), nothing is logged but the time of last interval is > advanced (there is only one such time for each log, shared by all directions). > > This makes sense for a case when you are only doing reads and don't want > write entries in your log. But it means that looking at a log file can be pretty > misleading. If you are logging with an interval of 100ms but only one read is > occurring per second, you get a log for iops like this: > > 149, 6, 0, 0, 0 > 1056, 10, 0, 0, 0 > 2064, 9, 0, 0, 0 > 3072, 10, 0, 0, 0 > 4081, 10, 0, 0, 0 > > If you turn the interval up to every 10 ms, you get this: > 149, 6, 0, 0, 0 > 1011, 100, 0, 0, 0 > 2011, 100, 0, 0, 0 > 3011, 100, 0, 0, 0 > 4011, 100, 0, 0, 0 > > Those zero entries are actually really important in a case like this. > Those logs do not look like logs for a job that did 1 read per second. > This is an extreme example to show the problem, and of course we know > what's going on here. > > But what if you're doing a practical test on storage with a fairly small log > interval, and the storage hiccups and processes no operations in an interval? > It can be impossible to know if an apparent "gap" in a log file is because there > was a zero entry or a system hiccup caused the logging process to run a little > late. > > Any thoughts? Just logging the zero entries would work, but at the expense > of noise in logs. > > (One option I can think of is remembering, per direction, that the last interval > was a zero-entry one, and then writing that if a non-zero interval is later > encountered. Also the last entry per direction must always be written, if any > non-zero interval for that direction was ever written.) > > Thanks, > Nick stat.c:__add_samples() is where log entries are added. It contains a continue statement which skips entries during log intervals with no IO. I would start poking around there if you want log entries for those intervals. Perhaps make the new behavior an option for those that depend on the current behavior. Vincent
