Sitsofe, Wow! It looks like you went the extra mile checking this out. It's going to take me a bit to grok this, I'll have to get back to you. Thanks much Kris Davis -----Original Message----- From: fio-owner@xxxxxxxxxxxxxxx [mailto:fio-owner@xxxxxxxxxxxxxxx] On Behalf Of Sitsofe Wheeler Sent: Thursday, March 22, 2018 9:47 AM To: Kris Davis <Kris.Davis@xxxxxxx> Cc: fio@xxxxxxxxxxxxxxx; Jens Axboe <axboe@xxxxxxxxx> Subject: Re: fiologparser_hist.py script patch and enhancements? On 20 March 2018 at 05:57, Kris Davis <Kris.Davis@xxxxxxx> wrote: > I created a pull request from github with the latest changes, both to fiologparser_hist.py and the associated man page. This includes a new "--directions=rwtm" option to allow independent directional results to be printed (with an added 'dir' column). I see Jens pulled it in so congrats on staying the course! > If there is a way to infer whether the histogram results are in 'ns' or 'ms' from the number of columns, please let me know. I wonder if the 'coarseness' option might be a factor they might complicate things. I'm mostly relying on the original script operation, rather than having a more complete understanding of the dependencies. The coarseness definitely complicates things but here are my thoughts: If you know the total columns and you're willing to assume fio compile defaults you can try and guess whether FIO_IO_U_PLAT_GROUP_NR is 19 (in which case you assume usec) or 29 (in which case you assume nsec). >From iolog.c: 723 int stride = 1 << hist_coarseness; [...] 734 s = __get_sample(samples, log_offset, i); 735 736 entry = s->data.plat_entry; 737 io_u_plat = entry->io_u_plat; 738 739 entry_before = flist_first_entry(&entry->list, struct io_u_plat_entry, list); 740 io_u_plat_before = entry_before->io_u_plat; 741 742 fprintf(f, "%lu, %u, %u, ", (unsigned long) s->time, 743 io_sample_ddir(s), s->bs); 744 for (j = 0; j < FIO_IO_U_PLAT_NR - stride; j += stride) { 745 fprintf(f, "%llu, ", (unsigned long long) 746 hist_sum(j, stride, io_u_plat, io_u_plat_before)); 747 } 748 fprintf(f, "%llu\n", (unsigned long long) 749 hist_sum(FIO_IO_U_PLAT_NR - stride, stride, io_u_plat, 750 io_u_plat_before)); So we can tell the number of columns is related to the 3 + (FIO_IO_U_PLAT_NR / (2 ^ coarseness)) + 1. The 3 comes from printing the time, the data direction and the blocksize columns. In python def main(): for pgn in [19, 29]: print("pgn=%d:" % (pgn)), for c in range(7): pn = (1 << 6) * pgn s = (1 << c) print(((pn - s) / s) + 4), print("") if __name__ == "__main__": main() prints the following: pgn=19: 1219 611 307 155 79 41 22 pgn=29: 1859 931 467 235 119 61 32 Quick comparison: $ ./fio --version fio-3.5-70-g40f1 $ ./fio --name=test --rw=read --runtime=200ms --time_based --filename=/tmp/fio.tmp --size=1M --write_hist_log=test --log_hist_msec=100 --log_hist_coarseness=0 $ head -1 test_clat_hist.1.log | awk -F',' '{print NF}' 1859 ./fio --name=test --rw=read --runtime=200ms --time_based --filename=/tmp/fio.tmp --size=1M --write_hist_log=test --log_hist_msec=100 --log_hist_coarseness=6 $ head -1 test_clat_hist.1.log | awk -F',' '{print NF}' 32 If we approximate the equation we get ((64 * FIO_IO_U_PLAT_GROUP_NR) / (2 ^ coarseness)) + 4 = total_columns; which can be rearranged to (64 * FIO_IO_U_PLAT_GROUP_NR) / (total_columns - 4) = (2 ^ coarseness) log2((64 * FIO_IO_U_PLAT_GROUP_NR) / (total_columns - 4) = coarseness This allows us to guess and check a bit quicker. For example with FIO_IO_U_PLAT_GROUP_NR=19, columns=611 import math def check_columns(column_actual, pgn_guess): approx_coarse_guess = int(math.log((64 * pgn_guess) / float(column_actual - 4), 2)) stride_guess = 1 << approx_coarse_guess column_guess = ((64 * pgn_guess - stride_guess) / stride_guess) + 4 return column_guess == column_actual >>> check_columns(611, 19) True A bigger test: >>> [check_columns(x, 19) for x in (1219, 611, 307, 155, 79, 41, 22)] [True, True, True, True, True, True, True] >>> [check_columns(x, 19) for x in (1859, 931, 467, 235, 119, 61, 32)] [False, False, False, False, False, False, False] If someone explicitly sets --group_nr they ought to set whether the time is in usec or nsec (or generally override the default) as we can't really guess any more. What do you think? -- Sitsofe | http://sucs.org/~sits/ -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html ��.n��������+%������w��{.n�������^n�r������&��z�ޗ�zf���h���~����������_��+v���)ߣ�
