You may want to consider using in place sort functions. list.sort() is more efficient than s=sorted(list) and will greatly reduce DRAM usage for large latency logs. On Linux you can check the python DRAM usage with resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, e.g. before and after your sort statements. Regards, Jeff -----Original Message----- From: fio-owner@xxxxxxxxxxxxxxx [mailto:fio-owner@xxxxxxxxxxxxxxx] On Behalf Of Mark Nelson Sent: Tuesday, May 24, 2016 9:20 AM To: Ben England <bengland@xxxxxxxxxx>; Jens Axboe <axboe@xxxxxxxxx> Cc: Martin Steigerwald <ms@xxxxxxxxx>; fio@xxxxxxxxxxxxxxx; Mark Nelson <mnelson@xxxxxxxxxx> Subject: Re: fiologparser.py I've got a version that removes the dependency and appears to return the same values: https://github.com/axboe/fio/pull/181 Going through the code though, it looks like the -A values are computed differently than in the other original functions. In the original get_contribution function, all samples within the bounds are counted, along with samples that are only partially within the bounds. Each sample is weighted based on the duration it overlapped with the sample period: https://github.com/axboe/fio/blob/master/tools/fiologparser.py#L195-L198 for -A, only the samples that are totally within the bounds are counted, and are weighted equally despite how much of the period was spent in that sample: https://github.com/axboe/fio/blob/master/tools/fiologparser.py#L173 Thus if you look at say the average from -a: fiologparser.py -a *clat* 1000, 11582.770 2000, 14033.844 3000, 17087.446 4000, 17946.245 5000, 14554.196 6000, 14407.804 7000, 15218.106 8000, 15157.951 the results are quite a bit different from -A: fiologparser.py -A *clat* | tr -s "," " " | cut -f1,4 -d" " 0.000000 11902.719298 1000.000000 13247.750000 2000.000000 14270.549020 3000.000000 15092.192308 4000.000000 14127.472727 5000.000000 12880.137931 6000.000000 15296.735849 7000.000000 14857.306122 8000.000000 14854.766667 Mark On 05/24/2016 10:35 AM, Ben England wrote: > OK we'll remove the dependencies, I still want to have the -A option supported. > -ben > > ----- Original Message ----- >> From: "Jens Axboe" <axboe@xxxxxxxxx> >> To: "Ben England" <bengland@xxxxxxxxxx>, "Mark Nelson" >> <mark.a.nelson@xxxxxxxxx> >> Cc: "Martin Steigerwald" <ms@xxxxxxxxx>, fio@xxxxxxxxxxxxxxx, "Mark >> Nelson" <mnelson@xxxxxxxxxx> >> Sent: Tuesday, May 24, 2016 11:28:39 AM >> Subject: Re: fiologparser.py >> >> On 05/24/2016 09:22 AM, Ben England wrote: >>> >>> >>> ----- Original Message ----- >>>> From: "Mark Nelson" <mark.a.nelson@xxxxxxxxx> >>>> To: "Ben England" <bengland@xxxxxxxxxx>, "Martin Steigerwald" >>>> <ms@xxxxxxxxx> >>>> Cc: fio@xxxxxxxxxxxxxxx, "Mark Nelson" <mnelson@xxxxxxxxxx>, "Jens Axboe" >>>> <axboe@xxxxxxxxx> >>>> Sent: Tuesday, May 24, 2016 10:04:14 AM >>>> Subject: Re: fiologparser.py >>>> >>>> Let's see if we can remove the numpy and scipy dependencies. It >>>> looks like we are just using it for >>>> min/average/median/max/percentile calculations. It would be nice >>>> if users didn't need anything other than argparse. >>>> >>> >>> Just curious, why is scipy a problem? Is it because CBT isn't a >>> package so you don't get dependencies handled when you install it? >>> You are correct, it's easy to remove the dependencies, I just didn't >>> know it was causing problems for people. You can get percentiles >>> from just sorting the sample values and indexing into the array at >>> the appropriate offset, I was just trying to re-use existing classes. >> >> It's not necessarily a problem, but the less dependencies you have, >> the easier it is for people to use. I do the same for fio, try to >> have as few external dependencies as possible. Remember, not >> everybody is running on Linux... >> >> -- >> Jens Axboe >> >> -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. ��.n��������+%������w��{.n�������^n�r������&��z�ޗ�zf���h���~����������_��+v���)ߣ�
