While debugging a custom program that can write large output files to
stdout, I noticed that the run time as displayed by the Bash 'time' prefix
wildly varied from run to run. It turns out that the issue isn't just with
my program. I can recreate it with 'cat':
$ time cat infile >outfile
If 'infile' is on the order of 140 MB, 'time' might show something as low
as:
real 0m0.146s
user 0m0.000s
sys 0m0.109s
CPU % 74.29
or as high as:
real 0m0.328s
user 0m0.000s
sys 0m0.109s
CPU % 33.31
If 'outfile' doesn't exist, the 'cat' runs much more quickly:
real 0m0.082s
user 0m0.000s
sys 0m0.081s
CPU % 99.77
Likewise, if I arrange to 'rm' the file right before running 'cat', it
runs consistently quickly:
$ time rm -f outfile
real 0m0.019s
user 0m0.000s
sys 0m0.019s
CPU % 99.93
$ time cat infile >outfile
real 0m0.081s
user 0m0.000s
sys 0m0.081s
CPU % 99.75
If I do these two commands in this order over and over, the timings for
'cat' are consistently fast.
In my custom C program, I have the same issue if I write to stdout or if I
'fopen' a named file like this:
fopen(outfile_name, "w")
If the file doesn't exist, the program runs consistently quickly:
real 0m0.191s
user 0m0.082s
sys 0m0.109s
CPU % 99.73
but if it exists the run times fluctuate quite a bit from run to run:
real 0m0.579s
user 0m0.072s
sys 0m0.147s
CPU % 37.85
real 0m0.473s
user 0m0.072s
sys 0m0.148s
CPU % 46.60
If I 'fopen' a named disk file right after doing unlink(outfile_name), I
get consistently fast run times.
Note: I don't think this is an issue with buffered I/O to stdout per se. I
say this because I'm seeing pretty consistent run times if I write to
stdout _as long as the output file didn't previously exist_. Rather, the
issue seems to be with _overwriting_ an existing disk file when I/O
redirection is used.
Are there any known issues with the Linux kernel, 'bash', glibc, or
whatever else might be involved with standard I/O that might cause this
issue? If this is not a known issue, what's the best way to open a bug
ticket?
I'm running Fedora 27 with the latest upgrades. The Linux kernel is
4.18.9-100.fc27.x86_64, glibc is 2.26-30, and bash is 4.4.23-1.
Thanks,
Dave
--
Dave Ulrick
d-ulrick@xxxxxxxxxxx
_______________________________________________
users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
