On Sun, 21 Jul 2024, at 3:38 PM, Slavko wrote: > Hi, > > Dňa 21. júla 2024 12:29:44 UTC používateľ Kerin Millar > <kfm@xxxxxxxxxxxxx> napísal: > >>You have a point. If on a severely RAM-constrained system, you may prefer to avoid the use of asynchronous pipelines and generate temporary files instead. Doing so would minimise RAM consumption. > > Correct me, if i am wrong, please... > > AFAIK memory is not problem in pipes. Yes, any pipe consumes some > memory, but AFAIK it is somewhat limited (i don't know exact numbers) > and if full, the writing process is paused, thus stops to produce more > output, and thus not consumes more memory. Yes, the default size of a pipe buffer is just 16 pages in Linux. However, one must also take into account that pipelines are fundamentally asynchronous. That is, all of the commands that comprise a shell pipeline are executed at approximately the same time and run in parallel to one another. In that particular respect, there is a potential bearing on memory consumption. I must emphasise that I am definitely not trying to discourage the use of pipelines - far from it! I only brought it up because I had the impression that you might be contending with a system that is severely short on RAM, in which case running programs serially can be helpful as a deliberate memory-saving measure. > > On other side, nowadays linux systems uses tmpfs for /tmp, thus storing > output in temp file can be even worse (in mean of RAM) than using pipes, > as whole output is in that file. Storing temporary file on disk is possible too, > but doing that too often can drain Flash's lifetime, as modern Flash storage > are really limited in that (in comparison with magnetic disks). IIRC the disk > was used for pipes in old Unixes... It depends on the exact circumstances but yes, it could easily be worse. If you have a mktemp(1) utility, it is probable that it responds to the TMPDIR variable, in which case you are free to point it to something other than a tmpfs-backed filesystem such as /var/tmp. > > But yes, without temp file, one cannot produce the same IP list multiple > times (reading it into variable is the same), but at least second temp file > have to be avoided... It is not quite the same but I understand your point. Assuming that TMPDIR is backed by a tmpfs, both cases would result in non-disk-backed pages being allocated that may later be paged out to a swap - if any - under sufficient memory pressure. -- Kerin Millar
