On Thu, Jan 16, 2020 at 10:33:32AM +0100, Miro Hrončok wrote: > On 15. 01. 20 23:59, Zbigniew Jędrzejewski-Szmek wrote: > >On Wed, Jan 15, 2020 at 06:05:42PM +0100, Miro Hrončok wrote: > >>### File types (and bytecode caches) > >> > >>The orthogonal dimension is the file type. Python standard library > >>contains directories with both "extension modules" (written in C > >>(usually) and compiled to `*.cpython-38-x86_64-linux-gnu.so` shared > >>object file) and "pure Python" modules (written in Python and saved > >>as `*.py` source file). > >> > >>Each pure Python module comes in 4 files: > >> > >>- `module.py` -- the source > >>- `__pycache__/module.cpython-38.pyc` -- regular (not optimized) bytecode cache > >>- `__pycache__/module.cpython-38.opt-1.pyc` -- optimized bytecode cache (level 1) > >>- `__pycache__/module.cpython-38.opt-2.pyc` -- optimized bytecode cache (level 2) > > > >I suspect that the difference in speed between loading various .pyc > >files is negligible. Do you have actual benchmarks for this? > > Loading time is theoretically faster for smaller files. Generality, > the opt-2 files in the stdlib are a bit smaller, but the opt-1 are > not. Technically, I agree that the loading time difference is > negligible. A quick benchmark: $ time python3 -c 'import importlib as i, pydoc_data.topics as t; [i.reload(t) for _ in range(10000)]' python3 -c 4.16s user 0.45s system 99% cpu 4.646 total $ time python3 -O -c 'import importlib as i, pydoc_data.topics as t; [i.reload(t) for _ in range(10000)]' python3 -O -c 4.01s user 0.45s system 99% cpu 4.492 total $ time python3 -OO -c 'import importlib as i, pydoc_data.topics as t; [i.reload(t) for _ in range(10000)]' python3 -OO -c 3.97s user 0.42s system 98% cpu 4.467 total sudo rm /usr/lib64/python3.7/pydoc_data/__pycache__/topics.cpython-37.* $ time python3 -c 'import importlib as i, pydoc_data.topics as t; [i.reload(t) for _ in range(1000)]' python3 -c 13.73s user 0.46s system 96% cpu 14.728 total $ time python3 -O -c 'import importlib as i, pydoc_data.topics as t; [i.reload(t) for _ in range(1000)]' python3 -O -c 13.01s user 0.33s system 98% cpu 13.480 total $ time python3 -OO -c 'import importlib as i, pydoc_data.topics as t; [i.reload(t) for _ in range(1000)]' python3 -OO -c 11.95s user 0.15s system 99% cpu 12.156 total So... the benefit from -O and -OO is negligible for most scenarios. (Note that here the page cache is hot, so what is being measured is the time Pythons spends doing CPU crunching. Normally, the latency to load the file from disk would be in play, and this latency would be the same for files of similar size. The difference of a few percent between opt levels would become negligible. And note that this is particularly big file, so the time required for parsing would be even less important for small files which are much more common.) But the effect of having *some* .pyc file is not. For this file (which is 600+kb), the difference is 147.28/4.646 ≈ 30 times. So we clearly need to keep the possibility of installing .pyc files, at least optionally. > But no, we didn't do any benchmarking (yet anyway) at the scale of > the current document, that would take a lot of time and energy. The > plan is to only do them for solutions we actually decide to go for > (but only if we anticipate a change -- for example not with the > hardlink-based deduplication, but yet with the zipped stdlib). > > >>### Solution 5: Stop shipping mandatory bytecode cache > >> > >>This solution sounds simple: We do no longer ship the bytecode cache > >>mandatorily. Technically, we move the `.pyc` files to a subpackage > >>of `python3-libs` (or three different subpackages, that is not > >>important here). And we only *Recommend* them from `python3-libs` -- > >>by default, the users get them, but for space critical Fedora > >>flavors (such as container images) the maintainers can opt-out and > >>so can the powerusers. > >> > >>This would **save 18.6 MiB / 50%** -- quite a lot. > >> > >>However, as said earlier, if the bytecode cache files are not there, > >>Python attempts to create them upon first import. That can result in > >>several problems, here we will try to propose how to workaround > >>them. > > > >Below using a flag file in each __pycache__ directory is suggested. > >What about a different route: having a flag file for all descendants > >of a directory? > > The idea was to avoid traversing up, as that can potentially slow > down Python invocation from a deep PATH. But yes, that is possible > as well. No, it will not (TTBMK). The file (or the lack of it) will be cached in the dentry cache, so the kernel will give an answer extremely quickly. And the python process can easily store the directories is checked in a lru_cache or something like that, to avoid the round trip to the kernel. > >For example, /usr/lib/python3.8/.dont_write_bytecode > >would cover all modules under /usr/lib/python3.8/. > >If a .pyc file is present, python could still make use of it. > > > >This would be a nicer solution because it wouldn't require modifying > >individual packages, but would still avoid the selinux issues and > >slowdowns from failed attempts to write the optimized files. > >The __pycache__ files wouldn't need to exist at all. > > Correct. Zbyszek _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
