On Thu, Jan 28, 2021, 11:11 AM Alexander Bokovoy <abokovoy@xxxxxxxxxx> wrote:
On to, 28 tammi 2021, Tomasz Torcz wrote:
>On Thu, Jan 28, 2021 at 11:04:34AM -0500, Rob Crittenden wrote:
>> Zbigniew Jędrzejewski-Szmek wrote:
>> > On Thu, Jan 28, 2021 at 03:20:38PM +0200, Alexander Bokovoy wrote:
>> >> With today's OpenQA tests I can point out that using zram on 2048MB RAM
>> >> VMs actually breaks FreeIPA deployment:
>> >> https://openqa.fedoraproject.org/tests/763006#step/role_deploy_domain_controller/35
>> >>
>> >> OpenQA uses 2048MB RAM for QEMU VMs and this was typically OK for
>> >> FreeIPA deployment with integrated CA and DNS server. Not anymore with
>> >> zram activated:
>> >>
>> >> Jan 27 21:17:47 fedora zram_generator::generator[25243]: Creating unit dev-zram0.swap (/dev/zram0 with 1384MB)
>> >>
>> >> which ends up eating 2/3rds of the whole memory budget and FreeIPA
>> >> installer fails:
>> >>
>> >> 2021-01-28T02:18:31Z DEBUG ipa-server-install was invoked with arguments [] and options: {'unattended': True, 'ip_addresses': None, 'domain_name': 'test.openqa.fedoraproject.org', 'realm_name': 'TEST.OPENQA.FEDORAPROJECT.ORG', 'host_name': None, 'ca_cert
>> >> 2021-01-28T02:18:31Z DEBUG IPA version 4.9.1-1.fc34
>> >> 2021-01-28T02:18:31Z DEBUG IPA platform fedora
>> >> 2021-01-28T02:18:31Z DEBUG IPA os-release Fedora 34 (Server Edition Prerelease)
>> >> 2021-01-28T02:18:31Z DEBUG Available memory is 823529472B
>> >> ...
>> >> 2021-01-28T02:18:31Z DEBUG The ipa-server-install command failed, exception: ScriptError: Less than the minimum 1.2GB of RAM is available, 0.77GB available
>> >> 2021-01-28T02:18:31Z ERROR Less than the minimum 1.2GB of RAM is available, 0.77GB available
>> >> 2021-01-28T02:18:31Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
>> >
>> > Enabling zram doesn't really "take away memory", because no pre-allocation happens.
>> > If there is no physical swap, then adding zram0 should just shown additional
>> > swap space, so I don't think it could cause the check to fail.
>> > But if there is physical swap, the zram device is used with higher preference
>> > than the physical swap. So I think the explanation could be that the VM has
>> > a swap partition. Before, some pages would be swapped out to zram, and some would
>> > be swapped out to the "real" swap. The fraction of RAM used for compressed zram
>> > would be on the order of 25% (zram-fraction=0.5 multiplied by typical compression 2:1).
>> >
>> > But now the kernel sees more zram swap, so it inserts pages there, taking away
>> > more of RAM, instead of saving pages to disk. So more memory (maybe 50% RAM) is
>> > used for the unswappable compressed pages. But this shouldn't break things:
>> > if there is enough pressure, pages would be swapped out to the physical swap device
>> > too.
>> >
>> > Assuming that this guess is correct, the check that ipa-server-install is
>> > doing should be adjusted. It should use the total available memory (ram + all kinds
>> > of swap) in the check, and not just available uncompressed pages.
>> > Or if it wants to ignore disk-based swap for some reason, it should use
>> > ram + zram + in-memory-zwap in the check.
>>
>> For bare metal IPA uses the python3-psutil call:
>> psutil.virtual_memory.available()
>>
>> I don't know how/if psutil reports zram (or cgroup v1 and v2 for that
>> matter).
>
> psutil (in general) reports data from /proc/meminfo; available come
> from MemAvailable: in that file. This is defined in kernel as:
>
>MemAvailable: An estimate of how much memory is available for starting new
> applications, without swapping. Calculated from MemFree,
> SReclaimable, the size of the file LRU lists, and the low
> watermarks in each zone.
> The estimate takes into account that the system needs some
> page cache to function well, and that not all reclaimable
> slab will be reclaimable, due to items being in use. The
> impact of those factors will vary from system to system.
>
>Notice "without swapping" in second line. Next question, how zram impacts
>reporting of MemAvailable by kernel?
This is a good note. If zram breaks kernel API promise to user space
(/proc/meminfo is one such API), how can it be enabled by default. I
also would question enabling zram by default if it does not play along
with cgroups. We do depend on cgroups being properly managed by systemd,
including resource allocation.
In my opinion, zram enablement in Fedora is quite premature.
It's the default Fedora wide since Fedora 33. It's been used by default in Fedora IoT since the beginning, and in openQA Anaconda tests for even longer than that.
What's premature about it?
Chris Murphy
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
