On 13.06.22 09:58, Vitaly Kuznetsov wrote: > "Michael Kelley (LINUX)" <mikelley@xxxxxxxxxxxxx> writes: > >> From: Florian Müller <max06.net@xxxxxxxxxxx> Sent: Saturday, June 11, 2022 12:45 PM >>> >>>>>>> >>>>>>> Issues showed up when I set up a Kali Linux Guest. I missed the >>>>>>> memory configuration before booting up the instance, so it started >>>>>>> with 1GB of memory, and ballooning active between 512MB and >>>>>>> several TB of memory. Hyper-V started to allocate more and more >>>>>>> memory to this guest since the reported memory requirements also >>>>>>> increased. The guest kernel didn't see any of that allocated memory, as >>>>>>> far as I can tell. >>>>>> >>>>>> Hot-adding new memory is done partially by the hv_balloon driver, >>>>>> but it also requires user space action. The user space action is provided >>>>>> by udev rules. >>>>>> In your Ubuntu Server 20.04 guest, there's a file >>>>>> /lib/udev/rules.d/40-vm- hotadd.rules. >>>>>> This file contains udev rules for hot-adding memory and CPUs. You >>>>>> should be able to copy this file with the udev rules onto your Kali >>>>>> system, and then the memory hot-add should work correctly. >>>>>> >>>>>> I'm not sure why Kali doesn't already have such udev rules. You >>>>>> might grep through all the files in /lib/udev/rules.d and if there >>>>>> are any rules for SUBSYSTEM==memory. >>>>>> Sometimes there are rules present, but commented out. >>>>>> >>>>> Thanks, I'll check these ones out! >>>>> >>>>> In the meantime, I was able to get it working, by compiling a kernel >>>>> with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y >>>>> Which was previously unset. It's enabled in ubuntu and it seems to >>>>> make hv_balloon work properly. >>>>> This seems to be the same case with Debian. >>>>> >>>> >>>> Yes, that looks like a good solution. I didn't remember that there is a kernel >>>> config option to automatically do the onlining. With this kernel option >>>> enabled, using a udev rule obviously isn't needed. The kernel option was >>>> added in Linux kernel version 4.7, which might be after the last time I looked >>>> at Hyper-V memory hot-add in detail. >>>> >>>> Michael >>>> >>> >>> Awesome! >>> >>> Last question: Since not having the kernel option by default and also not having the >>> udev rule in some distributions causes the Hyper-V host to eat up all the memory up to >>> the defined limit (and to die eventually), should this be considered as a bug? And if the >>> answer is no, how can I (or anyone) forward the requirement to the publishers to be >>> solved at the source? >>> >>> Thank you! >>> >> >> It's unclear whether this should be treated as a bug. We certainly want the >> "right" thing to happen as seamlessly as possible, but there are tradeoffs. Back >> when Vitaly Kuznetsov added CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE, >> I can see there was some debate about whether this option should be enabled >> by default. There was reluctance to do so because of potential backwards >> compatibility problems with other environments. When hot-adding real >> physical memory to a bare-metal server, apparently you don't want to >> automatically online the added memory. > > On x86/ARM you most likely do (as why plugging in memory in the first > place?) but there are not many bare metal systems which allow that. You actually do want to online in most cases, except, for example, s390x. There might be weird cases with PFN-mapping offline memory into VMs, but that is really a corner case when it comes to hotplugging memory. However, "how" you want to online memory (normal/movable) depends on the use case, the hotplug mechanism, and other factors (e.g., how much memory will we hotplug, what are the system demands regarding normal/movable memory, ...). Hyperv-balloon just wants to online memory "normal", so not to ZONE_MOVABLE. CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE should get the job done. > > Note, there were some developments after I've added > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE, namely > > commit 5f47adf762b78cae97de58d9ff01d2d44db09467 > Author: David Hildenbrand <david@xxxxxxxxxx> > Date: Mon Apr 6 20:07:44 2020 -0700 > > mm/memory_hotplug: allow to specify a default online_type > I was working on a systemd unit to just configure memory onlining semi-automatically, avoiding udev rules and to just configure auto-onlining in the kernel during system boot. Delaying memory onlining due to udev rules is a known issue when it comes to hotplugging a lot of memory via hyperv balloon. We want kernel to be onlined as fast as possible, directly when adding it in the kernel. I currently don't have the capacity to work on the systemd unit, but the latest version resides at: https://github.com/davidhildenbrand/systemd/tree/memoryhotplugd The idea is to let the unit figure out what might be best to do in the detected environment. In addition, the admin can adjust the configure/fine-tune handling without messing with udev rules, kernel parameters or the sysfs. >> >> The Ubuntu image you were testing is specifically an image configured for use >> in an Azure VM, so it makes sense to have memory automatically onlined by >> the kernel. So I looked at a generic Ubuntu 18.04 system, and it also has >> this kernel option set by default. But as another data point, RHEL/CentOS 8.4 >> does *not* have the kernel option set. So each distro evidently makes their >> own decision about this. > > Note: Fedora kernels come with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y as > well. IIRC, even on s390x. It's different on RHEL, where we leave standby memory under s390x offline via udev rules right now. > >> But both generic Ubuntu 18.04 and RHEL/CentOS 8.4 >> have the udev rules. The RHEL/CentOS 8.4 udev rule is more sophisticated >> in that it does not online the added memory when running on System/390 >> and PowerPC. > > I remember that with s390 we certainly don't want all memory to go > online automatically because but I don't remember the reason :-) The semantics of standby memory as you would find under z/VM and LPAR is different: the admin is supposed to online it on demand; until then, it's supposed to stay offline. Onlining will trigger actual backing of the VM with additional memory, so some weird form of on-demand memory hotplug. > >> >> I could envision changing the Linux kernel config rules to automatically set >> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE whenever >> CONFIG_HYPERV_BALLOON is selected. > > That could work for custom kernels but not for those shipped with > distributions as these will always have CONFIG_HYPERV_BALLOON if > Hyper-V is one of the supported targets (and it likely is). > Yeah, that would be broken and nacked by me. >> Alternatively, the Kali Linux folks could consider adding the >> appropriate udev rule. > > Or just enable CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE. We've enabled > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE in Fedora in 2016 and haven't heard > many (any?) complaints (besides ppc64 where it was disabled) since. The would work, most probably also for s390x. I assume most s390x users that run these distros (fedora, kali,...) most probably don't care about standby memory at all, meaning, not even having the option available in their LPARs/VMs. -- Thanks, David / dhildenb
