> Von: Michael Kelley (LINUX) <mikelley@xxxxxxxxxxxxx> > From: Florian Müller <max06.net@xxxxxxxxxxx> Sent: Saturday, June 11, > 2022 10:58 AM > > > > > Von: Michael Kelley (LINUX) <mikelley@xxxxxxxxxxxxx> > > > From: Florian Müller <max06.net@xxxxxxxxxxx> Sent: Saturday, June > > > 11, > > > 2022 8:40 AM > > > > > > > > Hello there, > > > > > > > > I'm trying to debug several, probably related issues to your hv_balloon > kernel module. > > > > > > > > All tests are done on Win11 21H2 (22000.675) pro, on a Ryzen 3950x > > > > with active AMD- V, and 64GB of memory. > > > > An updated Ubuntu Server 20.04 Guest is used for comparison. It's > > > > currently running kernel 5.13.0-1023-azure, configured with 1GB > > > > memory to start, and active ballooning between 512MB and 32GB of > memory. > > > > Memory hot-plugging and -unplugging works. > > > > > > > > Issues showed up when I set up a Kali Linux Guest. I missed the > > > > memory configuration before booting up the instance, so it started > > > > with 1GB of memory, and ballooning active between 512MB and > > > > several TB of memory. Hyper-V started to allocate more and more > > > > memory to this guest since the reported memory requirements also > > > > increased. The guest kernel didn't see any of that allocated memory, as > far as I can tell. > > > > > > Hot-adding new memory is done partially by the hv_balloon driver, > > > but it also requires user space action. The user space action is provided > by udev rules. > > > In your Ubuntu Server 20.04 guest, there's a file > > > /lib/udev/rules.d/40-vm- hotadd.rules. > > > This file contains udev rules for hot-adding memory and CPUs. You > > > should be able to copy this file with the udev rules onto your Kali > > > system, and then the memory hot-add should work correctly. > > > > > > I'm not sure why Kali doesn't already have such udev rules. You > > > might grep through all the files in /lib/udev/rules.d and if there > > > are any rules for SUBSYSTEM==memory. > > > Sometimes there are rules present, but commented out. > > > > > Thanks, I'll check these ones out! > > > > In the meantime, I was able to get it working, by compiling a kernel > > with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y > > Which was previously unset. It's enabled in ubuntu and it seems to > > make hv_balloon work properly. > > This seems to be the same case with Debian. > > > > Yes, that looks like a good solution. I didn't remember that there is a kernel > config option to automatically do the onlining. With this kernel option > enabled, using a udev rule obviously isn't needed. The kernel option was > added in Linux kernel version 4.7, which might be after the last time I looked > at Hyper-V memory hot-add in detail. > > Michael > Awesome! Last question: Since not having the kernel option by default and also not having the udev rule in some distributions causes the Hyper-V host to eat up all the memory up to the defined limit (and to die eventually), should this be considered as a bug? And if the answer is no, how can I (or anyone) forward the requirement to the publishers to be solved at the source? Thank you! > > > > > > > > This is clearly reproduceable on at least 2 Hyper-V hosts, with > > > > current live images of Debian (Bullseye) and Kali Linux, but not with > Ubuntu 20.04 or 22.04. > > > > (Get the Kali live image, create a new guest (version 10), turn > > > > off secure boot and boot from that image. It takes 3-5 minutes > > > > until the issue is visible in the hyper-v console.) > > > > > > > > After running more tests with different memory settings for > > > > ballooning, I am pretty sure: > > > > - Hyper-V respects the maximum setting for the memory balloon. > > > > Although it doesn't care if there's enough memory. > > > > - Guests can't use/see more memory than they had while booting up. > > > > - Guests can unplug memory. > > > > - Guests can hotplug previously unplugged memory up to their > > > > starting amount. > > > > > > Yes, adding this memory is done via the ballooning mechanism, and does > not > > > require user space participation. The hot-add mechanism is different > from > > > ballooning, even though both are packaging in the hv_balloon driver. > > > Hot- add is required only when adding memory that was not present > > > when the VM first booted, and that's when user space participation > > > is needed via the udev rule. > > > > > Gotcha, thanks for the clarification. I wasn't aware of that. > > > > > > - The reported values seem to be off: After compiling a kernel on > > > > Kali (and cooling down), the guest kernel shows a total of 3207MB > > > > memory, with 294MB used, 137MB free, 2775MB buffers/caches and > > > > 2611MB available. Hyper-V reports 4905MB required and 5840MB > allocated. > > > > - As of kernel 5.17.11, the issue is not solved. > > > > > > > > To sum up: I could use memory ballooning if I set the initial > > > > memory size to the maximum size and wait until it got freed up. > > > > > > > > There are several reports out there about what looks like a memory > > > > leak, without a solution. > > > > > > Could you point me at those reports? I'm not familiar with them. > > Yes, as much as I'm able to find them again. > > - > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcom > m > > unity.clearlinux.org%2Ft%2Fdynamic-memory-broken-on-hyper- > v%2F3891& > > > ;data=05%7C01%7C%7Cf4d2032b2a48483e8be508da4be0c95c%7C84df9e7fe9 > f640af > > > b435aaaaaaaaaaaa%7C1%7C0%7C637905726063903963%7CUnknown%7CTWF > pbGZsb3d8 > > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3 > D%7C3 > > > 000%7C%7C%7C&sdata=Q4HXxFY2UHIvBFZv1RyEfdX21BNzo8Bd6%2BDE > 6NhMDZM%3 > > D&reserved=0 ... and the rest is gone. I'll post them when I find > > them again. > > > > Thanks. > > > > > > > Michael > > > > > > > > > > > I'm currently comparing the kernel built by canonical with the > > > > kali kernel, but as I am not really a developer, I'm not sure if I > > > > could even find the difference if there is one. So, I'm calling > > > > (and hoping) for help, and offering any support when it comes to > > > > testing. I can apply > > > patches, build kernel packages and read logs if that helps. > > > > > > > > Thx-a-lot! > > > > Flo > > I hope I'm doing this mailing list thing right... > > Looks OK to me. :-) > > Michael
