From: Florian Müller <max06.net@xxxxxxxxxxx> Sent: Saturday, June 11, 2022 10:58 AM > > > Von: Michael Kelley (LINUX) <mikelley@xxxxxxxxxxxxx> > > From: Florian Müller <max06.net@xxxxxxxxxxx> Sent: Saturday, June 11, > > 2022 8:40 AM > > > > > > Hello there, > > > > > > I'm trying to debug several, probably related issues to your hv_balloon kernel module. > > > > > > All tests are done on Win11 21H2 (22000.675) pro, on a Ryzen 3950x > > > with active AMD- V, and 64GB of memory. > > > An updated Ubuntu Server 20.04 Guest is used for comparison. It's > > > currently running kernel 5.13.0-1023-azure, configured with 1GB memory > > > to start, and active ballooning between 512MB and 32GB of memory. > > > Memory hot-plugging and -unplugging works. > > > > > > Issues showed up when I set up a Kali Linux Guest. I missed the memory > > > configuration before booting up the instance, so it started with 1GB > > > of memory, and ballooning active between 512MB and several TB of > > > memory. Hyper-V started to allocate more and more memory to this guest > > > since the reported memory requirements also increased. The guest kernel > > > didn't see any of that allocated memory, as far as I can tell. > > > > Hot-adding new memory is done partially by the hv_balloon driver, but it also > > requires user space action. The user space action is provided by udev rules. > > In your Ubuntu Server 20.04 guest, there's a file /lib/udev/rules.d/40-vm- > > hotadd.rules. > > This file contains udev rules for hot-adding memory and CPUs. You should be > > able to copy this file with the udev rules onto your Kali system, and then the > > memory hot-add should work correctly. > > > > I'm not sure why Kali doesn't already have such udev rules. You might grep > > through all the files in /lib/udev/rules.d and if there are any rules for > > SUBSYSTEM==memory. > > Sometimes there are rules present, but commented out. > > > Thanks, I'll check these ones out! > > In the meantime, I was able to get it working, by compiling a kernel with > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y > Which was previously unset. It's enabled in ubuntu and it seems to make hv_balloon > work properly. > This seems to be the same case with Debian. > Yes, that looks like a good solution. I didn't remember that there is a kernel config option to automatically do the onlining. With this kernel option enabled, using a udev rule obviously isn't needed. The kernel option was added in Linux kernel version 4.7, which might be after the last time I looked at Hyper-V memory hot-add in detail. Michael > > > > > > This is clearly reproduceable on at least 2 Hyper-V hosts, with > > > current live images of Debian (Bullseye) and Kali Linux, but not with Ubuntu 20.04 or 22.04. > > > (Get the Kali live image, create a new guest (version 10), turn off > > > secure boot and boot from that image. It takes 3-5 minutes until the > > > issue is visible in the hyper-v console.) > > > > > > After running more tests with different memory settings for ballooning, I > > > am pretty sure: > > > - Hyper-V respects the maximum setting for the memory balloon. > > > Although it doesn't care if there's enough memory. > > > - Guests can't use/see more memory than they had while booting up. > > > - Guests can unplug memory. > > > - Guests can hotplug previously unplugged memory up to their starting > > > amount. > > > > Yes, adding this memory is done via the ballooning mechanism, and does not > > require user space participation. The hot-add mechanism is different from > > ballooning, even though both are packaging in the hv_balloon driver. Hot- > > add is required only when adding memory that was not present when the > > VM first booted, and that's when user space participation is needed via the > > udev rule. > > > Gotcha, thanks for the clarification. I wasn't aware of that. > > > > - The reported values seem to be off: After compiling a kernel on Kali > > > (and cooling down), the guest kernel shows a total of 3207MB memory, > > > with 294MB used, 137MB free, 2775MB buffers/caches and 2611MB > > > available. Hyper-V reports 4905MB required and 5840MB allocated. > > > - As of kernel 5.17.11, the issue is not solved. > > > > > > To sum up: I could use memory ballooning if I set the initial memory > > > size to the maximum size and wait until it got freed up. > > > > > > There are several reports out there about what looks like a memory > > > leak, without a solution. > > > > Could you point me at those reports? I'm not familiar with them. > Yes, as much as I'm able to find them again. > - https://community.clearlinux.org/t/dynamic-memory-broken-on-hyper-v/3891 > ... and the rest is gone. I'll post them when I find them again. > Thanks. > > > > Michael > > > > > > > > I'm currently comparing the kernel built by canonical with the kali > > > kernel, but as I am not really a developer, I'm not sure if I could > > > even find the difference if there is one. So, I'm calling (and hoping) > > > for help, and offering any support when it comes to testing. I can apply > > patches, build kernel packages and read logs if that helps. > > > > > > Thx-a-lot! > > > Flo > I hope I'm doing this mailing list thing right... Looks OK to me. :-) Michael
