Re: Error Cannot acquire state change lock from remoteDispatchDomainMigratePrepare3Params during live migration of domains

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


On Thu, Mar 07, 2024 at 03:30:30PM +0000, Daniel P. Berrangé wrote:
> I wonder if something is hitting the 'max_client_requests' limit and
> getting stalled.
> The initial thread message here says the lockup is happening during
> bulk concurrent live migrations of 200 VMs, 5 at a time.
> The default 'max_client_requests' is 5.... DANGER WILL ROBINSON...
> With live migration making requests across multiple libvirt daemons,
> if the target host has filled its 5 requests queue with long running
> operations, and then a "prepare migrate' call comes in, that'll get
> stalled behind a possibly slow operation at the RPC dispatch level.
> I'd suggest bumping 'max_client_requests' to 100 and seeing if the
> problem goes away.
> If so I wonder if we shouldn't raise our out of the box limits.
> '5' is pretty low considering the scale of virtualization hosts
> in the modern world, and where even my laptop has 20 CPUs and
> 64 GB of RAM.

FWIW I was running a simple workload inside KubeVirt (a test case
that's part of its functional test suite and involves spawning and
subsequently migrating a single VM) yesterday and I could see
warnings about hitting max_client_requests in the logs.

Andrea Bolognani / Red Hat / Virtualization
Users mailing list -- users@xxxxxxxxxxxxxxxxx
To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxx

[Index of Archives]     [Virt Tools]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux