Re: [PATCH] libxl: Fix domain shutdown

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2/22/21 10:34 AM, Michal Privoznik wrote:
On 2/22/21 6:02 PM, Jim Fehlig wrote:
On 2/22/21 5:46 AM, Michal Privoznik wrote:
On 2/20/21 1:19 AM, Jim Fehlig wrote:
Commit fa30ee04a2 caused a regression in normal domain shutown.
Initiating a shutdown from within the domain or via 'virsh shutdown'
does cause the guest OS running in the domain to shutdown, but libvirt
never reaps the domain so it is always shown in a running state until
calling 'virsh destroy'.

The shutdown thread is also an internal user of the driver shutdown
machinery and eventually calls libxlDomainDestroyInternal where
the ignoreDeathEvent inhibitor is set, but running in a thread
introduces the possibility of racing with the death event from
libxl. This can be prevented by setting ignoreDeathEvent before
running the shutdown thread.

An additional improvement is to handle the destroy event synchronously
instead of spawning a thread. The time consuming aspects of destroying
a domain have been completed when the destroy event is delivered.

Signed-off-by: Jim Fehlig <jfehlig@xxxxxxxx>
---
  src/libxl/libxl_domain.c | 115 ++++++++++++++++++---------------------
  1 file changed, 54 insertions(+), 61 deletions(-)

diff --git a/src/libxl/libxl_domain.c b/src/libxl/libxl_domain.c
index d59153fffa..32dc503089 100644
--- a/src/libxl/libxl_domain.c
+++ b/src/libxl/libxl_domain.c

@@ -685,42 +658,62 @@ libxlDomainEventHandler(void *data, VIR_LIBXL_EVENT_CONST libxl_event *event)
       * after calling libxl_domain_suspend() are handled by its callers.
       */
      if (xl_reason == LIBXL_SHUTDOWN_REASON_SUSPEND)
-        goto error;
+        goto cleanup;
-    /*
-     * Start a thread to handle shutdown.  We don't want to be tying up
-     * libxl's event machinery by doing a potentially lengthy shutdown.
-     */
-    shutdown_info = g_new0(struct libxlShutdownThreadInfo, 1);
+    vm = virDomainObjListFindByID(driver->domains, event->domid);
+    if (!vm) {
+        /* Nothing to do if we can't find the virDomainObj */
+        goto cleanup;
+    }
-    shutdown_info->driver = driver;
-    shutdown_info->event = (libxl_event *)event;
-    name = g_strdup_printf("ev-%d", event->domid);
-    if (event->type == LIBXL_EVENT_TYPE_DOMAIN_SHUTDOWN)
-        ret = virThreadCreateFull(&thread, false, libxlDomainShutdownThread,
-                                  name, false, shutdown_info);
-    else if (event->type == LIBXL_EVENT_TYPE_DOMAIN_DEATH)
-        ret = virThreadCreateFull(&thread, false, libxlDomainDeathThread,
-                                  name, false, shutdown_info);
+    if (event->type == LIBXL_EVENT_TYPE_DOMAIN_SHUTDOWN) {
+        libxlDomainObjPrivatePtr priv = vm->privateData;
+        struct libxlShutdownThreadInfo *shutdown_info = NULL;
+        virThread thread;
+        g_autofree char *name = NULL;
-    if (ret < 0) {
          /*
-         * Not much we can do on error here except log it.
+         * Start a thread to handle shutdown.  We don't want to be tying up
+         * libxl's event machinery by doing a potentially lengthy shutdown.
           */
-        VIR_ERROR(_("Failed to create thread to handle domain shutdown"));
-        goto error;
-    }
+        shutdown_info = g_new0(struct libxlShutdownThreadInfo, 1);
-    /*
-     * libxlShutdownThreadInfo and libxl_event are freed in shutdown thread
-     */
-    return;
+        shutdown_info->driver = driver;
+        shutdown_info->vm = vm;

Aaah, wanted to suggest that you increment @vm's refcounter here, because ..

+        shutdown_info->event = (libxl_event *)event;
+        name = g_strdup_printf("ev-%d", event->domid);
+        /*
+         * Cleanup will be handled by the shutdown thread.
+         * Ignore the forthcoming death event from libxl
+         */
+        priv->ignoreDeathEvent = true;
+        if (virThreadCreateFull(&thread, false, libxlDomainShutdownThread,
+                                name, false, shutdown_info) < 0) {

.. what if this thread is scheduled after [1]? But then noticed this subtle ..

+            /*
+             * Not much we can do on error here except log it.
+             */
+            VIR_ERROR(_("Failed to create thread to handle domain shutdown"));
+            VIR_FREE(shutdown_info);
+            goto cleanup;
+        }
+        /*
+         * virDomainObjEndAPI is called in the shutdown thread, where
+         * libxlShutdownThreadInfo and libxl_event are also freed.
+         */
+        return;

.. return. So the refcount is okay :-)

The subtle-ness definitely crossed my mind :-). Freeing the libxl event and shutdown info was already handled in the same way. The comment was a helpful reminder about the subtle behavior. I did consider options before settling on this approach, e.g. using g_autoptr to avoid the cleanup label altogether. For the shutdown info I would only have to create a free function, but the libxl_event would require defining a new struct to hold the event and libxl_ctx, along with a free func. It seemed a bit overkill, but I can pursue that if you'd like.

Not sure if it's worth it. But if you write the patch I'm more than happy to review it.

I'll look at it in conjunction with other cleanups and improvements I'm working on. In the meantime I've pushed this fix.

BTW, while reading your reply I noticed that ignoreDeathEvent should be set to false if the thread creation fails, although it probably doesn't matter much at that point. If the approach here is still acceptable, I'll squash in the below diff before pushing. Another BTW: is it fine to push this bug/regression fix even though we've now hit 7.1.0 freeze?

That's exactly what freeze is for! I mean, if we weren't allowed to push fixes then that's the point in having freeze?

Yes, thanks for point out the obvious. Another cup of coffee would have also done that :-).

Regards,
Jim





[Index of Archives]     [Virt Tools]     [Libvirt Users]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite News]     [KDE Users]     [Fedora Tools]

  Powered by Linux