On 01/20/2016 08:51 AM, Leno Hou wrote:
1. When switching CPUs to offline/online in a system more than 128 cpus
2. When using virsh to destroy domain in a system with more interface
All of above happens nl_recv returned with error: No buffer space available.
This patch sets the socket buffer size to 128K and turns on message peeking
on new function virNetlinkAlloc,as this would solve this problem totally
and permanetly.
Signed-off-by: Leno Hou <houqy@xxxxxxxxxxxxxxxxxx>
Cc: Wenyi Gao <wenyi@xxxxxxxxxxxxxxxxxx>
CC: Laine Stump <laine@xxxxxxxxx>
CC: Michal Privoznik <mprivozn@xxxxxxxxxx>
---
src/util/virnetlink.c | 42 ++++++++++++++++++++++++++++++++++++++----
src/util/virnetlink.h | 7 +++++++
2 files changed, 45 insertions(+), 4 deletions(-)
diff --git a/src/util/virnetlink.c b/src/util/virnetlink.c
index 679b48e..02f5215 100644
--- a/src/util/virnetlink.c
+++ b/src/util/virnetlink.c
@@ -64,13 +64,11 @@ struct virNetlinkEventHandle {
};
# ifdef HAVE_LIBNL1
-# define virNetlinkAlloc nl_handle_alloc
+# define virSocketSetBufferSize nl_set_buffer_size
# define virNetlinkFree nl_handle_destroy
-typedef struct nl_handle virNetlinkHandle;
# else
-# define virNetlinkAlloc nl_socket_alloc
+# define virSocketSetBufferSize nl_socket_set_buffer_size
# define virNetlinkFree nl_socket_free
-typedef struct nl_sock virNetlinkHandle;
# endif
typedef struct _virNetlinkEventSrvPrivate virNetlinkEventSrvPrivate;
@@ -108,6 +106,42 @@ static virNetlinkHandle *placeholder_nlhandle;
/* Function definitions */
/**
+ * virNetlinkAlloc
+ *
+ * Perform netlink allocation which was defined differently for libnl3
+ * vs libnl-1, and did all of three in one function
+ * 1) create socket
+ * 2) set larger buffer size
+ * 3) turn on message peeking
+ *
+ * see the following email message:
+ * https://www.redhat.com/archives/libvir-list/2016-January/msg00536.html
+ *
+ * Returns virNetlinkHandle
+ */
+virNetlinkHandle *virNetlinkAlloc(void)
+{
+ virNetlinkHandle *netlinknh;
+ #ifdef HAVE_LIBNL1
+ netlinknh = nl_handle_alloc();
+ #else
+ netlinknh = nl_socket_alloc();
+ #endif
+
+ if (virSocketSetBufferSize(netlinknh,131702,0) >= 0) {
+ nl_socket_enable_msg_peek(netlinknh);
+ return netlinknh;
+ }
+
+ virReportSystemError(errno,
+ "%s",_("can not set socket buffer size to 128k"));
+
+ virNetlinkFree(netlinknh);
+
+ return NULL;
+}
+
+/**
* virNetlinkStartup:
*
* Perform any initialization that needs to take place before the
diff --git a/src/util/virnetlink.h b/src/util/virnetlink.h
index 0664a7a..b4a2010 100644
--- a/src/util/virnetlink.h
+++ b/src/util/virnetlink.h
@@ -44,6 +44,13 @@ struct nlmsghdr;
# endif /* __linux__ */
+# ifdef HAVE_LIBNL1
+typedef struct nl_handle virNetlinkHandle;
+# else
+typedef struct nl_sock virNetlinkHandle;
+# endif
+virNetlinkHandle *virNetlinkAlloc(void);
+
int virNetlinkStartup(void);
void virNetlinkShutdown(void);
I had earlier tried modifying your previous patch to match this one, and
found that the attempt to set the buffer size failed if the socket
wasn't already connected. I solved this by making a separate function
virNetLinkCreateSocket(int protocol) which would create and connect a
socket, then set the buffer size and turn on message peeking. I then
called this function from all of the places that were previously calling
virNetlinkAlloc() (except one - the nlhandle created virNetlinkStartup()
is never connected, so has no protocol, but it also is never used to
receive anything, so it doesn't need a larger buffer or message
peeking). In my tests anyway, this worked properly.
I've attached my modified patch below. Since it started out from your
patch, I left it attributed to you and just added my name to a
Signed-off-by:. Assuming it fixes your problem, are you okay with this
patch going in?
>From 71a03fe6a78f144d2f21a46ce3b5796b94721599 Mon Sep 17 00:00:00 2001
From: Leno Hou <houqy@xxxxxxxxxxxxxxxxxx>
Date: Mon, 11 Jan 2016 14:59:00 +0800
Subject: [PATCH] util: increase libnl buffer size
In the following cases nl_recv() was returning the error "No buffer
space available":
* When switching CPUs to offline/online in a system more than 128 cpus
* When using virsh to destroy domain in a system with many interfaces
This patch sets the buffer size for all netlink sockets created by
libnl to 128K and turns on message peeking for nl_recv(). This
eliminates the "No buffer space available" errors seen in the cases
above, and also preempts other future errors the smaller buffers could
have caused.
Signed-off-by: Leno Hou <houqy@xxxxxxxxxxxxxxxxxx>
Signed-off-by: Laine Stump <laine@xxxxxxxxx>
---
src/util/virnetlink.c | 83 +++++++++++++++++++++++++++++++++++----------------
1 file changed, 57 insertions(+), 26 deletions(-)
diff --git a/src/util/virnetlink.c b/src/util/virnetlink.c
index 679b48e..5491ece 100644
--- a/src/util/virnetlink.c
+++ b/src/util/virnetlink.c
@@ -1,6 +1,6 @@
/*
- * Copyright (C) 2010-2015 Red Hat, Inc.
- * Copyright (C) 2010-2012 IBM Corporation
+ * Copyright (C) 2010-2016 Red Hat, Inc.
+ * Copyright (C) 2010-2012, 2016 IBM Corporation
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
@@ -65,10 +65,12 @@ struct virNetlinkEventHandle {
# ifdef HAVE_LIBNL1
# define virNetlinkAlloc nl_handle_alloc
+# define virNetlinkSetBufferSize nl_set_buffer_size
# define virNetlinkFree nl_handle_destroy
typedef struct nl_handle virNetlinkHandle;
# else
# define virNetlinkAlloc nl_socket_alloc
+# define virNetlinkSetBufferSize nl_socket_set_buffer_size
# define virNetlinkFree nl_socket_free
typedef struct nl_sock virNetlinkHandle;
# endif
@@ -160,6 +162,57 @@ virNetlinkShutdown(void)
}
}
+
+/**
+ * virNetLinkCreateSocket:
+ *
+ * @protocol: which protocol to connect to (e.g. NETLINK_ROUTE,
+ *
+ * Create a netlink socket, set its buffer size, and turn on message
+ * peeking (so the buffer size can be dynamically increased if
+ * needed).
+ *
+ * Returns a handle to the new netlink socket, or 0 if there was a failure.
+ *
+ */
+static virNetlinkHandle *
+virNetlinkCreateSocket(int protocol)
+{
+ virNetlinkHandle *nlhandle = NULL;
+
+ if (!(nlhandle = virNetlinkAlloc())) {
+ virReportSystemError(errno, "%s",
+ _("cannot allocate nlhandle for netlink"));
+ goto error;
+ }
+ if (nl_connect(nlhandle, protocol) < 0) {
+ virReportSystemError(errno,
+ _("cannot connect to netlink socket "
+ "with protocol %d"), protocol);
+ goto error;
+ }
+
+ if (virNetlinkSetBufferSize(nlhandle, 131702, 0) < 0) {
+ virReportSystemError(errno, "%s",
+ _("cannot set netlink socket buffer "
+ "size to 128k"));
+ goto error;
+ }
+ nl_socket_enable_msg_peek(nlhandle);
+
+ cleanup:
+ return nlhandle;
+
+ error:
+ if (nlhandle) {
+ nl_close(nlhandle);
+ virNetlinkFree(nlhandle);
+ nlhandle = NULL;
+ }
+ goto cleanup;
+}
+
+
/**
* virNetlinkCommand:
* @nlmsg: pointer to netlink message
@@ -200,19 +253,8 @@ int virNetlinkCommand(struct nl_msg *nl_msg,
goto cleanup;
}
- nlhandle = virNetlinkAlloc();
- if (!nlhandle) {
- virReportSystemError(errno,
- "%s", _("cannot allocate nlhandle for netlink"));
- goto cleanup;
- }
-
- if (nl_connect(nlhandle, protocol) < 0) {
- virReportSystemError(errno,
- _("cannot connect to netlink socket with protocol %d"),
- protocol);
+ if (!(nlhandle = virNetlinkCreateSocket(protocol)))
goto cleanup;
- }
fd = nl_socket_get_fd(nlhandle);
if (fd < 0) {
@@ -663,19 +705,8 @@ virNetlinkEventServiceStart(unsigned int protocol, unsigned int groups)
virNetlinkEventServerLock(srv);
/* Allocate a new socket and get fd */
- srv->netlinknh = virNetlinkAlloc();
-
- if (!srv->netlinknh) {
- virReportSystemError(errno,
- "%s", _("cannot allocate nlhandle for virNetlinkEvent server"));
+ if (!(srv->netlinknh = virNetlinkCreateSocket(protocol)))
goto error_locked;
- }
-
- if (nl_connect(srv->netlinknh, protocol) < 0) {
- virReportSystemError(errno,
- _("cannot connect to netlink socket with protocol %d"), protocol);
- goto error_server;
- }
fd = nl_socket_get_fd(srv->netlinknh);
if (fd < 0) {
--
2.5.0
--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list