Hi Luiz,
pe, 2024-03-08 kello 10:12 -0500, Luiz Augusto von Dentz kirjoitti:
> Hi Pauli,
>
> On Thu, Mar 7, 2024 at 4:05 PM Pauli Virtanen <pav@xxxxxx> wrote:
> >
> > Add special implementation of fd watcher GSource for audio use.
> >
> > For audio use cases, sound server may turn on TX timestamping on a
> > socket that we are watching. In this case, we shall not consider the TX
> > timestamping POLLERR as a socket error condition, nor read the TX
> > timestamps.
> >
> > When TX timestamps appear in errqueue, switch from fd poll wait to
> > polling the fd at regular intervals. This is because unread errqueue
> > causes poll() to wake up immediately, so the mainloop cannot block on
> > that, and we have to use a timer instead with some reasonable timeout
> > for the use case.
> >
> > This avoids consuming CPU waking up on timestamps we aren't going to
> > read, and also avoids busy looping if timestamping was left on but
> > errqueue is not flushed.
> >
> > Implement this only for io-glib; it is only needed for audio use cases
> > that anyway are using glib. Uses features from GLib 2.36 (from 2013) so
> > update configure.ac also.
>
> Do we really need to enable the TX timestamping on the daemon side
> though? Or that being enabled at pipewire enables it on all fd that
> have been duplicated? I'd assume that was not the case but perhaps the
> kernel code is actually doing TX timestamping regardless if the socket
> had it enabled or not? Btw, I'd consider disabling the POLLERR
> entirely if that happens and just rely on POLLHUP/shutdown to clean
> up, or perhaps we can set up a BPF filter that drops the errqueue
> messages that way we don't have to change anything on how we handle
> POLLERR since that should prevent the daemon to be wakeup.
setsockopt(), and errqueue are socket-level things, so what you do on
one fd handle affects all the others. E.g. getsockopt() in BlueZ sees
any setsockopt() done in Pipewire, similarly for recvmsg() from RX
queue or errqueue, or poll() wakeups on POLLIN / POLLERR / etc.
SO_TIMESTAMPING is only set on Pipewire side, and kernel sends
timestamps only when it is set, but you get all its effects also on the
fd BlueZ has when it's enabled on a fd PW has, since it's the same
socket.
Since errqueue is shared, any packets recv'd from it on BlueZ side, are
not received on Pipewire side.
poll() wakes up with POLLERR when something is in errqueue, see sk-
>sk_error_queue check in bt_sock_poll. This is the same for all of
net/* afaics. The poll() wakeup is again socket-wide.
POLLERR is always enabled in poll() / epoll() / select(), this is set
in fs/eventpoll.c side. AFAIK there's no way to stop waking up on it.
E.g. removing G_IO_ERR from the g_io_add_watch flags results to busy
loop inside GLib if something is left in the errqueue.
The net/* TX timestamping design is also such that after turning off
SO_TIMESTAMPING on a socket, it will still send TX timestamps for any
packets queued before the setsockopt().
These things seem to be design decisions in the net/* TX timestamping.
***
Re BPF: it looks hard, we'd have to make epoll() wait differently for
different fds. I'm not sure if fs/eventpoll.c has necessary BPF hooks?
I don't think we can drop messages from errqueue, since it'd be socket
level and we'd like some process to read them. It also looks like
sk_filter is not used on the errqueue in net/*
Can it really be done?
***
We probably could however have a custom setsockopt(SOL_BLUETOOTH,
BT_DISABLE_ERRQUEUE_POLL) that optionally disables POLLERR for non-
empty errqueue in bt_sock_poll(), which probably should work.
Then the TX timestamps would be need to be read by periodically waking
up to flush the queue, but that'd be natural to do here since we are
anyway waking up periodically to send packets.
Unless I'm missing something, the viable options are we (i) do that, or
(ii) do like in this patch, or (iii) use ioctl or something instead.
I'd maybe go with the new setsockopt(), if that works?
>
> > ---
> >
> > Notes:
> > This was the remaining BlueZ part of the TX timestamping additions.
> > Couldn't find a better way to do it, but it has to be done, so that the
> > TX timestamping can be used for the purpose it would be added.
> >
> > I'll probably send v2 for the others in a few days.
> >
> > acinclude.m4 | 3 +-
> > configure.ac | 2 +-
> > src/shared/io-glib.c | 153 ++++++++++++++++++++++++++++++++++++++++++-
> > src/shared/io-glib.h | 20 ++++++
> > 4 files changed, 174 insertions(+), 4 deletions(-)
> > create mode 100644 src/shared/io-glib.h
> >
> > diff --git a/acinclude.m4 b/acinclude.m4
> > index 4b73a5bfc..6c36c177e 100644
> > --- a/acinclude.m4
> > +++ b/acinclude.m4
> > @@ -62,8 +62,7 @@ AC_DEFUN([COMPILER_FLAGS], [
> > with_cflags="$with_cflags -Wswitch-enum"
> > with_cflags="$with_cflags -Wformat -Wformat-security"
> > with_cflags="$with_cflags -DG_DISABLE_DEPRECATED"
> > - with_cflags="$with_cflags -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_28"
> > - with_cflags="$with_cflags -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_32"
> > + with_cflags="$with_cflags -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_36"
> > fi
> > AC_SUBST([WARNING_CFLAGS], $with_cflags)
> > ])
> > diff --git a/configure.ac b/configure.ac
> > index 9ebc250cf..959a27b4d 100644
> > --- a/configure.ac
> > +++ b/configure.ac
> > @@ -72,7 +72,7 @@ AC_CHECK_LIB(dl, dlopen, dummy=yes,
> >
> > AC_CHECK_HEADERS(linux/types.h linux/if_alg.h linux/uinput.h linux/uhid.h sys/random.h)
> >
> > -PKG_CHECK_MODULES(GLIB, glib-2.0 >= 2.28)
> > +PKG_CHECK_MODULES(GLIB, glib-2.0 >= 2.36)
> >
> > if (test "${enable_threads}" = "yes"); then
> > AC_DEFINE(NEED_THREADS, 1, [Define if threading support is required])
> > diff --git a/src/shared/io-glib.c b/src/shared/io-glib.c
> > index 754043db1..8b76348dd 100644
> > --- a/src/shared/io-glib.c
> > +++ b/src/shared/io-glib.c
> > @@ -13,10 +13,12 @@
> > #endif
> >
> > #include <errno.h>
> > +#include <sys/socket.h>
> >
> > #include <glib.h>
> >
> > #include "src/shared/io.h"
> > +#include "src/shared/io-glib.h"
> >
> > struct io_watch {
> > struct io *io;
> > @@ -29,11 +31,19 @@ struct io_watch {
> > struct io {
> > int ref_count;
> > GIOChannel *channel;
> > + bool err_watch;
> > struct io_watch *read_watch;
> > struct io_watch *write_watch;
> > struct io_watch *disconnect_watch;
> > };
> >
> > +struct io_err_watch {
> > + GSource source;
> > + GIOChannel *io;
> > + GIOCondition events;
> > + gpointer tag;
> > +};
> > +
> > static struct io *io_ref(struct io *io)
> > {
> > if (!io)
> > @@ -179,10 +189,17 @@ static struct io_watch *watch_new(struct io *io, GIOCondition cond,
> >
> > prio = cond == G_IO_HUP ? G_PRIORITY_DEFAULT_IDLE : G_PRIORITY_DEFAULT;
> >
> > - watch->id = g_io_add_watch_full(io->channel, prio,
> > + if (!io->err_watch)
> > + watch->id = g_io_add_watch_full(io->channel, prio,
> > cond | G_IO_ERR | G_IO_NVAL,
> > watch_callback, watch,
> > watch_destroy);
> > + else
> > + watch->id = io_glib_add_err_watch_full(io->channel, prio,
> > + cond | G_IO_ERR | G_IO_NVAL,
> > + watch_callback, watch,
> > + watch_destroy);
> > +
> > if (watch->id == 0) {
> > watch_destroy(watch);
> > return NULL;
> > @@ -250,6 +267,15 @@ bool io_set_disconnect_handler(struct io *io, io_callback_func_t callback,
> > return io_set_handler(io, G_IO_HUP, callback, user_data, destroy);
> > }
> >
> > +bool io_set_use_err_watch(struct io *io, bool err_watch)
> > +{
> > + if (!io)
> > + return false;
> > +
> > + io->err_watch = err_watch;
> > + return true;
> > +}
> > +
> > ssize_t io_send(struct io *io, const struct iovec *iov, int iovcnt)
> > {
> > int fd;
> > @@ -278,3 +304,128 @@ bool io_shutdown(struct io *io)
> > return g_io_channel_shutdown(io->channel, TRUE, NULL)
> > == G_IO_STATUS_NORMAL;
> > }
> > +
> > +/*
> > + * GSource implementation that tolerates non-empty MSG_ERRQUEUE, without
> > + * attempting to flush it. This is intended for use with TX timestamping in
> > + * cases where someone else is reading the timestamps and we are only interested
> > + * in POLLHUP or socket errors.
> > + */
> > +
> > +static gint64 io_err_watch_wakeup;
> > +
> > +static gboolean io_err_watch_dispatch(GSource *source,
> > + GSourceFunc callback, gpointer user_data)
> > +{
> > + struct io_err_watch *watch = (void *)source;
> > + GIOFunc func = (void *)callback;
> > + gint64 timeout = 500 * G_TIME_SPAN_MILLISECOND;
> > + GIOCondition cond;
> > + int fd;
> > +
> > + if (!func)
> > + return FALSE;
> > +
> > + fd = g_io_channel_unix_get_fd(watch->io);
> > +
> > + /*
> > + * If woken up by POLLERR only, and SO_ERROR is not set, ignore this
> > + * event. Also disable polling for some time so that we don't busy loop
> > + * if nobody is reading from the errqueue.
> > + */
> > +
> > + if (watch->tag)
> > + cond = g_source_query_unix_fd(&watch->source, watch->tag);
> > + else
> > + cond = 0;
> > +
> > + if (cond == G_IO_ERR) {
> > + int err, ret;
> > + socklen_t len = sizeof(err);
> > +
> > + ret = getsockopt(fd, SOL_SOCKET, SO_ERROR, &err, &len);
> > + if (ret == 0 && err == 0) {
> > + g_source_remove_unix_fd(&watch->source, watch->tag);
> > + watch->tag = NULL;
> > +
> > + /* io_err watches all wake up at the same time */
> > + if (!io_err_watch_wakeup)
> > + io_err_watch_wakeup = g_get_monotonic_time()
> > + + timeout;
> > +
> > + g_source_set_ready_time(&watch->source,
> > + io_err_watch_wakeup);
> > + return TRUE;
> > + }
> > + }
> > +
> > + if (g_source_get_ready_time(&watch->source) != -1) {
> > + g_assert(!watch->tag);
> > + io_err_watch_wakeup = 0;
> > + watch->tag = g_source_add_unix_fd(&watch->source, fd,
> > + watch->events);
> > + g_source_set_ready_time(&watch->source, -1);
> > + }
> > +
> > + cond &= watch->events;
> > +
> > + if (cond)
> > + return func(watch->io, cond, user_data);
> > + else
> > + return TRUE;
> > +}
> > +
> > +static void io_err_watch_finalize(GSource *source)
> > +{
> > + struct io_err_watch *watch = (void *)source;
> > +
> > + if (watch->tag)
> > + g_source_remove_unix_fd(&watch->source, watch->tag);
> > +
> > + g_io_channel_unref(watch->io);
> > +}
> > +
> > +guint io_glib_add_err_watch_full(GIOChannel *io, gint priority,
> > + GIOCondition events,
> > + GIOFunc func, gpointer user_data,
> > + GDestroyNotify notify)
> > +{
> > + static GSourceFuncs source_funcs = {
> > + .dispatch = io_err_watch_dispatch,
> > + .finalize = io_err_watch_finalize,
> > + };
> > + GSourceFunc callback = (void *)func;
> > + struct io_err_watch *watch;
> > + gint fd;
> > + guint id;
> > +
> > + g_return_val_if_fail(!(events & (G_IO_IN | G_IO_OUT)), 0);
> > + g_return_val_if_fail(func, 0);
> > +
> > + fd = g_io_channel_unix_get_fd(io);
> > +
> > + watch = (void *)g_source_new(&source_funcs,
> > + sizeof(struct io_err_watch));
> > +
> > + watch->io = g_io_channel_ref(io);
> > + watch->events = events;
> > + watch->tag = g_source_add_unix_fd(&watch->source, fd, events);
> > +
> > + g_source_set_static_name((void *)watch, "io_glib_err_watch");
> > + g_source_set_callback(&watch->source, callback, user_data, notify);
> > +
> > + if (priority != G_PRIORITY_DEFAULT)
> > + g_source_set_priority(&watch->source, priority);
> > +
> > + id = g_source_attach(&watch->source, NULL);
> > + g_source_unref(&watch->source);
> > +
> > + return id;
> > +}
> > +
> > +guint io_glib_add_err_watch(GIOChannel *io, GIOCondition events, GIOFunc func,
> > + gpointer user_data)
> > +{
> > + return io_glib_add_err_watch_full(io, G_PRIORITY_DEFAULT, events,
> > + func, user_data, NULL);
> > +}
> > diff --git a/src/shared/io-glib.h b/src/shared/io-glib.h
> > new file mode 100644
> > index 000000000..1db6fd468
> > --- /dev/null
> > +++ b/src/shared/io-glib.h
> > @@ -0,0 +1,20 @@
> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > +/*
> > + *
> > + * BlueZ - Bluetooth protocol stack for Linux
> > + *
> > + * Copyright (C) 2012-2014 Intel Corporation. All rights reserved.
> > + *
> > + *
> > + */
> > +
> > +#include <glib.h>
> > +
> > +guint io_glib_add_err_watch(GIOChannel *io, GIOCondition events,
> > + GIOFunc func, gpointer user_data);
> > +guint io_glib_add_err_watch_full(GIOChannel *io, gint priority,
> > + GIOCondition events, GIOFunc func,
> > + gpointer user_data,
> > + GDestroyNotify notify);
> > +
> > +bool io_set_use_err_watch(struct io *io, bool err_watch);
> > --
> > 2.44.0
> >
> >
>
>
