On 5/4/20 11:37 PM, David Rientjes wrote:
On Mon, 4 May 2020, Emanuele Giuseppe Esposito wrote:
In this patch series I introduce statsfs, a synthetic ram-based virtual
filesystem that takes care of gathering and displaying statistics for the
Linux kernel subsystems.
This is exciting, we have been looking in the same area recently. Adding
Jonathan Adams <jwadams@xxxxxxxxxx>.
In your diffstat, one thing I notice that is omitted: an update to
Documentation/* :) Any chance of getting some proposed Documentation/
updates with structure of the fs, the per subsystem breakdown, and best
practices for managing the stats from the kernel level?
Yes, I will write some documentation. Thank you for the suggestion.
Values represent quantites that are gathered by the statsfs user. Examples
of values include the number of vm exits of a given kind, the amount of
memory used by some data structure, the length of the longest hash table
chain, or anything like that. Values are defined with the
statsfs_source_add_values function. Each value is defined by a struct
statsfs_value; the same statsfs_value can be added to many different
sources. A value can be considered "simple" if it fetches data from a
user-provided location, or "aggregate" if it groups all values in the
subordinates sources that include the same statsfs_value.
This seems like it could have a lot of overhead if we wanted to
periodically track the totality of subsystem stats as a form of telemetry
gathering from userspace. To collect telemetry for 1,000 different stats,
do we need to issue lseek()+read() syscalls for each of them individually
(or, worse, open()+read()+close())?
Any thoughts on how that can be optimized? A couple of ideas:
- an interface that allows gathering of all stats for a particular
interface through a single file that would likely be encoded in binary
and the responsibility of userspace to disseminate, or
- an interface that extends beyond this proposal and allows the reader to
specify which stats they are interested in collecting and then the
kernel will only provide these stats in a well formed structure and
also be binary encoded.
Are you thinking of another file, containing all the stats for the
directory in binary format?
We've found that the one-file-per-stat method is pretty much a show
stopper from the performance view and we always must execute at least two
syscalls to obtain a single stat.
Since this is becoming a generic API (good!!), maybe we can discuss
possible ways to optimize gathering of stats in mass?
Sure, the idea of a binary format was considered from the beginning in
[1], and it can be done either together with the current filesystem, or
as a replacement via different mount options.
Thank you,
Emanuele
[1] https://lore.kernel.org/kvm/5d6cdcb1-d8ad-7ae6-7351-3544e2fa366d@xxxxxxxxxx/?fbclid=IwAR18LHJ0PBcXcDaLzILFhHsl3qpT3z2vlG60RnqgbpGYhDv7L43n0ZXJY8M
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@xxxxxxxxxx>
v1->v2 remove unnecessary list_foreach_safe loops, fix wrong indentation,
change statsfs in stats_fs
Emanuele Giuseppe Esposito (5):
refcount, kref: add dec-and-test wrappers for rw_semaphores
stats_fs API: create, add and remove stats_fs sources and values
kunit: tests for stats_fs API
stats_fs fs: virtual fs to show stats to the end-user
kvm_main: replace debugfs with stats_fs
MAINTAINERS | 7 +
arch/arm64/kvm/Kconfig | 1 +
arch/arm64/kvm/guest.c | 2 +-
arch/mips/kvm/Kconfig | 1 +
arch/mips/kvm/mips.c | 2 +-
arch/powerpc/kvm/Kconfig | 1 +
arch/powerpc/kvm/book3s.c | 6 +-
arch/powerpc/kvm/booke.c | 8 +-
arch/s390/kvm/Kconfig | 1 +
arch/s390/kvm/kvm-s390.c | 16 +-
arch/x86/include/asm/kvm_host.h | 2 +-
arch/x86/kvm/Kconfig | 1 +
arch/x86/kvm/Makefile | 2 +-
arch/x86/kvm/debugfs.c | 64 --
arch/x86/kvm/stats_fs.c | 56 ++
arch/x86/kvm/x86.c | 6 +-
fs/Kconfig | 12 +
fs/Makefile | 1 +
fs/stats_fs/Makefile | 6 +
fs/stats_fs/inode.c | 337 ++++++++++
fs/stats_fs/internal.h | 35 +
fs/stats_fs/stats_fs-tests.c | 1088 +++++++++++++++++++++++++++++++
fs/stats_fs/stats_fs.c | 773 ++++++++++++++++++++++
include/linux/kref.h | 11 +
include/linux/kvm_host.h | 39 +-
include/linux/refcount.h | 2 +
include/linux/stats_fs.h | 304 +++++++++
include/uapi/linux/magic.h | 1 +
lib/refcount.c | 32 +
tools/lib/api/fs/fs.c | 21 +
virt/kvm/arm/arm.c | 2 +-
virt/kvm/kvm_main.c | 314 ++-------
32 files changed, 2772 insertions(+), 382 deletions(-)
delete mode 100644 arch/x86/kvm/debugfs.c
create mode 100644 arch/x86/kvm/stats_fs.c
create mode 100644 fs/stats_fs/Makefile
create mode 100644 fs/stats_fs/inode.c
create mode 100644 fs/stats_fs/internal.h
create mode 100644 fs/stats_fs/stats_fs-tests.c
create mode 100644 fs/stats_fs/stats_fs.c
create mode 100644 include/linux/stats_fs.h
--
2.25.2
