Currently, each local_storage type (sk, inode, task) has a 16-entry cache for local_storage data associated with a particular map. A local_storage map is assigned a fixed cache_idx when it is allocated. When looking in a local_storage for data associated with a map the cache entry at cache_idx is the only place the map can appear in cache. If the map's data is not in cache it is placed there after a search through the cache hlist. When there are >16 local_storage maps allocated for a local_storage type multiple maps have same cache_idx and thus may knock each other out of cache. BPF programs that use local_storage may require fast and consistent local storage access. For example, a BPF program using task local storage to make scheduling decisions would not be able to tolerate a long hlist search for its local_storage as this would negatively affect cycles available to applications. Providing a mechanism for such a program to ensure that its local_storage_data will always be in cache would ensure fast access. This series introduces a BPF_LOCAL_STORAGE_FORCE_CACHE flag that can be set on sk, inode, and task local_storage maps via map_extras. When a map with the FORCE_CACHE flag set is allocated it is assigned an 'exclusive' cache slot that it can't be evicted from until the map is free'd. If there are no slots available to exclusively claim, the allocation fails. BPF programs are expected to use BPF_LOCAL_STORAGE_FORCE_CACHE only if their data _must_ be in cache. The existing cache slots are used - as opposed to a separate cache - as exclusive caching is not expected to be used by the majority of local_storage BPF programs. So better to avoid adding a separate cache that will bloat memory and go unused. Patches: * Patch 1 implements kernel-side changes to support the feature * Patch 2 adds selftests validating functionality * Patch 3 is a oneline #define dedupe Dave Marchevsky (3): bpf: Introduce local_storage exclusive caching option selftests/bpf: Add local_storage exclusive cache test bpf: Remove duplicate define in bpf_local_storage.h include/linux/bpf_local_storage.h | 8 +- include/uapi/linux/bpf.h | 14 +++ kernel/bpf/bpf_inode_storage.c | 16 ++- kernel/bpf/bpf_local_storage.c | 42 ++++++-- kernel/bpf/bpf_task_storage.c | 16 ++- kernel/bpf/syscall.c | 7 +- net/core/bpf_sk_storage.c | 15 ++- .../test_local_storage_excl_cache.c | 52 +++++++++ .../bpf/progs/local_storage_excl_cache.c | 100 ++++++++++++++++++ .../bpf/progs/local_storage_excl_cache_fail.c | 36 +++++++ 10 files changed, 283 insertions(+), 23 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_local_storage_excl_cache.c create mode 100644 tools/testing/selftests/bpf/progs/local_storage_excl_cache.c create mode 100644 tools/testing/selftests/bpf/progs/local_storage_excl_cache_fail.c -- 2.30.2
