These series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature The feature details are documented in the APM listed below [1]. [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available at Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537 The patches are based on top of commit 1ac6b49423e83af2abed9be7fbdf2e491686c66b (tip/master) # Introduction AMD hardware can support 256 or more RMIDs. However, bandwidth monitoring feature only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not active. Users can create 256 or more monitor groups. But there can be only limited number of groups that can be give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups will be active for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between. The ABMC feature provides an option to the user to assign an RMID to the hardware counter and monitor the bandwidth for a longer duration. The assigned RMID will be active until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter. Without ABMC enabled, monitoring will work in current mode without assignment option. # Linux Implementation Linux resctrl subsystem provides the interface to count maximum of two memory bandwidth events per group, from a combination of available total and local events. Keeping the current interface, users can assign a maximum of 2 ABMC counters per group. User will also have the option to assign only one counter to the group. If the system runs out of assignable ABMC counters, kernel will display an error. Users need to unassign an already assigned counter to make space for new assignments. # Examples a. Check if ABMC support is available #mount -t resctrl resctrl /sys/fs/resctrl/ #cat /sys/fs/resctrl/info/L3_MON/mon_features llc_occupancy mbm_total_bytes mbm_total_bytes_config mbm_local_bytes mbm_local_bytes_config mbm_assign_capable ← Linux kernel detected ABMC feature b. Check if ABMC is enabled. By default, ABMC feature is disabled. Monitoring works in legacy monitor mode when ABMC is not enabled. #cat /sys/fs/resctrl/info/L3_MON/mbm_assign_enable 0 c. There will be new file "monitor_state" for each monitor group when ABMC feature is supported. However, monitor_state is not available if ABMC is disabled. #cat /sys/fs/resctrl/monitor_state Unsupported d. Read the event mbm_total_bytes and mbm_local_bytes. Without ABMC enabled, monitoring will work in current mode without assignment option. # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 779247936 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 765207488 e. Enable ABMC mode. #echo 1 > /sys/fs/resctrl/info/L3_MON/mbm_assign_enable #cat /sys/fs/resctrl/info/L3_MON/mbm_assign_enable 1 f. Read the monitor states. By default, both total and local MBM events are in "unassign" state. #cat /sys/fs/resctrl/monitor_state total=unassign;local=unassign g. Read the event mbm_total_bytes and mbm_local_bytes. In ABMC mode, the MBA events are not available until the user assigns the events explicitly. #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes Unsupported #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes Unsupported h. The event llc_occupancy is not affected by ABMC mode. Users can still read the llc_occupancy. #cat /sys/fs/resctrl/mon_data/mon_L3_00/llc_occupancy 557056 i. Now assign the total event and read the monitor_state. #echo total=assign > /sys/fs/resctrl/monitor_state #cat /sys/fs/resctrl/monitor_state total=assign;local=unassign j. Now that the total event is assigned. Read the total event. #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 6136000 k. Now assign the local event and read the monitor_state. #echo local=assign > /sys/fs/resctrl/monitor_state #cat /sys/fs/resctrl/monitor_state total=assign;local=assign Users can also assign both total and local events in one single command. l. Now that both total and local events are assigned, read the events. #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 6136000 #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 58694 m. Check the bandwidth configuration for the group. Note that bandwidth configuration has a domain scope. Total event defaults to 0x7F (to count all the events) and local event defaults to 0x15 (to count all the local numa events). The event bitmap decoding is available at https://www.kernel.org/doc/Documentation/x86/resctrl.rst in section "mbm_total_bytes_config", "mbm_local_bytes_config": #cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 0=0x7f;1=0x7f #cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x15;1=0xi15 n. Change the bandwidth source for domain 0 for the total event to count only reads. Note that this change effects lotal events on the domain 0. #echo 0=0x33 > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config #cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 0=0x33;1=0x7F o. Now read the total event again. The mbm_total_bytes should display only the read events. #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 314101 p. Unmount the resctrl #umount /sys/fs/resctrl/ NOTE: For simplicity these examples are run on a default resctrl group. Similar experiments are can be run non-defaults groups. --- v2: a. Major change is the way ABMC is enabled. Earlier, user needed to remount with -o abmc to enable ABMC feature. Removed that option now. Now users can enable ABMC by "$echo 1 to /sys/fs/resctrl/info/L3_MON/mbm_assign_enable". b. Added new word 21 to x86/cpufeatures.h. c. Display unsupported if user attempts to read the events when ABMC is enabled and event is not assigned. d. Display monitor_state as "Unsupported" when ABMC is disabled. e. Text updates and rebase to latest tip tree (as of Jan 18). f. This series is still work in progress. I am yet to hear from ARM developers. v1 : https://lore.kernel.org/lkml/20231201005720.235639-1-babu.moger@xxxxxxx/ Babu Moger (17): x86/cpufeatures: Add word 21 for scattered CPUID features x86/resctrl: Add support for Assignable Bandwidth Monitoring Counters (ABMC) x86/resctrl: Add ABMC feature in the command line options x86/resctrl: Detect Assignable Bandwidth Monitoring feature details x86/resctrl: Introduce resctrl_file_fflags_init x86/resctrl: Introduce interface to display number of ABMC counters x86/resctrl: Add support to enable/disable ABMC feature x86/resctrl: Introduce the interface to display ABMC state x86/resctrl: Introdruce rdtgroup_assign_enable_write x86/resctrl: Add interface to display monitor state of the group x86/resctrl: Report Unsupported when MBM events are read x86/resctrl: Initialize assignable counters bitmap x86/resctrl: Add data structures for ABMC assignment x86/resctrl: Introduce mbm_total_cfg and mbm_local_cfg x86/resctrl: Add the interface to assign the RMID x86/resctrl: Add the interface unassign the RMID x86/resctrl: Update RMID assignments on event configuration changes .../admin-guide/kernel-parameters.txt | 2 +- Documentation/arch/x86/resctrl.rst | 62 +++ arch/x86/include/asm/cpufeature.h | 6 +- arch/x86/include/asm/cpufeatures.h | 7 +- arch/x86/include/asm/disabled-features.h | 3 +- arch/x86/include/asm/msr-index.h | 2 + arch/x86/include/asm/required-features.h | 3 +- arch/x86/kernel/cpu/cpuid-deps.c | 3 + arch/x86/kernel/cpu/resctrl/core.c | 25 +- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 15 + arch/x86/kernel/cpu/resctrl/internal.h | 54 +- arch/x86/kernel/cpu/resctrl/monitor.c | 26 +- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 466 +++++++++++++++++- arch/x86/kernel/cpu/scattered.c | 1 + include/linux/resctrl.h | 2 + 15 files changed, 651 insertions(+), 26 deletions(-) -- 2.34.1
