Hi James,
On 8/16/24 11:29, James Morse wrote:
> Hi Babu,
>
> Some boring comments about where the code goes...
No worries. Lets address it when we can.
>
> On 06/08/2024 23:00, Babu Moger wrote:
>> Add the functionality to enable/disable AMD ABMC feature.
>>
>> AMD ABMC feature is enabled by setting enabled bit(0) in MSR
>> L3_QOS_EXT_CFG. When the state of ABMC is changed, the MSR needs
>> to be updated on all the logical processors in the QOS Domain.
>>
>> Hardware counters will reset when ABMC state is changed. Reset the
>> architectural state so that reading of hardware counter is not considered
>> as an overflow in next update.
>>
>> The ABMC feature details are documented in APM listed below [1].
>> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
>> Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
>> Monitoring (ABMC).
>
>> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
>> index 2bd207624eec..154983a67646 100644
>> --- a/arch/x86/kernel/cpu/resctrl/internal.h
>> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
>
>> @@ -536,6 +541,14 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
>>
>> void arch_mon_domain_online(struct rdt_resource *r, struct rdt_mon_domain *d);
>>
>> +static inline bool resctrl_arch_get_abmc_enabled(void)
>> +{
>> + return rdt_resources_all[RDT_RESOURCE_L3].mbm_cntr_assign_enabled;
>> +}
>
> Once the filesystem code moves to /fs/resctrl, this can't be inlined from the
> architectures internal.h. Accessing rdt_resources_all[] from asm/resctrl.h isn't something
> that is done today... could you move this to be a non-inline function in core.c?
Sure.
>
> (this saves me moving it later!)
>
>
>> +int resctrl_arch_mbm_cntr_assign_enable(void);
>> +void resctrl_arch_mbm_cntr_assign_disable(void);
>
> Please add these in linux/resctrl.h - it saves me moving them later!
>
Sure.
>
>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> index 7e76f8d839fc..6075b1e5bb77 100644
>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> @@ -2402,6 +2402,63 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable)
>
>> +static void _resctrl_abmc_enable(struct rdt_resource *r, bool enable)
>> +{
>> + struct rdt_mon_domain *d;
>
>
>> + /*
>> + * Hardware counters will reset after switching the monitor mode.
>> + * Reset the architectural state so that reading of hardware
>> + * counter is not considered as an overflow in the next update.
>> + */
>> + list_for_each_entry(d, &r->mon_domains, hdr.list) {
>> + on_each_cpu_mask(&d->hdr.cpu_mask,
>> + resctrl_abmc_set_one_amd, &enable, 1);
>> + resctrl_arch_reset_rmid_all(r, d);
>> + }
>
> Is there any mileage in getting resctrl_arch_mbm_cntr_assign_enable()'s caller to do this?
> Every architecture that supports this will have to do this, and neither x86 nor arm64 are
> able to do it atomically, or quicker than calling resctrl_arch_reset_rmid_all() for each
> domain.
Yes. I think it is better to it at at higher level(at
rdtgroup_mbm_mode_write). That way it is common across all the architectures.
>
>> +}
>
>
>> +int resctrl_arch_mbm_cntr_assign_enable(void)
>
> Could we pass the struct rdt_resource in - instead of hard coding it to be the L3 - you
> already check hw_res->mbm_cntr_assign_enabled so no additional check is needed...
>
> Background: I'd like to reduce the amount of "I magically know its the L3" to reduce the
> work for whoever has to add monitor support for something other than the L3.
> (I've currently no plans - but someone is going to build it!)
Yes. We can pass struct rdt_resource.
>
>
>> +{
>> + struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
>> + struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
>
>> + lockdep_assert_held(&rdtgroup_mutex);
>
> After the split between the architecture and filesystem code - this lock is private to the
> filesystem. If you need to prevent concurrent enable/disable calls the architecture should
> take its own mutex.
>
> | static DEFINE_MUTEX(abmc_lock);
> ?
These calls are originated from filesystem (in this case
rdtgroup_mbm_mode_write) which holds the mutex already. I don't think we
need a separate lock here. Let me know If I am missing something here.
>
>
>> + if (r->mon.mbm_cntr_assignable && !hw_res->mbm_cntr_assign_enabled) {
>> + _resctrl_abmc_enable(r, true);
>> + hw_res->mbm_cntr_assign_enabled = true;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +void resctrl_arch_mbm_cntr_assign_disable(void)
>> +{
>> + struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
>> + struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
>> +
>> + lockdep_assert_held(&rdtgroup_mutex);
>
> (same plea for passing the resource in, and not referring to the filesystem's locks)
Sure.
>
>
>> + if (hw_res->mbm_cntr_assign_enabled) {
>> + _resctrl_abmc_enable(r, false);
>> + hw_res->mbm_cntr_assign_enabled = false;
>> + }
>> +}
>
>
> The work you do in these functions is pretty symmetric. Is it worth combining them into:
> | resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable) {
> | struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
> |
> | if (hw_res->mbm_cntr_assign_enabled != enable) {
> | _resctrl_abmc_enable(r, enable
> | hw_res->mbm_cntr_assign_enabled = enable;
> | }
> | }
Yes. We can do it.
>
> I think you need a resctrl_arch_mbm_cntr_assign_test() too - I'll comment on that patch...
>
>
> Thanks,
>
> James
>
--
Thanks
Babu Moger
