Hi Peter,
On 8/19/24 11:27 AM, Peter Newman wrote:
On Mon, Aug 19, 2024 at 7:53 AM Reinette Chatre
<reinette.chatre@xxxxxxxxx> wrote:
On 8/16/24 11:09 AM, Reinette Chatre wrote:
On 8/16/24 10:16 AM, Peter Newman wrote:
On Fri, Aug 16, 2024 at 10:01 AM Reinette Chatre
<reinette.chatre@xxxxxxxxx> wrote:
On 8/16/24 9:31 AM, James Morse wrote:
On 06/08/2024 23:00, Babu Moger wrote:
Introduce interface to switch between ABMC and legacy modes.
By default ABMC is enabled on boot if the feature is available.
Provide the interface to go back to legacy mode if required.
I may have missed it on an earlier version ... why would anyone want the non-ABMC
behaviour on hardware that requires it: counters randomly reset and randomly return
'Unavailable'... is that actually useful?
You default this to on, so there isn't a backward compatibility argument here.
It seems like being able to disable this is a source of complexity - is it needed?
The ability to go back to legacy was added while looking ahead to support the next
"assignable counter" feature that is software based ("soft-RMID" .. "soft-ABMC"?).
This series adds support for ABMC on recent AMD hardware to address the issue described
in cover letter. This issue also exists on earlier AMD hardware that does not have the ABMC
feature and Peter is working on a software solution to address the issue on non-ABMC hardware.
This software solution is expected to have the same interface as the hardware solution but
earlier discussions revealed that it may introduce extra latency that users may only want to
accept during periods of active monitoring. Thus the option to disable the counter assignment
mode.
Sorry again for the soft-RMID/soft-ABMC confusion[1], it was soft-RMID
that impacted context switch latency. Soft-ABMC does not require any
additional work at context switch.
No problem. I did read [1] but I do not think I've seen soft-ABMC yet so
my understanding of what it does is vague.
The only disadvantage to soft-ABMC I can think of is that it also
limits reading llc_occupancy event counts to "assigned" groups,
whereas without it, llc_occupancy works reliably on all RMIDs on AMD
hardware.
hmmm ... keeping original llc_occupancy behavior does seem useful enough
as motivation to keep the "legacy"/"default" mbm_assign_mode? It does sound
to me as though soft-ABMC may not be as accurate when it comes to llc_occupancy.
As I understand the hardware may tag entries in cache with RMID and that has a longer
lifetime than the tasks that allocated that data into the cache. If soft-ABMC
permanently associates an RMID with a local and total counter pair but that
RMID is dynamically assigned to resctrl groups then a group may not always
get the same RMID ... and thus its llc_occupancy data would be a combination of
its cache allocations and all the cache allocations of resource groups that had
that RMID before it. This may need significantly enhanced "limbo" handling?
For the use case of soft-ABMC that I'm aware of, it would be better to
disable llc_occupancy events and accept it as a limitation as we're
not using this feature. I don't want to slow down the rate at which
MBM counters could be reassigned. Over the course of a multiple-second
bandwidth measurement window on a bandwidth-saturated host, a previous
group's initial cache occupancy isn't significant enough to justify a
limbo period, especially when padded out to 1 second.
This sounds fair. It also sounds like a motivation for user space to
be able to enable/disable soft-ABMC to be able to disable/enable
llc_occupancy.
I would feel differently if my users were more interested in
llc_occupancy counts and it was possible for the LLC to immediately
notify when the occupancy threshold for any of a set of groups has
been crossed.
To expand on this we may have to rework the interface if the counters can be
assigned to events other than MBM.
James: could you please elaborate how you plan to use this feature and if this
interface works for the planned usage?
Peter: considering the previous example [1] where soft-ABMC was using the "mbm_control"
interface I do not think it is ideal to only use the "t" and "l" flags while
llc_occupancy is also enabled/disabled via this interface. We should consider
(a) renaming the control file to indicate larger scope than MBM, (b) add flags
for llc_occupancy. What do you think? I believe this is in line with stated goal
from [1]: "I believe mbm_control should always accurately reflect which events
are being counted."
I should have said, "I believe mbm_control should always accurately
reflect which _MBM_ events are being counted."
In general, MBM requires maintaining cumulative, running counts, while
llc_occupancy is only a snapshot of cache usage. This is why MBM
results in contended resources (counters) which must be managed by the
user. In the MPAM implementations I've seen so far, a small number
(relative to the number of monitoring groups supported) of occupancy
monitors is sufficient for a large number of groups, because it only
limits the number of monitoring groups' occupancy counts which can be
read in parallel and can be adequately managed within the MPAM driver
without user interaction.
Thank you very much for keeping an eye on the MPAM requirements.
Because of this, broadening the scope of mbm_control to include
occupancy would only serve to remind the user whether occupancy is
supported, but would provide no new information beyond what's already
provided by mon_features.
Thank you.
Reinette
