On Thu, Oct 27, 2022, Wang, Wei W wrote:
> On Wednesday, October 26, 2022 11:44 PM, Sean Christopherson wrote:
> > > I think it would be better to do the thread pinning at the time when
> > > the thread is created by providing a pthread_attr_t attr, e.g. :
> > >
> > > pthread_attr_t attr;
> > >
> > > CPU_SET(vcpu->pcpu, &cpu_set);
> > > pthread_attr_setaffinity_np(&attr, sizeof(cpu_set_t), &cpu_set);
> > > pthread_create(thread, attr,...);
> > >
> > > Also, pinning a vCPU thread to a pCPU is a general operation which
> > > other users would need. I think we could make it more general and put
> > > it to kvm_util.
> >
> > We could, but it taking advantage of the pinning functionality would require
> > plumbing a command line option for every test,
>
> I think we could make this "pinning" be optional (no need to force everyone
> to use it).
Heh, it's definitely optional.
> > If we go this route in the future, we'd need to add a worker trampoline as the
> > pinning needs to happen in the worker task itself to guarantee that the pinning
> > takes effect before the worker does anything useful. That should be very
> > doable.
>
> The alternative way is the one I shared before, using this:
>
> /* Thread created with attribute ATTR will be limited to run only on
> the processors represented in CPUSET. */
> extern int pthread_attr_setaffinity_np (pthread_attr_t *__attr,
> size_t __cpusetsize,
> const cpu_set_t *__cpuset)
>
> Basically, the thread is created on the pCPU as user specified.
> I think this is better than "creating the thread on an arbitrary pCPU
> and then pinning it to the user specified pCPU in the thread's start routine".
Ah, yeah, that's better.
> > I do like the idea of extending __vcpu_thread_create(), but we can do that once
> > __vcpu_thread_create() lands to avoid further delaying this series.
>
> Sounds good. I can move some of those to vcpu_thread_create() once it's ready later.
>
> > struct perf_test_args {
> > @@ -43,8 +41,12 @@ struct perf_test_args {
> > bool nested;
> > /* True if all vCPUs are pinned to pCPUs */
> > bool pin_vcpus;
> > + /* The vCPU=>pCPU pinning map. Only valid if pin_vcpus is true. */
> > + uint32_t vcpu_to_pcpu[KVM_MAX_VCPUS];
>
> How about putting the pcpu id to "struct kvm_vcpu"? (please see below code
> posed to shows how that works). This is helpful when we later make this more generic,
> as kvm_vcpu is used by everyone.
I don't think "pcpu" belongs in kvm_vcpu, even in the long run. The vast, vast
majority of tests will never care about pinning, which means that vcpu->pcpu can't
be used for anything except the actual pinning. And for pinning, the "pcpu"
doesn't need to be persistent information, i.e. doesn't need to live in kvm_vcpu.
> Probably we also don't need "bool pin_vcpus".
Yeah, but for selftests shaving bytes is not exactly top priority, and having a
dedicated flag avoids the need for magic numbers. If Vipin had used -1, I'd
probably be fine with that, but I'm also totally fine using a dedicated flag too.
> We could initialize pcpu_id to -1 to indicate that the vcpu doesn't need
> pinning (this is also what I meant above optional for other users).
>
> Put the whole changes together (tested and worked fine), FYI:
The big downside of this is forcing all callers of perf_test_create_vm() to pass
in NULL. I really want to move away from this pattern as it makes what should be
simple code rather difficult to read due to having a bunch of "dead" params
dangling off the end of function calls.
