On 14/04/2021 14:43, Steven Rostedt wrote: >> This causes major problems for `perf` support under Xen, which assumes >> that the kernel's idea of CPUs matches that of the system. > Things are different with KernelShark. That is very encouraging to hear. >> When rendering a trace including Xen data, Xen can provide the real >> system CPUs, and dom0 wants to be rendered as a VM under Xen, similar to >> trace-Fedora21 in your screenshot above. (Obviously, if you're doing >> nested virt, things need to start nesting.) > Right. > > What I would envision how this would work, is that you would produce a > set of tracing files. One for each guest (including Dom0), and one for the > Xen hypervisor itself. The trick is to have a way to synchronize the time > stamps. What we just did with KVM is to have all the tracing record the > CPUs TSC, including the shift and multiplier that the CPU might change for > the guests. Then we have a way to convert the TSC to nanoseconds. This way > all tracing data has the same clock. It's somewhat complicated to get > right, and requires access to how the guests clocks are modified by the CPU. Hmm. In the past, I have had success by modifying Xen to refuse any shift/scale settings, at which point VMs and the hypervisor have directly-comparable raw TSC values. Xen certainly has enough information to describe what TSC rate/epoch each guest is seeing, but I doubt any of this is coherently exposed at the moment. > For KVM, each machine has a unique id and is stored in the trace.dat files. > We have the host store a mapping of what thread represents which guest VCPU > (virtual CPU). Then the "-a" option tells KernelShark to append the > tracing data as a dependency. I would imagine we can have something like > this: > > kernelshark xen.dat -a trace-dom0.dat -a trace-guest1.dat -a trace-guest2.dat > > The Xen plugin would then need to read the how the threads in xen.dat map > to the virtual CPUs of each of the guest files. Which would give you the > layering. Looks good. I suspect we might need to do a little work on Xen's trace data to make this mesh together nicely. In particular, Xen doesn't have a terribly good scheme on unique IDs for "a VM". We've got domain ID's which are Xen's unique instances of a running "thing", but they change across VM reboot/migrate/etc. I suspect we have some atomicity problems with unique identification information and VM-fork too. There is a UUID field but we leave that entirely up to the toolstack to manage. (A good test for naive toolstack code comes on the a localhost live migrate, because suddenly the toolstack is presented with one logical VM (=> one UUID) and two concurrent domid's.) I'll try to have a play with the plugin in some copious free time, but this work does look exciting. ~Andrew
|