On 13/04/2021 16:46, Steven Rostedt wrote: > Hi Giuseppe, > > On Tue, 13 Apr 2021 16:28:36 +0200 > Giuseppe Eletto <giuseppe.eletto@xxxxxxxxxxxx> wrote: > >> Hello, >> I want to share with you a new plugin developed by me, under the >> supervision of Dario Faggioli, which allows the new version of KernelShark >> (the v2-beta) to open and view the Xen traces created using the "xentrace" tool. >> >> In fact, KernelShark is a well known tool for graphical visualization >> Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks >> to its modular architecture, it is now possible to implement plugins which >> open and display traces with arbitrary format, for example, as in in >> this case, traces of the Xen hypervisor. > I'm guessing you have trace events coming from Xen itself? > > >> For more information on how to build the plugin and/or >> to view the source code I leave the repository below: >> https://github.com/giuseppe998e/kernelshark-xentrace-plugin >> >> >> In short: >> >> $ sudo apt install git build-essential libjson-c-dev >> $ git clone --recurse-submodules >> https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git >> $ cd kernelshark-xentrace-plugin/ >> $ make >> >> $ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz) >> $ kernelshark -p out/ks-xentrace.so trace.xen >> >> >> You will need the development version of KernelShark, available here: >> https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git > This will soon be the main repository, as we are going to deprecate the > version in the trace-cmd.git repo soon. And because libtracecmd 1.0 has > already been released. > > >> A screenshot of the plugin in action is available here: >> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png >> >> I'm happy to receive whatever feedback you may have about it, >> and to answer any question. >> > Thanks for doing this. What would be nice is to have the xen traces along > side the linux tracing. Perhaps we can update trace-cmd agent to work with > Xen as well. Does xen implement vsock or some other way to communicate > between the guests and the Dom0 kernel? If not, we should add one. The you > could do the following: > > 1. On each guest, run as root: trace-cmd agent --xen > 2. On Dom0 run: trace-cmd record -e (events on Dom0) \ > --xen (commands to do tracing in Xen HV) \ > -A <guest-name1> -e (events on guest) > > And then you would get a trace.dat file for Dom0 and the guest, and also > have a trace file for Xen (however that is done). And then on KernelShark, > we have a KVM plugin in development that does this. But you can do the same > with Xen. > > For KVM, we have: > > 1. On each guest: trace-cmd agent > 2. On the host: trace-cmd record -e kvm -e sched -e irq \ > -A guest-name -e all > The above produces trace.dat for the host trace, and > trace-<guest-name>.dat for the guest. > 3. kernelshark trace.dat -a trace-Fedora21.dat > > (I have a guest called Fedora21). > > http://rostedt.org/private/kernelshark-kvm.png > > Where you can see the kvm hypervisor task KVM-2356 is the host task running > the guest VCPU 0, and you see the switch between the two. > > Perhaps we can do something like that with Xen as well. The plugin is still > in the works, but should be published soon. And when it is, you could use > that as a template for Xen. A possibly tangential question. Where does KernelShark's idea of CPUs (i.e. real logical threads) come from? In a Xen system, dom0 is just a VM, and particularly on larger servers, may not be as many vcpus as the system has logical threads. This causes major problems for `perf` support under Xen, which assumes that the kernel's idea of CPUs matches that of the system. When rendering a trace including Xen data, Xen can provide the real system CPUs, and dom0 wants to be rendered as a VM under Xen, similar to trace-Fedora21 in your screenshot above. (Obviously, if you're doing nested virt, things need to start nesting.) ~Andrew
|