On Fri, Apr 23, 2021 at 11:37 AM Joel Fernandes <joelaf@xxxxxxxxxx> wrote:
>
> Apologies for the top post as I'm on Gmail mobile and only half awake at 4.30am. I should check but can we just scrape the crosvm PIDs from the host trace itself ? The vCPU threads are in scheduler events in the host trace. Of course that wouldn't work if we don't have events. Let me know if that works for you or if I should find another way.
>
> By the way if sync is supposed to fall back to the ptp algo, why did it not fallback for me?
Because of a bug in the code. It assumes if KVM is detected, then qemu
is used to run the VMs.
>
> Other thoughts:
> - it would be cool if trace cmd agent was run by the host directly on the guest. That might eliminate a step. I can try to see if that's possible with crosvm but it's not super high priority.
That will require to have some already established communication
channel between host and guest. We can use ssh for example, but the
user must set up in advance ssh server and pass credentials to
tracec-cmd on the host.
>
> Yes thanks for sending me any patches and happy to try.
Just sent a patch that should fix the problem. Please, test it when
you have time. The PTP-like algorithm will be used for trace
timestamps synchronization, the best accuracy can be achieved using
mono_raw trace clock:
host# trace-cmd record -e <host events> -C mono_raw -A test@32:823
-e <guest events>
Steven, may be mono_raw clock should be the default in this case, when
PTP is used for the synchronization ?
>
>
> On Fri, Apr 23, 2021, 4:17 AM Tzvetomir Stoyanov <tz.stoyanov@xxxxxxxxx> wrote:
>>
>> Hi Joel,
>>
>> On Fri, Apr 23, 2021 at 1:50 AM Joel Fernandes <joelaf@xxxxxxxxxx> wrote:
>> >
>> > Looks like my trace.dat files bounced from goodmis.org so I uploaded them here:
>> > https://drive.google.com/file/d/16wGsVo4PJ0kYQGAy195dfBW0RHSWVkx0/view?usp=sharing
>> >
>> > ---------- Forwarded message ---------
>> > From: Joel Fernandes <joelaf@xxxxxxxxxx>
>> > Date: Thu, Apr 22, 2021 at 6:46 PM
>> > Subject: Re: Instructions for clock sync for tracing host/guest
>> > To: Steven Rostedt <rostedt@xxxxxxxxxxx>, Yordan Karadzhov (VMware)
>> > <y.karadz@xxxxxxxxx>
>> > Cc: Tzvetomir Stoyanov <tz.stoyanov@xxxxxxxxx>
>> >
>> >
>> > On Thu, Apr 22, 2021 at 5:12 PM Joel Fernandes <joelaf@xxxxxxxxxx> wrote:
>> > >
>> > > On Thu, Apr 22, 2021 at 4:03 PM Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
>> > > >
>> > > > On Thu, 22 Apr 2021 15:49:21 -0400
>> > > > Joel Fernandes <joelaf@xxxxxxxxxx> wrote:
>> > > >
>> > > > > On Thu, Apr 22, 2021 at 3:48 PM Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
>> > > > > >
>> > > > > > On Thu, 22 Apr 2021 15:43:37 -0400
>> > > > > > Joel Fernandes <joelaf@xxxxxxxxxx> wrote:
>> > > > > >
>> > > > > > > > # trace-cmd record -e irq -e sched -e kvm -A test@32:823 -e sched
>> > > > > > >
>> > > > > > > OMG, that worked, thank you Steve and Tzvetomir !!!!!
>> > > > > >
>> > > > > > Technically, you can probably leave off "-e irq", but I do find it rather
>> > > > > > useful. But then, I add "-e irq" to the guest as well. It lets me see how
>> > > > > > interrupts transpire from the host to the guest.
>> > > > >
>> > > > > Got it thanks for the tip. Will report back any issues I see.
>> > > >
>> > > > And you know you can pull this up into KernelShark as well, right?
>> > > >
>> > > > Checkout the development version from Yordan's repository:
>> > > >
>> > > >
>> > > > $ git clone https://github.com/yordan-karadzhov/kernel-shark.git
>> > > > $ cd kernel-shark
>> > > > $ git checkout origin/yordan_devel
>> > > > $ cmake .
>> > > > $ make
>> > > > $ sudo make install
>> > > >
>> > > > Then run kernelshark with:
>> > > >
>> > > > $ kernelshark trace-host.dat -a trace-guest.dat
>> > > >
>> > > > (obviously, using the actual names of the trace.dat files for the host and
>> > > > the guest)
>> > > >
>> > > > Then you can select: Plots -> KVM Combo Plots
>> > > >
>> > > > Then select the "all" box, and then apply. Then you get something like this:
>> > > >
>> > > > http://rostedt.org/private/ks-host-guest.png
>> > > >
>> > > > Where you see how the tasks mapping to the host and guest are aligned.
>> > > >
>> > > > That is, if everything works fine.
>> > >
>> > > Yes, I sort of knew KernelShark had this support, but so glad you sent
>> > > me more details on how to do it, I'll try it out. I am also wondering
>> > > how the bars of the vCPUs will look like because you can have vCPU
>> > > threads migrated to different physical CPUs. I will go try it out and
>> > > see what comes up :-)
>> >
>> > Hi all, Just wanted to report my progress today:
>> >
>> > When I try to open the guest+host trace with the yordan_devel branch and command
>> > kernelshark ~/vm-host-trace/trace.dat -a ~/vm-host-trace/trace-test.dat
>> >
>> > I get the following in the console: https://pastebin.com/raw/EmbsyuB8
>> >
>> > And the GUI looks like the attached picture, I don't see any mention
>> > of KVM like Steve sees in
>> > http://rostedt.org/private/ks-host-guest.png . What am I missing?
>> >
>> > I also zipped and attached my trace files to this email. Could you
>> > take a look at them?
>>
>> Looks like there is a gap in our implementation, affecting your use
>> case. We rely too much on the quemu guest information that we gather,
>> which is missing on your setup. As this information is not mandatory,
>> the implementation should not rely on it. What we need:
>> - PID of the process, running the guest VM. In case of KVM, we use
>> this PID to get the KVM guest TSC clock parameters, needed for better
>> host and guest trace timestamps synchronization. Without this PID, KVM
>> cannot be used for timestamps synchronization. The logic should fail
>> back to a PTP-like algorithm, which is more generic, works in all
>> environments, but is less accurate than KVM logic.
>> - PIDs of each thread, running a guest virtual CPU. This is not
>> required for the tracing, but for better trace visualisation in
>> KernelShark. It helps to map the host task to a vCPU and to visualise
>> them together.
>>
>> Currently we collect that information from quemu, is there a way to
>> get it from crosvm ? If yes, a crosvm support can be implemented in
>> trace-cmd. But as I said, it is not mandatory to have it for the
>> trace, I can send you a patch next week addressing this gap in the
>> implementation.
>>
>> Thanks for testing this code!
>>
>> >
>> > My kernelshark HEAD commit is: 070d657 ("kernel-shark: Add KVMCombo
>> > plugin") . Let me know if I should be building some other commit sha.
>> >
>> > Adding Yordan to this email as well.
>> >
>> > Thanks a lot!
>> > -Joel
>>
>>
>>
>> --
>> Tzvetomir (Ceco) Stoyanov
>> VMware Open Source Technology Center
--
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
|