Dear all
I continue watching a mailing list whether a similar problem is reported because a problem does not seem to happen to others.
Any information, however small, would be appreciated.
regards
> -----Original Message-----
> From: Uehara, Shouta (Shouta.Uehara@xxxxxxxxxxxxxxx)
> Sent: Friday, November 09, 2012 6:52 PM
> To: 'kvm@xxxxxxxxxxxxxxx'; 'Xiao Guangrong
> (xiaoguangrong@xxxxxxxxxxxxxxxxxx)'
> Subject: RE: Guest performance is reduced after live migration
>
> I've analysed the problem with migration using perf-events, and it is
> confirmed cost of glibc is significantly increased.
> I made a simple test code to execute read system call on the guest as follows,
> and examined the performance from host and guest.
>
> fd = open("/dev/zero", O_RDONLY);
> while (1) { read(fd &ch, 1); }
>
> ---
> [Source host]
>
> $ sudo perf kvm --host --guest record -a -o src_host.perf sleep 10
>
> Events: 10K cycles
> 89.36% qemu-system-x86 [unknown] [g]
> 0xffffffff810a36ef
> 9.04% qemu-system-x86 [unknown] [u]
> 0x3fd20e2020
> 0.16% swapper [kernel.kallsyms] [k] intel_idle
> 0.11% sleep [kernel.kallsyms] [k] page_fault
>
> [Guest on source host]
>
> $ perf record -o src_guest.perf ./loop_read
>
> Events: 29K cpu-clock
> 11.71% loop_read [kernel.kallsyms] [k] system_call_after_swapgs
> 9.58% loop_read libc-2.14.90.so [.] __GI___libc_read
> 6.92% loop_read [kernel.kallsyms] [k] vfs_read
> 5.53% loop_read [kernel.kallsyms] [k] fsnotify
>
> __GI___libc_read
> : 0000003fd20e2010 <__read>:
> 2.31 : 3fd20e2010: 83 3d 1d 22 2d 00 00 cmpl
> $0x0,0x2d221d(
> 5.67 : 3fd20e2017: 75 10 jne
> 3fd20e2029 <__
> :
> : 0000003fd20e2019 <__read_nocancel>:
> 1.82 : 3fd20e2019: b8 00 00 00 00 mov
> $0x0,%eax
> 0.00 : 3fd20e201e: 0f 05 syscall
> 87.78 : 3fd20e2020: 48 3d 01 f0 ff ff cmp
> $0xfffffffffff
> 0.00 : 3fd20e2026: 73 31 jae
> 3fd20e2059 <__
> 2.42 : 3fd20e2028: c3 retq
>
> [Destination host]
>
> $ sudo perf kvm --host --guest record -a -o dst_host.perf sleep 10
>
> Events: 10K cycles
> 58.39% qemu-system-x86 [unknown] [g]
> 0xffffffff810a3a6e
> 40.14% qemu-system-x86 [unknown] [u]
> 0x3fd20e2017
> 0.13% gnome-shell nouveau_dri.so [.] 0xbd7c9
> 0.11% swapper [kernel.kallsyms] [k] intel_idle
>
> [Guest on destination host]
>
> $ perf record -o dst_guest.perf ./loop_read
>
> Events: 29K cpu-clock
> 41.95% loop_read libc-2.14.90.so [.] __GI___libc_read
> 7.90% loop_read [kernel.kallsyms] [k] system_call_after_swapgs
> 4.61% loop_read [kernel.kallsyms] [k] vfs_read
> 3.72% loop_read [kernel.kallsyms] [k] fsnotify
>
> __GI___libc_read
> : 0000003fd20e2010 <__read>:
> 0.41 : 3fd20e2010: 83 3d 1d 22 2d 00 00 cmpl
> $0x0,0x2d221d(
> 86.10 : 3fd20e2017: 75 10 jne
> 3fd20e2029 <__
> :
> : 0000003fd20e2019 <__read_nocancel>:
> 0.33 : 3fd20e2019: b8 00 00 00 00 mov
> $0x0,%eax
> 0.00 : 3fd20e201e: 0f 05 syscall
> 12.84 : 3fd20e2020: 48 3d 01 f0 ff ff cmp
> $0xfffffffffff
> 0.00 : 3fd20e2026: 73 31 jae
> 3fd20e2059 <__
> 0.33 : 3fd20e2028: c3 retq
>
> ---
>
> After the migration, "jne" instruction in __read became to take a long time.
> This is due to increase the overhead of read system call. The performance
> was not improved even if thp/hugetlb was disabled.
> Does this problem happen only to me? Should I also ask other community about
> this problem?
>
> Shota
>
> > -----Original Message-----
> > From: Xiao Guangrong [mailto:xiaoguangrong@xxxxxxxxxxxxxxxxxx]
> > Sent: Thursday, November 01, 2012 1:45 PM
> > To: Uehara, Shouta (Shouta.Uehara@xxxxxxxxxxxxxxx)
> > Cc: kvm@xxxxxxxxxxxxxxx
> > Subject: Re: Guest performance is reduced after live migration
> >
> > Shouta,
> >
> > Can it be reproduced if thp/hugetlbfs is disabled on both source and
> > destination?
> >
> > On 11/01/2012 08:12 AM, Shouta.Uehara@xxxxxxxxxxxxxxx wrote:
> > > Hello.
> > >
> > > I have a problem with the performance of the guest Linux after live
> > migration.
> > > When I analyze the file I/O latency of the guest using LMbench3, the
> > > latency of the guest on the destination host is about 2 times bigger
> > > than the guest on the source host. As a result that I investigated
> > > it, this problem occurs if three following conditions are right.
> > >
> > > 1. Use the kernel version 2.6.38.6-26.fc15.x86_64 or later.
> > > 2. Execute system calls for low level file I/O (read, write, open
> > > etc.)
> > on the
> > > guest.
> > > 3. Enable EPT.
> > >
> > > Performance cannot decrease on other tests of LMbench3; numerical
> > > operation, context switching, memory latencies etc.
> > >
> > > Is this an existing problem ? What should I do to resolve it ?
> > >
> > > ---
> > > The situation the problem occurs
> > >
> > > CPU models : Intel Xeon E3-1245 (Both hosts have same cpu.)
> > > Host kernel : 3.6.4
> > > Host OS : Fedora 16 64bit ver. (Both hosts have same OS.)
> > > Guest OS : Fedora 16 64bit ver., kernel 3.1.0-7.fc16.x86_64
> > > QEMU version : qemu-kvm 1.2.0
> > > QEMU parameter : qemu-kvm -m 1024 -drive file=guest.qcow2,if=virtio
> \
> > > -boot order=c,menu=on -k ja
> > > (-incoming tcp:192.168.0.51:4444 set only
> > destination)
> > > QEMU command : migrate -d tcp:192.168.0.51:4444
> > > Analyses : LMbench version 3
> > >
> > > [Guest on the source host]
> > > $ ./lat_syscall open
> > > Simple open/close: 1.4281 microseconds
> > > $ ./lat_syscall read
> > > Simple read: 0.1416 microseconds
> > > $ ./lat_syscall write
> > > Simple write: 0.1194 microseconds
> > > $ ./lat_fs -s 0
> > > 0k 550 98726 247712
> > > 1k 383 69751 193985
> > > 4k 381 69348 193383
> > > 10k 313 54369 155697
> > >
> > > [Guest on the source host]
> > > $ ./lat_syscall open
> > > Simple open/close: 1.6126 microseconds
> > > $ ./lat_syscall read
> > > Simple read: 0.2206 microseconds
> > > $ ./lat_syscall write
> > > Simple write: 0.1980 microseconds
> > > $ ./lat_fs -s 0
> > > 0k 427 83252 243222
> > > 1k 315 60530 191650
> > > 4k 184 34535 190947
> > > 10k 65 11922 152608
> > > ---
> > >
> > > Thanks.
> > >
> > > Shota Uehara <Shouta.Uehara()jp.yokogawa.com>
-----
CONFIDENTIAL: This e-mail may contain information that is confidential or otherwise protected from disclosure and intended only for the party to whom it is addressed. If you are not the intended recipient, please notify the sender by return and delete this e-mail. You are hereby formally advised that any unauthorized use, disclosure or copying of this email is strictly prohibited and may be unlawful.
��.n��������+%������w��{.n�����o�^n�r������&��z�ޗ�zf���h���~����������_��+v���)ߣ�
