On Thu, Feb 20, 2014 at 06:56:24AM -0800, Dirk Brandewie wrote:
> On 02/19/2014 04:51 PM, Greg KH wrote:
> >On Wed, Feb 19, 2014 at 04:35:37PM -0800, Greg KH wrote:
> >>On Wed, Feb 19, 2014 at 04:03:40PM -0800, Dirk Brandewie wrote:
> >>>On 02/19/2014 02:47 PM, Greg KH wrote:
> >>>>Hi Dirk,
> >>>>
> >>>>I've been having some huge slowdowns on my box building kernels, and I
> >>>>took the time to bisect it down to commit
> >>>>fcb6a15c2e7e76d493e6f91ea889ab40e1c643a4 (intel_pstate: Take core C0
> >>>>time into account for core busy calculation). With that patch reverted
> >>>>on Linus's current tree, my build speeds are back up to the normal rate.
> >>>>
> >>>>The difference is huge, 2 minutes to do a kernel build with that patch
> >>>>reverted, 8-10 minutes with it applied! With all of the stable kernel
> >>>>builds and other trees, this is a huge problem for my workload (all I do
> >>>>is kernel builds it seems...)
> >>>>
> >>>>I see some patches you marked as "fixes" that you sent to Rafael, do you
> >>>>want me to test any of those? How am I the only one seeing this
> >>>>problem, do you need my cpu information or anything else?
> >>>
> >>>Can you give me a description of you build system? CPU, number of sockets,
> >>
> >>The last processor in /proc/cpuinfo:
> >>
> >>processor : 7
> >>vendor_id : GenuineIntel
> >>cpu family : 6
> >>model : 60
> >>model name : Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
> >>stepping : 3
> >>microcode : 0x9
> >>cpu MHz : 3556.464
> >>cache size : 8192 KB
> >>physical id : 0
> >>siblings : 8
> >>core id : 3
> >>cpu cores : 4
> >>apicid : 7
> >>initial apicid : 7
> >>fpu : yes
> >>fpu_exception : yes
> >>cpuid level : 13
> >>wp : yes
> >>flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
> >>bogomips : 7000.78
> >>clflush size : 64
> >>cache_alignment : 64
> >>address sizes : 39 bits physical, 48 bits virtual
> >>
> >>>building from/to local media. Any special setup I should use here for my test?
> >>
> >>I build on a SSD-like disk (flash media on PCI Express slot), so I
> >>should have almost no i/o time.
> >>
> >>I build with 'make -j16'
> >>
> >>>If you have time having the output of turbostat for a build with and
> >>>without would be very useful.
> >>
> >> $ ./turbostat
> >> turbostat: no /dev/cpu/0/msr
> >> Try "# modprobe msr": No such file or directory
> >>
> >>Bah, I'll rebuild with msr and get you that info...
> >
> >Attached is two files, "fast_build.txt" is when your patch is reverted,
> >and building a kernel takes only 2 minutes. "slow_build.txt" is when
> >the patch is not reverted, using Linus's latest tree, and I stopped it
> >after a while because I got bored :)
> >
> >I think the numbers in these two files shows the real problem that is
> >happening...
>
> For some reason intel_pstate is deciding you have no load and no increasing
> the P state :-(
> >
> >If there's anything else you need me to test, please let me know.
>
> I are running on Linus's tree could you run the following script with the patch
> applied and send the output of "perf script".
>
> #! /bin/sh
>
> sudo perf record -C 5 -c 1 -e power:pstate_sample&
> sleep .5
> foo=$!
> taskset -c 5 cat /dev/zero > /dev/null&
> bar=$!
> sleep 1
> kill $bar
> sleep .5
> sudo kill $foo
The output of this is attached below as "bad.perf", and for comparison,
I ran this with the patch reverted, which is the result in the
"good.perf" file below.
Anything else I can test?
thanks,
greg k-h
# ========
# captured on: Thu Feb 20 10:08:29 2014
# hostname : thunder
# os release : 3.14.0-rc3-00168-g960dfc4eb23a
# perf version : 3.13.3
# arch : x86_64
# nrcpus online : 8
# nrcpus avail : 8
# cpudesc : Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
# cpuid : GenuineIntel,6,60,3
# total memory : 7843716 kB
# cmdline : /usr/bin/perf record -C 5 -c 1 -e power:pstate_sample
# event : name = power:pstate_sample, type = 2, config = 0x97, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1, precise_ip = 0, attr_mmap2 = 0, attr_mmap = 1, attr_mmap_data = 0
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, software = 1, tracepoint = 2, breakpoint = 5
# ========
#
:821 821 [005] 33.459535: power:pstate_sample: core_busy=0 scaled=0 state=8 mperf=967942 aperf=221343 freq=770000
cat 821 [005] 33.469539: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34967293 aperf=7992496 freq=770000
cat 821 [005] 33.479547: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34982605 aperf=7996061 freq=770000
cat 821 [005] 33.489558: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989377 aperf=7997575 freq=770000
cat 821 [005] 33.499570: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990148 aperf=7997758 freq=770000
cat 821 [005] 33.509581: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34988992 aperf=7997486 freq=770000
cat 821 [005] 33.519593: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989728 aperf=7997651 freq=770000
cat 821 [005] 33.529605: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989955 aperf=7997704 freq=770000
cat 821 [005] 33.539617: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989832 aperf=7997673 freq=770000
cat 821 [005] 33.549628: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989763 aperf=7997664 freq=770000
cat 821 [005] 33.559640: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990795 aperf=7997894 freq=770000
cat 821 [005] 33.569652: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34988817 aperf=7997446 freq=770000
cat 821 [005] 33.579663: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990130 aperf=7997740 freq=770000
cat 821 [005] 33.589675: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989955 aperf=7997706 freq=770000
cat 821 [005] 33.599687: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989378 aperf=7997572 freq=770000
cat 821 [005] 33.609699: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989867 aperf=7997683 freq=770000
cat 821 [005] 33.619711: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34990218 aperf=7997766 freq=770000
cat 821 [005] 33.629722: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989570 aperf=7997613 freq=770000
cat 821 [005] 33.639734: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989395 aperf=7997579 freq=770000
cat 821 [005] 33.649746: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990077 aperf=7997730 freq=770000
cat 821 [005] 33.659757: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989920 aperf=7997696 freq=770000
cat 821 [005] 33.669769: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989570 aperf=7997617 freq=770000
cat 821 [005] 33.679781: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990008 aperf=7997713 freq=770000
cat 821 [005] 33.689792: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989517 aperf=7997609 freq=770000
cat 821 [005] 33.699804: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990463 aperf=7997806 freq=770000
cat 821 [005] 33.709816: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34990060 aperf=7997738 freq=770000
cat 821 [005] 33.719828: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989570 aperf=7997619 freq=770000
cat 821 [005] 33.729840: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989762 aperf=7997660 freq=770000
cat 821 [005] 33.739852: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990393 aperf=7997789 freq=770000
cat 821 [005] 33.749865: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34993507 aperf=7998508 freq=770000
cat 821 [005] 33.759876: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989518 aperf=7997615 freq=770000
cat 821 [005] 33.769887: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34985720 aperf=7996737 freq=770000
cat 821 [005] 33.779898: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989920 aperf=7997708 freq=770000
cat 821 [005] 33.789910: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989710 aperf=7997646 freq=770000
cat 821 [005] 33.799922: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34990322 aperf=7997789 freq=770000
cat 821 [005] 33.809933: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989063 aperf=7997499 freq=770000
cat 821 [005] 33.819945: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990112 aperf=7997736 freq=770000
cat 821 [005] 33.829957: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989955 aperf=7997707 freq=770000
cat 821 [005] 33.839969: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34990305 aperf=7997786 freq=770000
cat 821 [005] 33.849980: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989763 aperf=7997662 freq=770000
cat 821 [005] 33.859992: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989377 aperf=7997568 freq=770000
cat 821 [005] 33.870004: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989675 aperf=7997643 freq=770000
cat 821 [005] 33.880015: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989658 aperf=7997635 freq=770000
cat 821 [005] 33.890027: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34990112 aperf=7997738 freq=770000
cat 821 [005] 33.900039: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989833 aperf=7997673 freq=770000
cat 821 [005] 33.910051: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989692 aperf=7997645 freq=770000
cat 821 [005] 33.920063: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990130 aperf=7997744 freq=770000
cat 821 [005] 33.930074: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989815 aperf=7997671 freq=770000
cat 821 [005] 33.940086: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989675 aperf=7997641 freq=770000
cat 821 [005] 33.950098: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989675 aperf=7997641 freq=770000
cat 821 [005] 33.960109: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989745 aperf=7997661 freq=770000
cat 821 [005] 33.970121: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990445 aperf=7997806 freq=770000
cat 821 [005] 33.980133: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989763 aperf=7997667 freq=770000
cat 821 [005] 33.990145: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989465 aperf=7997590 freq=770000
cat 821 [005] 34.000156: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990165 aperf=7997754 freq=770000
cat 821 [005] 34.010168: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989430 aperf=7997581 freq=770000
cat 821 [005] 34.020180: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990200 aperf=7997757 freq=770000
cat 821 [005] 34.030192: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989745 aperf=7997660 freq=770000
cat 821 [005] 34.040203: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990025 aperf=7997724 freq=770000
cat 821 [005] 34.050215: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34990532 aperf=7997837 freq=770000
cat 821 [005] 34.060227: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34988643 aperf=7997398 freq=770000
cat 821 [005] 34.070239: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990287 aperf=7997782 freq=770000
cat 821 [005] 34.080250: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989465 aperf=7997590 freq=770000
cat 821 [005] 34.090262: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34990375 aperf=7997804 freq=770000
cat 821 [005] 34.100274: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989308 aperf=7997557 freq=770000
cat 821 [005] 34.110286: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990060 aperf=7997729 freq=770000
cat 821 [005] 34.120297: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989727 aperf=7997652 freq=770000
cat 821 [005] 34.130309: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989990 aperf=7997709 freq=770000
cat 821 [005] 34.140321: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989868 aperf=7997680 freq=770000
cat 821 [005] 34.150333: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989745 aperf=7997658 freq=770000
cat 821 [005] 34.160344: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989902 aperf=7997695 freq=770000
cat 821 [005] 34.170356: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989903 aperf=7997689 freq=770000
cat 821 [005] 34.180368: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989780 aperf=7997669 freq=770000
cat 821 [005] 34.190380: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989797 aperf=7997666 freq=770000
cat 821 [005] 34.200391: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989938 aperf=7997699 freq=770000
cat 821 [005] 34.210403: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989727 aperf=7997650 freq=770000
cat 821 [005] 34.220415: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989938 aperf=7997704 freq=770000
cat 821 [005] 34.230426: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989675 aperf=7997639 freq=770000
cat 821 [005] 34.240438: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989832 aperf=7997665 freq=770000
cat 821 [005] 34.250450: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989693 aperf=7997653 freq=770000
cat 821 [005] 34.260462: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989745 aperf=7997660 freq=770000
cat 821 [005] 34.270473: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989972 aperf=7997707 freq=770000
cat 821 [005] 34.280485: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989885 aperf=7997689 freq=770000
cat 821 [005] 34.290497: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989535 aperf=7997605 freq=770000
cat 821 [005] 34.300509: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989763 aperf=7997662 freq=770000
cat 821 [005] 34.310520: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34990007 aperf=7997717 freq=770000
cat 821 [005] 34.320532: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990113 aperf=7997738 freq=770000
cat 821 [005] 34.330544: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989832 aperf=7997678 freq=770000
cat 821 [005] 34.340556: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989675 aperf=7997639 freq=770000
cat 821 [005] 34.350568: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990113 aperf=7997735 freq=770000
cat 821 [005] 34.360579: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989360 aperf=7997568 freq=770000
cat 821 [005] 34.370591: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34990182 aperf=7997758 freq=770000
cat 821 [005] 34.380602: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989535 aperf=7997610 freq=770000
cat 821 [005] 34.390614: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989780 aperf=7997664 freq=770000
cat 821 [005] 34.400627: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34991985 aperf=7998164 freq=770000
cat 821 [005] 34.410638: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34987750 aperf=7997204 freq=770000
cat 821 [005] 34.420649: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989728 aperf=7997652 freq=770000
cat 821 [005] 34.430662: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34990602 aperf=7997837 freq=770000
cat 821 [005] 34.440673: power:pstate_sample: core_busy=22 scaled=96 state=8 mperf=34989570 aperf=7997630 freq=770000
cat 821 [005] 34.450685: power:pstate_sample: core_busy=22 scaled=97 state=8 mperf=34989675 aperf=7997637 freq=770000
swapper 0 [005] 34.470751: power:pstate_sample: core_busy=11 scaled=48 state=8 mperf=34905746 aperf=7978460 freq=770000
swapper 0 [005] 34.484100: power:pstate_sample: core_busy=0 scaled=0 state=8 mperf=129641 aperf=29655 freq=770000
swapper 0 [005] 34.497456: power:pstate_sample: core_busy=0 scaled=0 state=8 mperf=170450 aperf=38984 freq=770000
swapper 0 [005] 34.510793: power:pstate_sample: core_busy=0 scaled=0 state=8 mperf=415013 aperf=94909 freq=770000
swapper 0 [005] 34.971331: power:pstate_sample: core_busy=0 scaled=0 state=8 mperf=385821 aperf=88241 freq=770000
# ========
# captured on: Thu Feb 20 10:04:27 2014
# hostname : thunder
# os release : 3.14.0-rc3-00169-g61d81fafdf07
# perf version : 3.13.3
# arch : x86_64
# nrcpus online : 8
# nrcpus avail : 8
# cpudesc : Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
# cpuid : GenuineIntel,6,60,3
# total memory : 7843720 kB
# cmdline : /usr/bin/perf record -C 5 -c 1 -e power:pstate_sample
# event : name = power:pstate_sample, type = 2, config = 0x97, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1, precise_ip = 0, attr_mmap2 = 0, attr_mmap = 1, attr_mmap_data = 0
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, software = 1, tracepoint = 2, breakpoint = 5
# ========
#
swapper 0 [005] 61435.228822: power:pstate_sample: core_busy=105 scaled=96 state=38 mperf=117675 aperf=124347 freq=3698378
cat 20481 [005] 61435.672634: power:pstate_sample: core_busy=101 scaled=92 state=38 mperf=6401489 aperf=6483508 freq=3544707
cat 20481 [005] 61435.682644: power:pstate_sample: core_busy=108 scaled=97 state=39 mperf=34983090 aperf=37981660 freq=3799960
cat 20481 [005] 61435.692654: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34886623 aperf=38868334 freq=3899355
cat 20481 [005] 61435.702666: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34890066 aperf=38872130 freq=3899355
cat 20481 [005] 61435.712678: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34897018 aperf=38884312 freq=3899902
cat 20481 [005] 61435.722690: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989661 aperf=38988486 freq=3899902
cat 20481 [005] 61435.732701: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34898491 aperf=38885509 freq=3899765
cat 20481 [005] 61435.742713: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989780 aperf=38988611 freq=3899902
cat 20481 [005] 61435.752725: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989735 aperf=38988548 freq=3899902
cat 20481 [005] 61435.762737: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989941 aperf=38988800 freq=3899902
cat 20481 [005] 61435.772748: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989789 aperf=38988624 freq=3899902
cat 20481 [005] 61435.782760: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989908 aperf=38988754 freq=3899902
cat 20481 [005] 61435.792772: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989759 aperf=38988583 freq=3899902
cat 20481 [005] 61435.802783: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989938 aperf=38988790 freq=3899902
cat 20481 [005] 61435.812795: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989708 aperf=38988514 freq=3899902
cat 20481 [005] 61435.822807: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989858 aperf=38988721 freq=3899902
cat 20481 [005] 61435.832819: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989929 aperf=38988778 freq=3899902
cat 20481 [005] 61435.842830: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989765 aperf=38988592 freq=3899902
cat 20481 [005] 61435.852842: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989789 aperf=38988620 freq=3899902
cat 20481 [005] 61435.862854: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989884 aperf=38988726 freq=3899902
cat 20481 [005] 61435.872866: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989801 aperf=38988640 freq=3899902
cat 20481 [005] 61435.882877: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989828 aperf=38988670 freq=3899902
cat 20481 [005] 61435.892889: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989955 aperf=38988789 freq=3899902
cat 20481 [005] 61435.902901: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989697 aperf=38988530 freq=3899902
cat 20481 [005] 61435.912913: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989795 aperf=38988633 freq=3899902
cat 20481 [005] 61435.922924: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989857 aperf=38988683 freq=3899902
cat 20481 [005] 61435.932936: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989992 aperf=38988862 freq=3899902
cat 20481 [005] 61435.942948: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989741 aperf=38988560 freq=3899902
cat 20481 [005] 61435.952960: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989807 aperf=38988651 freq=3899902
cat 20481 [005] 61435.962972: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989890 aperf=38988726 freq=3899902
cat 20481 [005] 61435.972983: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989837 aperf=38988689 freq=3899902
cat 20481 [005] 61435.982995: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34990590 aperf=38989482 freq=3899902
cat 20481 [005] 61435.993007: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34990051 aperf=38988923 freq=3899902
cat 20481 [005] 61436.003018: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34924586 aperf=38914564 freq=3899765
cat 20481 [005] 61436.013030: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989685 aperf=38988521 freq=3899902
cat 20481 [005] 61436.023042: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989902 aperf=38988741 freq=3899902
cat 20481 [005] 61436.033054: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34990355 aperf=38989256 freq=3899902
cat 20481 [005] 61436.043065: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989244 aperf=38988021 freq=3899902
cat 20481 [005] 61436.053077: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989988 aperf=38988830 freq=3899902
cat 20481 [005] 61436.063089: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989807 aperf=38988648 freq=3899902
cat 20481 [005] 61436.073100: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989783 aperf=38988614 freq=3899902
cat 20481 [005] 61436.083112: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989756 aperf=38988581 freq=3899902
cat 20481 [005] 61436.093124: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989905 aperf=38988745 freq=3899902
cat 20481 [005] 61436.103136: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989774 aperf=38988606 freq=3899902
cat 20481 [005] 61436.113147: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989926 aperf=38988775 freq=3899902
cat 20481 [005] 61436.123159: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989709 aperf=38988551 freq=3899902
cat 20481 [005] 61436.133171: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989887 aperf=38988719 freq=3899902
cat 20481 [005] 61436.143183: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34991031 aperf=38989992 freq=3899902
cat 20481 [005] 61436.153195: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989449 aperf=38988243 freq=3899902
cat 20481 [005] 61436.163206: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989089 aperf=38987846 freq=3899902
cat 20481 [005] 61436.173218: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989715 aperf=38988556 freq=3899902
cat 20481 [005] 61436.183230: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989920 aperf=38988771 freq=3899902
cat 20481 [005] 61436.193241: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989857 aperf=38988680 freq=3899902
cat 20481 [005] 61436.203253: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989956 aperf=38988819 freq=3899902
cat 20481 [005] 61436.213265: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989589 aperf=38988388 freq=3899902
cat 20481 [005] 61436.223276: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989417 aperf=38988215 freq=3899902
cat 20481 [005] 61436.233288: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34990236 aperf=38989130 freq=3899902
cat 20481 [005] 61436.243300: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989926 aperf=38988776 freq=3899902
cat 20481 [005] 61436.253312: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989675 aperf=38988470 freq=3899902
cat 20481 [005] 61436.263324: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34990087 aperf=38988963 freq=3899902
cat 20481 [005] 61436.273335: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989652 aperf=38988472 freq=3899902
cat 20481 [005] 61436.283347: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989771 aperf=38988601 freq=3899902
cat 20481 [005] 61436.293359: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989849 aperf=38988703 freq=3899902
cat 20481 [005] 61436.303370: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989887 aperf=38988725 freq=3899902
cat 20481 [005] 61436.313382: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989813 aperf=38988651 freq=3899902
cat 20481 [005] 61436.323394: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989881 aperf=38988715 freq=3899902
cat 20481 [005] 61436.333406: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989825 aperf=38988678 freq=3899902
cat 20481 [005] 61436.343418: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34993041 aperf=38992225 freq=3899902
cat 20481 [005] 61436.353429: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34986632 aperf=38985120 freq=3899902
cat 20481 [005] 61436.363441: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989747 aperf=38988566 freq=3899902
cat 20481 [005] 61436.373453: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989914 aperf=38988762 freq=3899902
cat 20481 [005] 61436.383464: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989923 aperf=38988763 freq=3899902
cat 20481 [005] 61436.393476: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989759 aperf=38988595 freq=3899902
cat 20481 [005] 61436.403488: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34931709 aperf=38825285 freq=3890058
cat 20481 [005] 61436.413500: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989726 aperf=38988542 freq=3899902
cat 20481 [005] 61436.423512: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34993703 aperf=38992984 freq=3899902
cat 20481 [005] 61436.433523: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34986060 aperf=38984480 freq=3899902
cat 20481 [005] 61436.443535: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34990086 aperf=38988929 freq=3899902
cat 20481 [005] 61436.453547: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989646 aperf=38988460 freq=3899902
cat 20481 [005] 61436.463558: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989733 aperf=38988584 freq=3899902
cat 20481 [005] 61436.473570: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989824 aperf=38988639 freq=3899902
cat 20481 [005] 61436.483582: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989837 aperf=38988691 freq=3899902
cat 20481 [005] 61436.493593: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989825 aperf=38988671 freq=3899902
cat 20481 [005] 61436.503605: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989827 aperf=38988646 freq=3899902
cat 20481 [005] 61436.513617: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989819 aperf=38988662 freq=3899902
cat 20481 [005] 61436.523629: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989878 aperf=38988713 freq=3899902
cat 20481 [005] 61436.533640: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989783 aperf=38988615 freq=3899902
cat 20481 [005] 61436.543652: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989882 aperf=38988749 freq=3899902
cat 20481 [005] 61436.553664: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989809 aperf=38988620 freq=3899902
cat 20481 [005] 61436.563676: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989885 aperf=38988752 freq=3899902
cat 20481 [005] 61436.573687: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989842 aperf=38988660 freq=3899902
cat 20481 [005] 61436.583699: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989807 aperf=38988645 freq=3899902
cat 20481 [005] 61436.593711: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989807 aperf=38988656 freq=3899902
cat 20481 [005] 61436.603723: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989786 aperf=38988618 freq=3899902
cat 20481 [005] 61436.613734: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989857 aperf=38988686 freq=3899902
cat 20481 [005] 61436.623746: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989950 aperf=38988804 freq=3899902
cat 20481 [005] 61436.633758: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989741 aperf=38988564 freq=3899902
cat 20481 [005] 61436.643770: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989855 aperf=38988706 freq=3899902
cat 20481 [005] 61436.653781: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989973 aperf=38988801 freq=3899902
cat 20481 [005] 61436.663793: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=34989655 aperf=38988487 freq=3899902
swapper 0 [005] 61436.683852: power:pstate_sample: core_busy=111 scaled=99 state=39 mperf=28745848 aperf=32026974 freq=3899492
swapper 0 [005] 61436.697202: power:pstate_sample: core_busy=100 scaled=91 state=38 mperf=124360 aperf=124457 freq=3502597
swapper 0 [005] 61436.710551: power:pstate_sample: core_busy=100 scaled=94 state=37 mperf=129452 aperf=129543 freq=3502324
