Re: Windows slow boot: contractor wanted

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Rik,

Are there any more tests which I can usefully do for you?

I notice that 3.6.0-rc4 is out - are there changes from rc3 which are worth
me retesting?

Cheers,

Richard.


Richard Davies wrote:
> Rik van Riel wrote:
> > Can you get a backtrace to that _raw_spin_lock_irqsave, to see
> > from where it is running into lock contention?
> >
> > It would be good to know whether it is isolate_freepages_block,
> > yield_to, kvm_vcpu_on_spin or something else...
>
> Hi Rik,
>
> I got into a slow boot situation on 3.6.0-rc3, ran "perf record -g -a" for a
> while, then ran perf report with the output below.
>
> This trace looks more like the second perf top trace that I sent on Saturday
> (there were two in my email and they were different from each other as well
> as different from on 3.5.2).
>
> The symptoms were a bit different too - the VM boots appeared to be
> completely locked up rather than just slow, and I couldn't quit qemu-kvm at
> the monitor - I had to restart the host.
>
> So perhaps this one is actually a deadlock rather than just slow?
>
> Cheers,
>
> Richard.
>
>
> # ========
> # captured on: Sun Aug 26 10:08:28 2012
> # os release : 3.6.0-rc3-elastic
> # perf version : 3.5.2
> # arch : x86_64
> # nrcpus online : 16
> # nrcpus avail : 16
> # cpudesc : AMD Opteron(tm) Processor 6128
> # cpuid : AuthenticAMD,16,9,1
> # total memory : 131971760 kB
> # cmdline : /home/root/bin/perf record -g -a 
> # event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, id = { 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64 }
> # HEADER_CPU_TOPOLOGY info available, use -I to display
> # HEADER_NUMA_TOPOLOGY info available, use -I to display
> # ========
> #
> # Samples: 2M of event 'cycles'
> # Event count (approx.): 1040676441385
> #
> # Overhead          Command         Shared Object                                          Symbol
> # ........  ...............  ....................  ..............................................
> #
>     90.01%         qemu-kvm  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave                    
>                    |
>                    --- _raw_spin_lock_irqsave
>                       |          
>                       |--99.99%-- isolate_migratepages_range
>                       |          compact_zone
>                       |          compact_zone_order
>                       |          try_to_compact_pages
>                       |          __alloc_pages_direct_compact
>                       |          __alloc_pages_nodemask
>                       |          alloc_pages_vma
>                       |          do_huge_pmd_anonymous_page
>                       |          handle_mm_fault
>                       |          __get_user_pages
>                       |          get_user_page_nowait
>                       |          hva_to_pfn.isra.33
>                       |          __gfn_to_pfn
>                       |          gfn_to_pfn_async
>                       |          try_async_pf
>                       |          tdp_page_fault
>                       |          kvm_mmu_page_fault
>                       |          pf_interception
>                       |          handle_exit
>                       |          kvm_arch_vcpu_ioctl_run
>                       |          kvm_vcpu_ioctl
>                       |          do_vfs_ioctl
>                       |          sys_ioctl
>                       |          system_call_fastpath
>                       |          ioctl
>                       |          |          
>                       |          |--54.91%-- 0x10100000002
>                       |          |          
>                       |           --45.09%-- 0x10100000006
>                        --0.01%-- [...]
>      4.66%         qemu-kvm  [kernel.kallsyms]     [k] sub_preempt_count                         
>                    |
>                    --- sub_preempt_count
>                       |          
>                       |--99.77%-- _raw_spin_unlock_irqrestore
>                       |          |          
>                       |          |--99.99%-- compact_checklock_irqsave
>                       |          |          isolate_migratepages_range
>                       |          |          compact_zone
>                       |          |          compact_zone_order
>                       |          |          try_to_compact_pages
>                       |          |          __alloc_pages_direct_compact
>                       |          |          __alloc_pages_nodemask
>                       |          |          alloc_pages_vma
>                       |          |          do_huge_pmd_anonymous_page
>                       |          |          handle_mm_fault
>                       |          |          __get_user_pages
>                       |          |          get_user_page_nowait
>                       |          |          hva_to_pfn.isra.33
>                       |          |          __gfn_to_pfn
>                       |          |          gfn_to_pfn_async
>                       |          |          try_async_pf
>                       |          |          tdp_page_fault
>                       |          |          kvm_mmu_page_fault
>                       |          |          pf_interception
>                       |          |          handle_exit
>                       |          |          kvm_arch_vcpu_ioctl_run
>                       |          |          kvm_vcpu_ioctl
>                       |          |          do_vfs_ioctl
>                       |          |          sys_ioctl
>                       |          |          system_call_fastpath
>                       |          |          ioctl
>                       |          |          |          
>                       |          |          |--51.94%-- 0x10100000002
>                       |          |          |          
>                       |          |           --48.06%-- 0x10100000006
>                       |           --0.01%-- [...]
>                        --0.23%-- [...]
>      1.23%             ksmd  [kernel.kallsyms]     [k] memcmp                                    
>                        |
>                        --- memcmp
>                           |          
>                           |--99.83%-- memcmp_pages
>                           |          |          
>                           |          |--78.46%-- ksm_scan_thread
>                           |          |          kthread
>                           |          |          kernel_thread_helper
>                           |          |          
>                           |           --21.54%-- try_to_merge_with_ksm_page
>                           |                     ksm_scan_thread
>                           |                     kthread
>                           |                     kernel_thread_helper
>                            --0.17%-- [...]
>      0.91%             ksmd  [kernel.kallsyms]     [k] smp_call_function_many                    
>                        |
>                        --- smp_call_function_many
>                           |          
>                           |--99.98%-- native_flush_tlb_others
>                           |          |          
>                           |          |--99.86%-- flush_tlb_page
>                           |          |          ptep_clear_flush
>                           |          |          try_to_merge_with_ksm_page
>                           |          |          ksm_scan_thread
>                           |          |          kthread
>                           |          |          kernel_thread_helper
>                           |           --0.14%-- [...]
>                            --0.02%-- [...]
>      0.34%         qemu-kvm  [kernel.kallsyms]     [k] _raw_spin_unlock_irqrestore               
>                    |
>                    --- _raw_spin_unlock_irqrestore
>                       |          
>                       |--96.08%-- compact_checklock_irqsave
>                       |          isolate_migratepages_range
>                       |          compact_zone
>                       |          compact_zone_order
>                       |          try_to_compact_pages
>                       |          __alloc_pages_direct_compact
>                       |          __alloc_pages_nodemask
>                       |          alloc_pages_vma
>                       |          do_huge_pmd_anonymous_page
>                       |          handle_mm_fault
>                       |          __get_user_pages
>                       |          get_user_page_nowait
>                       |          hva_to_pfn.isra.33
>                       |          __gfn_to_pfn
>                       |          gfn_to_pfn_async
>                       |          try_async_pf
>                       |          tdp_page_fault
>                       |          kvm_mmu_page_fault
>                       |          pf_interception
>                       |          handle_exit
>                       |          kvm_arch_vcpu_ioctl_run
>                       |          kvm_vcpu_ioctl
>                       |          do_vfs_ioctl
>                       |          sys_ioctl
>                       |          system_call_fastpath
>                       |          ioctl
>                       |          |          
>                       |          |--65.19%-- 0x10100000006
>                       |          |          
>                       |           --34.81%-- 0x10100000002
>                       |          
>                       |--2.68%-- isolate_migratepages_range
>                       |          compact_zone
>                       |          compact_zone_order
>                       |          try_to_compact_pages
>                       |          __alloc_pages_direct_compact
>                       |          __alloc_pages_nodemask
>                       |          alloc_pages_vma
>                       |          do_huge_pmd_anonymous_page
>                       |          handle_mm_fault
>                       |          __get_user_pages
>                       |          get_user_page_nowait
>                       |          hva_to_pfn.isra.33
>                       |          __gfn_to_pfn
>                       |          gfn_to_pfn_async
>                       |          try_async_pf
>                       |          tdp_page_fault
>                       |          kvm_mmu_page_fault
>                       |          pf_interception
>                       |          handle_exit
>                       |          kvm_arch_vcpu_ioctl_run
>                       |          kvm_vcpu_ioctl
>                       |          do_vfs_ioctl
>                       |          sys_ioctl
>                       |          system_call_fastpath
>                       |          ioctl
>                       |          |          
>                       |          |--52.08%-- 0x10100000002
>                       |          |          
>                       |           --47.92%-- 0x10100000006
>                       |          
>                       |--0.56%-- ntp_tick_length
>                       |          do_timer
>                       |          tick_do_update_jiffies64
>                       |          tick_sched_timer
>                       |          __run_hrtimer
>                       |          hrtimer_interrupt
>                       |          smp_apic_timer_interrupt
>                       |          apic_timer_interrupt
>                       |          compact_checklock_irqsave
>                       |          isolate_migratepages_range
>                       |          compact_zone
>                       |          compact_zone_order
>                       |          try_to_compact_pages
>                       |          __alloc_pages_direct_compact
>                       |          __alloc_pages_nodemask
>                       |          alloc_pages_vma
>                       |          do_huge_pmd_anonymous_page
>                       |          handle_mm_fault
>                       |          __get_user_pages
>                       |          get_user_page_nowait
>                       |          hva_to_pfn.isra.33
>                       |          __gfn_to_pfn
>                       |          gfn_to_pfn_async
>                       |          try_async_pf
>                       |          tdp_page_fault
>                       |          kvm_mmu_page_fault
>                       |          pf_interception
>                       |          handle_exit
>                       |          kvm_arch_vcpu_ioctl_run
>                       |          kvm_vcpu_ioctl
>                       |          do_vfs_ioctl
>                       |          sys_ioctl
>                       |          system_call_fastpath
>                       |          ioctl
>                       |          0x10100000002
>                        --0.68%-- [...]
>      0.30%          swapper  [kernel.kallsyms]     [k] default_idle                              
>                     |
>                     --- default_idle
>                        |          
>                        |--99.95%-- cpu_idle
>                        |          start_secondary
>                         --0.05%-- [...]
>      0.15%         qemu-kvm  [kernel.kallsyms]     [k] isolate_migratepages_range                
>                    |
>                    --- isolate_migratepages_range
>                       |          
>                       |--97.41%-- compact_zone
>                       |          compact_zone_order
>                       |          try_to_compact_pages
>                       |          __alloc_pages_direct_compact
>                       |          __alloc_pages_nodemask
>                       |          alloc_pages_vma
>                       |          do_huge_pmd_anonymous_page
>                       |          handle_mm_fault
>                       |          __get_user_pages
>                       |          get_user_page_nowait
>                       |          hva_to_pfn.isra.33
>                       |          __gfn_to_pfn
>                       |          gfn_to_pfn_async
>                       |          try_async_pf
>                       |          tdp_page_fault
>                       |          kvm_mmu_page_fault
>                       |          pf_interception
>                       |          handle_exit
>                       |          kvm_arch_vcpu_ioctl_run
>                       |          kvm_vcpu_ioctl
>                       |          do_vfs_ioctl
>                       |          sys_ioctl
>                       |          system_call_fastpath
>                       |          ioctl
>                       |          |          
>                       |          |--54.02%-- 0x10100000002
>                       |          |          
>                       |           --45.98%-- 0x10100000006
>                       |          
>                        --2.59%-- compact_zone_order
>                                  try_to_compact_pages
>                                  __alloc_pages_direct_compact
>                                  __alloc_pages_nodemask
>                                  alloc_pages_vma
>                                  do_huge_pmd_anonymous_page
>                                  handle_mm_fault
>                                  __get_user_pages
>                                  get_user_page_nowait
>                                  hva_to_pfn.isra.33
>                                  __gfn_to_pfn
>                                  gfn_to_pfn_async
>                                  try_async_pf
>                                  tdp_page_fault
>                                  kvm_mmu_page_fault
>                                  pf_interception
>                                  handle_exit
>                                  kvm_arch_vcpu_ioctl_run
>                                  kvm_vcpu_ioctl
>                                  do_vfs_ioctl
>                                  sys_ioctl
>                                  system_call_fastpath
>                                  ioctl
>                                  |          
>                                  |--56.10%-- 0x10100000002
>                                  |          
>                                   --43.90%-- 0x10100000006
>      0.12%         qemu-kvm  [kernel.kallsyms]     [k] compact_zone                              
>                    |
>                    --- compact_zone
>                        compact_zone_order
>                        try_to_compact_pages
>                        __alloc_pages_direct_compact
>                        __alloc_pages_nodemask
>                        alloc_pages_vma
>                        do_huge_pmd_anonymous_page
>                        handle_mm_fault
>                        __get_user_pages
>                        get_user_page_nowait
>                        hva_to_pfn.isra.33
>                        __gfn_to_pfn
>                        gfn_to_pfn_async
>                        try_async_pf
>                        tdp_page_fault
>                        kvm_mmu_page_fault
>                        pf_interception
>                        handle_exit
>                        kvm_arch_vcpu_ioctl_run
>                        kvm_vcpu_ioctl
>                        do_vfs_ioctl
>                        sys_ioctl
>                        system_call_fastpath
>                        ioctl
>                       |          
>                       |--52.09%-- 0x10100000002
>                       |          
>                        --47.91%-- 0x10100000006
>      0.11%         qemu-kvm  [kernel.kallsyms]     [k] flush_tlb_func                            
>                    |
>                    --- flush_tlb_func
>                       |          
>                       |--99.58%-- generic_smp_call_function_interrupt
>                       |          smp_call_function_interrupt
>                       |          call_function_interrupt
>                       |          |          
>                       |          |--94.65%-- compact_checklock_irqsave
>                       |          |          isolate_migratepages_range
>                       |          |          compact_zone
>                       |          |          compact_zone_order
>                       |          |          try_to_compact_pages
>                       |          |          __alloc_pages_direct_compact
>                       |          |          __alloc_pages_nodemask
>                       |          |          alloc_pages_vma
>                       |          |          do_huge_pmd_anonymous_page
>                       |          |          handle_mm_fault
>                       |          |          __get_user_pages
>                       |          |          get_user_page_nowait
>                       |          |          hva_to_pfn.isra.33
>                       |          |          __gfn_to_pfn
>                       |          |          gfn_to_pfn_async
>                       |          |          try_async_pf
>                       |          |          tdp_page_fault
>                       |          |          kvm_mmu_page_fault
>                       |          |          pf_interception
>                       |          |          handle_exit
>                       |          |          kvm_arch_vcpu_ioctl_run
>                       |          |          kvm_vcpu_ioctl
>                       |          |          do_vfs_ioctl
>                       |          |          sys_ioctl
>                       |          |          system_call_fastpath
>                       |          |          ioctl
>                       |          |          |          
>                       |          |          |--78.04%-- 0x10100000006
>                       |          |          |          
>                       |          |           --21.96%-- 0x10100000002
>                       |          |          
>                       |          |--4.67%-- sub_preempt_count
>                       |          |          _raw_spin_unlock_irqrestore
>                       |          |          compact_checklock_irqsave
>                       |          |          isolate_migratepages_range
>                       |          |          compact_zone
>                       |          |          compact_zone_order
>                       |          |          try_to_compact_pages
>                       |          |          __alloc_pages_direct_compact
>                       |          |          __alloc_pages_nodemask
>                       |          |          alloc_pages_vma
>                       |          |          do_huge_pmd_anonymous_page
>                       |          |          handle_mm_fault
>                       |          |          __get_user_pages
>                       |          |          get_user_page_nowait
>                       |          |          hva_to_pfn.isra.33
>                       |          |          __gfn_to_pfn
>                       |          |          gfn_to_pfn_async
>                       |          |          try_async_pf
>                       |          |          tdp_page_fault
>                       |          |          kvm_mmu_page_fault
>                       |          |          pf_interception
>                       |          |          handle_exit
>                       |          |          kvm_arch_vcpu_ioctl_run
>                       |          |          kvm_vcpu_ioctl
>                       |          |          do_vfs_ioctl
>                       |          |          sys_ioctl
>                       |          |          system_call_fastpath
>                       |          |          ioctl
>                       |          |          |          
>                       |          |          |--78.18%-- 0x10100000006
>                       |          |          |          
>                       |          |           --21.82%-- 0x10100000002
>                       |           --0.68%-- [...]
>                        --0.42%-- [...]
>      0.09%         qemu-kvm  [kernel.kallsyms]     [k] mod_zone_page_state                       
>                    |
>                    --- mod_zone_page_state
>                       |          
>                       |--80.84%-- isolate_migratepages_range
>                       |          compact_zone
>                       |          compact_zone_order
>                       |          try_to_compact_pages
>                       |          __alloc_pages_direct_compact
>                       |          __alloc_pages_nodemask
>                       |          alloc_pages_vma
>                       |          do_huge_pmd_anonymous_page
>                       |          handle_mm_fault
>                       |          __get_user_pages
>                       |          get_user_page_nowait
>                       |          hva_to_pfn.isra.33
>                       |          __gfn_to_pfn
>                       |          gfn_to_pfn_async
>                       |          try_async_pf
>                       |          tdp_page_fault
>                       |          kvm_mmu_page_fault
>                       |          pf_interception
>                       |          handle_exit
>                       |          kvm_arch_vcpu_ioctl_run
>                       |          kvm_vcpu_ioctl
>                       |          do_vfs_ioctl
>                       |          sys_ioctl
>                       |          system_call_fastpath
>                       |          ioctl
>                       |          |          
>                       |          |--53.90%-- 0x10100000002
>                       |          |          
>                       |           --46.10%-- 0x10100000006
>                       |          
>                        --19.16%-- compact_zone
>                                  compact_zone_order
>                                  try_to_compact_pages
>                                  __alloc_pages_direct_compact
>                                  __alloc_pages_nodemask
>                                  alloc_pages_vma
>                                  do_huge_pmd_anonymous_page
>                                  handle_mm_fault
>                                  __get_user_pages
>                                  get_user_page_nowait
>                                  hva_to_pfn.isra.33
>                                  __gfn_to_pfn
>                                  gfn_to_pfn_async
>                                  try_async_pf
>                                  tdp_page_fault
>                                  kvm_mmu_page_fault
>                                  pf_interception
>                                  handle_exit
>                                  kvm_arch_vcpu_ioctl_run
>                                  kvm_vcpu_ioctl
>                                  do_vfs_ioctl
>                                  sys_ioctl
>                                  system_call_fastpath
>                                  ioctl
>                                  |          
>                                  |--55.04%-- 0x10100000002
>                                  |          
>                                   --44.96%-- 0x10100000006
>      0.09%         qemu-kvm  [kernel.kallsyms]     [k] migrate_pages                             
>                    |
>                    --- migrate_pages
>                       |          
>                       |--96.21%-- compact_zone
>                       |          compact_zone_order
>                       |          try_to_compact_pages
>                       |          __alloc_pages_direct_compact
>                       |          __alloc_pages_nodemask
>                       |          alloc_pages_vma
>                       |          do_huge_pmd_anonymous_page
>                       |          handle_mm_fault
>                       |          __get_user_pages
>                       |          get_user_page_nowait
>                       |          hva_to_pfn.isra.33
>                       |          __gfn_to_pfn
>                       |          gfn_to_pfn_async
>                       |          try_async_pf
>                       |          tdp_page_fault
>                       |          kvm_mmu_page_fault
>                       |          pf_interception
>                       |          handle_exit
>                       |          kvm_arch_vcpu_ioctl_run
>                       |          kvm_vcpu_ioctl
>                       |          do_vfs_ioctl
>                       |          sys_ioctl
>                       |          system_call_fastpath
>                       |          ioctl
>                       |          |          
>                       |          |--52.94%-- 0x10100000002
>                       |          |          
>                       |           --47.06%-- 0x10100000006
>                       |          
>                        --3.79%-- compact_zone_order
>                                  try_to_compact_pages
>                                  __alloc_pages_direct_compact
>                                  __alloc_pages_nodemask
>                                  alloc_pages_vma
>                                  do_huge_pmd_anonymous_page
>                                  handle_mm_fault
>                                  __get_user_pages
>                                  get_user_page_nowait
>                                  hva_to_pfn.isra.33
>                                  __gfn_to_pfn
>                                  gfn_to_pfn_async
>                                  try_async_pf
>                                  tdp_page_fault
>                                  kvm_mmu_page_fault
>                                  pf_interception
>                                  handle_exit
>                                  kvm_arch_vcpu_ioctl_run
>                                  kvm_vcpu_ioctl
>                                  do_vfs_ioctl
>                                  sys_ioctl
>                                  system_call_fastpath
>                                  ioctl
>                                  |          
>                                  |--50.72%-- 0x10100000002
>                                  |          
>                                   --49.28%-- 0x10100000006
>      0.09%         qemu-kvm  [kernel.kallsyms]     [k] __zone_watermark_ok                       
>                    |
>                    --- __zone_watermark_ok
>                       |          
>                       |--95.81%-- zone_watermark_ok
>                       |          compact_zone
>                       |          compact_zone_order
>                       |          try_to_compact_pages
>                       |          __alloc_pages_direct_compact
>                       |          __alloc_pages_nodemask
>                       |          alloc_pages_vma
>                       |          do_huge_pmd_anonymous_page
>                       |          handle_mm_fault
>                       |          __get_user_pages
>                       |          get_user_page_nowait
>                       |          hva_to_pfn.isra.33
>                       |          __gfn_to_pfn
>                       |          gfn_to_pfn_async
>                       |          try_async_pf
>                       |          tdp_page_fault
>                       |          kvm_mmu_page_fault
>                       |          pf_interception
>                       |          handle_exit
>                       |          kvm_arch_vcpu_ioctl_run
>                       |          kvm_vcpu_ioctl
>                       |          do_vfs_ioctl
>                       |          sys_ioctl
>                       |          system_call_fastpath
>                       |          ioctl
>                       |          |          
>                       |          |--51.21%-- 0x10100000002
>                       |          |          
>                       |           --48.79%-- 0x10100000006
>                       |          
>                        --4.19%-- compact_zone
>                                  compact_zone_order
>                                  try_to_compact_pages
>                                  __alloc_pages_direct_compact
>                                  __alloc_pages_nodemask
>                                  alloc_pages_vma
>                                  do_huge_pmd_anonymous_page
>                                  handle_mm_fault
>                                  __get_user_pages
>                                  get_user_page_nowait
>                                  hva_to_pfn.isra.33
>                                  __gfn_to_pfn
>                                  gfn_to_pfn_async
>                                  try_async_pf
>                                  tdp_page_fault
>                                  kvm_mmu_page_fault
>                                  pf_interception
>                                  handle_exit
>                                  kvm_arch_vcpu_ioctl_run
>                                  kvm_vcpu_ioctl
>                                  do_vfs_ioctl
>                                  sys_ioctl
>                                  system_call_fastpath
>                                  ioctl
>                                  |          
>                                  |--50.00%-- 0x10100000006
>                                  |          
>                                   --50.00%-- 0x10100000002
>      0.06%             perf  [kernel.kallsyms]     [k] copy_user_generic_string                  
>                        |
>                        --- copy_user_generic_string
>                            generic_file_buffered_write
>                            __generic_file_aio_write
>                            generic_file_aio_write
>                            ext4_file_write
>                            do_sync_write
>                            vfs_write
>                            sys_write
>                            system_call_fastpath
>                            write
>                            run_builtin
>                            main
>                            __libc_start_main
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux