On Fri, Mar 22, 2013 at 08:37:33PM +0800, Xiao Guangrong wrote: > On 03/22/2013 08:12 PM, Gleb Natapov wrote: > > On Fri, Mar 22, 2013 at 08:03:04PM +0800, Xiao Guangrong wrote: > >> On 03/22/2013 07:47 PM, Gleb Natapov wrote: > >>> On Fri, Mar 22, 2013 at 07:39:24PM +0800, Xiao Guangrong wrote: > >>>> On 03/22/2013 07:28 PM, Gleb Natapov wrote: > >>>>> On Fri, Mar 22, 2013 at 07:10:44PM +0800, Xiao Guangrong wrote: > >>>>>> On 03/22/2013 06:54 PM, Marcelo Tosatti wrote: > >>>>>> > >>>>>>>> > >>>>>>>>> > >>>>>>>>> And then have codepaths that nuke shadow pages break from the spinlock, > >>>>>>>> > >>>>>>>> I think this is not needed any more. We can let mmu_notify use the generation > >>>>>>>> number to invalid all shadow pages, then we only need to free them after > >>>>>>>> all vcpus down and mmu_notify unregistered - at this point, no lock contention, > >>>>>>>> we can directly free them. > >>>>>>>> > >>>>>>>>> such as kvm_mmu_slot_remove_write_access does now (spin_needbreak). > >>>>>>>> > >>>>>>>> BTW, to my honest, i do not think spin_needbreak is a good way - it does > >>>>>>>> not fix the hot-lock contention and it just occupies more cpu time to avoid > >>>>>>>> possible soft lock-ups. > >>>>>>>> > >>>>>>>> Especially, zap-all-shadow-pages can let other vcpus fault and vcpus contest > >>>>>>>> mmu-lock, then zap-all-shadow-pages release mmu-lock and wait, other vcpus > >>>>>>>> create page tables again. zap-all-shadow-page need long time to be finished, > >>>>>>>> the worst case is, it can not completed forever on intensive vcpu and memory > >>>>>>>> usage. > >>>>>>> > >>>>>>> Yes, but the suggestion is to use spin_needbreak on the VM shutdown > >>>>>>> cases, where there is no detailed concern about performance. Such as > >>>>>>> mmu_notifier_release, kvm_destroy_vm, etc. In those cases what matters > >>>>>>> most is that host remains unaffected (and that it finishes in a > >>>>>>> reasonable time). > >>>>>> > >>>>>> Okay. I agree with you, will give a try. > >>>>>> > >>>>>>> > >>>>>>>> I still think the right way to fix this kind of thing is optimization for > >>>>>>>> mmu-lock. > >>>>>>> > >>>>>>> And then for the cases where performance matters just increase a > >>>>>>> VM global generetion number, zap the roots and then on kvm_mmu_get_page: > >>>>>>> > >>>>>>> kvm_mmu_get_page() { > >>>>>>> sp = lookup_hash(gfn) > >>>>>>> if (sp->role = role) { > >>>>>>> if (sp->mmu_gen_number != kvm->arch.mmu_gen_number) { > >>>>>>> kvm_mmu_commit_zap_page(sp); (no need for TLB flushes as its unreachable) > >>>>>>> kvm_mmu_init_page(sp); > >>>>>>> proceed as if the page was just allocated > >>>>>>> } > >>>>>>> } > >>>>>>> } > >>>>>>> > >>>>>>> It makes the kvm_mmu_zap_all path even faster than you have now. > >>>>>>> I suppose this was your idea correct with the generation number correct? > >>>>>> > >>>>>> Wow, great minds think alike, this is exactly what i am doing. ;) > >>>>>> > >>>>> Not that I disagree with above code, but why not make mmu_gen_number to be > >>>>> part of a role and remove old pages in kvm_mmu_free_some_pages() whenever > >>>>> limit is reached like we looks to be doing with role.invalid pages now. > >>>> > >>>> These pages can be reused after purge its entries and delete it from parents > >>>> list, it can reduce the pressure of memory allocator. Also, we can move it to > >>>> the head of active_list so that the pages with invalid_gen can be reclaimed first. > >>>> > >>> You mean tail of the active_list, since kvm_mmu_free_some_pages() > >>> removes pages from tail? Since pages with new mmu_gen_number will be put > >> > >> I mean purge the invalid-gen page first, then update its valid-gen to current-gen, > >> then move it to the head of active_list: > >> > >> kvm_mmu_get_page() { > >> sp = lookup_hash(gfn) > >> if (sp->role = role) { > >> if (sp->mmu_gen_number != kvm->arch.mmu_gen_number) { > >> kvm_mmu_purge_page(sp); (no need for TLB flushes as its unreachable) > >> sp->mmu_gen_number = kvm->arch.mmu_gen_number; > >> @@@@@@ move sp to the head of active list @@@@@@ > >> } > >> } > >> } > >> > >> > > And I am saying that if you make mmu_gen_number part of the role you do > > not need to change kvm_mmu_get_page() at all. It will just work. > > Oh, i got what your said. But i want to reuse these page (without > free and re-allocate). What do you think about this? > We did not do that for sp->role.invalid pages although we could do what is proposed above for them too (am I right?). If there is measurable advantage of reusing invalid pages in kvm_mmu_get_page() lets do it like that, but if not then less code is better. > > > >>> at the head of the list it is natural that tail will contain pages with > >>> outdated generation numbers without need to explicitly move them. > >> > >> Currently, only the new allocated page can be moved to the head of > >> active_list. The existing pages are not moved by kvm_mmu_get_page. > >> It seems a bug. > > Ideally it needs to be LRU list based on accessed bit scanning. > > Yes, but unfortunately, A bit does not be supported on some intel cpus... > Yes, but there is not much we can do there. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html