On Wed, Mar 07, 2012 at 05:07:45PM +0900, Takuya Yoshikawa wrote: > Marcelo Tosatti <mtosatti@xxxxxxxxxx> wrote: > > > > Partly yes: my method mainly depends on the number of dirty pages, > > > not slot size. > > > > > > But it is not a new problem: traversing all shadow pages for that > > > also takes linearly increasing time. > > > > It was not necessary to read the bitmap under mmu_lock previously. > > Let's check actual data! > > Below I pasted a simple test result to show that reading bitmap is not > a problem at all compared to traversing shadow pages. > > ** During doing the same live migration test as: > > For real workloads, both VGA and live migration, we have observed pure > improvements: when the guest was reading a file during live migration, > we originally saw a few ms of latency, but with the new method the > latency was less than 200us. > > I measured how long the current method takes to just write protect sptes > with the mmu_lock held - kvm_mmu_slot_remove_write_access() time. > > > You can see many ms order of protection times from this result: for me this > is more problematic than downtime problem many people like to improve. > > In contrast my method only took 200us in the worst case: actually what I > measured for that was the entire kvm_vm_ioctl_get_dirty_log() time which > contained more extra tasks, e.g. copy_to_user(). > > FYI: changing the guest memory size from 4GB to 8GB did not show any > siginificant change to my method, but, as you can guess, traversing > shadow pages will need more time for increased shadow pages. > > > If we have 4K shadow pages in the slot, kvm_mmu_slot_remove_write_access() > have to traverse all of them, checking all 512 entries in them. > > Compared to that, the bitmap size of 4GB memory slot is 1M bits = 128KB. > Reading this 8 pages is negligible. Right, thanks for checking it. > My unit-test experiment has also showed that xchg overheads is not so much, > compared to others: > > 493900.4 15911.9 60.2 125% 5% 8K > 760268.2 5929.6 46.4 63% 199% 16K > 1238709.6 7123.5 37.8 23% 173% 32K > 2359523.6 3121.7 36.0 -9% 87% 64K > 4540780.6 10155.6 34.6 -27% 30% 128K > 8747274.0 10973.3 33.3 -31% -3% 256K > > Note that these cases need to xchg the entire dirty bitmap because at least > one bit is set for each unsigned-long-word. > > The big difference came from the number of sptes to protect alone. > > Takuya What is worrying are large memory cases: think of the 50GB slot case. 100ms hold time is pretty bad (and reacquiring the lock is relatively simple). > > === > funcgraph_entry: + 25.123 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: + 35.746 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 922.886 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: + 20.153 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: + 20.424 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: + 17.595 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: + 20.240 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 9783.060 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1992.718 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1312.128 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2028.900 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1455.889 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1382.795 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2030.321 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1407.248 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2189.321 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1444.344 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2291.976 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1801.848 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1993.104 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1531.858 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2394.283 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1613.203 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1699.472 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2416.467 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1566.451 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1772.670 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1700.544 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1590.114 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2311.419 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1923.888 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2534.780 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2083.623 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1664.170 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2867.553 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2684.615 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1706.371 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2655.976 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1720.777 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2993.758 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1924.842 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3091.190 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1776.427 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2808.984 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2669.008 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2359.525 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2703.617 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2623.198 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1942.833 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1906.551 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2981.093 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2168.301 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1949.932 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2992.925 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3360.511 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1993.321 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3187.857 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 1989.417 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2001.865 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2047.220 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3107.808 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2039.732 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2057.575 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2417.748 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2076.445 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2308.323 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3216.713 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2148.263 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2269.673 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 2133.566 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3757.388 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3372.302 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3679.316 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3516.200 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 630.067 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 3191.830 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: ! 658.717 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: + 66.683 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: + 31.027 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: 0.274 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: 0.568 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: 0.460 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: 0.358 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: 0.197 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: 0.306 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: 0.259 us | kvm_mmu_slot_remove_write_access(); > funcgraph_entry: 0.181 us | kvm_mmu_slot_remove_write_access(); -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
