On 01/08/2015 11:49 PM, Luigi Semenzato wrote: > I am taking a closer look at the performance of the Linux MM in the > context of heavy zram usage. The bottom line is that there is > surprisingly high overhead (35-40%) from MM code other than > compression/decompression routines. I'd like to share some results in > the hope they will be helpful in planning future development. > > SETUP > > I am running on an ASUS Chromebox with about 2GB RAM (actually 4GB, > but with mem=1931M). The zram block device size is approx. 2.8GB > (uncompressed size). > > http://www.amazon.com/Asus-CHROMEBOX-M004U-ASUS-Desktop/dp/B00IT1WJZQ > > Intel(R) Celeron(R) 2955U @ 1.40GHz > MemTotal: 1930456 kB > SwapTotal: 2827816 kB > > I took the kernel from Linus's tree a few days ago: Linux localhost > 3.19.0-rc2+ (...) x86_64. I also set maxcpus=1. The kernel > configuration is available if needed. > > EXPERIMENTS > > I wrote a page walker (historically called "balloon") which allocates > a lot of memory, more than physical RAM, and fills it with a dump of > /dev/mem from a Chrome OS system running at capacity. The memory > compresses down to about 35%. I ran two main experiments. > > 1. Compression/decompression. After filling the memory, the program > touches the first byte of all pages in a random permutation (I tried > sequentially too, it makes little difference). At steady state, this > forces one page decompression and one compression (on average) at each > step of the walk. > > 2. Decompression only. After filling the memory, the program walks > all pages sequentially. Then it frees the second half of the pages > (the ones most recently touched), and walks the first half. This > causes one page decompression at each step, and almost no > compressions. > > RESULTS > > The average time (real time) to walk a page in microseconds is > > experiment 1 (compress + decompress): 26.5 us/page > experiment 2 (decompress only): 9.3 us/page > > I ran "perf record -ag"during the relevant parts of the experiment. > (CAVEAT: the version of perf I used doesn't match the kernel, it's > quite a bit older, but that should be mostly OK). I put the output of > "perf report" in this Google Drive folder: > > https://drive.google.com/folderview?id=0B6kmZ3mOd0bzVzJKeTV6eExfeFE&usp=sharing > > (You shouldn't need a Google ID to access it. You may have to re-join > the link if the plain text mailer splits it into multiple lines.) > > I also tried to analyze cumulative graph profiles. Interestingly the > only tool I found to do this is gprof2dot (any other suggestion? I I think this could be useful for better graphs here: http://www.brendangregg.com/flamegraphs.html > would prefer a text-based tool). The output is in the .png files in > the same folder. The interesting numbers are: > > experiment 1 > compression 43.2% > decompression 20.4% > everything else 36.4% > > experiment 2 > decompression 61.7% > everything else 38.3% > > The graph profiles don't seem to show low-hanging fruits on any path. > > CONCLUSION > > Before zram, in a situation involving swapping, the MM overhead was > probably nearly invisible, especially with rotating disks. But with > zram the MM is surprisingly close to being the main bottleneck. > Compression/decompression speeds will likely improve, and they are > tuneable (tradeoff between compression ratio and speed). Compression > can happen often in the background, so decompression speed is more > important for latency, and LZ4 decompression can already be a lot > faster than LZO (the experiments use LZO, and LZ4 can be 2x faster). > This suggests that simplifying and speeding up the relevant code paths > in the Linux MM may be worth the effort. > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@xxxxxxxxx. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a> > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>