On (23/02/23 15:53), Minchan Kim wrote: > > TEST > > ==== > > > > It's very challenging to reliably test this series. I ended up > > developing my own synthetic test that has 100% reproducibility. > > The test generates significan fragmentation (for each size class) > > and then performs compaction for each class individually and tracks > > the number of memcpy() in zs_object_copy(), so that we can compare > > the amount work compaction does on per-class basis. > > > > Total amount of work (zram mm_stat objs_moved) > > ---------------------------------------------- > > > > Old fullness grouping, old compaction algorithm: > > 323977 memcpy() in zs_object_copy(). > > > > Old fullness grouping, new compaction algorithm: > > 262944 memcpy() in zs_object_copy(). > > > > New fullness grouping, new compaction algorithm: > > 213978 memcpy() in zs_object_copy(). > > > > > > Per-class compaction memcpy() comparison (T-test) > > Just curiosity: What's the T-test? T-test is a statistical method used to compare the means of two independent groups or samples and determine if the difference between them is statistically significant. > > x Old fullness grouping, old compaction algorithm > > + Old fullness grouping, new compaction algorithm > > > > N Min Max Median Avg Stddev > > x 140 349 3513 2461 2314.1214 806.03271 > > + 140 289 2778 2006 1878.1714 641.02073 > > Difference at 95.0% confidence > > -435.95 +/- 170.595 > > -18.8387% +/- 7.37193% > > (Student's t, pooled s = 728.216) > > > > > > x Old fullness grouping, old compaction algorithm > > + New fullness grouping, new compaction algorithm > > > > N Min Max Median Avg Stddev > > x 140 349 3513 2461 2314.1214 806.03271 > > + 140 226 2279 1644 1528.4143 524.85268 > > Difference at 95.0% confidence > > -785.707 +/- 159.331 > > -33.9527% +/- 6.88516% > > (Student's t, pooled s = 680.132) > > What's the different with result above? Did you just run two times and > shows they are consistent or this is new result based on different > testing? The test is exactly the same, it is designed to have 0 variability, it creates exactly same fragmentation during each run, so we always compare apples to apples. What is being changed (and hence tested) are fullness grouping and compaction algorithm. The first one tests the effect of new compaction algorithm alone: old fullness grouping and old compaction algorithm VS old fullness grouping and new compaction algorithm. The data show that with sufficient level of confidence (95%) we can claim that new compaction does make a statstically significant improvement and reduce the number of memcpy() calls (by 18.3% in this particular case). The second one tests the effect of new fullness grouping and new compaction algorithm. The data show that with sufficient level of confidence we can claim that new fullness grouping and new compaction do make a statstically significant improvement and reduce the number of memcpy() calls (by 33.9% in this particular case).
