On Thu, Mar 26, 2020 at 06:48:37AM -0400, Derrick Stolee wrote: > > - the point of the t/perf suite is to find performance regressions, > > and it won't help with that. We don't compare the numbers between > > two tests (which the perf suite has no idea are even related), and > > any change in its numbers would have nothing to do with bitmaps. > > This does make me think if there is a way to adjust "./run" to test two > different config settings or command-line options instead of just two > different build versions. Perhaps something like > > ./run "HEAD -c core.commitGraph=true" "HEAD -c core.commitGraph=false" -- p4200-line-log.sh > > But that's just musing on my part. I think that's the right direction, but it would require tests knowing how to make use of those extra parameters. What I'd _really_ like is a way to parameterize a whole bunch of things: - test repo - git version - particular config settings do runs with whatever combinations of parameters you choose, and then create tables showing differences across various settings of a parameter. Some parameters would be automatically set up, but test scripts would have to advertise other parameters they know about. I don't think it's conceptually hard, but the existing perf code is pretty hacked-together and relies on the filesystem for storage. So the notion that we'd only ever compare different versions runs deep; that's how the on-disk storage is organized. > > So let's just drop it. It's not useful and is adding minutes to perf > > runs. > > I agree with your reasoning. This is not a critical path for clients, > and all servers should be using bitmaps. Yeah, I should have noted that we're not actually testing the performance of a stock repack anywhere else. It wouldn't belong in p5310, but in theory it might be of interest in another script. But I agree for the reasons you gave that it's not all that interesting a timing. -Peff
