On Fri, 2009-06-12 at 19:05 -0700, Adam Williamson wrote: > On Sat, 2009-06-13 at 04:33 +0300, Gilboa Davara wrote: > > > You don't like Phoronix' benchmark? Why? What should they have done > > differently? Have you ever contacted Phoronix (E.g. Using their forums) > > and tried to resolve these issues? Did they refuse? > > They should use distribution-compiled binaries - or at least record and > explain the fact that they don't, and check whether there are > significant differences between the compiled binaries they use on each > distro. Up until 30 minutes ago, I was unaware of the fact that they use test-suite compiled binaries. Though I'd imagine that in Phoronix' view, having (far) different compile options in the distribution supplied binaries might generate invalid results. (Due to missing features, non-standard optimization, etc) Of-cause, the best solution would have been to test -both- versions - read: Phoronix-compiled binaries next to distribution supplied binaries This should generate far cleaner (and far more interesting) results. > And when they observe anomalies, they should try and do some > kind of research to confirm the result and figure out _why_, not just > note the fact of the anomaly. I fear that you're expecting far too much from popular website. I'd rather see an open dialog between Phoronix and the different distributions an in effort to gain usable test-data out of their benchmarks. > Multiple people have pointed this out to them in the past, but they > haven't really made a concerted effort to change. Has anyone attempted to start an open dialog with them using their forums? [1]. At least the past, Micheal (Phoronix founder) was very responsive. > I have a kind of > love/hate relationship with Phoronix - they're a popular site and do > some really good stuff, but they also make a lot of frustratingly lazy > mistakes and shorthand contractions in many articles. I believe we should praise Phoronix for their work, even if we do not agree with their methodology. As I said in my previous post, Phoronix completely changed the landscape of OSS websites and OSS benchmarking. - Gilboa [1] http://www.phoronix.com/forums/forumdisplay.php?f=49 -- fedora-devel-list mailing list fedora-devel-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-devel-list
