Greg Smith wrote:
This thread reminds me of Jignesh's "Proposal of tunable fix for scalability of 8.4" thread from March, except with only a fraction of the real-world detail. There are multiple high-profile locks causing scalability concerns at quadruple digit high user counts in the PostgreSQL code base, finding them is easy. Shoot, I know exactly where a couple are, and I didn't have to think about it at all--just talked with Jignesh a couple of times, led me right to them. Fixing them without causing regressions in low client count cases, now that's the hard part. No amount of theoretical discussion advances that any until you're at least staring at a very specific locking problem you've already characterized extensively via profiling. And even then, profiling trumps theory every time. This is why I stay out of these discussions and work on boring benchmark tools instead.
I disagree that profiling trumps theory every time. Profiling is useful for identifying places where the existing architecture exhibits the best and worst behaviour. It doesn't tell you whether a different architecture (even a slightly different architecture) would work better or worse. It might help identify architecture problems. It does not provide you with architectural solutions.
I think it would be more correct to say that prototyping trumps theory. That is, if somebody has a theory, and they invest time into a proof-of-concept patch, and post actual results to show you that "by changing this code over here to that, I get a N% improvement when using thousands of connections, at no measurable cost for the single connection case", these results will be far more compelling than theory.
Still, it has to involve theory, as not everybody has the time to run off and prototype every wild idea. Discussion can determine whether an idea has enough merit to be worth investing in a prototype.
I think several valuable theories have been discussed, many of which directly apply to the domain that PostgreSQL fits within. The question isn't about how valuable these theories are - they ARE valuable. The question is how much support from the team can be gathered to bring about change, and how willing the team is to accept or invest in architectural changes that might take PostgreSQL to the next level. The real problem here is the words "invest" and "might". That is, people are not going to invest on a "might" - people need to be convinced, and for people that don't have a problem today, the motivation to make the investment is far less.
In my case, all I have to offer you is theory at this time. I don't have the time to work on PostgreSQL, and I have not invested the time to learn the internals of PostgreSQL well enough to comfortably and effectively make changes to implement a theory I might have. I want to get there - but there are so many other projects and ideas to pursue, and I only have a few hours a day to decide what to spend it on.
You can tell me "sorry, your contribution of theory isn't welcome". In fact, that looks like exactly what you have done. :-)
If the general community agrees with you, I'll stop my contributions of theories. :-)
I think, though, that some of the PostgreSQL architecture is "old theory". I have this silly idea that PostgreSQL could one day be better than Oracle (in terms of features and performance - PostgreSQL already beats Oracle on cost :-) ). It won't get there without some significant changes. In only the last few years, I have watched as some pretty significant changes were introduced into PostgreSQL that significantly improved its performance and feature set. Many of these changes might have started with profiling - but the real change came from applied theory, not from profiling. Bitmap indexes are an example of this. Profiling tells you what - that large joins involving OR are slow? It takes theory to answer "why" and "so, what do we do about it?"
Cheers, mark -- Mark Mielke <mark@xxxxxxxxx> -- Sent via pgsql-performance mailing list (pgsql-performance@xxxxxxxxxxxxxx) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
