Matthew wrote:
On Tue, 4 Dec 2007, Gregory Stark wrote:
Fwiw, what made you bring up this topic now? You're the second person in about
two days to bring up precisely this issue and it was an issue I had been
planning to bring up on -hackers as it was.
I only just joined the performance mailing list to talk about R-trees. I
would probably have brought it up earlier if I had been here earlier.
However, we're thinking of buying this large machine, and that reminded
me.
I have been biting at the bit for my bosses to allow me time to write an
indexing system for transient data - a lookup table backed by disc,
looking up from an integer to get an object, native in Java. Our system
often needs to fetch a list of a thousand different objects by a key like
that, and Postgres just doesn't do that exact thing fast. I was going to
implement it with full asynchronous IO, to do that particular job very
fast, so I have done a reasonable amount of research into the topic. In
Java, that is. It would add a little bit more performance for our system.
That wouldn't cover us - we still need to do complex queries with the same
problem, and that'll have to stay in Postgres
So much excitement and zeal - refreshing to see. And yet, no numbers!
:-)
You describe a new asynchronous I/O system to map integers to Java
objects above. Why would you write this? Have you tried BerkeleyDB or
BerkeleyDB JE? BerkeleyDB with BDB as a backend or their full Java
backend gives you a Java persistence API that will allow you to map
Java objects (including integers) to other Java objects. They use
generated Java run time instructions instead of reflection to store and
lock your Java objects. If it came to a bet, I would bet that their
research and tuning over
several years, and many people, would beat your initial implementation,
asynchronous I/O or not.
Asynchronous I/O is no more a magic bullet than threading. It requires
a lot of work to get it right, and if one gets it wrong, it can be
slower than the regular I/O or single threaded scenarios. Both look
sexy on paper, neither may be the solution to your problem. Or they may
be. We wouldn't know without numbers.
Cheers,
mark
--
Mark Mielke <mark@xxxxxxxxx>
|
