Firstly thank you to all who have taken the time to reply so far.
I need to clarify a few things based on the feedback I have received.
1. I understand the concerns you have about people using internal APIs
that the developers are free to change. I also understand the risks I
take if I use an undocumented API. I would prefer to use a supported
public API with full support of the community. I want a clean API as
you have put it. My understanding is that the level of abstraction at
which I want to use Postgres is not supported by a public API today.
This API may exist at a less official level for internal use. Lets
revisit the argument as to whether I NEED this level of abstraction in
a later point as people have made good suggestions in this regard to
counter my opinion.
2. I want to base my development on a fully fledged relational DB (not
just an embedded DB). This is because I believe it will increase the
appeal of my product.
3. We see cleanly defined layers in Networking Software and Operating
Systems Software. For example, I can send stuff over a network as
Ethernet Frames if I want to use a library at that level of
abstraction, OR I can send stuff as TCP packets, this assuming I have
at my disposal libraries exposing these levels of abstraction. I
believe there is a case for DBMS software to support this concept of
layering. I'm not aware of any standards based database software stack
but I believe that if it existed it could result in better reuse of
database core technology. This stack would maybe begin with a buffer
cache at level 0, a data structures layer at level 1 (Btrees, heaps
etc), a data model layer at level 2...eventually you work up to client
APIs and so on. Whatever this stack looks like, I would like to be able
to link to a library that exposed the data structure layer of a fully
fledged relational DB. I'm not going to try to implement that from
scratch and I want what I develop to have full SQL support.
4. Why I think I need to work at this level of abstraction. I believe,
certain types of queries can be realized more efficiently by code that
can seek to specific parts of an index and immediately returning a
small number of rows to the client. The query would remain active and
the client would only retrieve the next rows after the previous cursor
position when the client was ready to do so. With a huge underlying
table set this would enable very free incremental browsing of the data.
Imagine a table browser application sitting on top of a multi
million-row table. The user types in a partial name match ("Mal").
They want the table to scroll to the first name matching the name they
have typed ("Malcolm"). It should then be cheap to interactively
scroll downwards from here. It should also be cheap to scroll upwards
to view records immediately preceding "Malcolm". The most natural
way I can think of doing this is an API to cursor seek to "Malcolm"
and navigate forwards or backwards from that cursor pos. I feel a
B-Tree cursor API matches the way you would implement a table like
this, only retrieving data as the user scrolls to it.
