On Mon, Nov 04, 2019 at 06:33:15PM -0500, Gunther wrote:
The time has come.
FPGA optimization is in the palm of our hands (literally a 2 TB 40
GB/s IO PostgreSQL server fits into less than a shoe box), and on
Amazon AWS F1 instances.
Some demos are beginning to exist: https://github.com/Xilinx/data-analytics.
<https://github.com/Xilinx/data-analytics>
But a lot more could be done. How about linear sort performance at
O(N)? https://hackaday.com/2016/01/20/a-linear-time-sorting-algorithm-for-fpgas/.
And how about https://people.csail.mit.edu/wjun/papers/fccm2017.pdf,
the following four sorting accelerators are used:
* Tuple Sorter : Sorts an N-tuple using a sorting network.
* Page Sorter : Sorts an 8KB (a flash page) chunk of sorted N-tuples
in on-chip memory.
* Super-Page Sorter : Sorts 16 8K-32MB sorted chunks in DRAM.
* Storage-to-Storage Sorter: Sorts 16 512MB or larger sorted chunks in
flash.
Order of magnitude speed improvements? Better than Hadoop clusters on
a single chip? 40 GB/s I/O throughput massive full table scan, blazing
fast sort-merge joins? Here it is. Anybody working more on that?
Should be an ideal project for a student or a group of students.
For the record, this is not exactly a new thing. Netezza (a PostgreSQL
fork started in 1999 IBM) used FPGAs. Now there's swarm64 [1], another
PostgreSQL fork, also using FPGAs with newer PostgreSQL releases.
Those are proprietary forks, though. The main reason why the community
itself is not working on this directly (at least not on pgsql-hackers)
is exactly that it requires specialized hardware, which the devs
probably don't have, making development impossible, and the regular
customers are not asking for it either (one of the reasons being limited
availability of such hardware, especially for customers running in the
cloud and not being even able to deploy custom appliances).
I don't think this will change, unless the access to systems with FPGAs
becomes much easier (e.g. if AWS introduces such instance type).
Is there a PostgreSQL foundation I could donate to, 501(c)(3) tax
exempt? I can donate and possibly find some people at Purdue
University who might take this on. Interest?
I don't think there's any such non-profit, managing/funding development.
At least I'm not avare of it. There are various non-profits around the
world, but those are organizing events and local communities.
I'd say the best way to do something like this is to either talk to one
of the companies participating in PostgreSQL devopment (pgsql-hackers is
probably a good starting point), or - if you absolutely need to go
through a non-profit - approach a university (which does not mean people
from pgsql-hackers can't be involved, of course). I've been involved in
a couple of such research projects in Europe, not sure what exactly is
the situation/rules in US.
regards
[1] https://swarm64.com/netezza-replacement/
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
