On 2021-05-02 14:49:44 +0100, Pól Ua Laoínecháin wrote: > While perusing the interweb, I stumbled on this very interesting blog > post from TiDB. > > > https://pingcap.com/blog/tikv-and-spdk-pushing-the-limits-of-storage-performance > > It talks about the Storage Performance Development Kit (SPDK) (spdk.io). This sounds certainly interesting. Without reading up on the details, however, I notice that it implements a file system in user space which means that the kernel cannot enforce any permissions. This may be ok for a database (which typically runs as a single OS user anyway), but it is something to consider. It looks somewhat similar to Oracle's "raw device tablespaces" of the 1980s. By the time I got involved in database programming in the late 1990s these were considered obsolete (negligible performance advantage, but a hassle for the DBA). Maybe with NVME SSDs and persistent memory like Intel Optane it is time to revisit that idea. There are less intrusive possibilities, though: Linux has recently (kernel 5.1 - oh, that is already 2 years old) aquired a new async I/O API named io_uring, which eliminates the system call overhead. I haven't played around with it myself, but some blog posts report quite substantial performance improvements, in some cases approaching the theoretical limits of the used (very fast) SSDs. > Will this have any implications for PostgreSQL, given that it is a db > that compiles/runs on a large number of systems - or can subsystems > such as this be integrated/included for those chips which support it? > This particular SDK appears to be Intel specific, but if one chip > manufacturer can do it, can't they all (eventually)? I don't think the chipset makes much of a difference. It's an open source library written in C - it can almost certainly be recompiled for ARM or whatever. It will almost certainly be linux-specific, though. It's probably possible to write something similar for Windows, MacOS, FreeBSD, etc. but I have no idea how hard that may be (maybe you just have to change a few system calls - maybe you have to rewrite 80 percent of it). More important for PostgreSQL is whether something like this can be incorporated without changing the overall architecture: If you just have to change a handful of functions performing low-level I/O, it may be worthwhile even if only a few systems (Linux systems where the DBA is willing to set up devices for direct access from user space) benefit from it. If it means rewriting large parts of postgres and then some platforms cannot be supported at all or only at reduced performance, this is not an option. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp@xxxxxx | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!"
Attachment:
signature.asc
Description: PGP signature
