Is sqlite is a constraint?
From big data presctive is better to find a more natural format like parquet or even json. For example what if you've different sqlite version
בתאריך יום ה׳, 23 בדצמ׳ 2021, 17:02, מאת Дмитрий Иванов <firstdismay@xxxxxxxxx>:
Or, if you want to extend this theme, you can use a PostgreSQL-based "SQLite file player" with
PostgreSQL + Python[sqlite3] extension.This way you can provide direct access to SQLite files without duplicating data in PostgreSQL cluster tables.
PS: It may seem that this will reduce performance. When I started my project, I had some preconceptions about Python. But analyzing projects like Patroni changed my mind.--Regards, Dmitry!ср, 22 дек. 2021 г. в 10:24, David G. Johnston <david.g.johnston@xxxxxxxxx>:On Tue, Dec 21, 2021 at 10:06 PM David Gauthier <davegauthierpg@xxxxxxxxx> wrote:I'll have to read more about sqlite_fdw. Thanks for that Steve !Each SQLite isn't that big (billions of records), more like 30K records or so. But there are lots and lots of these SQLite DBs which add up over time to perhaps billions of records.This is for a big corp with an IT dept. Maybe I can get them to upgrade the DB itself.Thank You too David !So, more similar to the image storage question than I first thought, but still large enough where the specific usage patterns and needs end up being the deciding factor (keeping in mind you can pick multiple solutions - so that really old data, ideally on a partition, can be removed from the DB while still remaining accessible if just more slowly or laboriously).One possibility to consider - ditch the SQLite dependency and just store CSV (but maybe with a funky delimiter sequence). You can then us "string_to_table(...)" on that delimiter to materialize a table out of the data right in a query.David J.
