> On 09/06/2023 16:00 CEST Wim Bertels <wim.bertels@xxxxxxx> wrote: > > Joe Conway schreef op vr 09-06-2023 om 09:16 [-0400]: > > On 6/8/23 22:17, Pat Trainor wrote: > > > I need to have a very large matrix to maintain & query, and if not > > > (1,600 column limit), then how could such data be broken down to > > > work? > > > > 100,000 rows * > > 100,000 columns * > > 8 bytes (assuming float8) > > = about 80 GB per matrix if I got the math correct. > > based on my personal experience i would not use postgres in the case > where you need many columns, u can work around this with json for > example, but it will likely end up being less easy to work with > > as Joe replied: R or Python are probably a better fit, > or another database that can easily handle a lot of columns, > postgres is a great database, but not when you need a lot of columns > > (as you noted+: > there might be another backend storage for postgres that can handle > this better (or in the future?), but i don't think there is one; > also there is the header for which standard 8K is provisioned anyway, > so that is the first bottleneck (you can change this value, if you > compile postgres yourself) > https://www.postgresql.org/docs/current/limits.html ) Rasdaman may also be an option. Saw it a few weeks ago on this very list. https://rasdaman.org https://www.postgresql.org/message-id/CAFj8pRDjE0mdL6_b86ZDawHtNeRPQLciWos3m3PGJueJ5COSjQ%40mail.gmail.com -- Erik
