best ways to handle large matrix data

["Wenjian Yang" <wenjiany@xxxxxxxxx>]

Hi, this maybe a more general question and not limited to postgresql.

We are generating some matrix like data, e.g. one object will have a fixed number 100000 values, and we have currently 300 objects. We can visualized the data as a 100000 by 300 matrix, and each object/column has an identifier, each row has an identifier as well. The number of rows are fixed, but columns will increase by time. The data is type double.

The queries that we encounter most often are that given a set of rowids, e.g. (1000, 2000, 2001, 3241), retrieve the slice of the data. (the set of rowids are usually generated from SQL)

I know that NetCDF and HDF5 can be used to strore such information, but is there any way we can save the matrix in postgres database and have easy retrieval and addition without sacrificing too much space? If not, I guess I'll try to write some functions in postgres to do some retrieval from netcdf. 

Thanks.

Wenjian