On Mon, Jan 18, 2021 at 04:25:09PM +0100, Pierre-Yves Chibon wrote: > Good Morning Everyone, ...snip... > Currently our ideas are: > - for datanommer: > - port it to fedora-messaging > - adjust it to whichever solution we chose to replace datagrepper > > - for datagrepper: > - keep it as is > - Replace by > - postgres https://postgrest.org/ > - prest https://github.com/prest/prest > - kinto https://docs.kinto-storage.org/en/stable/ > - Swagger/OpenAPI https://swagger.io/ Doing any of those means existing queries no longer work right? Thats kind of a pain. ;( > - Add support for Graphql > > - for the postgresql server > - Split messages per year in different table > - Unite them using a postgresql view I've long wanted to do this. ;) It might be worth making it non default to query more than the most recent year. Most queries won't need anything that old... > - Kick out the old messages per year > - Keep the current year + n-1 in the current DB > - Kick the other to another DB? > - Kick the other to a tarball somewhere? I would prefer to keep it queryable... there's some things that may want to query the entire backhistory. > - Output the database daily dump to file / year > - TimescaleDB a postgresql plugin for time-series data > - https://alibaba-cloud.medium.com/postgresql-time-series-database-plug-in-timescaledb-deployment-practices-6a07e246eb0d > - https://dev.t-matix.com/blog/postgresql-as-a-time-series-database/ > - https://docs.timescale.com/latest/introduction > - Make the msg field in the message table be a JSON field If you do this would you convert all the old messages? > Would you have any other ideas of things we could look at? I'm not sure if there's any compression we could use here, but it would be nice if the data took up less room. :) Thanks for looking into this! kevin
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
