On Mon, Jan 18, 2021 at 05:41:35PM -0800, Kevin Fenzi wrote: > On Mon, Jan 18, 2021 at 04:25:09PM +0100, Pierre-Yves Chibon wrote: > > Good Morning Everyone, > ...snip... > > > Currently our ideas are: > > - for datanommer: > > - port it to fedora-messaging > > - adjust it to whichever solution we chose to replace datagrepper > > > > - for datagrepper: > > - keep it as is > > - Replace by > > - postgres https://postgrest.org/ > > - prest https://github.com/prest/prest > > - kinto https://docs.kinto-storage.org/en/stable/ > > - Swagger/OpenAPI https://swagger.io/ > > Doing any of those means existing queries no longer work right? > Thats kind of a pain. ;( Yes, that will need to be taken into account when a decision is made on which of these solutions to pick. > > - Add support for Graphql > > > > - for the postgresql server > > - Split messages per year in different table > > - Unite them using a postgresql view > > I've long wanted to do this. ;) > > It might be worth making it non default to query more than the most > recent year. Most queries won't need anything that old... > > > - Kick out the old messages per year > > - Keep the current year + n-1 in the current DB > > - Kick the other to another DB? > > - Kick the other to a tarball somewhere? > > I would prefer to keep it queryable... there's some things that may want > to query the entire backhistory. Fair. Again, the idea was to list every ideas here and then we can weight them again each other. > > - Output the database daily dump to file / year > > - TimescaleDB a postgresql plugin for time-series data > > - https://alibaba-cloud.medium.com/postgresql-time-series-database-plug-in-timescaledb-deployment-practices-6a07e246eb0d > > - https://dev.t-matix.com/blog/postgresql-as-a-time-series-database/ > > - https://docs.timescale.com/latest/introduction > > - Make the msg field in the message table be a JSON field > > If you do this would you convert all the old messages? Yes > > Would you have any other ideas of things we could look at? > > I'm not sure if there's any compression we could use here, but it would > be nice if the data took up less room. :) Worth looking into as well. > Thanks for looking into this! Hopefully it'll be productive and successful! Pierre
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
