Generally, I'd suggest you think carefully about the nature of the jobs, and draw up a list of must-have properties (performance of course, but also things like whether jobs have to survive planned or unplanned outages, be visible across a WAN, numbers of readers and writers, delivery guarantees, etc etc) and then decide on make versus "buy". Distributed systems are hard, and hard to get right.
On Fri, 22 Mar 2024, 16:17 Thiemo Kellner, <thiemo@xxxxxxxxxxxxxxxxxxxx> wrote:
Am 22.03.2024 um 14:15 schrieb Fred Habash:
> We developed a home-grown queue system using Postgres, but its
> performance was largely hindered by que tables bloating and the need to
> continuously vacuum them. It did not scale whatsoever. With some
> workarounds, we ended up designing three sets of queue tables, switching
> between them based on some queue stats, vacuum the inactive set, and repeat.
> We kept this queue system for low SLA app components. For others, we
> switched to Kafka. Investing in learning and implementing purpose built
> queue systems pays off for the long term.
I wonder whether one should https://youtu.be/VEWXmdjzIpQ&t=543 not to
scale either.
