You don't mention shared_buffers, which is quite low by default. Not sure of the memory of your docker container, but it might be prudent to increase shared_buffers to keep as much data as possible in memory rather than needing to read from disk by a second run. To test the possibility Tom Lane suggested, do a manual analyze after data insert to see if it is also slow. Explain (analyze, buffers) select... and then using https://explain.depesz.com/ or https://tatiyants.com/pev/#/plans/new would be a good option to have some visualization on where the query is going off the rails.
