Re: how to monitor the progress of really large bulk operations?

[Melvin Davidson <melvin6925@xxxxxxxxx>]

On Tue, Sep 27, 2016 at 5:03 PM, Mike Sofen <msofen@xxxxxxxxxx> wrote:

Hi gang,

 

On PG 9.5.1, linux, I’m running some large ETL operations, migrate data from a legacy mysql system into PG, upwards of 250m rows in a transaction (it’s on a big box).  It’s always a 2 step operation – extract raw mysql data and pull it to the target big box into staging tables that match the source, the second step being read the landed dataset and transform it into the final formats, linking to newly generated ids, compressing big subsets into jsonb documents, etc.

 

While I could break it into smaller chunks, it hasn’t been necessary, and it doesn’t eliminate my need:  how to view the state of a transaction in flight, seeing how many rows have been read or inserted (possible for a transaction in flight?), memory allocations across the various PG processes, etc.

 

Possible or a hallucination?

 

Mike Sofen (Synthetic Genomics)

AFAIK, it is not currently possible to monitor the progress/status of a query. However, I do see that this is planned for "sometime in the  future".
https://postgresql.uservoice.com/forums/21853-general/suggestions/249745-query-progress-reported-by-percentage-in-pg-stat

--

Melvin Davidson
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.