I’m continuing my upgrade journey, this time from 11 to 13, and the process is dying in the copy phase, always on the same DB:
—
Performing Upgrade
------------------
Analyzing all rows in the new cluster ok
Freezing all rows in the new cluster ok
Deleting files from new pg_xact ok
Copying old pg_xact to new server ok
Setting next transaction ID and epoch for new cluster ok
Deleting files from new pg_multixact/offsets ok
Copying old pg_multixact/offsets to new server ok
Deleting files from new pg_multixact/members ok
Copying old pg_multixact/members to new server ok
Setting next multixact ID and offset for new cluster ok
Resetting WAL archives ok
Setting frozenxid and minmxid counters in new cluster ok
Restoring global objects in the new cluster ok
Restoring database schemas in the new cluster
messages
*failure*
Consult the last few lines of "pg_upgrade_dump_16387.log" for
the probable cause of the failure.
Failure, exiting
—
The log contains (which is different each time):
—
pg_restore: WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
pg_restore: creating COMMENT "public.FUNCTION "st_isempty"("rast" "public"."raster")"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 5338; 0 0 COMMENT FUNCTION "st_isempty"("rast" "public"."raster") postgres
pg_restore: error: could not execute query: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Command was: COMMENT ON FUNCTION "public"."st_isempty"("rast" "public"."raster") IS 'args: rast - Returns true if the raster is empty (width = 0 and height = 0). Otherwise, returns false.’;
—
And the pgsql13 server log contains:
—
2020-11-17 11:51:40.953 EST [96545] LOG: database system is ready to accept connections
free(): invalid pointer
2020-11-17 11:51:42.880 EST [96545] LOG: server process (PID 96575) was terminated by signal 6: Aborted
2020-11-17 11:51:42.880 EST [96545] LOG: terminating any other active server processes
2020-11-17 11:51:42.880 EST [96582] WARNING: terminating connection because of crash of another server process
2020-11-17 11:51:42.880 EST [96582] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-11-17 11:51:42.880 EST [96582] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-11-17 11:51:42.884 EST [96545] LOG: all server processes terminated; reinitializing
2020-11-17 11:51:42.904 EST [96545] LOG: received fast shutdown request
2020-11-17 11:51:42.905 EST [96585] LOG: database system was interrupted; last known up at 2020-11-17 11:51:42 EST
2020-11-17 11:51:42.906 EST [96585] LOG: database system was not properly shut down; automatic recovery in progress
2020-11-17 11:51:42.906 EST [96585] LOG: redo starts at E0/DB6B2960
2020-11-17 11:51:42.907 EST [96545] LOG: abnormal database system shutdown
2020-11-17 11:51:42.909 EST [96545] LOG: database system is shut down
—
So I’m assuming it’s that free() call. Servers have PostGIS 3.0 on them, all installed from repo, and running CentOS 8.
