Hi,
Can you guide me how to debug postgresql crash?
I have a development VM (kvm) running Centos 8.1(up to date with all patches, kernel 4.18.0-147.8.1.el8_1.x86_64, postgresql12-server-12.2-2PGDG.rhel8.x86_64 ) with a very large partitioned table (few hundred GBs) . Creating an index on partition tables crashed the postgresql instance. I have in the log:
2020-04-26 17:35:50.065 EEST [8385] LOG: background worker "parallel worker" (PID 32152) was terminated by signal 11: Segmentation fault
2020-04-26 17:35:50.065 EEST [8385] DETAIL: Failed process was running: CREATE INDEX IDX_instrid
ON ach.achiteminfo USING btree
(instrid);
2020-04-26 17:35:50.065 EEST [8385] LOG: terminating any other active server processes
2020-04-26 17:35:50.065 EEST [32134] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32134] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32134] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.065 EEST [32146] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32146] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32146] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.065 EEST [32126] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32126] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32126] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.066 EEST [32130] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.066 EEST [32130] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.066 EEST [32130] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.066 EEST [32129] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.066 EEST [32129] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.066 EEST [32129] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.067 EEST [32132] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.067 EEST [32132] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.067 EEST [32132] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.067 EEST [32142] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.067 EEST [32142] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.067 EEST [32142] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.068 EEST [32143] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.068 EEST [32143] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.068 EEST [32143] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.068 EEST [32140] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.068 EEST [32140] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.068 EEST [32140] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.069 EEST [32141] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.069 EEST [32141] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.069 EEST [32141] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.307 EEST [8385] LOG: all server processes terminated; reinitializing
2020-04-26 17:35:52.355 EEST [32164] LOG: database system was interrupted; last known up at 2020-04-26 17:30:29 EEST
2020-04-26 17:35:52.703 EEST [32164] LOG: database system was not properly shut down; automatic recovery in progress
2020-04-26 17:35:52.705 EEST [32164] LOG: redo starts at 6FD/A53E0110
2020-04-26 17:35:53.885 EEST [32165] FATAL: the database system is in recovery mode
2020-04-26 17:35:53.894 EEST [32166] FATAL: the database system is in recovery mode
2020-04-26 17:35:55.285 EEST [32167] FATAL: the database system is in recovery mode
2020-04-26 17:35:56.346 EEST [32168] FATAL: the database system is in recovery mode
2020-04-26 17:35:57.416 EEST [32169] FATAL: the database system is in recovery mode
2020-04-26 17:35:58.270 EEST [32164] LOG: redo done at 6FD/ED969B98
2020-04-26 17:35:58.466 EEST [32170] FATAL: the database system is in recovery mode
2020-04-26 17:35:59.555 EEST [32171] FATAL: the database system is in recovery mode
2020-04-26 17:36:00.606 EEST [32172] FATAL: the database system is in recovery mode
2020-04-26 17:36:02.746 EEST [32174] FATAL: the database system is in recovery mode
2020-04-26 17:36:03.409 EEST [8385] LOG: database system is ready to accept connections
2020-04-26 17:36:03.805 EEST [32175] FATAL: the database system is in recovery mode
2020-04-26 17:35:50.065 EEST [8385] DETAIL: Failed process was running: CREATE INDEX IDX_instrid
ON ach.achiteminfo USING btree
(instrid);
2020-04-26 17:35:50.065 EEST [8385] LOG: terminating any other active server processes
2020-04-26 17:35:50.065 EEST [32134] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32134] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32134] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.065 EEST [32146] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32146] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32146] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.065 EEST [32126] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32126] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32126] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.066 EEST [32130] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.066 EEST [32130] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.066 EEST [32130] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.066 EEST [32129] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.066 EEST [32129] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.066 EEST [32129] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.067 EEST [32132] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.067 EEST [32132] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.067 EEST [32132] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.067 EEST [32142] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.067 EEST [32142] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.067 EEST [32142] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.068 EEST [32143] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.068 EEST [32143] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.068 EEST [32143] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.068 EEST [32140] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.068 EEST [32140] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.068 EEST [32140] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.069 EEST [32141] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.069 EEST [32141] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.069 EEST [32141] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.307 EEST [8385] LOG: all server processes terminated; reinitializing
2020-04-26 17:35:52.355 EEST [32164] LOG: database system was interrupted; last known up at 2020-04-26 17:30:29 EEST
2020-04-26 17:35:52.703 EEST [32164] LOG: database system was not properly shut down; automatic recovery in progress
2020-04-26 17:35:52.705 EEST [32164] LOG: redo starts at 6FD/A53E0110
2020-04-26 17:35:53.885 EEST [32165] FATAL: the database system is in recovery mode
2020-04-26 17:35:53.894 EEST [32166] FATAL: the database system is in recovery mode
2020-04-26 17:35:55.285 EEST [32167] FATAL: the database system is in recovery mode
2020-04-26 17:35:56.346 EEST [32168] FATAL: the database system is in recovery mode
2020-04-26 17:35:57.416 EEST [32169] FATAL: the database system is in recovery mode
2020-04-26 17:35:58.270 EEST [32164] LOG: redo done at 6FD/ED969B98
2020-04-26 17:35:58.466 EEST [32170] FATAL: the database system is in recovery mode
2020-04-26 17:35:59.555 EEST [32171] FATAL: the database system is in recovery mode
2020-04-26 17:36:00.606 EEST [32172] FATAL: the database system is in recovery mode
2020-04-26 17:36:02.746 EEST [32174] FATAL: the database system is in recovery mode
2020-04-26 17:36:03.409 EEST [8385] LOG: database system is ready to accept connections
2020-04-26 17:36:03.805 EEST [32175] FATAL: the database system is in recovery mode
Best regards,
Radu
