Wow, Tom, thanks, that gives us a lot to go on.
The first sign of trouble on the client is just ECONNRESET:
[[37m2021-04-23T07:21:01.073Z[39m] [35m WARN[39m: main/6716 on [redacted]:
[37m [36mPubSub lost connection to datatabase, retrying connection. Error: read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:209:20)
at TCP.callbackTrampoline (internal/async_hooks.js:131:14) {
errno: -4077,
code: 'ECONNRESET',
syscall: 'read'
}[39m
[39m[[37m2021-04-23T07:21:01.079Z[39m] [35m WARN[39m: main/6716 on [redacted]:
[37m [36mPubSub lost connection to datatabase, retrying connection. Error: Connection terminated unexpectedly
(Times in that log are GMT, so 07:21am here === 03:21am in the psql log).
I don’t believe we have any unusual extensions. We do have triggers, and the VM does have antivirus protection. I’ll work on exclusions for the AV, and we’ll look into our triggers a bit.
Thanks!
Eric
From:
Tom Lane <tgl@xxxxxxxxxxxxx>
Date: Friday, April 23, 2021 at 12:36 PM
To: Eric Hill <Eric.Hill@xxxxxxx>
Cc: pgsql-general@xxxxxxxxxxxxxxxxxxxx <pgsql-general@xxxxxxxxxxxxxxxxxxxx>
Subject: Re: server process exited with code 1
EXTERNAL
Eric Hill <Eric.Hill@xxxxxxx> writes:
> 2021-04-23 03:20:57 EDT [5324]: LOG: connection received: host=10.120.80.162 port=54017
> 2021-04-23 03:20:57 EDT [5324]: LOG: connection authorized: user=dba_webjmp database=webjmp
> 2021-04-23 03:21:00 EDT [15776]: LOG: server process (PID 14820) exited with exit code 1
> 2021-04-23 03:21:00 EDT [15776]: DETAIL: Failed process was running: SET client_min_messages TO warning;SET TIME ZONE INTERVAL '+00:00' HOUR TO MINUTE;
> 2021-04-23 03:21:00 EDT [15776]: LOG: terminating any other active server processes
Hm. That's fairly odd. Exit code 1 ordinarily means that the backend
hit a "FATAL" error, which is one that causes process termination but
*not* a database-wide restart. The fact that the postmaster is treating
it that way anyway implies that the backend failed to disarm the "dead
man switch" mechanism that exists to check that backends cleaned up
their shared-memory state before exiting. Another thing that doesn't
square with this being an ordinary FATAL exit is that there's no message
visible in the log.
(I wonder what this looks like from the client's end --- is it getting
any sort of termination message, or just a lost network connection?)
The implication is that something in the backend did _exit(1) to force
process termination without any of the normal atexit cleanup. There
is not, or at least isn't supposed to be, anything in Postgres proper
that does that. So I'm wondering if you have any nonstandard code
in there, such as unusual extensions. A badly-written event trigger
could perhaps do it too.
This being Windows, a certain amount of suspicion has to be directed
towards bogus antivirus code, too.
regards, tom lane
