On 11 Mar 2024, at 16:04, Adrian Klaver wrote: > On 3/11/24 03:11, Nick Renders wrote: >> Thank you for your reply Laurenz. >> I don't think it is related to any third party security software. We have several other machines with a similar setup, but this is the only server that has this issue. >> >> The one thing different about this machine however, is that it runs 2 instances of Postgres: >> - cluster A on port 165 >> - cluster B on port 164 >> Cluster A is actually a backup from another Postgres server that is restored on a daily basis via Barman. This means that we login remotely from the Barman server over SSH, stop cluster A's service (port 165), clear the Data folder, restore the latest back into the Data folder, and start up the service again. >> Cluster B's Data and service (port 164) remain untouched during all this time. This is the cluster that experiences the intermittent "operation not permitted" issue. >> >> Over the past 2 weeks, I have suspended our restore script and the issue did not occur. >> I have just performed another restore on cluster A and now cluster B is throwing errors in the log again. > > Since it seems to be the trigger, what are the contents of the restore script? > >> >> Any idea why this is happening? It does not occur with every restore, but it seems to be related anyway. >> >> Thanks, >> >> Nick Renders >> > > > -- > Adrian Klaver > adrian.klaver@xxxxxxxxxxx > ...how are A and B connected? The 2 cluster are not connected. They run on the same macOS 14 machine with a single Postgres installation ( /Library/PostgreSQL/16/ ) and their respective Data folders are located on the same volume ( /Volumes/Postgres_Data/PostgreSQL/16/data and /Volumes/Postgres_Data/PostgreSQL/16-DML/data ). Beside that, they run independently on 2 different ports, specified in the postgresql.conf. > ...run them under different users on the system. Are you referring to the "postgres" user / role? Does that also mean setting up 2 postgres installation directories? > ...what are the contents of the restore script? ## stop cluster A ssh postgres@10.0.0.1 '/Library/PostgreSQL/16/bin/pg_ctl -D /Volumes/Postgres_Data/PostgreSQL/16/data stop' ## save config files (ARC_postgresql_16.conf is included in postgresql.conf and contains cluster-specific information like the port number) ssh postgres@10.0.0.1 'cd /Volumes/Postgres_Data/PostgreSQL/16/data && cp ARC_postgresql_16.conf ../ARC_postgresql_16.conf' ssh postgres@10.0.0.1 'cd /Volumes/Postgres_Data/PostgreSQL/16/data && cp pg_hba.conf ../pg_hba.conf' ## clear data directory ssh postgres@10.0.0.1 'rm -r /Volumes/Postgres_Data/PostgreSQL/16/data/*' ## transfer recovery (this will copy the backup "20240312T040106" and any lingering WAL files into the Data folder) barman recover --remote-ssh-command 'ssh postgres@10.0.0.1' pg 20240312T040106 /Volumes/Postgres_Data/PostgreSQL/16/data ## restore config files ssh postgres@10.0.0.1 'cd /Volumes/Postgres_Data/PostgreSQL/16/data && cd .. && mv ARC_postgresql_16.conf /Volumes/Postgres_Data/PostgreSQL/16/data/ARC_postgresql_16.conf' ssh postgres@10.0.0.1 'cd /Volumes/Postgres_Data/PostgreSQL/16/data && cd .. && mv pg_hba.conf /Volumes/Postgres_Data/PostgreSQL/16/data/pg_hba.conf' ## start cluster A ssh postgres@10.0.0.1 '/Library/PostgreSQL/16/bin/pg_ctl -D /Volumes/Postgres_Data/PostgreSQL/16/data start > /dev/null' This script runs on a daily basis at 4:30 AM. It did so this morning and there was no issue with cluster B. So even though the issue is most likely related to the script, it does not cause it every time. Best regards, Nick Renders
