On Wed, 28 May 2008 18:37:06 -0700 "Medi Montaseri" <montaseri@xxxxxxxxx> wrote: > Hi, > > I am faced with a database disapperance and seeking some explanations > outside of gremlins. > I had a database running at > > cat /etc/sysconfig/pgsl/postmaster > PGDATA=/qmsvol/pg_8.1.9/data > PGLOG=/var/log/pgsql/pgstartup.log > > Where /qmsvol is an iSCSI block device > A couple of days ago, my server was rebooted and by the time I got to it my > database was deleted, gone, zapped, not there any more. > > I looked at my pgstartup.log where I see the following.... > > postmaster cannot access the server configuration file > "/qmsvol/pg_8.1.9/postgresql.conf": Permission denied > over 17 times and then following by... > The database cluster will be initialized with locale en_US.UTF-8. > > I think the following happend... > Since my PGDATA was on an iSCSI device, by the time /etc/rc3.d/S64postgresql > was executed, the device below it was not available.....question...why the > error says "permission denied" vs "file not found". In the meantime, pg_ctl > kept trying and finally concluded that the data directory is blank, and > hence this must be a out-of-box case and he is good to initdb the PGDATA and > as it called initdb to do the job... the iSCSI volume below it came online > and by then the bomb had already been dropped. > > Now I need to find some facts to support this... When you mount a partition on linux, it does this by overlaying it's root directory with the existing one on the parent volume. Ownerships and permissions are also replaced. I expect that the /qmsvol directory will be owned by root, with fairly restrictive access rights. This will not be the case the root ( . ) directory on the external device, which will be postgres-friendly. > Where else can I look for forensics I don't think you need any more! To fix this, I'd do 2 things. First, start postgres much later in the boot sequence: cd /etc/rc3.d ; mv S64postgresql S99postgresql ( and the same in rc5.d if you're using a gui at all ). and do the converse to whichever script mounts your external devices. Also add in a test that the device is mounted in the start) block of /etc/init.d/postgresql... something simple like while [ ! -d /qmsvol/pg_8.1.9/data ] do sleep 5 done ( well, something that can't hang forever would be preferable! ). > > Thanks > Medi > hth, Steve -- Steve Holdoway <steve.holdoway@xxxxxxxxxxxxx>
