One node goes offline, the other node can't see the replicated volume anymore

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hmmm - I mount my firewall-scripts volume like this now in fstab:

localhost:/firewall-scripts /firewall-scripts glusterfs defaults,_netdev 0 0

But gluster volume info still looks like this on both nodes:

[root at chicago-fw1 system]# gluster volume info

Volume Name: firewall-scripts
Type: Replicate
Volume ID: 239b6401-e873-449d-a2d3-1eb2f65a1d4c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 192.168.253.1:/gluster-fw1
Brick2: 192.168.253.2:/gluster-fw2
[root at chicago-fw1 system]#

Seems to me, my Gluster volume would still have some dependency on networking, even though I mount it with localhost in fstab, right?  

In all our trial and error, the only thing that's worked so far is putting in that 30 second delay in my custom rc.local file.  So instead of fighting this, why not try my hand at a systemd service file named, say, glustermount.service.  

So here is the glustermout.service file I made.  It runs a script named /etc/rc.d/glustermount.sh pasted in below.

[root at chicago-fw1 rc.d]# cd /usr/lib/systemd/system
[root at chicago-fw1 system]# more glustermount.service
# This unit is Greg's attempt to mount my Gluster filesystem.
[Unit]
Description=Set up Gluster mounts
ConditionFileIsExecutable=/etc/rc.d/glustermount.sh
After=network.target glusterd.service

[Service]
Type=oneshot
ExecStart=/etc/rc.d/glustermount.sh
TimeoutSec=0
RemainAfterExit=no
SysVStartPriority=99

[Install]
WantedBy=multi-user.target
[root at chicago-fw1 system]#

Here is glustermount.sh as it exists right now.   Really just a renamed rc.local I've been using the whole time, now with some debug echo lines.  

[root at chicago-fw1 rc.d]# more glustermount.sh
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
#
# Note removed by default starting in Fedora 16.

touch /var/lock/subsys/local

#***********************************
# Local stuff below

date
echo "Sleeping 30 seconds."
### sleep 30
date
echo "Making sure the Gluster stuff is mounted"
echo "Mounted before mount -av"
df -h
mount -av
echo "Mounted after mount -av"
df -h
# The fstab mounts happen early in startup, then Gluster starts up later.
# By now, Gluster should be up and running and the mounts should work.
# That _netdev option is supposed to account for the delay but doesn't seem
# to work right.

echo "Starting up firewall common items"
## /firewall-scripts/etc/rc.d/common-rc.local
[root at chicago-fw1 rc.d]#

I should see mounts before and after when I look at a systemctl status.  But I don't.  And systemctl status shows my /firewall-scripts filesystem successfully mounted, but when I go to look with df -h, it's not mounted.  Yet another midnight mystery - take a look.

[root at chicago-fw1 system]# systemctl start glustermount.service
[root at chicago-fw1 system]# systemctl status glustermount.service -n 50
glustermount.service - Set up Gluster mounts
   Loaded: loaded (/usr/lib/systemd/system/glustermount.service; enabled)
   Active: inactive (dead) since Tue 2013-07-16 00:27:36 CDT; 5s ago
  Process: 2464 ExecStart=/etc/rc.d/glustermount.sh (code=exited, status=0/SUCCESS)

Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Tue Jul 16 00:27:35 CDT 2013
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Sleeping 30 seconds.
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Tue Jul 16 00:27:35 CDT 2013
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Making sure the Gluster stuff is mounted
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Mounted before mount -av
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Filesystem                       Size  Used Avail Use% Mounted on
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: /dev/mapper/fedora-root           14G  3.9G  8.7G  31% /
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: devtmpfs                         990M     0  990M   0% /dev
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: tmpfs                            996M     0  996M   0% /dev/shm
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: tmpfs                            996M  888K  996M   1% /run
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: tmpfs                            996M     0  996M   0% /sys/...roup
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: tmpfs                            996M     0  996M   0% /tmp
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: /dev/sda2                        477M   87M  365M  20% /boot
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: /dev/sda1                        200M  9.4M  191M   5% /boot/efi
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: /dev/mapper/fedora-gluster--fw1  7.9G   33M  7.8G   1% /gluster-fw1
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: extra arguments at end (ignored)
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: /                        : ignored
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: /boot                    : already mounted
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: /boot/efi                : already mounted
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: /gluster-fw1             : already mounted
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: swap                     : ignored
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: /firewall-scripts        : successfully mounted
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: Mounted after mount -av
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: Filesystem                       Size  Used Avail Use% Mounted on
Jul 16 00:27:36 chicago-fw1 systemd[1]: Started Set up Gluster mounts.
[root at chicago-fw1 system]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root           14G  3.9G  8.7G  31% /
devtmpfs                         990M     0  990M   0% /dev
tmpfs                            996M     0  996M   0% /dev/shm
tmpfs                            996M  888K  996M   1% /run
tmpfs                            996M     0  996M   0% /sys/fs/cgroup
tmpfs                            996M     0  996M   0% /tmp
/dev/sda2                        477M   87M  365M  20% /boot
/dev/sda1                        200M  9.4M  191M   5% /boot/efi
/dev/mapper/fedora-gluster--fw1  7.9G   33M  7.8G   1% /gluster-fw1
[root at chicago-fw1 system]#
Finally, running the same glustermount.sh script by hand works just fine.  It  **only** breaks when run from systemd.  Here is what happens when I run it by hand.  Compare below to what happens above when I tell systemd to run it.  

[root at chicago-fw1 system]#
[root at chicago-fw1 system]# /etc/rc.d/glustermount.sh
Tue Jul 16 00:23:15 CDT 2013
Sleeping 30 seconds.
Tue Jul 16 00:23:15 CDT 2013
Making sure the Gluster stuff is mounted
Mounted before mount -av
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root           14G  3.9G  8.7G  31% /
devtmpfs                         990M     0  990M   0% /dev
tmpfs                            996M     0  996M   0% /dev/shm
tmpfs                            996M  888K  996M   1% /run
tmpfs                            996M     0  996M   0% /sys/fs/cgroup
tmpfs                            996M     0  996M   0% /tmp
/dev/sda2                        477M   87M  365M  20% /boot
/dev/sda1                        200M  9.4M  191M   5% /boot/efi
/dev/mapper/fedora-gluster--fw1  7.9G   33M  7.8G   1% /gluster-fw1
/                        : ignored
/boot                    : already mounted
/boot/efi                : already mounted
/gluster-fw1             : already mounted
swap                     : ignored
extra arguments at end (ignored)
/firewall-scripts        : successfully mounted
Mounted after mount -av
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root           14G  3.9G  8.7G  31% /
devtmpfs                         990M     0  990M   0% /dev
tmpfs                            996M     0  996M   0% /dev/shm
tmpfs                            996M  888K  996M   1% /run
tmpfs                            996M     0  996M   0% /sys/fs/cgroup
tmpfs                            996M     0  996M   0% /tmp
/dev/sda2                        477M   87M  365M  20% /boot
/dev/sda1                        200M  9.4M  191M   5% /boot/efi
/dev/mapper/fedora-gluster--fw1  7.9G   33M  7.8G   1% /gluster-fw1
localhost:/firewall-scripts      7.6G   19M  7.2G   1% /firewall-scripts
Starting up firewall common items
[root at chicago-fw1 system]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root           14G  3.9G  8.7G  31% /
devtmpfs                         990M     0  990M   0% /dev
tmpfs                            996M     0  996M   0% /dev/shm
tmpfs                            996M  888K  996M   1% /run
tmpfs                            996M     0  996M   0% /sys/fs/cgroup
tmpfs                            996M     0  996M   0% /tmp
/dev/sda2                        477M   87M  365M  20% /boot
/dev/sda1                        200M  9.4M  191M   5% /boot/efi
/dev/mapper/fedora-gluster--fw1  7.9G   33M  7.8G   1% /gluster-fw1
localhost:/firewall-scripts      7.6G   19M  7.2G   1% /firewall-scripts
[root at chicago-fw1 system]#

- Greg


[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux