One node goes offline, the other node can't see the replicated volume anymore

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I think we're making progress.  I put in a sleep 30 in my rc.local, rebooted, and my filesystem is now mounted after my first logon.  

Still some stuff I don't understand in /var/log/messages, but my before and after mounts look much better.   And notice how messages from all kinds of things get mixed in together.  So systemd must fire up a bunch of concurrent threads to do its thing.  And that's why F19 boots so fast.  But the tradeoff is you can't count on things happening in sequence.  

I wonder if I can set up one of those service doo-dad files, where I want glusterd started first and then have it run a script to mount my stuff?  That would be a more deterministic way to do it versus sleeping 30 seconds in rc.local.  I have to go out for a couple of hours.  I'll see what I can put together and report results here.

Jul 15 17:13:18 chicago-fw1 audispd: queue is full - dropping event
Jul 15 17:13:18 chicago-fw1 audispd: queue is full - dropping event
Jul 15 17:13:18 chicago-fw1 audispd: queue is full - dropping event
Jul 15 17:13:20 chicago-fw1 systemd[1]: Started GlusterFS an clustered file-system server.
Jul 15 17:13:22 chicago-fw1 mount[1001]: Mount failed. Please check the log file for more details.
Jul 15 17:13:22 chicago-fw1 systemd[1]: firewall\x2dscripts.mount mount process exited, code=exited status=1
Jul 15 17:13:22 chicago-fw1 systemd[1]: Unit firewall\x2dscripts.mount entered failed state.
.
.
. a bazillion meaningless selinux warnings (because selinux=permissive here)
.
.
Jul 15 17:13:40 chicago-fw1 setroubleshoot: SELinux is preventing /usr/sbin/glusterfsd from name_bind access on the tcp_socket .
 For complete SELinux messages. run sealert -l 221b72d0-d5d8-4a70-bedd-697a6b9e0f03
Jul 15 17:13:40 chicago-fw1 setroubleshoot: SELinux is preventing /usr/sbin/glusterfsd from name_bind access on the tcp_socket .
 For complete SELinux messages. run sealert -l 22b9b899-3fe2-47fc-8c5d-7bd5ed0e1f17
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: Mon Jul 15 17:13:40 CDT 2013
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: Making sure the Gluster stuff is mounted
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: Mounted before mount -av
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: Filesystem                       Size  Used Avail Use% Mounted on
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /dev/mapper/fedora-root           14G  3.9G  8.7G  31% /
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: devtmpfs                         990M     0  990M   0% /dev
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: tmpfs                            996M     0  996M   0% /dev/shm
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: tmpfs                            996M  884K  996M   1% /run
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: tmpfs                            996M     0  996M   0% /sys/fs/cgroup
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: tmpfs                            996M     0  996M   0% /tmp
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /dev/sda2                        477M   87M  365M  20% /boot
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /dev/sda1                        200M  9.4M  191M   5% /boot/efi
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /dev/mapper/fedora-gluster--fw1  7.9G   33M  7.8G   1% /gluster-fw1
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: extra arguments at end (ignored)
Jul 15 17:13:40 chicago-fw1 setroubleshoot: SELinux is preventing /usr/sbin/glusterfsd from name_bind access on the tcp_socket .
 For complete SELinux messages. run sealert -l 225efbe9-0ea3-4f5b-8791-c325d2f0eed6
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /                        : ignored
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /boot                    : already mounted
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /boot/efi                : already mounted
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /gluster-fw1             : already mounted
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: swap                     : ignored
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /firewall-scripts        : successfully mounted
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: Mounted after mount -av
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: Filesystem                       Size  Used Avail Use% Mounted on
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /dev/mapper/fedora-root           14G  3.9G  8.7G  31% /
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: devtmpfs                         990M     0  990M   0% /dev
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: tmpfs                            996M     0  996M   0% /dev/shm
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: tmpfs                            996M  884K  996M   1% /run
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: tmpfs                            996M     0  996M   0% /sys/fs/cgroup
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: tmpfs                            996M     0  996M   0% /tmp
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /dev/sda2                        477M   87M  365M  20% /boot
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /dev/sda1                        200M  9.4M  191M   5% /boot/efi
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: /dev/mapper/fedora-gluster--fw1  7.9G   33M  7.8G   1% /gluster-fw1
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: 192.168.253.1:/firewall-scripts  7.6G   19M  7.2G   1% /firewall-scripts
Jul 15 17:13:40 chicago-fw1 rc.local[1005]: Starting up firewall common items
Jul 15 17:13:40 chicago-fw1 systemd[1]: Started /etc/rc.d/rc.local Compatibility.
Jul 15 17:13:40 chicago-fw1 systemd[1]: Starting Terminate Plymouth Boot Screen...


Greg Scott
Infrasupport Corporation
GregScott at Infrasupport.com

Direct 1-651-260-1051


-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Greg Scott
Sent: Monday, July 15, 2013 5:12 PM
To: 'Marcus Bointon'; gluster-users at gluster.org List
Subject: Re: One node goes offline, the other node can't see the replicated volume anymore

And for what it's worth, I just now looked and noticed rc.local does not really run last in the startup sequence anymore.  According to below, it only depends on the network being started.   So I could easily be trying my mounts before gluster ever gets fired up.  

[root at chicago-fw1 system]# pwd
/usr/lib/systemd/system
[root at chicago-fw1 system]# more rc-local.service #  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it #  under the terms of the GNU Lesser General Public License as published by #  the Free Software Foundation; either version 2.1 of the License, or #  (at your option) any later version.


# This unit gets pulled automatically into multi-user.target by # systemd-rc-local-generator if /etc/rc.d/rc.local is executable.
[Unit]
Description=/etc/rc.d/rc.local Compatibility ConditionFileIsExecutable=/etc/rc.d/rc.local
After=network.target

[Service]
Type=forking
ExecStart=/etc/rc.d/rc.local start
TimeoutSec=0
RemainAfterExit=yes
SysVStartPriority=99
[root at chicago-fw1 system]#

- Greg


-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Marcus Bointon
Sent: Monday, July 15, 2013 5:05 PM
To: gluster-users at gluster.org List
Subject: Re: One node goes offline, the other node can't see the replicated volume anymore

On 15 Jul 2013, at 21:22, Greg Scott <GregScott at infrasupport.com> wrote:

> # The fstab mounts happen early in startup, then Gluster starts up later.
> # By now, Gluster should be up and running and the mounts should work.
> # That _netdev option is supposed to account for the delay but doesn't 
> seem # to work right.

It's interesting to see that script - that's what's happens to me with 3.3.0. If I set gluster mounts to mount from fstab with _netdev, it hangs the boot completely and I have to go into single user mode (and edit it out of fstab) to recover, though gluster logs nothing at all. Autofs fails too (though I think that's autofs not understanding mounting an NFS volume from localhost), yet it all works on a manual mount.

Sad to see you're having trouble with 3.4. I hope you can make it work!

Marcus
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux