2.0.2 - Fast First Access and SQLite Affecting Bugs Still Present

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



As the subject line says - the above two issues, as mentioned before, are still present.

Fast First Access Bug
=====================
To reproduce, use a script that mounts a glusterfs cluster/replicate share from the local node with only the local node being up, and then immediately tries to bind mount a subdirectory from that share into another directory, e.g.

8K-----8K-----8K-----8K-----8K-----8K-----8K-----
#!/bin/bash
mount -t glusterfs \
-o defaults,noatime,nodiratime,direct-io-mode=off,log-file=/dev/null\
,log-level=NONE /etc/glusterfs/root.vol /mnt/newroot
mount --bind /mnt/newroot/cluster/cdsl/2 /mnt/newroot/cdsl.local
8K-----8K-----8K-----8K-----8K-----8K-----8K-----

The bind mount will reliably fail. I'm not sure if this makes any difference WRT the amount of content in the directory being mounted, but in case it does, path that root.vol points at should contain something resembling a Linux root file system (i.e. not that many directories in the root).

Here is the root.vol I'm using:
8K-----8K-----8K-----8K-----8K-----8K-----8K-----
volume root1
        type protocol/client
        option transport-type socket
        option address-family inet
        option remote-host 10.1.0.10
        option remote-subvolume root1
end-volume

volume root-store
        type storage/posix
        option directory /mnt/tmproot/gluster/root/x86_64
end-volume

volume root2
        type features/posix-locks
        subvolumes root-store
end-volume

volume server
        type protocol/server
        option transport-type socket
        option address-family inet
        subvolumes root2
        option auth.addr.root2.allow 127.0.0.1,10.*
end-volume

volume root
        type cluster/replicate
        subvolumes root1 root2
        option read-subvolume root2
end-volume
8K-----8K-----8K-----8K-----8K-----8K-----8K-----

Note that 10.1.0.10 node isn't up, only the local node is up. I haven't tested with the 2nd node up since I haven't built the 2nd node yet.

If I modify the mounting script to do something like this instead:

8K-----8K-----8K-----8K-----8K-----8K-----8K-----
#!/bin/bash
mount -t glusterfs \
-o defaults,noatime,nodiratime,direct-io-mode=off,log-file=/dev/null\
,log-level=NONE /etc/glusterfs/root.vol /mnt/newroot
# Note - added sleep and ls
sleep 2
ls -la /mnt/newroot > /dev/null
sleep 2
ls -laR /mnt/newroot/cluster > /dev/null
sleep 2
mount --bind /mnt/newroot/cluster/cdsl/2 /mnt/newroot/cdsl.local
8K-----8K-----8K-----8K-----8K-----8K-----8K-----

then it works.

SQLite Affecting Bugs
=====================
There seems to be an issue that reliably (but very subtly) affects some of the SQLite functionality. This is evident in the way the RPM database behaves (converted to SQLite because as far as I can tell BDB needs writable mmap() which means it won't work on any fuse based fs) - for example, it just won't find some of the packages even though they are installed. Here is an example (a somewhat ironic one, you might say):

# ls -la /usr/lib64/libfuse.so.*
lrwxrwxrwx 1 root root 16 May 25 12:39 /usr/lib64/libfuse.so.2 -> libfuse.so.2.7.4
-rwxr-xr-x 1 root root 134256 Feb 19 21:40 /usr/lib64/libfuse.so.2.7.4

# rpm -q --whatprovides /usr/lib64/libfuse.so.2
fuse-libs-2.7.4glfs11-1

# rpm -Uvh glusterfs-client-2.0.2-1.el5.x86_64.rpm glusterfs-server-2.0.2-1.el5.x86_64.rpm glusterfs-client-2.0.2-1.el5.x86_64.rpm warning: package glusterfs-client = 2.0.2-1.el5 was already added, skipping glusterfs-client < 2.0.2-1.el5
error: Failed dependencies:
libfuse.so.2()(64bit) is needed by glusterfs-client-2.0.2-1.el5.x86_64 libfuse.so.2(FUSE_2.4)(64bit) is needed by glusterfs-client-2.0.2-1.el5.x86_64 libfuse.so.2(FUSE_2.5)(64bit) is needed by glusterfs-client-2.0.2-1.el5.x86_64 libfuse.so.2(FUSE_2.6)(64bit) is needed by glusterfs-client-2.0.2-1.el5.x86_64

So libfuse is there, RPM knows that fuse-libs-2.7.4glfs11-1 package provides, and yet when glusterfs tries to install, it fails to find it. This _only_ happens when the RPM DB (/var/lib/rpm) is on glusterfs. The sama package sets on machines that aren't rooted on glusterfs deal with this package combination just fine. rpm --rebuilddb doesn't alter the situation at all, the issue is still present after the DB rebuild.

If the above is deemed difficult to set up, there is another way to easily recreate an SQLite related issue. Mount /home via glusterfs, log into X, and fire up Firefox 3.0.x (I'm using 3.0.10 on x86_64, but this has been reproducible for a very long time with older versions, too). Add a bookmark. It'll show up in the bookmarks menu. Now exit firefox, wait a few seconds for it to shut down, and fire it up again. Check the bookmarks - the page you have just added won't be there.

I only tested this (/home) with both nodes being up, I haven't tried it with one node being down.

Has anybody got any ideas on what could be causing this or any workarounds? In the RPM DB case, the FS is mounted with the following parameters (from ps, after startup): /usr/sbin/glusterfs --log-level=NONE --log-file=/dev/null --disable-direct-io-mode --volfile=/etc/glusterfs/root.vol /mnt/newroot

Home is mounted with the following:
/usr/sbin/glusterfs --log-level=NORMAL --volfile=/etc/glusterfs/home.vol /home

If these are the same bug, then this implies that direct-io-mode has no effect on it.

Has anybody got a clue where the root causes of these two may be, and more importantly, when a fix might be available? The bind mount issue is particularly annoying because it means that startup of glusterfs root requires nasty init script bodges to work around timing/settling issues (as mentioned above), the likes of which really shouldn't find their way into a production environment.

If any further debug/reproduction info is needed, please do tell and I'll do my best to provide it.

Best regards.

Gordan




[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux