Hi,
I'm rather new to this project, having stumbled across it earlier this
afternoon, so forgive me if I'm still trying to find my way around. I
was in the need of an alternative to NFS that would let me spread the
task of sharing my downloaded source code files across a couple of
boxes, and GlusterFS looked like a great candidate, having had no luck
with Coda or OpenAFS.
I also want to share some files on the web server here with various
network clients (some x86, some MIPS) to make maintenance of web pages
easier, and to allow backups to my file server (a MIPS-based Cobalt Qube2).
All the boxes are running Gentoo Linux... Gentoo Hardened on the x86
systems (mostly with vanilla kernels, SELinux disabled, but with stack
protection enabled in the compiler), and plain Gentoo on MIPS.
For the record, the following setup is in use:
Web Server:
Intel PIII 550MHz running Gentoo Linux with Kernel 2.6.23.1 (vanilla
sources). fuse 2.7.0, glusterfs-1.3.11 (client+server).
Laptop:
Intel PII 300MHz running Gentoo Linux with Kernel 2.6.25 (vanilla
sources). fuse 2.7.0, glusterfs-1.3.11 (client+server)
Cobalt Qube2:
QED RM5231 250MHz (MIPS4 Little Endian) running Gentoo Linux with
kernel 2.6.27-rc4 (latest linux-mips.org git sources). glusterfs-1.3.11
(server only for now, but will be installing fuse and the client soon).
All boxes are synced to the local NTP server which runs at Stratum 2.
Other machines to be added into the mix:
o Two Lemote Fulong computers (Loongson 2E 660MHz CPU; MIPS3 Little Endian)
o SGI O2 (RM527x 300MHz; MIPS4 Big Endian) -- should be good for
endianness checks
o A few more x86 boxes, and one or two SGI boxes when I get around to
it. ;-)
At this time, I have the server, glusterfsd, running nicely on the
Qube2, performing the job of sharing out a repository stored there. The
web server (x86) currently connects to this without issues as does my
x86-based laptop. The problem arose this afternoon when I tried to set
up glusterfsd on the web server to share out Apache's vhost roots (in
/home/httpd) so I could mount them elsewhere.
On the server, I have the following configuration file:
> www ~ # cat /etc/glusterfs/glusterfs-server.vol
> volume brick
> type storage/posix
> option directory /home/httpd/
> end-volume
>
> volume server
> type protocol/server
> subvolumes brick
> option transport-type tcp/server # For TCP/IP transport
> option auth.ip.brick.allow *
> end-volume
Save a couple of slight differences, it's more or less a direct copy
from the 10min setup guide. For debugging purposes, I started the
server with the following command (as root) under strace:
# strace -o /tmp/glusterfsd-strace.txt glusterfsd -f \
> /etc/glusterfs/glusterfs-server.vol -L DEBUG -l \
> /tmp/glusterfsd-log.txt -N
On my laptop, I have a similar config file:
> stuartl at portege ~ $ cat /etc/glusterfs/glusterfs-client.vol
> volume client
> type protocol/client
> option transport-type tcp/client
> option remote-host 10.0.0.254
> option remote-subvolume brick
> end-volume
I attempt to mount the files using the following command:
# strace -o /tmp/glusterfs-strace.txt glusterfs -f \
> /etc/glusterfs/glusterfs-client.vol -L DEBUG -l \
> /tmp/glusterfs-log.txt -N
On the client side, the following is generated in the logfiles:
> portege ~ # cat /tmp/glusterfs-log.txt
> 2008-08-24 18:20:33 D [glusterfs.c:167:get_spec_fp] glusterfs: loading spec from /etc/glusterfs/glusterfs-client.vol
> 2008-08-24 18:20:33 D [spec.y:107:new_section] parser: New node for 'client'
> 2008-08-24 18:20:33 D [xlator.c:115:xlator_set_type] xlator: attempt to load file /usr/lib/glusterfs/1.3.11/xlator/protocol/client.so
> 2008-08-24 18:20:33 D [spec.y:127:section_type] parser: Type:client:protocol/client
> 2008-08-24 18:20:33 D [spec.y:141:section_option] parser: Option:client:transport-type:tcp/client
> 2008-08-24 18:20:33 D [spec.y:141:section_option] parser: Option:client:remote-host:10.0.0.254
> 2008-08-24 18:20:33 D [spec.y:141:section_option] parser: Option:client:remote-subvolume:brick
> 2008-08-24 18:20:33 D [spec.y:209:section_end] parser: end:client
> 2008-08-24 18:20:33 D [glusterfs.c:128:fuse_graph] glusterfs: setting option mount-point to /mnt/net/
> 2008-08-24 18:20:33 D [xlator.c:115:xlator_set_type] xlator: attempt to load file /usr/lib/glusterfs/1.3.11/xlator/mount/fuse.so
> 2008-08-24 18:20:34 D [client-protocol.c:5013:init] client: defaulting transport-timeout to 42
> 2008-08-24 18:20:34 D [transport.c:80:transport_load] transport: attempt to load file /usr/lib/glusterfs/1.3.11/transport/tcp/client.so
> 2008-08-24 18:20:34 D [client-protocol.c:5040:init] client: defaulting limits.transaction-size to 268435456
> 2008-08-24 18:20:34 D [client-protocol.c:5340:notify] client: got GF_EVENT_PARENT_UP, attempting connect on transport
> 2008-08-24 18:20:34 D [client-protocol.c:4757:client_protocol_reconnect] client: attempting reconnect
> 2008-08-24 18:20:34 D [tcp-client.c:77:tcp_connect] client: socket fd = 6
> 2008-08-24 18:20:34 D [tcp-client.c:107:tcp_connect] client: finalized on port `1023'
> 2008-08-24 18:20:34 D [tcp-client.c:128:tcp_connect] client: defaulting remote-port to 6996
> 2008-08-24 18:20:34 D [common-utils.c:179:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: 10.0.0.254
> 2008-08-24 18:20:34 D [common-utils.c:204:gf_resolve_ip] resolver: returning IP:10.0.0.254[0] for hostname: 10.0.0.254
> 2008-08-24 18:20:34 D [common-utils.c:212:gf_resolve_ip] resolver: flushing DNS cache
> 2008-08-24 18:20:34 D [tcp-client.c:161:tcp_connect] client: connect on 6 in progress (non-blocking)
> 2008-08-24 18:20:34 D [tcp-client.c:205:tcp_connect] client: connection on 6 success
> 2008-08-24 18:20:34 D [client-protocol.c:5362:notify] client: got GF_EVENT_CHILD_UP
> 2008-08-24 18:20:34 W [client-protocol.c:4784:client_protocol_cleanup] client: cleaning up state in transport object 0x8b32da8
> 2008-08-24 18:20:34 D [tcp.c:87:tcp_disconnect] client: connection disconnected
> 2008-08-24 18:20:35 D [client-protocol.c:4757:client_protocol_reconnect] client: attempting reconnect
> 2008-08-24 18:20:35 D [tcp-client.c:77:tcp_connect] client: socket fd = 6
> 2008-08-24 18:20:35 D [tcp-client.c:107:tcp_connect] client: finalized on port `1023'
> 2008-08-24 18:20:35 D [tcp-client.c:128:tcp_connect] client: defaulting remote-port to 6996
> 2008-08-24 18:20:35 D [common-utils.c:179:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: 10.0.0.254
> 2008-08-24 18:20:35 D [common-utils.c:204:gf_resolve_ip] resolver: returning IP:10.0.0.254[0] for hostname: 10.0.0.254
> 2008-08-24 18:20:35 D [common-utils.c:212:gf_resolve_ip] resolver: flushing DNS cache
> 2008-08-24 18:20:35 D [tcp-client.c:161:tcp_connect] client: connect on 6 in progress (non-blocking)
> 2008-08-24 18:20:35 D [tcp-client.c:198:tcp_connect] client: connection on 6 still in progress - try later
> 2008-08-24 18:20:37 D [client-protocol.c:4757:client_protocol_reconnect] client: attempting reconnect
> 2008-08-24 18:20:37 D [tcp-client.c:198:tcp_connect] client: connection on 6 still in progress - try later
> 2008-08-24 18:20:40 D [client-protocol.c:4757:client_protocol_reconnect] client: attempting reconnect
> 2008-08-24 18:20:40 D [tcp-client.c:205:tcp_connect] client: connection on 6 success
> 2008-08-24 18:20:40 D [client-protocol.c:5362:notify] client: got GF_EVENT_CHILD_UP
> 2008-08-24 18:20:40 W [client-protocol.c:4784:client_protocol_cleanup] client: cleaning up state in transport object 0x8b32da8
> 2008-08-24 18:20:40 D [tcp.c:87:tcp_disconnect] client: connection disconnected
> 2008-08-24 18:20:45 D [client-protocol.c:4757:client_protocol_reconnect] client: attempting reconnect
> 2008-08-24 18:20:45 D [tcp-client.c:77:tcp_connect] client: socket fd = 6
> 2008-08-24 18:20:45 D [tcp-client.c:107:tcp_connect] client: finalized on port `1023'
> 2008-08-24 18:20:45 D [tcp-client.c:128:tcp_connect] client: defaulting remote-port to 6996
> 2008-08-24 18:20:45 D [common-utils.c:179:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: 10.0.0.254
> 2008-08-24 18:20:45 D [common-utils.c:204:gf_resolve_ip] resolver: returning IP:10.0.0.254[0] for hostname: 10.0.0.254
> 2008-08-24 18:20:45 D [common-utils.c:212:gf_resolve_ip] resolver: flushing DNS cache
> 2008-08-24 18:20:45 D [tcp-client.c:161:tcp_connect] client: connect on 6 in progress (non-blocking)
> 2008-08-24 18:20:46 D [tcp-client.c:198:tcp_connect] client: connection on 6 still in progress - try later
> 2008-08-24 18:20:54 D [client-protocol.c:4757:client_protocol_reconnect] client: attempting reconnect
> 2008-08-24 18:20:54 D [tcp-client.c:205:tcp_connect] client: connection on 6 success
> 2008-08-24 18:20:54 D [client-protocol.c:5362:notify] client: got GF_EVENT_CHILD_UP
> 2008-08-24 18:20:54 W [client-protocol.c:4784:client_protocol_cleanup] client: cleaning up state in transport object 0x8b32da8
> 2008-08-24 18:20:54 D [tcp.c:87:tcp_disconnect] client: connection disconnected
> 2008-08-24 18:21:07 D [client-protocol.c:4757:client_protocol_reconnect] client: attempting reconnect
> 2008-08-24 18:21:07 D [tcp-client.c:77:tcp_connect] client: socket fd = 6
> 2008-08-24 18:21:07 D [tcp-client.c:107:tcp_connect] client: finalized on port `1023'
> 2008-08-24 18:21:07 D [tcp-client.c:128:tcp_connect] client: defaulting remote-port to 6996
> 2008-08-24 18:21:07 D [common-utils.c:179:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: 10.0.0.254
> 2008-08-24 18:21:07 D [common-utils.c:204:gf_resolve_ip] resolver: returning IP:10.0.0.254[0] for hostname: 10.0.0.254
> 2008-08-24 18:21:07 D [common-utils.c:212:gf_resolve_ip] resolver: flushing DNS cache
> 2008-08-24 18:21:07 D [tcp-client.c:161:tcp_connect] client: connect on 6 in progress (non-blocking)
> 2008-08-24 18:21:07 D [tcp-client.c:205:tcp_connect] client: connection on 6 success
> 2008-08-24 18:21:07 D [client-protocol.c:5362:notify] client: got GF_EVENT_CHILD_UP
> 2008-08-24 18:21:07 W [client-protocol.c:4784:client_protocol_cleanup] client: cleaning up state in transport object 0x8b32da8
> 2008-08-24 18:21:07 D [tcp.c:87:tcp_disconnect] client: connection disconnected
> 2008-08-24 18:21:28 D [client-protocol.c:4757:client_protocol_reconnect] client: attempting reconnect
> 2008-08-24 18:21:28 D [tcp-client.c:77:tcp_connect] client: socket fd = 6
> 2008-08-24 18:21:28 D [tcp-client.c:107:tcp_connect] client: finalized on port `1023'
> 2008-08-24 18:21:28 D [tcp-client.c:128:tcp_connect] client: defaulting remote-port to 6996
> 2008-08-24 18:21:28 D [common-utils.c:179:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: 10.0.0.254
> 2008-08-24 18:21:28 D [common-utils.c:204:gf_resolve_ip] resolver: returning IP:10.0.0.254[0] for hostname: 10.0.0.254
> 2008-08-24 18:21:28 D [common-utils.c:212:gf_resolve_ip] resolver: flushing DNS cache
> 2008-08-24 18:21:28 D [tcp-client.c:161:tcp_connect] client: connect on 6 in progress (non-blocking)
> 2008-08-24 18:21:28 D [tcp-client.c:205:tcp_connect] client: connection on 6 success
> 2008-08-24 18:21:28 D [client-protocol.c:5362:notify] client: got GF_EVENT_CHILD_UP
> 2008-08-24 18:21:28 W [client-protocol.c:4784:client_protocol_cleanup] client: cleaning up state in transport object 0x8b32da8
> 2008-08-24 18:21:28 D [tcp.c:87:tcp_disconnect] client: connection disconnected
> 2008-08-24 18:21:32 W [glusterfs.c:416:glusterfs_cleanup_and_exit] glusterfs: shutting down server
> portege ~ # cat /tmp/glusterfs-
> glusterfs-1.3.11/ glusterfs-CURRENT.tar.gz glusterfs-log.txt glusterfs-strace.txt
> portege ~ # cat /tmp/glusterfs-strace.txt
> execve("/usr/sbin/glusterfs", ["glusterfs", "-f", "/etc/glusterfs/glusterfs-client."..., "-L"..., "DEBUG"..., "-l"..., "/tmp/glusterfs-log.txt"..., "-N"..., "/mnt/net/"...], [/* 67 vars */]) = 0
> brk(0) = 0x8b2d000
> access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
> open("/etc/ld.so.cache", O_RDONLY) = 3
> fstat64(3, {st_mode=S_IFREG|0644, st_size=168554, ...}) = 0
> mmap2(NULL, 168554, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f2b000
> close(3) = 0
> open("/usr/lib/libglusterfs.so.0", O_RDONLY) = 3
> read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0pW\0\0004\0\0\0"..., 512) = 512
> fstat64(3, {st_mode=S_IFREG|0755, st_size=387428, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f2a000
> mmap2(NULL, 115488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7f0d000
> mmap2(0xb7f28000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b) = 0xb7f28000
> close(3) = 0
> open("/lib/libdl.so.2", O_RDONLY) = 3
> read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\f\0\0004\0\0\0"..., 512) = 512
> fstat64(3, {st_mode=S_IFREG|0755, st_size=10652, ...}) = 0
> mmap2(NULL, 12344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7f09000
> mmap2(0xb7f0b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0xb7f0b000
> close(3) = 0
> open("/lib/tls/libpthread.so.0", O_RDONLY) = 3
> read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340K\0\0004\0\0\0"..., 512) = 512
> fstat64(3, {st_mode=S_IFREG|0755, st_size=129332, ...}) = 0
> mmap2(NULL, 90336, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7ef2000
> mmap2(0xb7f05000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12) = 0xb7f05000
> mmap2(0xb7f07000, 4320, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7f07000
> close(3) = 0
> open("/lib/tls/libc.so.6", O_RDONLY) = 3
> read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220Y\1\0004\0\0\0"..., 512) = 512
> fstat64(3, {st_mode=S_IFREG|0755, st_size=1299536, ...}) = 0
> mmap2(NULL, 1226876, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7dc6000
> mmap2(0xb7eec000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x125) = 0xb7eec000
> mmap2(0xb7eef000, 10364, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7eef000
> close(3) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7dc5000
> set_thread_area({entry_number:-1 -> 6, base_addr:0xb7dc5ac0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
> open("/dev/urandom", O_RDONLY) = 3
> read(3, "\31\223Uu", 4) = 4
> close(3) = 0
> mprotect(0xb7eec000, 4096, PROT_READ) = 0
> mprotect(0xb7f05000, 4096, PROT_READ) = 0
> mprotect(0xb7f0b000, 4096, PROT_READ) = 0
> mprotect(0xb7f28000, 4096, PROT_READ) = 0
> mprotect(0x804c000, 4096, PROT_READ) = 0
> mprotect(0xb7f70000, 4096, PROT_READ) = 0
> munmap(0xb7f2b000, 168554) = 0
> set_tid_address(0xb7dc5b08) = 11503
> set_robust_list(0xb7dc5b10, 0xc) = 0
> rt_sigaction(SIGRTMIN, {0xb7ef66d0, [], SA_SIGINFO}, NULL, 8) = 0
> rt_sigaction(SIGRT_1, {0xb7ef6750, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
> rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
> getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
> uname({sys="Linux", node="portege", ...}) = 0
> brk(0) = 0x8b2d000
> brk(0x8b4e000) = 0x8b4e000
> setrlimit(RLIMIT_CORE, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
> open("/tmp/glusterfs-log.txt", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 3
> fstat64(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f54000
> fstat64(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> _llseek(3, 0, [0], SEEK_SET) = 0
> open("/etc/glusterfs/glusterfs-client.vol", O_RDONLY|O_LARGEFILE) = 4
> time(NULL) = 1219566033
> open("/etc/localtime", O_RDONLY) = 5
> fstat64(5, {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> fstat64(5, {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f53000
> read(5, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\0"..., 4096) = 413
> close(5) = 0
> munmap(0xb7f53000, 4096) = 0
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [glusterfs"..., 117) = 117
> rt_sigaction(SIGUSR1, {0xb7e2d6d0, [USR1], SA_RESTART}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGUSR2, {0xb7f1bfa0, [USR2], SA_RESTART}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGSEGV, {0xb7f1c8e0, [SEGV], SA_RESTART}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGABRT, {0xb7f1c8e0, [ABRT], SA_RESTART}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_IGN}, 8) = 0
> rt_sigaction(SIGHUP, {0xb7f18350, [HUP], SA_RESTART}, {SIG_IGN}, 8) = 0
> rt_sigaction(SIGTERM, {0x8049b30, [TERM], SA_RESTART}, {SIG_IGN}, 8) = 0
> _llseek(-1, 0, 0xbfd6f090, SEEK_SET) = -1 EBADF (Bad file descriptor)
> ftruncate64(-1, 0) = -1 EBADF (Bad file descriptor)
> write(-1, "11503\n", 6) = -1 EBADF (Bad file descriptor)
> close(-1) = -1 EBADF (Bad file descriptor)
> mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb75c4000
> mprotect(0xb75c4000, 4096, PROT_NONE) = 0
> clone(child_stack=0xb7dc44c4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xb7dc4bd8, {entry_number:6, base_addr:0xb7dc4b90, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0xb7dc4bd8) = 11515
> ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfd6e9d8) = -1 ENOTTY (Inappropriate ioctl for device)
> fstat64(4, {st_mode=S_IFREG|0644, st_size=143, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f53000
> read(4, "volume client\n\ttype protocol/cli"..., 8192) = 143
> read(4, "", 4096) = 0
> time(NULL) = 1219566033
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [spec.y:10"..., 77) = 77
> time(NULL) = 1219566033
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [xlator.c:"..., 134) = 134
> futex(0xb7f0c02c, FUTEX_WAKE, 2147483647) = 0
> open("/usr/lib/glusterfs/1.3.11/xlator/protocol/client.so", O_RDONLY) = 5
> read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 at +\0\0004\0\0\0"..., 512) = 512
> fstat64(5, {st_mode=S_IFREG|0755, st_size=141051, ...}) = 0
> mmap2(NULL, 74660, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0xb7f40000
> mmap2(0xb7f51000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x10) = 0xb7f51000
> close(5) = 0
> mprotect(0xb7f51000, 4096, PROT_READ) = 0
> time(NULL) = 1219566033
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [spec.y:12"..., 84) = 84
> time(NULL) = 1219566033
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [spec.y:14"..., 98) = 98
> time(NULL) = 1219566033
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [spec.y:14"..., 95) = 95
> time(NULL) = 1219566033
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [spec.y:14"..., 95) = 95
> time(NULL) = 1219566033
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [spec.y:20"..., 66) = 66
> read(4, "", 8192) = 0
> ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfd6e9d8) = -1 ENOTTY (Inappropriate ioctl for device)
> close(4) = 0
> munmap(0xb7f53000, 4096) = 0
> time(NULL) = 1219566033
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [glusterfs"..., 102) = 102
> time(NULL) = 1219566033
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:33 D [xlator.c:"..., 129) = 129
> open("/usr/lib/glusterfs/1.3.11/xlator/mount/fuse.so", O_RDONLY) = 4
> read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\30\0\0004\0\0\0"..., 512) = 512
> fstat64(4, {st_mode=S_IFREG|0755, st_size=101283, ...}) = 0
> mmap2(NULL, 50028, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0xb7f33000
> mmap2(0xb7f3e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0xa) = 0xb7f3e000
> close(4) = 0
> open("/etc/ld.so.cache", O_RDONLY) = 4
> fstat64(4, {st_mode=S_IFREG|0644, st_size=168554, ...}) = 0
> mmap2(NULL, 168554, PROT_READ, MAP_PRIVATE, 4, 0) = 0xb759a000
> close(4) = 0
> open("/usr/lib/libfuse.so.2", O_RDONLY) = 4
> read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 N\0\0004\0\0\0"..., 512) = 512
> fstat64(4, {st_mode=S_IFREG|0755, st_size=85344, ...}) = 0
> mmap2(NULL, 88248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0xb7584000
> mmap2(0xb7598000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x13) = 0xb7598000
> close(4) = 0
> open("/lib/tls/librt.so.1", O_RDONLY) = 4
> read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P\35\0\0004\0\0\0"..., 512) = 512
> fstat64(4, {st_mode=S_IFREG|0755, st_size=33492, ...}) = 0
> mmap2(NULL, 33100, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0xb757b000
> mmap2(0xb7582000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x6) = 0xb7582000
> close(4) = 0
> mprotect(0xb7582000, 4096, PROT_READ) = 0
> mprotect(0xb7598000, 4096, PROT_READ) = 0
> mprotect(0xb7f3e000, 4096, PROT_READ) = 0
> munmap(0xb759a000, 168554) = 0
> lstat64("/mnt/net/", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
> open("/dev/fuse", O_RDWR|O_LARGEFILE) = 4
> getgid32() = 0
> getuid32() = 0
> mount("glusterfs", "/mnt/net/", "fuse"..., MS_NOSUID|MS_NODEV, "allow_other,default_permissions,"...) = 0
> geteuid32() = 0
> lstat64("/mnt", {st_mode=S_IFDIR|0755, st_size=102, ...}) = 0
> clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7dc5b08) = 11565
> waitpid(11565, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 11565
> --- SIGCHLD (Child exited) @ 0 (0) ---
> getuid32() = 0
> rt_sigaction(SIGHUP, NULL, {0xb7f18350, [HUP], SA_RESTART}, 8) = 0
> rt_sigaction(SIGINT, NULL, {SIG_IGN}, 8) = 0
> rt_sigaction(SIGTERM, NULL, {0x8049b30, [TERM], SA_RESTART}, 8) = 0
> rt_sigaction(SIGPIPE, NULL, {SIG_IGN}, 8) = 0
> time(NULL) = 1219566034
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:34 D [client-pr"..., 95) = 95
> time(NULL) = 1219566034
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:34 D [transport"..., 136) = 136
> open("/usr/lib/glusterfs/1.3.11/transport/tcp/client.so", O_RDONLY) = 5
> read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\f\0\0004\0\0\0"..., 512) = 512
> fstat64(5, {st_mode=S_IFREG|0755, st_size=46004, ...}) = 0
> mmap2(NULL, 16580, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0xb75bf000
> mmap2(0xb75c2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x2) = 0xb75c2000
> close(5) = 0
> mprotect(0xb75c2000, 4096, PROT_READ) = 0
> time(NULL) = 1219566034
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:34 D [client-pr"..., 108) = 108
> setrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
> time(NULL) = 1219566034
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:34 D [client-pr"..., 118) = 118
> gettimeofday({1219566034, 561531}, NULL) = 0
> epoll_create(1024) = 5
> futex(0x8b342cc, FUTEX_WAIT, 1, NULL) = 0
> futex(0x8b342b0, FUTEX_WAKE, 1) = 0
> epoll_wait(5, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=145960360, u64=145960360}}}, 1, -1) = 1
> time(NULL) = 1219566034
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:34 W [client-pr"..., 127) = 127
> time(NULL) = 1219566034
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:34 D [tcp.c:87:"..., 80) = 80
> epoll_ctl(5, EPOLL_CTL_DEL, 6, {EPOLLOUT|EPOLLMSG|EPOLLERR|EPOLLET|0x37f0e820, {u32=3086127092, u64=626792034022363124}}) = 0
> close(6) = 0
> futex(0x8b342cc, FUTEX_WAIT, 3, NULL) = 0
> futex(0x8b342b0, FUTEX_WAKE, 1) = 0
> epoll_wait(5, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=145960360, u64=145960360}}}, 1, -1) = 1
> time(NULL) = 1219566040
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:40 W [client-pr"..., 127) = 127
> time(NULL) = 1219566040
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:40 D [tcp.c:87:"..., 80) = 80
> epoll_ctl(5, EPOLL_CTL_DEL, 6, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLMSG|0x19000, {u32=3218534156, u64=13250337929468243724}}) = 0
> close(6) = 0
> futex(0x8b342cc, FUTEX_WAIT, 5, NULL) = 0
> futex(0x8b342b0, FUTEX_WAKE, 1) = 0
> epoll_wait(5, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=145960360, u64=145960360}}}, 1, -1) = 1
> time(NULL) = 1219566054
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:54 W [client-pr"..., 127) = 127
> time(NULL) = 1219566054
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:20:54 D [tcp.c:87:"..., 80) = 80
> epoll_ctl(5, EPOLL_CTL_DEL, 6, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLMSG|0x19000, {u32=3218534156, u64=13250337929468243724}}) = 0
> close(6) = 0
> futex(0x8b342cc, FUTEX_WAIT, 7, NULL) = 0
> futex(0x8b342b0, FUTEX_WAIT, 2, NULL) = 0
> futex(0x8b342b0, FUTEX_WAKE, 1) = 0
> epoll_wait(5, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=145960360, u64=145960360}}}, 1, -1) = 1
> time(NULL) = 1219566067
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:21:07 W [client-pr"..., 127) = 127
> time(NULL) = 1219566067
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:21:07 D [tcp.c:87:"..., 80) = 80
> epoll_ctl(5, EPOLL_CTL_DEL, 6, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLMSG|0x19000, {u32=3218534156, u64=13250337929468243724}}) = 0
> close(6) = 0
> futex(0x8b342cc, FUTEX_WAIT, 9, NULL) = ? ERESTARTSYS (To be restarted)
> --- SIGINT (Interrupt) @ 0 (0) ---
> futex(0x8b342cc, FUTEX_WAIT, 9, NULL) = ? ERESTARTSYS (To be restarted)
> --- SIGINT (Interrupt) @ 0 (0) ---
> futex(0x8b342cc, FUTEX_WAIT, 9, NULL) = 0
> futex(0x8b342b0, FUTEX_WAKE, 1) = 0
> epoll_wait(5, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=145960360, u64=145960360}}}, 1, -1) = 1
> time(NULL) = 1219566088
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:21:28 W [client-pr"..., 127) = 127
> time(NULL) = 1219566088
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:21:28 D [tcp.c:87:"..., 80) = 80
> epoll_ctl(5, EPOLL_CTL_DEL, 6, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLMSG|0x19000, {u32=3218534156, u64=13250337929468243724}}) = 0
> close(6) = 0
> futex(0x8b342cc, FUTEX_WAIT, 11, NULL) = ? ERESTARTSYS (To be restarted)
> --- SIGTERM (Terminated) @ 0 (0) ---
> time(NULL) = 1219566092
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:21:32 W [glusterfs"..., 99) = 99
> exit_group(0) = ?
> portege ~ #
On the server, the following messages are logged (and yes, at the end I
do manually try to shut it down with SIGINT before resorting to SIGTERM).
> www ~ # cat /tmp/glusterfsd-log.txt
> 2008-08-24 18:10:56 D [glusterfs.c:167:get_spec_fp] glusterfs: loading spec from /etc/glusterfs/glusterfs-server.vol
> 2008-08-24 18:10:56 D [spec.y:107:new_section] parser: New node for 'brick'
> 2008-08-24 18:10:56 D [xlator.c:115:xlator_set_type] xlator: attempt to load file /usr/lib/glusterfs/1.3.11/xlator/storage/posix.so
> 2008-08-24 18:10:56 D [spec.y:127:section_type] parser: Type:brick:storage/posix
> 2008-08-24 18:10:56 D [spec.y:141:section_option] parser: Option:brick:directory:/home/httpd/
> 2008-08-24 18:10:56 D [spec.y:209:section_end] parser: end:brick
> 2008-08-24 18:10:56 D [spec.y:107:new_section] parser: New node for 'server'
> 2008-08-24 18:10:56 D [xlator.c:115:xlator_set_type] xlator: attempt to load file /usr/lib/glusterfs/1.3.11/xlator/protocol/server.so
> 2008-08-24 18:10:56 D [spec.y:127:section_type] parser: Type:server:protocol/server
> 2008-08-24 18:10:56 D [spec.y:196:section_sub] parser: child:server->brick
> 2008-08-24 18:10:56 D [spec.y:141:section_option] parser: Option:server:transport-type:tcp/server
> 2008-08-24 18:10:56 D [spec.y:141:section_option] parser: Option:server:auth.ip.brick.allow:*
> 2008-08-24 18:10:56 D [spec.y:209:section_end] parser: end:server
> 2008-08-24 18:10:56 D [server-protocol.c:6309:init] server: protocol/server xlator loaded
> 2008-08-24 18:10:56 D [transport.c:80:transport_load] transport: attempt to load file /usr/lib/glusterfs/1.3.11/transport/tcp/server.so
> 2008-08-24 18:10:56 D [server-protocol.c:6350:init] server: defaulting limits.transaction-size to 4194304
> 2008-08-24 18:11:45 D [tcp-server.c:145:tcp_server_notify] server: Registering socket (6) for new transport object of 10.0.0.87
> 2008-08-24 18:11:45 E [protocol.c:342:gf_block_unserialize_transport] server: frame size (168) > max (-2147458000)
> 2008-08-24 18:11:45 D [tcp.c:87:tcp_disconnect] server: connection disconnected
> 2008-08-24 18:11:45 D [server-protocol.c:6269:server_protocol_cleanup] server: cleaned up transport state for client 10.0.0.87:1022
> 2008-08-24 18:11:45 D [tcp-server.c:257:gf_transport_fini] server: destroying transport object for 10.0.0.87:1022 (fd=6)
> 2008-08-24 18:11:49 D [tcp-server.c:145:tcp_server_notify] server: Registering socket (6) for new transport object of 10.0.0.87
> 2008-08-24 18:11:51 E [protocol.c:342:gf_block_unserialize_transport] server: frame size (168) > max (-2147458000)
> 2008-08-24 18:11:51 D [tcp.c:87:tcp_disconnect] server: connection disconnected
> 2008-08-24 18:11:51 D [server-protocol.c:6269:server_protocol_cleanup] server: cleaned up transport state for client 10.0.0.87:1022
> 2008-08-24 18:11:51 D [tcp-server.c:257:gf_transport_fini] server: destroying transport object for 10.0.0.87:1022 (fd=6)
> 2008-08-24 18:11:59 D [tcp-server.c:145:tcp_server_notify] server: Registering socket (6) for new transport object of 10.0.0.87
> 2008-08-24 18:12:05 E [protocol.c:342:gf_block_unserialize_transport] server: frame size (168) > max (-2147458000)
> 2008-08-24 18:12:05 D [tcp.c:87:tcp_disconnect] server: connection disconnected
> 2008-08-24 18:12:05 D [server-protocol.c:6269:server_protocol_cleanup] server: cleaned up transport state for client 10.0.0.87:1022
> 2008-08-24 18:12:05 D [tcp-server.c:257:gf_transport_fini] server: destroying transport object for 10.0.0.87:1022 (fd=6)
> 2008-08-24 18:13:13 W [glusterfs.c:416:glusterfs_cleanup_and_exit] glusterfs: shutting down server
> www ~ # cat /tmp/glusterfsd-strace.txt
> execve("/usr/sbin/glusterfsd", ["glusterfsd", "-f", "/etc/glusterfs/glusterfs-server."..., "-L"..., "DEBUG"..., "-l"..., "/tmp/glusterfsd-log.txt"..., "-N"...], [/* 45 vars */]) = 0
> brk(0) = 0x80006000
> access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
> open("/etc/ld.so.cache", O_RDONLY) = 3
> fstat64(3, {st_mode=S_IFREG|0644, st_size=62235, ...}) = 0
> mmap2(NULL, 62235, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f08000
> close(3) = 0
> open("/usr/lib/libglusterfs.so.0", O_RDONLY) = 3
> read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320W\0\0004\0\0\0"..., 512) = 512
> fstat64(3, {st_mode=S_IFREG|0755, st_size=416299, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f07000
> mmap2(NULL, 143628, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7ee3000
> mmap2(0xb7f05000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21) = 0xb7f05000
> close(3) = 0
> open("/lib/libdl.so.2", O_RDONLY) = 3
> read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\f\0\0004\0\0\0"..., 512) = 512
> fstat64(3, {st_mode=S_IFREG|0755, st_size=10652, ...}) = 0
> mmap2(NULL, 12344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7edf000
> mmap2(0xb7ee1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0xb7ee1000
> close(3) = 0
> open("/lib/libpthread.so.0", O_RDONLY) = 3
> read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340K\0\0004\0\0\0"..., 512) = 512
> fstat64(3, {st_mode=S_IFREG|0755, st_size=129424, ...}) = 0
> mmap2(NULL, 90336, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7ec8000
> mmap2(0xb7edb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12) = 0xb7edb000
> mmap2(0xb7edd000, 4320, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7edd000
> close(3) = 0
> open("/lib/libc.so.6", O_RDONLY) = 3
> read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220Y\1\0004\0\0\0"..., 512) = 512
> fstat64(3, {st_mode=S_IFREG|0755, st_size=1299536, ...}) = 0
> mmap2(NULL, 1226876, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7d9c000
> mmap2(0xb7ec2000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x125) = 0xb7ec2000
> mmap2(0xb7ec5000, 10364, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7ec5000
> close(3) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d9b000
> set_thread_area({entry_number:-1 -> 6, base_addr:0xb7d9bac0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
> open("/dev/urandom", O_RDONLY) = 3
> read(3, ",\367\25\214", 4) = 4
> close(3) = 0
> mprotect(0xb7ec2000, 4096, PROT_READ) = 0
> mprotect(0xb7edb000, 4096, PROT_READ) = 0
> mprotect(0xb7ee1000, 4096, PROT_READ) = 0
> mprotect(0xb7f05000, 4096, PROT_READ) = 0
> mprotect(0x80004000, 4096, PROT_READ) = 0
> mprotect(0xb7f33000, 4096, PROT_READ) = 0
> munmap(0xb7f08000, 62235) = 0
> set_tid_address(0xb7d9bb08) = 9299
> set_robust_list(0xb7d9bb10, 0xc) = 0
> rt_sigaction(SIGRTMIN, {0xb7ecc6d0, [], SA_SIGINFO}, NULL, 8) = 0
> rt_sigaction(SIGRT_1, {0xb7ecc750, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
> rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
> getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
> uname({sys="Linux", node="www", ...}) = 0
> brk(0) = 0x80006000
> brk(0x80027000) = 0x80027000
> setrlimit(RLIMIT_CORE, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
> open("/tmp/glusterfsd-log.txt", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 3
> fstat64(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f17000
> fstat64(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> _llseek(3, 0, [0], SEEK_SET) = 0
> open("/etc/glusterfs/glusterfs-server.vol", O_RDONLY|O_LARGEFILE) = 4
> time(NULL) = 1219565456
> open("/etc/localtime", O_RDONLY) = 5
> fstat64(5, {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> fstat64(5, {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f16000
> read(5, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\0"..., 4096) = 413
> close(5) = 0
> munmap(0xb7f16000, 4096) = 0
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [glusterfs"..., 117) = 117
> rt_sigaction(SIGUSR1, {0xb7e03560, [USR1], SA_RESTART}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGUSR2, {0xb7ef4d20, [USR2], SA_RESTART}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGSEGV, {0xb7ef5680, [SEGV], SA_RESTART}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGABRT, {0xb7ef5680, [ABRT], SA_RESTART}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_IGN}, 8) = 0
> rt_sigaction(SIGHUP, {0xb7eef6d0, [HUP], SA_RESTART}, {SIG_IGN}, 8) = 0
> rt_sigaction(SIGTERM, {0x80001e60, [TERM], SA_RESTART}, {SIG_IGN}, 8) = 0
> _llseek(-1, 0, 0xbfd2a560, SEEK_SET) = -1 EBADF (Bad file descriptor)
> ftruncate64(-1, 0) = -1 EBADF (Bad file descriptor)
> write(-1, "9299\n", 5) = -1 EBADF (Bad file descriptor)
> close(-1) = -1 EBADF (Bad file descriptor)
> mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb759a000
> mprotect(0xb759a000, 4096, PROT_NONE) = 0
> clone(child_stack=0xb7d9a4c4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xb7d9abd8, {entry_number:6, base_addr:0xb7d9ab90, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0xb7d9abd8) = 9300
> ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfd29de8) = -1 ENOTTY (Inappropriate ioctl for device)
> fstat64(4, {st_mode=S_IFREG|0644, st_size=228, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f16000
> read(4, "volume brick\n\ttype storage/posix"..., 8192) = 228
> read(4, "", 4096) = 0
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:10"..., 76) = 76
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [xlator.c:"..., 132) = 132
> futex(0xb7ee202c, FUTEX_WAKE, 2147483647) = 0
> open("/usr/lib/glusterfs/1.3.11/xlator/storage/posix.so", O_RDONLY) = 5
> read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\37\0\0004\0\0\0"..., 512) = 512
> fstat64(5, {st_mode=S_IFREG|0755, st_size=90384, ...}) = 0
> mmap2(NULL, 49552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0xb7f09000
> mmap2(0xb7f14000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0xa) = 0xb7f14000
> close(5) = 0
> mprotect(0xb7f14000, 4096, PROT_READ) = 0
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:12"..., 81) = 81
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:14"..., 94) = 94
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:20"..., 65) = 65
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:10"..., 77) = 77
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [xlator.c:"..., 134) = 134
> open("/usr/lib/glusterfs/1.3.11/xlator/protocol/server.so", O_RDONLY) = 5
> read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P=\0\0004\0\0\0"..., 512) = 512
> fstat64(5, {st_mode=S_IFREG|0755, st_size=165201, ...}) = 0
> mmap2(NULL, 86664, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0xb7584000
> mmap2(0xb7598000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x14) = 0xb7598000
> close(5) = 0
> mprotect(0xb7598000, 4096, PROT_READ) = 0
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:12"..., 84) = 84
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:19"..., 75) = 75
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:14"..., 98) = 98
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:14"..., 94) = 94
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [spec.y:20"..., 66) = 66
> read(4, "", 8192) = 0
> ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfd29e08) = -1 ENOTTY (Inappropriate ioctl for device)
> close(4) = 0
> munmap(0xb7f16000, 4096) = 0
> umask(0) = 022
> mkdir("/home/httpd/", 0777) = -1 EEXIST (File exists)
> stat64("/home/httpd/", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
> gettimeofday({1219565456, 60809}, NULL) = 0
> gettimeofday({1219565456, 60930}, NULL) = 0
> setrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [server-pr"..., 90) = 90
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [transport"..., 136) = 136
> open("/usr/lib/glusterfs/1.3.11/transport/tcp/server.so", O_RDONLY) = 4
> read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\16\0\0004\0\0\0"..., 512) = 512
> fstat64(4, {st_mode=S_IFREG|0755, st_size=46330, ...}) = 0
> mmap2(NULL, 16420, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0xb757f000
> mmap2(0xb7582000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x2) = 0xb7582000
> close(4) = 0
> mprotect(0xb7582000, 4096, PROT_READ) = 0
> socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 4
> setsockopt(4, SOL_SOCKET, SO_SNDTIMEO, "*\0\0\0\0\0\0\0", 8) = 0
> setsockopt(4, SOL_SOCKET, SO_RCVTIMEO, "*\0\0\0\0\0\0\0", 8) = 0
> setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> bind(4, {sa_family=AF_INET, sin_port=htons(6996), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> listen(4, 10) = 0
> epoll_create(1024) = 5
> epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=2147509296, u64=2147509296}}) = 0
> open("/usr/lib/glusterfs/1.3.11/auth/ip.so", O_RDONLY) = 6
> read(6, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\5\0\0004\0\0\0"..., 512) = 512
> fstat64(6, {st_mode=S_IFREG|0755, st_size=10122, ...}) = 0
> mmap2(NULL, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 6, 0) = 0xb757d000
> mmap2(0xb757e000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 6, 0) = 0xb757e000
> close(6) = 0
> mprotect(0xb757e000, 4096, PROT_READ) = 0
> time(NULL) = 1219565456
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:10:56 D [server-pr"..., 106) = 106
> setrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
> mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6d7c000
> mprotect(0xb6d7c000, 4096, PROT_NONE) = 0
> clone(child_stack=0xb757c4c4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xb757cbd8, {entry_number:6, base_addr:0xb757cb90, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0xb757cbd8) = 9301
> epoll_wait(5, {{EPOLLIN, {u32=2147509296, u64=2147509296}}}, 1, -1) = 1
> accept(4, {sa_family=AF_INET, sin_port=htons(1022), sin_addr=inet_addr("10.0.0.87")}, [16]) = 6
> futex(0xb7ec6cf0, FUTEX_WAKE, 2147483647) = 0
> getpeername(6, {sa_family=AF_INET, sin_port=htons(1022), sin_addr=inet_addr("10.0.0.87")}, [16]) = 0
> time(NULL) = 1219565505
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:45 D [tcp-serve"..., 128) = 128
> epoll_ctl(5, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=2147531600, u64=2147531600}}) = 0
> epoll_wait(5, {{EPOLLIN, {u32=2147531600, u64=2147531600}}}, 2, -1) = 1
> read(6, "Block Start\n0000000000067932\n000"..., 113) = 113
> time(NULL) = 1219565505
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:45 E [protocol."..., 115) = 115
> time(NULL) = 1219565505
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:45 D [tcp.c:87:"..., 80) = 80
> epoll_ctl(5, EPOLL_CTL_DEL, 6, {EPOLLRDNORM|EPOLLWRNORM|EPOLLHUP|EPOLLET|0xc020, {u32=3218252780, u64=13822290701277373420}}) = 0
> close(6) = 0
> time(NULL) = 1219565505
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:45 D [server-pr"..., 132) = 132
> time(NULL) = 1219565505
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:45 D [tcp-serve"..., 121) = 121
> epoll_wait(5, {{EPOLLIN, {u32=2147509296, u64=2147509296}}}, 2, -1) = 1
> accept(4, {sa_family=AF_INET, sin_port=htons(1022), sin_addr=inet_addr("10.0.0.87")}, [16]) = 6
> getpeername(6, {sa_family=AF_INET, sin_port=htons(1022), sin_addr=inet_addr("10.0.0.87")}, [16]) = 0
> time(NULL) = 1219565509
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:49 D [tcp-serve"..., 128) = 128
> epoll_ctl(5, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=2147531584, u64=2147531584}}) = 0
> epoll_wait(5, {{EPOLLIN, {u32=2147531584, u64=2147531584}}}, 2, -1) = 1
> read(6, "Block Start\n0000000000067932\n000"..., 113) = 113
> time(NULL) = 1219565511
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:51 E [protocol."..., 115) = 115
> time(NULL) = 1219565511
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:51 D [tcp.c:87:"..., 80) = 80
> epoll_ctl(5, EPOLL_CTL_DEL, 6, {EPOLLWRNORM|EPOLLERR|EPOLLHUP|EPOLLET|0xc020, {u32=3218252780, u64=13822290701277373420}}) = 0
> close(6) = 0
> time(NULL) = 1219565511
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:51 D [server-pr"..., 132) = 132
> time(NULL) = 1219565511
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:51 D [tcp-serve"..., 121) = 121
> epoll_wait(5, {{EPOLLIN, {u32=2147509296, u64=2147509296}}}, 2, -1) = 1
> accept(4, {sa_family=AF_INET, sin_port=htons(1022), sin_addr=inet_addr("10.0.0.87")}, [16]) = 6
> getpeername(6, {sa_family=AF_INET, sin_port=htons(1022), sin_addr=inet_addr("10.0.0.87")}, [16]) = 0
> time(NULL) = 1219565519
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:11:59 D [tcp-serve"..., 128) = 128
> epoll_ctl(5, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=2147532080, u64=2147532080}}) = 0
> epoll_wait(5, {{EPOLLIN, {u32=2147532080, u64=2147532080}}}, 2, -1) = 1
> read(6, "Block Start\n0000000000067932\n000"..., 113) = 113
> time(NULL) = 1219565525
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:12:05 E [protocol."..., 115) = 115
> time(NULL) = 1219565525
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:12:05 D [tcp.c:87:"..., 80) = 80
> epoll_ctl(5, EPOLL_CTL_DEL, 6, {EPOLLRDNORM|EPOLLWRNORM|EPOLLWRBAND|EPOLLERR|EPOLLET|0xb820, {u32=3218252780, u64=13822290701277373420}}) = 0
> close(6) = 0
> time(NULL) = 1219565525
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:12:05 D [server-pr"..., 132) = 132
> time(NULL) = 1219565525
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:12:05 D [tcp-serve"..., 121) = 121
> epoll_wait(5, 8000bcc8, 2, -1) = -1 EINTR (Interrupted system call)
> --- SIGINT (Interrupt) @ 0 (0) ---
> epoll_wait(5, 8000bcc8, 2, -1) = -1 EINTR (Interrupted system call)
> --- SIGINT (Interrupt) @ 0 (0) ---
> epoll_wait(5, 8000bcc8, 2, -1) = -1 EINTR (Interrupted system call)
> --- SIGINT (Interrupt) @ 0 (0) ---
> epoll_wait(5, 8000bcc8, 2, -1) = -1 EINTR (Interrupted system call)
> --- SIGINT (Interrupt) @ 0 (0) ---
> epoll_wait(5, 8000bcc8, 2, -1) = -1 EINTR (Interrupted system call)
> --- SIGINT (Interrupt) @ 0 (0) ---
> epoll_wait(5, 8000bcc8, 2, -1) = -1 EINTR (Interrupted system call)
> --- SIGINT (Interrupt) @ 0 (0) ---
> epoll_wait(5, 8000bcc8, 2, -1) = -1 EINTR (Interrupted system call)
> --- SIGQUIT (Quit) @ 0 (0) ---
> epoll_wait(5, 8000bcc8, 2, -1) = -1 EINTR (Interrupted system call)
> --- SIGINT (Interrupt) @ 0 (0) ---
> epoll_wait(5, 8000bcc8, 2, -1) = -1 EINTR (Interrupted system call)
> --- SIGTERM (Terminated) @ 0 (0) ---
> time(NULL) = 1219565593
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=413, ...}) = 0
> write(3, "2008-08-24 18:13:13 W [glusterfs"..., 99) = 99
> exit_group(0) = ?
The key error message seems to be this one (note the high negative
maximum number):
> 2008-08-24 18:11:45 E [protocol.c:342:gf_block_unserialize_transport] server: frame size (168) > max (-2147458000)
Speaking in the IRC channel, Anand Babu (aka rooty) had reason to
suspect the following segment of code:
> [2008-08-24 18:28] <rooty> if (max_block_size && (blk->size > max_block_size)) {
> [2008-08-24 18:28] <rooty> gf_log (trans->xl->name, GF_LOG_ERROR,
> [2008-08-24 18:28] <rooty> "frame size (%"PRId32") > max (%"PRId32")",
> [2008-08-24 18:28] <rooty> blk->size, max_block_size);
> [2008-08-24 18:28] <rooty> /* block size exceeds the maximum block size permitted by the protocol controlling
> [2008-08-24 18:28] <rooty> * this transport */
> [2008-08-24 18:28] <rooty> goto herr;
> [2008-08-24 18:28] <rooty> }
I've recompiled it with debugging symbols, tried running it in gdb, but
so far (quite likely because I'm not very experienced with gdb) haven't
had any luck tracking the problem down.
The problem arises regardless of whether the client exists on a remote
computer, or on the server itself connecting via the loopback device.
The web server is successfully able to mount GlusterFS shares on other
computers, but isn't able to share its own.
It's an unusual bug, and seems specific to just this machine, so
exactly what is going on is anyone's guess at this point. Clearly
there's some edge case that I'm triggering here. If there's any further
information I can provide to aid tracking down this bug, let me know.
Thanks in advance.
Regards,
--
Stuart Longland (aka Redhatter) .'''.
Gentoo Linux/MIPS Cobalt and Docs Developer '.'` :
. . . . . . . . . . . . . . . . . . . . . . .'.'
http://dev.gentoo.org/~redhatter :.'
I haven't lost my mind...
...it's backed up on a tape somewhere.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
Url : http://zresearch.com/pipermail/gluster-users/attachments/20080824/0ffdcbd5/attachment-0001.pgp
