here are our logs when nfs is crashing.... [2011-06-10 08:54:14.900049] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: f8851fc2, ACCESS: NFS: 0(Call completed successfully.), POSIX: 0(Success) [2011-06-10 08:54:14.902002] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: f9851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:14.902037] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:14.902062] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: f9851fc2, GETATTR: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 74c48fb3-d065-462b-83a9-e4558b042465 [2011-06-10 08:54:14.920579] D [afr-transaction.c:976:afr_post_nonblocking_inodelk_cbk] 0-ksc-replicate-0: Non blocking inodelks done. Proceeding to FOP [2011-06-10 08:54:14.921099] D [client-lk.c:442:delete_granted_locks_fd] 0-ksc-client-0: Number of locks cleared=0 [2011-06-10 08:54:14.921155] D [client-lk.c:442:delete_granted_locks_fd] 0-ksc-client-1: Number of locks cleared=0 [2011-06-10 08:54:14.932629] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: f9851fc2, GETATTR: NFS: 0(Call completed successfully.), POSIX: 0(Success) [2011-06-10 08:54:14.932863] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fa851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:14.932890] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:14.932907] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fa851fc2, ACCESS: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 74c48fb3-d065-462b-83a9-e4558b042465 [2011-06-10 08:54:14.961700] D [socket.c:193:__socket_rwv] 0-ksc-client-0: EOF from peer 10.0.1.198:24031 [2011-06-10 08:54:14.961741] W [socket.c:1494:__socket_proto_state_machine] 0-ksc-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.1.198:24031) [2011-06-10 08:54:14.961757] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now [2011-06-10 08:54:14.961858] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-0: forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-10 08:54:14.920644 [2011-06-10 08:54:14.961880] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-0: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.961880] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-0: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.961940] I [client.c:1883:client_rpc_notify] 0-ksc-client-0: disconnected [2011-06-10 08:54:14.988468] W [socket.c:204:__socket_rwv] 0-ksc-client-1: readv failed (Connection reset by peer) [2011-06-10 08:54:14.988490] W [socket.c:1494:__socket_proto_state_machine] 0-ksc-client-1: reading from socket failed. Error (Connection reset by peer), peer (10.0.1.199:24027) [2011-06-10 08:54:14.988501] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now [2011-06-10 08:54:14.988501] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now [2011-06-10 08:54:14.988551] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-1: forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-10 08:54:14.920655 [2011-06-10 08:54:14.988568] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.988568] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.988599] D [client.c:77:client_submit_request] 0-ksc-client-0: connection in disconnected state [2011-06-10 08:54:14.988630] W [client3_1-fops.c:4379:client3_1_xattrop] 0-ksc-client-0: failed to send the fop: Transport endpoint is not connected [2011-06-10 08:54:14.988657] D [name.c:157:client_fill_address_family] 0-ksc-client-1: address-family not specified, guessing it to be inet/inet6 [2011-06-10 08:54:14.991719] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.0.1.199 (port-24007) for hostname: 10.0.1.199 and port: 24007 [2011-06-10 08:54:14.991786] I [socket.c:2272:socket_submit_request] 0-ksc-client-1: not connected (priv->connected = 0) [2011-06-10 08:54:14.991803] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-ksc-client-1: failed to submit rpc-request (XID: 0x70357x Program: GlusterFS 3.1, ProgVers: 310, Proc: 33) to rpc-transport (ksc-client-1) [2011-06-10 08:54:14.991819] D [afr-lk-common.c:409:transaction_lk_op] 0-ksc-replicate-0: lk op is for a transaction [2011-06-10 08:54:14.991833] D [client.c:77:client_submit_request] 0-ksc-client-0: connection in disconnected state [2011-06-10 08:54:14.991845] W [client3_1-fops.c:4735:client3_1_inodelk] 0-ksc-client-0: failed to send the fop: Transport endpoint is not connected [2011-06-10 08:54:14.991862] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-ksc-client-1: failed to submit rpc-request (XID: 0x70358x Program: GlusterFS 3.1, ProgVers: 310, Proc: 29) to rpc-transport (ksc-client-1) [2011-06-10 08:54:14.991876] I [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.991904] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: c71a69ff, CREATE: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 4, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid f3e21f99-0aed-4042-8b81-16a804497b5b [2011-06-10 08:54:14.991945] D [client.c:129:client_submit_request] 0-ksc-client-1: rpc_clnt_submit failed [2011-06-10 08:54:14.991967] D [client.c:129:client_submit_request] 0-ksc-client-1: rpc_clnt_submit failed [2011-06-10 08:54:14.992014] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-1: forced unwinding frame type(GlusterFS 3.1) op(RELEASE(41)) called at 2011-06-10 08:54:14.921180 [2011-06-10 08:54:14.992046] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-1: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2011-06-10 08:54:14.932952 [2011-06-10 08:54:14.992060] I [client3_1-fops.c:411:client3_1_stat_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.992073] D [client.c:77:client_submit_request] 0-ksc-client-0: connection in disconnected state [2011-06-10 08:54:14.992091] W [client3_1-fops.c:2658:client3_1_stat] 0-ksc-client-0: failed to send the fop Transport endpoint is not connected [2011-06-10 08:54:14.992101] D [afr-inode-read.c:204:afr_stat_cbk] 0-ksc-replicate-0: /pimp/htdocs/mountpoints.sh: all subvolumes tried, going out [2011-06-10 08:54:14.992115] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fa851fc2, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:14.992181] I [client.c:1883:client_rpc_notify] 0-ksc-client-1: disconnected [2011-06-10 08:54:14.992197] E [afr-common.c:2546:afr_notify] 0-ksc-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-06-10 08:54:15.5285] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fb851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.5312] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.5312] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.5344] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fb851fc2, GETATTR: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 74c48fb3-d065-462b-83a9-e4558b042465 [2011-06-10 08:54:15.5381] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /pimp/htdocs/mountpoints.sh: no child is up [2011-06-10 08:54:15.5399] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fb851fc2, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.5783] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fc851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.5810] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.5833] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: fc851fc2, LOOKUP: args: FH: hashcount 2, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid ff18fc56-c352-4caf-b98d-4eff71494acc, name: mountpoints.sh [2011-06-10 08:54:15.5897] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.5954] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: fc851fc2, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.11858] E [socket.c:1685:socket_connect_finish] 0-ksc-client-1: connection to 10.0.1.199:24027 failed (Connection refused) [2011-06-10 08:54:15.11886] D [socket.c:289:__socket_disconnect] 0-ksc-client-1: shutdown() returned -1. Transport endpoint is not connected [2011-06-10 08:54:15.11921] D [socket.c:193:__socket_rwv] 0-ksc-client-1: EOF from peer 10.0.1.199:24027 [2011-06-10 08:54:15.11938] D [socket.c:1494:__socket_proto_state_machine] 0-ksc-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.1.199:24027) [2011-06-10 08:54:15.11954] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now [2011-06-10 08:54:15.158995] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e1f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.159038] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.159038] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.159065] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: e1f47a24, GETATTR: args: FH: hashcount 6, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 9b62d8e0-cf19-487e-b56a-9ac980294801 [2011-06-10 08:54:15.159105] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /vertrieb/htdocs/vertrieb_imrich/fetischtopliste.de/www/button.php: no child is up [2011-06-10 08:54:15.159126] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: e1f47a24, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.159501] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e2f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.159528] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.159550] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: e2f47a24, LOOKUP: args: FH: hashcount 5, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid dd60aa24-78fa-4509-abee-acf32f12542f, name: button.php [2011-06-10 08:54:15.159604] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.159654] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: e2f47a24, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.160014] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e3f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.160041] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.160080] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: e3f47a24, GETATTR: args: FH: hashcount 5, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid dd60aa24-78fa-4509-abee-acf32f12542f [2011-06-10 08:54:15.160113] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /vertrieb/htdocs/vertrieb_imrich/fetischtopliste.de/www: no child is up [2011-06-10 08:54:15.160130] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: e3f47a24, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.160532] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e4f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.160554] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.160554] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.160569] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: e4f47a24, LOOKUP: args: FH: hashcount 4, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 434fc86c-5bed-4a38-b919-8822d38bd1a3, name: www [2011-06-10 08:54:15.160611] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.160657] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: e4f47a24, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.192962] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: c81a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.193000] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.193019] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: c81a69ff, GETATTR: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 61273392-07cb-4427-bb48-2cf869031802 [2011-06-10 08:54:15.193050] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /tomorrowwinners.com/typo3temp/locks: no child is up [2011-06-10 08:54:15.193069] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: c81a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.193526] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: c91a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.193547] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.193564] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: c91a69ff, LOOKUP: args: FH: hashcount 2, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 96012827-9101-40a1-b540-b6c47fe49e59, name: locks [2011-06-10 08:54:15.193607] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.193654] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: c91a69ff, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.193987] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: ca1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.194015] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.194037] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: ca1a69ff, GETATTR: args: FH: hashcount 2, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 96012827-9101-40a1-b540-b6c47fe49e59 [2011-06-10 08:54:15.194070] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /tomorrowwinners.com/typo3temp: no child is up [2011-06-10 08:54:15.194086] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: ca1a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.194566] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cb1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.194587] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.194614] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: cb1a69ff, LOOKUP: args: FH: hashcount 1, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 7fecc27e-a8de-4fe5-adce-c5e00d35c7ac, name: typo3temp [2011-06-10 08:54:15.194659] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.194703] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: cb1a69ff, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.396934] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cc1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.396989] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.396989] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.397017] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: cc1a69ff, GETATTR: args: FH: hashcount 1, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 7fecc27e-a8de-4fe5-adce-c5e00d35c7ac [2011-06-10 08:54:15.397064] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /tomorrowwinners.com: no child is up [2011-06-10 08:54:15.397087] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: cc1a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.397441] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cd1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.397468] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.397492] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: cd1a69ff, LOOKUP: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001, name: tomorrowwinners.com [2011-06-10 08:54:15.397551] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.397600] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: cd1a69ff, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.397953] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: ce1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.397981] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.398004] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: ce1a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.398038] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.398055] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: ce1a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.600726] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cf1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.600769] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.600769] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.600789] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: cf1a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.600857] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: cf1a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.601525] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d01a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.601553] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.601576] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d01a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.601627] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.601646] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d01a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.725647] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fd851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.725689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.725689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.725719] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fd851fc2, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.725749] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.725769] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fd851fc2, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.726209] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fe851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.726236] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.726252] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fe851fc2, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.726276] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.726292] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fe851fc2, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.804546] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d11a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.804580] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.804580] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.804606] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d11a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.804658] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d11a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.805060] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d21a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.805088] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.805111] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d21a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.805143] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.805159] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d21a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.8639] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d31a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:16.8698] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.20711] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d31a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.20821] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d31a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.21517] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d41a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:16.21556] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.21575] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d41a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.21606] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:16.21625] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d41a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.225038] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d51a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:16.225096] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.225096] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.225118] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d51a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.225193] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d51a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.225595] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d61a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:16.225622] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.225639] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d61a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.225669] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:16.225688] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d61a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.243210] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d71a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:16.243232] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.243232] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.243249] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d71a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.243274] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:16.243291] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d71a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.243670] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d81a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:16.243697] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.243717] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d81a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.243756] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d81a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.428628] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d91a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:16.428689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.428689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.428711] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d91a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.428787] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d91a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.429129] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: da1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:16.429157] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.429182] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: da1a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 Am 09.06.2011 17:09, schrieb anthony garnier: > Hi, > > I got the same problem as Juergen, > My volume is a simple replicated volume with 2 host and GlusterFS 3.2.0 > > Volume Name: poolsave > Type: Replicate > Status: Started > Number of Bricks: 2 > Transport-type: tcp > Bricks: > Brick1: ylal2950:/soft/gluster-data > Brick2: ylal2960:/soft/gluster-data > Options Reconfigured: > diagnostics.brick-log-level: DEBUG > network.ping-timeout: 20 > performance.cache-size: 512MB > nfs.port: 2049 > > I'm running this command : > > I get those error : > tar: ./uvs00: owner not changed > tar: could not stat ./uvs00/log/0906uvsGESEC.log > tar: ./uvs00: group not changed > tar: could not stat ./uvs00/log/0306uvsGESEC.log > tar: ./uvs00/log: Input/output error > cannot change back?: Unknown error 526 > tar: ./uvs00/log: owner not changed > tar: ./uvs00/log: group not changed > tar: tape blocksize error > > And then I tried to "ls" in gluster mount : > /bin/ls: .: Input/output error > > only way is to restart the volume > > > Here is the logfile in Debug mod : > > > Given volfile: > +------------------------------------------------------------------------------+ > 1: volume poolsave-client-0 > 2: type protocol/client > 3: option remote-host ylal2950 > 4: option remote-subvolume /soft/gluster-data > 5: option transport-type tcp > 6: option ping-timeout 20 > 7: end-volume > 8: > 9: volume poolsave-client-1 > 10: type protocol/client > 11: option remote-host ylal2960 > 12: option remote-subvolume /soft/gluster-data > 13: option transport-type tcp > 14: option ping-timeout 20 > 15: end-volume > 16: > 17: volume poolsave-replicate-0 > 18: type cluster/replicate > 19: subvolumes poolsave-client-0 poolsave-client-1 > 20: end-volume > 21: > 22: volume poolsave-write-behind > 23: type performance/write-behind > 24: subvolumes poolsave-replicate-0 > 25: end-volume > 26: > 27: volume poolsave-read-ahead > 28: type performance/read-ahead > 29: subvolumes poolsave-write-behind > 30: end-volume > 31: > 32: volume poolsave-io-cache > 33: type performance/io-cache > 34: option cache-size 512MB > 35: subvolumes poolsave-read-ahead > 36: end-volume > 37: > 38: volume poolsave-quick-read > 39: type performance/quick-read > 40: option cache-size 512MB > 41: subvolumes poolsave-io-cache > 42: end-volume > 43: > 44: volume poolsave-stat-prefetch > 45: type performance/stat-prefetch > 46: subvolumes poolsave-quick-read > 47: end-volume > 48: > 49: volume poolsave > 50: type debug/io-stats > 51: option latency-measurement off > 52: option count-fop-hits off > 53: subvolumes poolsave-stat-prefetch > 54: end-volume > 55: > 56: volume nfs-server > 57: type nfs/server > 58: option nfs.dynamic-volumes on > 59: option rpc-auth.addr.poolsave.allow * > 60: option nfs3.poolsave.volume-id 71e0dabf-4620-4b6d-b138-3266096b93b6 > 61: option nfs.port 2049 > 62: subvolumes poolsave > 63: end-volume > > +------------------------------------------------------------------------------+ > [2011-06-09 16:52:23.709018] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-poolsave-client-0: changing > port to 24014 (from 0) > [2011-06-09 16:52:23.709211] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-poolsave-client-1: changing > port to 24011 (from 0) > [2011-06-09 16:52:27.716417] I [client-handshake.c:1080:select_server_supported_programs] > 0-poolsave-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) > [2011-06-09 16:52:27.716650] I [client-handshake.c:913:client_setvolume_cbk] 0-poolsave-client-0: > Connected to 10.68.217.85:24014, attached to remote volume '/soft/gluster-data'. > [2011-06-09 16:52:27.716679] I [afr-common.c:2514:afr_notify] 0-poolsave-replicate-0: Subvolume > 'poolsave-client-0' came back up; going online. > [2011-06-09 16:52:27.717020] I [afr-common.c:836:afr_fresh_lookup_cbk] 0-poolsave-replicate-0: added > root inode > [2011-06-09 16:52:27.729719] I [client-handshake.c:1080:select_server_supported_programs] > 0-poolsave-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) > [2011-06-09 16:52:27.730014] I [client-handshake.c:913:client_setvolume_cbk] 0-poolsave-client-1: > Connected to 10.68.217.86:24011, attached to remote volume '/soft/gluster-data'. > [2011-06-09 17:01:35.537084] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_mkdir+0x1cc) [0x2aaaab3b88fc] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_mkdir+0x151) [0x2aaaab2948e1] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_mkdir+0xd2) > [0x2aaaab1856c2]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 > gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present > [2011-06-09 17:01:35.546601] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) > [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 > gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present > [2011-06-09 17:01:35.569755] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: > remote operation failed: Directory not empty > [2011-06-09 17:01:35.569881] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: > remote operation failed: Directory not empty > [2011-06-09 17:01:35.579674] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_mkdir+0x1cc) [0x2aaaab3b88fc] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_mkdir+0x151) [0x2aaaab2948e1] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_mkdir+0xd2) > [0x2aaaab1856c2]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 > gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present > [2011-06-09 17:01:35.587907] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) > [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 > gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present > [2011-06-09 17:01:35.612918] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) > [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 > gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present > [2011-06-09 17:01:35.645357] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) > [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 > gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present > [2011-06-09 17:01:35.660873] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: > remote operation failed: Directory not empty > [2011-06-09 17:01:35.660955] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: > remote operation failed: Directory not empty > [2011-06-09 17:01:35.665933] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: > remote operation failed: Directory not empty > [2011-06-09 17:01:35.666057] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: > remote operation failed: Directory not empty > [2011-06-09 17:01:35.671199] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: > remote operation failed: Directory not empty > [2011-06-09 17:01:35.671241] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: > remote operation failed: Directory not empty > [2011-06-09 17:01:35.680959] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) > [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 > gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present > [2011-06-09 17:01:35.715633] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) > [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 > gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present > [2011-06-09 17:01:35.732798] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: > remote operation failed: Permission denied > [2011-06-09 17:01:35.733044] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: > remote operation failed: Permission denied > [2011-06-09 17:01:35.750009] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] > (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) > [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 > gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present > [2011-06-09 17:01:35.784610] W [socket.c:1494:__socket_proto_state_machine] 0-poolsave-client-0: > reading from socket failed. Error (Transport endpoint is not connected), peer (10.68.217.85:24014) > [2011-06-09 17:01:35.784745] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab58145f7f9] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab58145ef8e] > (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab58145eefe]))) 0-poolsave-client-0: > forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-09 17:01:35.752080 > [2011-06-09 17:01:35.784770] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-poolsave-client-0: > remote operation failed: Transport endpoint is not connected > [2011-06-09 17:01:35.784811] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab58145f7f9] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab58145ef8e] > (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab58145eefe]))) 0-poolsave-client-0: > forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2011-06-09 17:01:35.752414 > [2011-06-09 17:01:35.784828] I [client3_1-fops.c:411:client3_1_stat_cbk] 0-poolsave-client-0: remote > operation failed: Transport endpoint is not connected > [2011-06-09 17:01:35.784875] I [client.c:1883:client_rpc_notify] 0-poolsave-client-0: disconnected > [2011-06-09 17:01:35.785400] W [socket.c:204:__socket_rwv] 0-poolsave-client-1: readv failed > (Connection reset by peer) > [2011-06-09 17:01:35.785435] W [socket.c:1494:__socket_proto_state_machine] 0-poolsave-client-1: > reading from socket failed. Error (Connection reset by peer), peer (10.68.217.86:24011) > [2011-06-09 17:01:35.785496] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab58145f7f9] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab58145ef8e] > (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab58145eefe]))) 0-poolsave-client-1: > forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-09 17:01:35.752089 > [2011-06-09 17:01:35.785516] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-poolsave-client-1: > remote operation failed: Transport endpoint is not connected > [2011-06-09 17:01:35.785542] W [client3_1-fops.c:4379:client3_1_xattrop] 0-poolsave-client-0: failed > to send the fop: Transport endpoint is not connected > [2011-06-09 17:01:35.817662] I [socket.c:2272:socket_submit_request] 0-poolsave-client-1: not > connected (priv->connected = 0) > [2011-06-09 17:01:35.817698] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-poolsave-client-1: failed to > submit rpc-request (XID: 0x576x Program: GlusterFS 3.1, ProgVers: 310, Proc: 33) to rpc-transport > (poolsave-client-1) > [2011-06-09 17:01:35.817721] W [client3_1-fops.c:4735:client3_1_inodelk] 0-poolsave-client-0: failed > to send the fop: Transport endpoint is not connected > [2011-06-09 17:01:35.817744] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-poolsave-client-1: failed to > submit rpc-request (XID: 0x577x Program: GlusterFS 3.1, ProgVers: 310, Proc: 29) to rpc-transport > (poolsave-client-1) > [2011-06-09 17:01:35.817780] I [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-poolsave-client-1: > remote operation failed: Transport endpoint is not connected > [2011-06-09 17:01:35.817897] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab58145f7f9] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab58145ef8e] > (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab58145eefe]))) 0-poolsave-client-1: > forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2011-06-09 17:01:35.784870 > [2011-06-09 17:01:35.817918] I [client3_1-fops.c:411:client3_1_stat_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2011-06-09 17:01:35.817969] I [client.c:1883:client_rpc_notify] 0-poolsave-client-1: disconnected > [2011-06-09 17:01:35.817988] E [afr-common.c:2546:afr_notify] 0-poolsave-replicate-0: All subvolumes > are down. Going offline until atleast one of them comes back up. > [2011-06-09 17:01:35.818007] E [socket.c:1685:socket_connect_finish] 0-poolsave-client-1: connection > to 10.68.217.86:24011 failed (Connection refused) > [2011-06-09 17:01:35.818606] I [afr.h:838:AFR_LOCAL_INIT] 0-poolsave-replicate-0: no subvolumes up > [2011-06-09 17:01:35.819129] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /uvs00/log: > no child is up > [2011-06-09 17:01:35.819354] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /uvs00/log: > no child is up > [2011-06-09 17:01:35.820090] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /uvs00: no > child is up > [2011-06-09 17:01:35.820760] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:35.821212] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:35.821600] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:35.822123] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:35.822511] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:35.822975] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:35.823286] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:35.823583] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:35.823857] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:47.518006] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > [2011-06-09 17:01:49.39204] E [socket.c:1685:socket_connect_finish] 0-poolsave-client-0: connection > to 10.68.217.85:24014 failed (Connection refused) > [2011-06-09 17:01:49.136932] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up > > > > > Message: 7 > > Date: Thu, 9 Jun 2011 12:56:39 +0530 > > From: Shehjar Tikoo <shehjart at gluster.com> > > Subject: Re: Glusterfs 3.2.0 NFS Problem > > To: J?rgen Winkler <juergen.winkler at xidras.com> > > Cc: gluster-users at gluster.org > > Message-ID: <4DF075AF.3040509 at gluster.com> > > Content-Type: text/plain; charset="us-ascii"; format=flowed > > > > This can happen if all your servers were unreachable for a few seconds. The > > situation must have rectified during the restart. We could confirm if you > > change the log level on nfs to DEBUG and send us the log. > > > > Thanks > > -Shehjar > > > > Ju"rgen Winkler wrote: > > > Hi, > > > > > > i noticed a strange behavior with NFS and Glusterfs 3.2.0 , 3 of our > > > Servers are loosing the Mount but when you restart the Volume on the > > > Server it works again without a remount. > > > > > > On the server i noticed this entries in the Glusterfs/Nfs log-file when > > > the mount on the Client becomes unavailable : > > > > > > [2011-06-08 14:37:02.568693] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:02.569212] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:02.611910] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:02.624477] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.288272] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.296150] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.309247] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.320939] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.321786] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.333609] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.334089] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.344662] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.352666] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.354195] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.360446] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.369331] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.471556] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:04.480013] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:05.639700] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:05.652535] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:07.578469] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:07.588949] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:07.590395] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:07.591414] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:07.591932] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:07.592596] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:07.639317] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:07.652919] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:09.332435] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:09.340622] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:09.349360] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:09.349550] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:09.360445] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:09.369497] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:09.369752] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:09.382097] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > [2011-06-08 14:37:09.382387] I [afr-inode-read.c:270:afr_stat] > > > 0-ksc-replicate-0: /: no child is up > > > > > > > > > Thx for the help > > > > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > > > > > > ------------------------------ > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > > > > End of Gluster-users Digest, Vol 38, Issue 14 > > ********************************************* > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- Mag. Christopher Anderlik Leiter Technik ________________________________________________________________________________ Xidras GmbH Stockern 47 3744 Stockern Austria Tel: 0043 2983 201 30 5 01 Fax: 0043 2983 201 30 5 01 9 Email: christopher.anderlik at xidras.com Web: http://www.xidras.com FN 317036 f | Landesgericht Krems | ATU64485024 ________________________________________________________________________________ VERTRAULICHE INFORMATIONEN! Diese eMail enth?lt vertrauliche Informationen und ist nur f?r den berechtigten Empf?nger bestimmt. Wenn diese eMail nicht f?r Sie bestimmt ist, bitten wir Sie, diese eMail an uns zur?ckzusenden und anschlie?end auf Ihrem Computer und Mail-Server zu l?schen. Solche eMails und Anlagen d?rfen Sie weder nutzen, noch verarbeiten oder Dritten zug?nglich machen, gleich in welcher Form. Wir danken f?r Ihre Kooperation! CONFIDENTIAL! This email contains confidential information and is intended for the authorised recipient only. If you are not an authorised recipient, please return the email to us and then delete it from your computer and mail-server. You may neither use nor edit any such emails including attachments, nor make them accessible to third parties in any manner whatsoever. Thank you for your cooperation ________________________________________________________________________________
