Experiencing errors after adding new nodes

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks for the reply Pranith.

Unfortunately I am not able to see the bug in the system.  Can you give 
me more detail to a workaround?

Much appreciated.

Regards,

John

On 02/24/2011 10:11 PM, Pranith Kumar. Karampuri wrote:
> Rebalance and EPERM, looks like you are hitting http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2369
>
> Pranith.
>
> ----- Original Message -----
> From: "John Lao"<jlao at cloud9analytics.com>
> To: gluster-users at gluster.org
> Sent: Thursday, February 24, 2011 11:55:52 PM
> Subject: Experiencing errors after adding new nodes
>
> Hi,
>
> I had a 2 node distributed cluster running on 3.1.1 and I added 2 more nodes.  I then ran a rebalance on the cluster.
>
> Now I am getting permission denied errors and I see the following in the client logs:
>
> [2011-02-24 09:59:10.210166] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
> [2011-02-24 09:59:11.851656] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
> [root at qe-loader1 glusterfs]# tail -100 mnt-qe-filer01.log
> [2011-02-24 09:32:50.844211] I [dht-common.c:415:dht_revalidate_cbk] loader-dht: mismatching layouts for /
> [2011-02-24 09:32:50.845312] I [dht-layout.c:588:dht_layout_normalize] loader-dht: found anomalies in /. holes=1 overlaps=0
> [2011-02-24 09:32:50.846417] I [dht-layout.c:689:dht_layout_dir_mismatch] loader-dht: subvol: loader-client-0; inode layout - 0 - 2147483646; disk layout - 2147483646 - 3221225468
> [2011-02-24 09:32:50.846434] I [dht-common.c:415:dht_revalidate_cbk] loader-dht: mismatching layouts for /
> [2011-02-24 09:32:50.846561] I [dht-layout.c:689:dht_layout_dir_mismatch] loader-dht: subvol: loader-client-1; inode layout - 2147483647 - 4294967295; disk layout - 3221225469 - 4294967295
> [2011-02-24 09:32:50.846577] I [dht-common.c:415:dht_revalidate_cbk] loader-dht: mismatching layouts for /
> [2011-02-24 09:32:50.847650] I [dht-layout.c:588:dht_layout_normalize] loader-dht: found anomalies in /. holes=1 overlaps=0
> [2011-02-24 09:39:08.159175] I [fuse-bridge.c:3138:fuse_thread_proc] fuse: unmounting /mnt/qe-filer01
> [2011-02-24 09:39:08.160180] I [glusterfsd.c:672:cleanup_and_exit] glusterfsd: shutting down
> [2011-02-24 09:39:14.184554] W [io-stats.c:1644:init] loader: dangling volume. check volfile
> [2011-02-24 09:39:14.184599] W [dict.c:1204:data_to_str] dict: @data=(nil)
> [2011-02-24 09:39:14.184614] W [dict.c:1204:data_to_str] dict: @data=(nil)
> Given volfile:
> +------------------------------------------------------------------------------+
>    1: volume loader-client-0
>    2:     type protocol/client
>    3:     option remote-host qe-filer03
>    4:     option remote-subvolume /data1
>    5:     option transport-type tcp
>    6: end-volume
>    7:
>    8: volume loader-client-1
>    9:     type protocol/client
>   10:     option remote-host qe-filer04
>   11:     option remote-subvolume /data1
>   12:     option transport-type tcp
>   13: end-volume
>   14:
>   15: volume loader-client-2
>   16:     type protocol/client
>   17:     option remote-host 172.24.8.25
>   18:     option remote-subvolume /data1
>   19:     option transport-type tcp
>   20: end-volume
>   21:
>   22: volume loader-client-3
>   23:     type protocol/client
>   24:     option remote-host qe-filer02
>   25:     option remote-subvolume /data1
>   26:     option transport-type tcp
>   27: end-volume
>   28:
>   29: volume loader-dht
>   30:     type cluster/distribute
>   31:     subvolumes loader-client-0 loader-client-1 loader-client-2 loader-client-3
>   32: end-volume
>   33:
>   34: volume loader-write-behind
>   35:     type performance/write-behind
>   36:     subvolumes loader-dht
>   37: end-volume
>   38:
>   39: volume loader-read-ahead
>   40:     type performance/read-ahead
>   41:     subvolumes loader-write-behind
>   42: end-volume
>   43:
>   44: volume loader-io-cache
>   45:     type performance/io-cache
>   46:     subvolumes loader-read-ahead
>   47: end-volume
>   48:
>   49: volume loader-quick-read
>   50:     type performance/quick-read
>   51:     subvolumes loader-io-cache
>   52: end-volume
>   53:
>   54: volume loader-stat-prefetch
>   55:     type performance/stat-prefetch
>   56:     subvolumes loader-quick-read
>   57: end-volume
>   58:
>   59: volume loader
>   60:     type debug/io-stats
>   61:     subvolumes loader-stat-prefetch
>   62: end-volume
>
> +------------------------------------------------------------------------------+
> [2011-02-24 09:39:18.132954] I [client-handshake.c:993:select_server_supported_programs] loader-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
> [2011-02-24 09:39:18.136077] I [client-handshake.c:829:client_setvolume_cbk] loader-client-2: Connected to 172.24.8.25:24009, attached to remote volume '/data1'.
> [2011-02-24 09:39:18.136498] I [client-handshake.c:993:select_server_supported_programs] loader-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
> [2011-02-24 09:39:18.137083] I [client-handshake.c:829:client_setvolume_cbk] loader-client-1: Connected to 172.24.8.222:24009, attached to remote volume '/data1'.
> [2011-02-24 09:39:18.140002] I [client-handshake.c:993:select_server_supported_programs] loader-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
> [2011-02-24 09:39:18.140599] I [client-handshake.c:829:client_setvolume_cbk] loader-client-0: Connected to 172.24.8.221:24009, attached to remote volume '/data1'.
> [2011-02-24 09:39:18.144060] I [client-handshake.c:993:select_server_supported_programs] loader-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
> [2011-02-24 09:39:18.144689] I [client-handshake.c:829:client_setvolume_cbk] loader-client-3: Connected to 172.24.8.220:24009, attached to remote volume '/data1'.
> [2011-02-24 09:39:18.150335] I [fuse-bridge.c:2817:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
> [2011-02-24 09:54:12.904354] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse: 4094: UNLINK() / c9.lock =>  -1 (Permission denied)
> [2011-02-24 09:54:34.716535] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
> [2011-02-24 09:54:36.470275] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
> [2011-02-24 09:55:24.588912] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
> [2011-02-24 09:55:26.180690] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
> [2011-02-24 09:55:57.816431] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
> [2011-02-24 09:55:59.242725] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
> [2011-02-24 09:56:54.92112] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23558: / c9_refresh.lock =>  -1 (Permission denied)
> [2011-02-24 09:57:01.792819] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23564: /Cloud9c9_refresh.lock =>  -1 (Permission denied)
> [2011-02-24 09:57:31.812728] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23570: / c9_refresh.lock =>  -1 (Permission denied)
> [2011-02-24 09:58:12.738834] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23582: / c9_refresh.lock =>  -1 (Permission denied)
> [2011-02-24 09:59:10.210166] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
> [2011-02-24 09:59:11.851656] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
>
> I am running the rebalance again to see if it will fix the layout issue.
>
> Any ideas why I'm seeing these errors after adding new nodes?
>
> Regards,
>
> John Lao
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users



[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux