Mark Staudinger wrote:
Hi Folks,
I'm trying to debug an issue that I've found while attempting to
qualify GlusterFS for potential distributed storage projects on the
FreeBSD-11.1 server platform - using the existing package of GlusterFS
v3.11.1_4
The main issue I've encountered is that I cannot add new bricks while
setting/increasing the replica count.
If I create a replicated volume "poc" on two hosts, say
s1:/gluster/1/poc and s2:/gluster/1/poc, the volume is created
properly and shows replicated status, files are written to both volumes.
If I create a single volume: s1:/gluster/1/poc as a single /
distributed brick, and then try to run
gluster volume add-brick poc replica 2 s2:/gluster/1/poc
it will always fail (sometimes after a pause, sometimes not.) The
only error I'm seeing on the server hosting the new brick, aside from
the generic "Unable to add bricks" message, is like so:
I [MSGID: 106578]
[glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks]
0-management: replica-count is set 2
I [MSGID: 106578]
[glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks]
0-management: type is set 2, need to change it
E [MSGID: 106054]
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops]
0-management: Failed to set extended attribute trusted.add-brick :
Operation not supported [Operation not supported]
E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]
0-glusterd: Unable to add bricks
E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn]
0-management: Add-brick commit failed.
I was initially using ZFS and noted that ZFS on FreeBSD does not
support xattr, so I reverted to using UFS as the storage type for the
brick, and still encounter this behavior.
I also recompiled the port (again, GlusterFS v3.11.1) with the patch
from https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals
specifically with xattr handling in FreeBSD.
To recap - I'm able to create any type of volume (2 or 3-way
replicated or distributed), but I'm unable to add replicated bricks to
a volume.
I was, however, able to add a second distributed brick ( gluster
volume add-brick poc s2:/gluster/1/poc ) - so the issue seems specific
to adding and/or changing the replica count while adding a new brick.
Please let me know if there are any other issues in addition to bug
#1452961 I should be aware of, or additional log or debug info I can
provide.
Hi Folks,
As a follow-up to this, and it not making sense that xattrs would not be
supported on UFS2, I am attaching the 'truss' output of the extattr
syscalls from two operations. In both situations, the command was
initiated on server1, and traced on server2.
The first is a working "create-brick" issued from server1, traced from
server2: ( gluster volume create poc transport tcp replica 2
s1:/gluster/1/poc s2:/gluster/1/poc )
extattr_set_link(0x807a55440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x807a55440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_set_link(0x807a55440,0x1,0x803b24c17,0x807dfe250,0x10) = 16 (0x10)
And the second is from the not-working situation where the brick from s2
is being added to the volume ( gluster volume add-brick poc replica 2
s2:/gluster/1/poc commit )
extattr_set_link(0x80545c440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x80545c440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_set_link(0x80545c440,0x1,0x803b24c17,0x8051e9c20,0x10) = 16 (0x10)
extattr_set_link(0x80545c440,0x1,0x803b2774c,0x807bff068,0xc) = 12 (0xc)
write(6," 31: option afr-pending-xattr
poc-client-0,poc-client-1\n",64) = 64 (0x40)
extattr_set_link(0x807bff890,0x1,0x803b27886,0x80545d440,0x400) ERR#45
'Operation not supported'
write(6,"[2018-04-28 15:21:51.271551] E [MSGID: 106054]
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops]
0-management: Failed to set extended attribute trusted.add-brick :
Operation not supported [Operation not supported]\n",225) = 225 (0xe1)
Cheers,
-=Mark
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
