Hi gluster-users!
I’ve got an issue
currently with my 2x brick replicate volume. The volume is fine,
and both bricks are replicating fine, but there is 1 file which
refuses to be deleted or moved (and now can’t be viewed via a
glusterfs mount)
We just upgraded
from 3.7.13-ubuntu1~trusty1
to 3.7.16-ubuntu1~trusty1,
whether this has anything to do with it, I am not sure.
The following tests
have been performed on the mount points.
- Doing a stat on the file works
- Moving(renaming) the folder above
works
- Opening the file
vi file
and doing a cat file
use to work before I did:
mv /var/www/site/file /home/ubuntu/file
This mv command
never completed and I had to kill -9 it after an hour waiting.
It seems whenever
the file is opened now, it results in the process halting on a
stat command. The following is a stack trace when trying to run
the code (which calls the file, its php)
<snip>
16:17:34.358528 lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php", {st_mode=S_IFREG|0664, st_size=11878, ...}) = 0
16:17:34.358613 lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
16:17:34.358699 lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
16:17:34.358810 lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php", {st_mode=S_IFREG|0664, st_size=11878, ...}) = 0
16:17:34.358870 lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
16:17:34.358966 lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
16:17:34.359046 open("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php", O_RDONLY) = 5
16:17:34.362692 fstat(5, {st_mode=S_IFREG|0664, st_size=11878, ...}) = 0
16:17:34.362769 fstat(5, {st_mode=S_IFREG|0664, st_size=11878, ...}) = 0
16:17:34.362848 fstat(5, {st_mode=S_IFREG|0664, st_size=11878, ...}) = 0
16:17:34.362910 fstat(5, {st_mode=S_IFREG|0664, st_size=11878, ...}) = 0
16:17:34.363008 mmap(NULL, 11878, PROT_READ, MAP_SHARED, 5, 0) = 0x7eff135fd000
16:17:34.363102 stat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php", {st_mode=S_IFREG|0664, st_size=11878, ...}) = 0
This is where the
stack trace fails everytime.
sudo gluster volume
heal gv-www info
Brick demo-gluster01:/gluster
Status: Connected
Number of entries: 0
Brick
demo-gluster02:/gluster
Status: Connected
Number of entries: 0
sudo gluster volume
info
Volume Name: gv-www
Type: Replicate
Volume ID: 6163f185-d58b-45f2-8bf0-eeb3a18142d9
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: demo-gluster01:/gluster
Brick2: demo-gluster02:/gluster
Options Reconfigured:
cluster.self-heal-daemon: enable
features.scrub: Active
features.bitrot: on
diagnostics.client-log-level: INFO
diagnostics.brick-log-level: DEBUG
performance.readdir-ahead: on
I just enabled
bitrot and scrubbing. (To help with these possible issues)
The only real error
I can see in this timeframe is the following:
Brick1:
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.160099] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.174907] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.176708] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.178831] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.181356] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.183602] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.186835] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.189262] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.191969] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
Brick2:
/var/log/glusterfs/var-www.log.1:[2016-10-15 00:16:48.163353] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 6b0b2d6a-f6ea-4ab8-89fc-d776afc17c35. (Possible split-brain)
But I cannot find
any log relating to this on the gluster servers.
Also I can cat the
file no problem if I login to the bricks. What could be a reason
of it not working on the mount point…
Any help would be
greatly appreciated.
Cheers, Tim
