we have a machine that previously had centos 6.8 and
gluster 3.7.10-1 with 2 bricks. The machine had to be
rebuilt with centos 6.8 and the 2 bricks were not formatted.
Gluster 3.7.11 was installed with the new OS, and we can
start the service, create the volume with the 2 bricks and
mount the gluster share.
ls /data
ls: cannot access /data/gluster-data: No such file or
directory
gluster-data
The data and directories are still there (i.e. we can still
see them looking at the underlying file systems) but gluster
isn't serving them.
Looking in the log file for each brick we sea the same
errors:
[2016-06-03 04:30:07.494068] I [MSGID: 100030]
[glusterfsd.c:2332:main] 0-/usr/sbin/glusterfsd: Started
running /usr/sbin/glusterfsd version 3.7.11 (args:
/usr/sbin/glusterfsd -s mseas-data2 --volfile-id
data-volume.mseas-data2.mnt-brick1 -p
/var/lib/glusterd/vols/data-volume/run/mseas-data2-mnt-brick1.pid
-S
/var/run/gluster/aa572e87933c930cb53983de35bdccbe.socket
--brick-name /mnt/brick1 -l
/var/log/glusterfs/bricks/mnt-brick1.log --xlator-option
*-posix.glusterd-uuid=c1110fd9-cb99-4ca1-b18a-536a122d67ef
--brick-port 49152 --xlator-option
data-volume-server.listen-port=49152)
[2016-06-03 04:30:07.510671] I [MSGID: 101190]
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll:
Started thread with index 1
[2016-06-03 04:30:07.519040] I
[graph.c:269:gf_add_cmdline_options] 0-data-volume-server:
adding option 'listen-port' for volume
'data-volume-server' with value '49152'
[2016-06-03 04:30:07.519089] I
[graph.c:269:gf_add_cmdline_options] 0-data-volume-posix:
adding option 'glusterd-uuid' for volume
'data-volume-posix' with value
'c1110fd9-cb99-4ca1-b18a-536a122d67ef'
[2016-06-03 04:30:07.519479] I [MSGID: 101190]
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll:
Started thread with index 2
[2016-06-03 04:30:07.519486] I [MSGID: 115034]
[server.c:403:_check_for_auth_option] 0-/mnt/brick1: skip
format check for non-addr auth option
auth.login./mnt/brick1.allow
[2016-06-03 04:30:07.519537] I [MSGID: 115034]
[server.c:403:_check_for_auth_option] 0-/mnt/brick1: skip
format check for non-addr auth option
auth.login.0016d59c-9691-4bb2-bc44-b1d8b19dd230.password
[2016-06-03 04:30:07.520926] I
[rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit]
0-rpc-service: Configured rpc.outstanding-rpc-limit with
value 64
[2016-06-03 04:30:07.521003] W [MSGID: 101002]
[options.c:957:xl_opt_validate] 0-data-volume-server:
option 'listen-port' is deprecated, preferred is
'transport.socket.listen-port', continuing with correction
[2016-06-03 04:30:07.523056] I [MSGID: 121050]
[ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore:
CTR Xlator is disabled.
[2016-06-03 04:30:07.523077] W [MSGID: 101105]
[gfdb_sqlite3.h:239:gfdb_set_sql_params]
0-data-volume-changetimerecorder: Failed to retrieve
sql-db-pagesize from params.Assigning default value: 4096
[2016-06-03 04:30:07.523086] W [MSGID: 101105]
[gfdb_sqlite3.h:239:gfdb_set_sql_params]
0-data-volume-changetimerecorder: Failed to retrieve
sql-db-journalmode from params.Assigning default value:
wal
[2016-06-03 04:30:07.523095] W [MSGID: 101105]
[gfdb_sqlite3.h:239:gfdb_set_sql_params]
0-data-volume-changetimerecorder: Failed to retrieve
sql-db-sync from params.Assigning default value: off
[2016-06-03 04:30:07.523102] W [MSGID: 101105]
[gfdb_sqlite3.h:239:gfdb_set_sql_params]
0-data-volume-changetimerecorder: Failed to retrieve
sql-db-autovacuum from params.Assigning default value:
none
[2016-06-03 04:30:07.523280] I [trash.c:2369:init]
0-data-volume-trash: no option specified for 'eliminate',
using NULL
[2016-06-03 04:30:07.523910] W
[graph.c:357:_log_if_unknown_option] 0-data-volume-server:
option 'rpc-auth.auth-glusterfs' is not recognized
[2016-06-03 04:30:07.523937] W
[graph.c:357:_log_if_unknown_option] 0-data-volume-server:
option 'rpc-auth.auth-unix' is not recognized
[2016-06-03 04:30:07.523955] W
[graph.c:357:_log_if_unknown_option] 0-data-volume-server:
option 'rpc-auth.auth-null' is not recognized
[2016-06-03 04:30:07.523989] W
[graph.c:357:_log_if_unknown_option] 0-data-volume-quota:
option 'timeout' is not recognized
[2016-06-03 04:30:07.524031] W
[graph.c:357:_log_if_unknown_option] 0-data-volume-trash:
option 'brick-path' is not recognized
[2016-06-03 04:30:07.529994] W [MSGID: 113036]
[posix.c:2211:posix_rename] 0-data-volume-posix: found
directory at /mnt/brick1/.trashcan/ while expecting ENOENT
[File exists]
Final graph:
+------------------------------------------------------------------------------+
1: volume data-volume-posix
2: type storage/posix
3: option glusterd-uuid
c1110fd9-cb99-4ca1-b18a-536a122d67ef
4: option directory /mnt/brick1
5: option volume-id
c54b2a60-ffdc-4d82-9db1-890e41002e28
6: end-volume
7:
8: volume data-volume-trash
9: type features/trash
10: option trash-dir .trashcan
11: option brick-path /mnt/brick1
12: option trash-internal-op off
13: subvolumes data-volume-posix
14: end-volume
15:
16: volume data-volume-changetimerecorder
17: type features/changetimerecorder
18: option db-type sqlite3
19: option hot-brick off
20: option db-name brick1.db
21: option db-path /mnt/brick1/.glusterfs/
22: option record-exit off
23: option ctr_link_consistency off
24: option ctr_lookupheal_link_timeout 300
25: option ctr_lookupheal_inode_timeout 300
26: option record-entry on
27: option ctr-enabled off
28: option record-counters off
29: option ctr-record-metadata-heat off
30: option sql-db-cachesize 1000
31: option sql-db-wal-autocheckpoint 1000
32: subvolumes data-volume-trash
33: end-volume
34:
35: volume data-volume-changelog
36: type features/changelog
37: option changelog-brick /mnt/brick1
38: option changelog-dir
/mnt/brick1/.glusterfs/changelogs
39: option changelog-barrier-timeout 120
40: subvolumes data-volume-changetimerecorder
41: end-volume
42:
43: volume data-volume-bitrot-stub
44: type features/bitrot-stub
45: option export /mnt/brick1
46: subvolumes data-volume-changelog
47: end-volume
48:
49: volume data-volume-access-control
50: type features/access-control
51: subvolumes data-volume-bitrot-stub
52: end-volume
53:
54: volume data-volume-locks
55: type features/locks
56: subvolumes data-volume-access-control
57: end-volume
58:
59: volume data-volume-upcall
60: type features/upcall
61: option cache-invalidation off
62: subvolumes data-volume-locks
63: end-volume
64:
65: volume data-volume-io-threads
66: type performance/io-threads
67: subvolumes data-volume-upcall
68: end-volume
69:
70: volume data-volume-marker
71: type features/marker
72: option volume-uuid
c54b2a60-ffdc-4d82-9db1-890e41002e28
73: option timestamp-file
/var/lib/glusterd/vols/data-volume/marker.tstamp
74: option quota-version 0
75: option xtime off
76: option gsync-force-xtime off
77: option quota off
78: option inode-quota off
79: subvolumes data-volume-io-threads
80: end-volume
81:
82: volume data-volume-barrier
83: type features/barrier
84: option barrier disable
85: option barrier-timeout 120
86: subvolumes data-volume-marker
87: end-volume
88:
89: volume data-volume-index
90: type features/index
91: option index-base /mnt/brick1/.glusterfs/indices
92: subvolumes data-volume-barrier
93: end-volume
94:
95: volume data-volume-quota
96: type features/quota
97: option volume-uuid data-volume
98: option server-quota off
99: option timeout 0
100: option deem-statfs off
101: subvolumes data-volume-index
102: end-volume
103:
104: volume data-volume-worm
105: type features/worm
106: option worm off
107: subvolumes data-volume-quota
108: end-volume
109:
110: volume data-volume-read-only
111: type features/read-only
112: option read-only off
113: subvolumes data-volume-worm
114: end-volume
115:
116: volume /mnt/brick1
117: type debug/io-stats
118: option log-level INFO
119: option latency-measurement off
120: option count-fop-hits off
121: subvolumes data-volume-read-only
122: end-volume
123:
124: volume data-volume-server
125: type protocol/server
126: option transport.socket.listen-port 49152
127: option rpc-auth.auth-glusterfs on
128: option rpc-auth.auth-unix on
129: option rpc-auth.auth-null on
130: option rpc-auth-allow-insecure on
131: option transport-type tcp
132: option auth.login./mnt/brick1.allow
0016d59c-9691-4bb2-bc44-b1d8b19dd230
133: option
auth.login.0016d59c-9691-4bb2-bc44-b1d8b19dd230.password
b021dbcf-e114-4c23-ad9f-968a2d93dd61
134: option auth.addr./mnt/brick1.allow *
135: subvolumes /mnt/brick1
136: end-volume
137:
+------------------------------------------------------------------------------+
[2016-06-03 04:30:07.583590] I [login.c:81:gf_auth]
0-auth/login: allowed user names:
0016d59c-9691-4bb2-bc44-b1d8b19dd230
[2016-06-03 04:30:07.583640] I [MSGID: 115029]
[server-handshake.c:690:server_setvolume]
0-data-volume-server: accepted client from
mseas-data2-2383-2016/06/03-04:30:07:127671-data-volume-client-0-0-0
(version: 3.7.11)
[2016-06-03 04:30:40.124584] I [login.c:81:gf_auth]
0-auth/login: allowed user names:
0016d59c-9691-4bb2-bc44-b1d8b19dd230
[2016-06-03 04:30:40.124628] I [MSGID: 115029]
[server-handshake.c:690:server_setvolume]
0-data-volume-server: accepted client from
mseas-data2-2500-2016/06/03-04:30:40:46064-data-volume-client-0-0-0
(version: 3.7.11)
[2016-06-03 04:30:43.265342] W [MSGID: 101182]
[inode.c:174:__foreach_ancestor_dentry]
0-data-volume-server: per dentry fn returned 1
[2016-06-03 04:30:43.265393] C [MSGID: 101184]
[inode.c:228:__is_dentry_cyclic] 0-/mnt/brick1/inode:
detected cyclic loop formation during inode linkage. inode
(00000000-0000-0000-0000-000000000001) linking under
itself as gluster-data
[2016-06-03 04:30:43.269197] W [MSGID: 101182]
[inode.c:174:__foreach_ancestor_dentry]
0-data-volume-server: per dentry fn returned 1
[2016-06-03 04:30:43.269241] C [MSGID: 101184]
[inode.c:228:__is_dentry_cyclic] 0-/mnt/brick1/inode:
detected cyclic loop formation during inode linkage. inode
(00000000-0000-0000-0000-000000000001) linking under
itself as gluster-data
[2016-06-03 04:30:43.270689] W [MSGID: 101182]
[inode.c:174:__foreach_ancestor_dentry]
0-data-volume-server: per dentry fn returned 1
[2016-06-03 04:30:43.270733] C [MSGID: 101184]
[inode.c:228:__is_dentry_cyclic] 0-/mnt/brick1/inode:
detected cyclic loop formation during inode linkage. inode
(00000000-0000-0000-0000-000000000001) linking under
itself as gluster-data
This is a distributed volume, not replicated. Can we delete
the gluster volume, remove the .glusterfs folders from each
brick and recreate the volume? Will it re-index the files on
both bricks?
Note:
From the last lines of log file,
there is a soft link at /mnt/brick1/.glusterfs/00/00/
00000000-0000-0000-0000-000000000001
>>>> ../../..
We have tried removing the link and restarting the service,
no change in behavior. It replaces/rebuilds the link on
service startup.
Any advice you can give will be appreciated.
Thanks
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley@xxxxxxx
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
