|
Hi Kris, These failures are mostly while transferring data from Master to Slave. Geo-replication is not intelligent to find exact number and reason for failures. I sent patch to effectively handle the rsync/tar failures. This patch is expected to be available with 3.7.8 release. http://review.gluster.org/#/c/12856/ Above patch identifies the reason for failure and logs in /var/log/glusterfs/geo-replication/<MASTERVOL>/*.log. Answers inline. regards Aravinda On 02/03/2016 07:16 AM, Kris Laib
wrote:
Looks like these issues can be ignored. Errors may be due to xattrs syncing. With the above patch, logging is improved to log the failure reasons.I recently setup geo-replication between two sites, and everything appears to be working, however, if I do a "gluster volume geo-replication SourceVol DestVol status detail", I see that the "failures" column seems to be incrementing for each file it transfers. I've looked at the logs in /var/log/glusterfs/geo-replication/SourceVol/* , but nothing stood out to me. I've verified that the MD5 sum matches on the files at the destination after it's been replicated, so I'm unsure what is triggering the failure. Aside from the incrementing failure column, it all looks good to me. Possible reasons for Sync failures,All boxes in both clusters are CentOS 6.7, running glusterfs 3.7.5 My questions: 1) What's causing the failures to be logged (please let me know what additional logs to provide)? - May be failure to sync xattrs/acls to Slave. - I/O error on Slave - Files unlinked in Slave - GFID Conflict Will fix this issue in upcoming releases.2) How do I reset the failures counter? It stays at the current value even once the files are deleted from the source Entry - Pending Entry operations - Entry operations like CREATE, RENAME, UNLINK etc, which are executed as RPC calls.3) Is there documentation somewhere I missed that explains what Entry, Data, Meta, and Failures are in the status detail view for geo-replication? Data - Pending Data operations - Rsync/Tar is used to transfer data from Master to Slave. Meta - Pending Meta operations - Changing ownership, permissions are executed as RPC calls. Failures - Going to be obsolete since Errors will be logged in log files. Geo-replication status only available from Master.4) Should "gluster volume geo-replication status" show me anything on the destination? Mine simply outputs "No active geo-replication sessions", though the command on the source works fine. Thanks, Kris ----------- Here's an example of what I'm seeing: [root@xxxxxxxxxxxxxx temp]$ mount | grep "gfs-prd.mgmt:/prd_items" gfs-prd.mgmt:/prd_items on /mnt/temp type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) [root@xxxxxxxxxxxxxx temp]$ pwd /mnt/temp [root@xxxxxxxxxxxxxx temp]$ gluster volume geo-replication prd_items georep@xxxxxxxxxxxxxx::dr_items status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- gfs-prd02.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep@xxxxxxxxxxxxxx::dr_items gfs-slc03.mgmt Active Changelog Crawl 2016-02-03 01:28:26 0 0 0 2 N/A N/A N/A gfs-prd03.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep@xxxxxxxxxxxxxx::dr_items gfs-slc02.mgmt Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A [root@xxxxxxxxxxxxxx temp]$ dd if=/dev/urandom of=test01 bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.179957 s, 5.8 MB/s [root@xxxxxxxxxxxxxx temp]$ dd if=/dev/urandom of=test02 bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.181704 s, 5.8 MB/s [root@xxxxxxxxxxxxxx temp]$ dd if=/dev/urandom of=test03 bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.181575 s, 5.8 MB/s [root@xxxxxxxxxxxxxx temp]$ gluster volume geo-replication prd_items georep@xxxxxxxxxxxxxx::dr_items status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- gfs-prd02.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep@xxxxxxxxxxxxxx::dr_items gfs-slc03.mgmt Active Changelog Crawl 2016-02-03 01:29:11 0 0 0 5 N/A N/A N/A gfs-prd03.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep@xxxxxxxxxxxxxx::dr_items gfs-slc02.mgmt Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.log ============== [2016-02-03 01:26:14.702147] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns [2016-02-03 01:26:14.718751] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462711, 0) [2016-02-03 01:27:15.371404] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns [2016-02-03 01:27:15.379936] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462771, 0) [2016-02-03 01:27:58.15177] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462831, 0) [2016-02-03 01:27:59.54204] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:00.101433] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:01.104391] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:02.312911] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:03.356967] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:04.364518] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:05.565407] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:06.610732] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:07.620142] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:08.818785] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462877 could not be processed - moving on... [2016-02-03 01:28:08.826875] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = f5a44b23-cdd8-4278-92d1-1b2a16e8c9d5 [2016-02-03 01:28:15.94074] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462876, 0) [2016-02-03 01:28:18.669955] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 17 crawls, 3 turns [2016-02-03 01:28:27.720731] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462891, 0) [2016-02-03 01:28:58.387859] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462906, 0) [2016-02-03 01:28:59.360420] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:00.430566] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:01.610398] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:02.667408] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:03.692369] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:04.947613] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:05.932200] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:06.947702] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:08.186101] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:09.203042] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462937 could not be processed - moving on... [2016-02-03 01:29:09.205854] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = c99a05c9-9e7d-41cf-b148-9bec91f527c8 [2016-02-03 01:29:15.432744] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462936, 0) [2016-02-03 01:29:16.743381] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:17.944102] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:18.970493] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:20.77699] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:21.197900] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:22.259825] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:23.349780] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:24.469357] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:25.526793] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:26.612977] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462952 could not be processed - moving on... [2016-02-03 01:29:26.615757] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = a7dc6c28-9239-44af-909f-a2938bb387b6,550ffdd3-4636-45b8-b278-fbfe120b9d85 [2016-02-03 01:29:29.828999] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 16 crawls, 3 turns [2016-02-03 01:30:15.98556] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462951, 0) [2016-02-03 01:30:30.709733] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns [2016-02-03 01:31:12.918898] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454463011, 0) [2016-02-03 01:31:31.423204] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1-changes.log =============== [2016-02-03 01:26:12.472345] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:26:12.472396] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1721, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:26:12.472415] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:26:12.472426] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:26:12.472440] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2478 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462772.472206), (vec: 1, len: 4104) [2016-02-03 01:26:12.472467] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:26:12.472481] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:26:12.472502] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6b9, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:26:14.952267] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462772 to processed directory [2016-02-03 01:27:12.531138] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:27:12.531177] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1722, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:27:12.531195] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:27:12.531207] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:27:12.531220] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2479 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462832.531035), (vec: 1, len: 4104) [2016-02-03 01:27:12.531241] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:27:12.531256] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:27:12.531274] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6ba, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:27:15.660249] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462832 to processed directory [2016-02-03 01:27:57.576147] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:27:57.576198] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1723, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:27:57.576217] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:27:57.576230] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:27:57.576246] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2480 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462877.576035), (vec: 1, len: 4104) [2016-02-03 01:27:57.576282] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:27:57.576297] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:27:57.576327] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bb, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:28:08.833040] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462877 to processed directory [2016-02-03 01:28:12.589377] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:28:12.589420] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1724, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:28:12.589439] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:28:12.589451] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:28:12.589466] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2481 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462892.589270), (vec: 1, len: 4104) [2016-02-03 01:28:12.589487] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:28:12.589500] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:28:12.589521] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bc, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:28:15.398144] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462892 to processed directory [2016-02-03 01:28:27.604453] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:28:27.604510] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1725, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:28:27.604530] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:28:27.604542] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:28:27.604557] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2482 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462907.604293), (vec: 1, len: 4104) [2016-02-03 01:28:27.604580] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:28:27.604594] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:28:27.604615] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bd, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:28:28.066455] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462907 to processed directory [2016-02-03 01:28:57.634864] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:28:57.634938] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1726, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:28:57.635006] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:28:57.635028] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:28:57.635049] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2483 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462937.634652), (vec: 1, len: 4104) [2016-02-03 01:28:57.635079] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:28:57.635098] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:28:57.635127] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6be, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:29:09.211888] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462937 to processed directory [2016-02-03 01:29:12.650651] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:29:12.650705] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1727, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:29:12.650726] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:29:12.650739] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:29:12.650755] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2484 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462952.650534), (vec: 1, len: 4104) [2016-02-03 01:29:12.650787] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:29:12.650802] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:29:12.650824] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bf, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:29:26.622432] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462952 to processed directory [2016-02-03 01:30:12.693458] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:30:12.693499] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1728, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:30:12.693528] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:30:12.693544] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:30:12.693562] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2485 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463012.693353), (vec: 1, len: 4104) [2016-02-03 01:30:12.693587] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:30:12.693610] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:30:12.693638] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c0, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:30:15.432599] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463012 to processed directory [2016-02-03 01:31:12.754211] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:31:12.754243] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1729, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:31:12.754260] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:31:12.754273] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:31:12.754294] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2486 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463072.754125), (vec: 1, len: 4104) [2016-02-03 01:31:12.754323] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:31:12.754338] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:31:12.754357] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c1, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:31:13.156812] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463072 to processed directory [2016-02-03 01:32:12.816173] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:32:12.816219] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1730, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:32:12.816236] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:32:12.816248] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:32:12.816261] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2487 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463132.816061), (vec: 1, len: 4104) [2016-02-03 01:32:12.816298] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:32:12.816319] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:32:12.816361] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c2, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:32:13.864741] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463132 to processed directory [2016-02-03 01:33:12.877209] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:33:12.877247] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1731, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:33:12.877276] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:33:12.877308] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:33:12.877332] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2488 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463192.877131), (vec: 1, len: 4104) [2016-02-03 01:33:12.877360] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:33:12.877380] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:33:12.877408] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c3, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:33:14.577963] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463192 to processed directory Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1.gluster.log ============= [2016-02-01 20:45:23.591390] I [MSGID: 108031] [afr-common.c:1783:afr_local_discovery_cbk] 0-prd_items-replicate-0: selecting local read_child prd_items-client-0 [2016-02-03 01:27:58.464278] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage [2016-02-03 01:29:03.558047] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage [2016-02-03 01:29:25.386117] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
