The “java.io.FileNotFound” error happened at 0:00 Jan 1st 2017, That’s Beijing time, while glusterfs use UTC time，the time should be 2017-01-01 16:00:00. Here is the log of three server at that time: ab09 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log … [2017-01-01 16:00:00.778962] E [MSGID: 113039] [posix.c:3011:posix_open] 0-gvol0-posix: open on /mnt/brick3/vol/.glusterfs/a6/1d/a61d7b2f-d53f-4eb9-be7a-656b0915dbea, flags: 1025 [No such file or directory] [2017-01-01 16:00:04.864163] W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link /mnt/brick3/vol/gps/data/2017-01/02/84.txt -> /mnt/brick3/vol/.glusterfs/02/a6/02a67715-47ba-4552-ae84-35cd660f38e5failed [File exists] The message "E [MSGID: 113039] [posix.c:3011:posix_open] 0-gvol0-posix: open on /mnt/brick3/vol/.glusterfs/a6/1d/a61d7b2f-d53f-4eb9-be7a-656b0915dbea, flags: 1025 [No such file or directory]" repeated 5 times between [2017-01-01 16:00:00.778962] and [2017-01-01 16:00:00.779308] The message "W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link /mnt/brick3/vol/gps/data/2017-01/02/84.txt -> /mnt/brick3/vol/.glusterfs/02/a6/02a67715-47ba-4552-ae84-35cd660f38e5failed [File exists]" repeated 4 times between [2017-01-01 16:00:04.864163] and [2017-01-01 16:00:04.864286] … ac08 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log … [2017-01-01 16:00:00.371922] W [MSGID: 113096] [posix-handle.c:828:posix_handle_soft] 0-gvol0-posix: symlink ../../e1/b5/e1b5fbec-f861-4c6a-99e8-8d3c28cce76a/02 -> /mnt/brick3/vol/.glusterfs/05/9b/059b889e-5e7f-429c-a208-922226b329a2 failed [File exists] [2017-01-01 16:00:00.778274] W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link /mnt/brick3/vol/gps/up_data/2017-01/02/4.txt -> /mnt/brick3/vol/.glusterfs/a6/1d/a61d7b2f-d53f-4eb9-be7a-656b0915dbeafailed [File exists] [2017-01-01 16:00:01.264791] W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link /mnt/brick3/vol/gps/down_data/2017-01/02/8.txt -> /mnt/brick3/vol/.glusterfs/f4/58/f458e364-1772-4047-8edd-4aa48a096126failed [File exists] [2017-01-01 16:00:04.834138] W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link /mnt/brick3/vol/gps/data/2017-01/02/84.txt -> /mnt/brick3/vol/.glusterfs/02/a6/02a67715-47ba-4552-ae84-35cd660f38e5failed [File exists] The message "W [MSGID: 113096] [posix-handle.c:828:posix_handle_soft] 0-gvol0-posix: symlink ../../e1/b5/e1b5fbec-f861-4c6a-99e8-8d3c28cce76a/02 -> /mnt/brick3/vol/.glusterfs/05/9b/059b889e-5e7f-429c-a208-922226b329a2 failed [File exists]" repeated 5 times between[2017-01-01 16:00:00.371922] and [2017-01-01 16:00:00.372030] The message "W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link /mnt/brick3/vol/gps/up_data/2017-01/02/4.txt -> /mnt/brick3/vol/.glusterfs/a6/1d/a61d7b2f-d53f-4eb9-be7a-656b0915dbeafailed [File exists]" repeated 4 times between [2017-01-01 16:00:00.778274] and [2017-01-01 16:00:00.778341] The message "W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link /mnt/brick3/vol/gps/data/2017-01/02/84.txt -> /mnt/brick3/vol/.glusterfs/02/a6/02a67715-47ba-4552-ae84-35cd660f38e5failed [File exists]" repeated 4 times between [2017-01-01 16:00:04.834138] and [2017-01-01 16:00:04.834314] … ad08 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log … [2017-01-01 16:00:00.399017] W [MSGID: 113096] [posix-handle.c:828:posix_handle_soft] 0-gvol0-posix: symlink ../../92/c6/92c699d6-2670-4547-9845-ec0450107a2e/02 -> /mnt/brick3/vol/.glusterfs/36/3d/363d3c2d-5ccb-497e-9ecc-6eaf5f435026 failed [File exists] [2017-01-01 16:00:02.057190] W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link /mnt/brick3/vol/gps/up_data/2017-01/02/46.txt -> /mnt/brick3/vol/.glusterfs/93/ea/93ea8b72-e208-468d-812c-88d7f80bc204failed [File exists] [2017-01-01 16:00:02.057221] E [MSGID: 113020] [posix.c:2895:posix_create] 0-gvol0-posix: setting gfid on /mnt/brick3/vol/gps/up_data/2017-01/02/46.txt failedThe message "W [MSGID: 113096] [posix-handle.c:828:posix_handle_soft] 0-gvol0-posix: symlink ../../92/c6/92c699d6-2670-4547-9845-ec0450107a2e/02 -> /mnt/brick3/vol/.glusterfs/36/3d/363d3c2d-5ccb-497e-9ecc-6eaf5f435026 failed [File exists]" repeated 5 times between [2017-01-01 16:00:00.399017] and [2017-01-01 16:00:00.399097] … And the “cat: read error: Input/output error” happened at about 2017-0103 10:30, 2:30 at UTC time. Here is the log: ab09 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log … [2017-01-03 02:30:57.684122] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 94bedfd1-619d-402a-9826-67dab7600f43 [2017-01-03 02:30:57.684147] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-gvol0-server: accepted client from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-51-0-0 (version: 3.8.5) [2017-01-03 02:30:58.368750] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-51-0-0 [2017-01-03 02:30:58.368797] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-51-0-0 [2017-01-03 02:46:29.361991] W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link /mnt/brick3/vol/gps/up_data/2017-01/03/91.txt -> /mnt/brick3/vol/.glusterfs/e6/82/e6823e82-04c5-4e81-a3a8-6534d809d3f1failed [File exists] [2017-01-03 02:48:55.302428] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection from ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-51-0-0 [2017-01-03 02:48:55.302487] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-51-0-0 … ac08 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log … [2017-01-03 02:30:57.684317] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 94bedfd1-619d-402a-9826-67dab7600f43 [2017-01-03 02:30:57.684338] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-gvol0-server: accepted client from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-52-0-0 (version: 3.8.5) [2017-01-03 02:30:58.368991] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-52-0-0 [2017-01-03 02:30:58.369016] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-52-0-0 [2017-01-03 02:48:55.302622] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection from ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-52-0-0 [2017-01-03 02:48:55.302657] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-52-0-0 … ad08 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log … [2017-01-03 02:30:57.693935] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 94bedfd1-619d-402a-9826-67dab7600f43 [2017-01-03 02:30:57.693956] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-gvol0-server: accepted client from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-53-0-0 (version: 3.8.5) [2017-01-03 02:30:58.369055] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-53-0-0 [2017-01-03 02:30:58.369074] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-53-0-0 [2017-01-03 02:48:55.302720] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection from ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-53-0-0 [2017-01-03 02:48:55.302773] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-53-0-0 … Likun 发件人: Ravishankar N [mailto:ravishankar@xxxxxxxxxx] On 01/03/2017 09:03 AM, likun wrote:
Regards, Ravi |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
