Hi, we use a bunch of replicated gluster volumes as a backend for our backup. Yesterday I noticed that some synthetic backups failed because of I/O errors. Today I ran "find /gluster_vol -type f | xargs md5sum" and got loads of I/O errors. The brick log file shows the below errors [2017-06-19 13:42:33.554875] E [MSGID: 116020] [bit-rot-stub.c:566:br_stub_check_bad_object] 0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba is a bad object. Returning [2017-06-19 13:42:33.554923] E [MSGID: 116020] [bit-rot-stub.c:566:br_stub_check_bad_object] 0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba is a bad object. Returning [2017-06-19 13:42:33.554931] E [MSGID: 115081] [server-rpc-fops.c:1201:server_fstat_cbk] 0-Server_Standard_05-server: 21461: FSTAT -2 (c75016a9-95c1-4819-b24a-e5d77107c4ba) ==> (Input/output error) [Input/output error] [2017-06-19 13:42:33.554940] E [MSGID: 115081] [server-rpc-fops.c:1201:server_fstat_cbk] 0-Server_Standard_05-server: 21462: FSTAT -2 (c75016a9-95c1-4819-b24a-e5d77107c4ba) ==> (Input/output error) [Input/output error] [2017-06-19 13:42:33.555655] E [MSGID: 116020] [bit-rot-stub.c:566:br_stub_check_bad_object] 0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba is a bad object. Returning [2017-06-19 13:42:33.555697] E [MSGID: 115081] [server-rpc-fops.c:1201:server_fstat_cbk] 0-Server_Standard_05-server: 21463: FSTAT -2 (c75016a9-95c1-4819-b24a-e5d77107c4ba) ==> (Input/output error) [Input/output error] [2017-06-19 13:42:33.555950] E [MSGID: 116020] [bit-rot-stub.c:566:br_stub_check_bad_object] 0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba is a bad object. Returning [2017-06-19 13:42:33.555983] E [MSGID: 115081] [server-rpc-fops.c:1201:server_fstat_cbk] 0-Server_Standard_05-server: 21464: FSTAT -2 (c75016a9-95c1-4819-b24a-e5d77107c4ba) ==> (Input/output error) [Input/output error] [2017-06-19 13:42:33.556604] E [MSGID: 116020] [bit-rot-stub.c:566:br_stub_check_bad_object] 0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba is a bad object. Returning Any idea what's wrong? BTW: I'm running gluster 3.8.12 on Ubuntu 16.04 - 4.4.0-79 many thanks for your help Bernhard _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
