Re: libgfapi failover problem on replica bricks

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




----- Original Message -----
| From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx>
| To: "Humble Chirammal" <hchiramm@xxxxxxxxxx>
| Cc: "Roman" <romeo.r@xxxxxxxxx>, gluster-users@xxxxxxxxxxx, "Niels de Vos" <ndevos@xxxxxxxxxx>
| Sent: Wednesday, August 27, 2014 12:34:22 PM
| Subject: Re:  libgfapi failover problem on replica bricks
| 
| 
| On 08/27/2014 12:24 PM, Roman wrote:
| > root@stor1:~# ls -l /usr/sbin/glfsheal
| > ls: cannot access /usr/sbin/glfsheal: No such file or directory
| > Seems like not.
| Humble,
|       Seems like the binary is still not packaged?

Checking with Kaleb on this.

| 
| 
| >
| >
| > 2014-08-27 9:50 GMT+03:00 Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx
| > <mailto:pkarampu@xxxxxxxxxx>>:
| >
| >
| >     On 08/27/2014 11:53 AM, Roman wrote:
| >>     Okay.
| >>     so here are first results:
| >>
| >>     after I disconnected the first server, I've got this:
| >>
| >>     root@stor2:~# gluster volume heal HA-FAST-PVE1-150G info
| >>     Volume heal failed
| >     Can you check if the following binary is present?
| >     /usr/sbin/glfsheal
| >
| >     Pranith
| >>
| >>
| >>     but
| >>     [2014-08-26 11:45:35.315974] I
| >>     [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
| >>     0-HA-FAST-PVE1-150G-replicate-0:  foreground data self heal  is
| >>     successfully completed,  data self heal from
| >>     HA-FAST-PVE1-150G-client-1  to sinks  HA-FAST-PVE1-150G-client-0,
| >>     with 16108814336 bytes on HA-FAST-PVE1-150G-client-0, 16108814336
| >>     bytes on HA-FAST-PVE1-150G-client-1,  data - Pending matrix:  [ [
| >>     0 0 ] [ 348 0 ] ]  on <gfid:e3ede9c6-28d6-4755-841a-d8329e42ccc4>
| >>
| >>     something wrong during upgrade?
| >>
| >>     I've got two VM-s on different volumes: one with HD on and other
| >>     with HD off.
| >>     Both survived the outage and both seemed synced.
| >>
| >>     but today I've found kind of a bug with log rotation.
| >>
| >>     logs rotated both on server and client sides, but logs are being
| >>     written in *.log.1 file :)
| >>
| >>     /var/log/glusterfs/mnt-pve-HA-MED-PVE1-1T.log.1
| >>     /var/log/glusterfs/glustershd.log.1
| >>
| >>     such behavior came after upgrade.
| >>
| >>     logrotate.d conf files include the HUP for gluster pid-s.
| >>
| >>     client:
| >>     /var/log/glusterfs/*.log {
| >>             daily
| >>             rotate 7
| >>             delaycompress
| >>             compress
| >>             notifempty
| >>             missingok
| >>             postrotate
| >>                     [ ! -f /var/run/glusterd.pid ] || kill -HUP `cat
| >>     /var/run/glusterd.pid`
| >>             endscript
| >>     }
| >>
| >>     but I'm not able to ls the pid on client side (should it be
| >>     there?) :(
| >>
| >>     and servers:
| >>     /var/log/glusterfs/*.log {
| >>             daily
| >>             rotate 7
| >>             delaycompress
| >>             compress
| >>             notifempty
| >>             missingok
| >>             postrotate
| >>                     [ ! -f /var/run/glusterd.pid ] || kill -HUP `cat
| >>     /var/run/glusterd.pid`
| >>             endscript
| >>     }
| >>
| >>
| >>     /var/log/glusterfs/*/*.log {
| >>             daily
| >>             rotate 7
| >>             delaycompress
| >>             compress
| >>             notifempty
| >>             missingok
| >>             copytruncate
| >>             postrotate
| >>                     [ ! -f /var/run/glusterd.pid ] || kill -HUP `cat
| >>     /var/run/glusterd.pid`
| >>             endscript
| >>     }
| >>
| >>     I do have /var/run/glusterd.pid on server side.
| >>
| >>     Should I change something? Logrotation seems to be broken.
| >>
| >>
| >>
| >>
| >>
| >>
| >>     2014-08-26 9:29 GMT+03:00 Pranith Kumar Karampuri
| >>     <pkarampu@xxxxxxxxxx <mailto:pkarampu@xxxxxxxxxx>>:
| >>
| >>
| >>         On 08/26/2014 11:55 AM, Roman wrote:
| >>>         Hello all again!
| >>>         I'm back from vacation and I'm pretty happy with 3.5.2
| >>>         available for wheezy. Thanks! Just made my updates.
| >>>         For 3.5.2 do I still have to set cluster.self-heal-daemon to
| >>>         off?
| >>         Welcome back :-). If you set it to off, the test case you
| >>         execute will work(Validate please :-) ). But we need to test
| >>         it with self-heal-daemon 'on' and fix any bugs if the test
| >>         case does not work?
| >>
| >>         Pranith.
| >>
| >>>
| >>>
| >>>         2014-08-06 12:49 GMT+03:00 Humble Chirammal
| >>>         <hchiramm@xxxxxxxxxx <mailto:hchiramm@xxxxxxxxxx>>:
| >>>
| >>>
| >>>
| >>>
| >>>             ----- Original Message -----
| >>>             | From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>
| >>>             | To: "Roman" <romeo.r@xxxxxxxxx <mailto:romeo.r@xxxxxxxxx>>
| >>>             | Cc: gluster-users@xxxxxxxxxxx
| >>>             <mailto:gluster-users@xxxxxxxxxxx>, "Niels de Vos"
| >>>             <ndevos@xxxxxxxxxx <mailto:ndevos@xxxxxxxxxx>>, "Humble
| >>>             Chirammal" <hchiramm@xxxxxxxxxx
| >>>             <mailto:hchiramm@xxxxxxxxxx>>
| >>>             | Sent: Wednesday, August 6, 2014 12:09:57 PM
| >>>             | Subject: Re:  libgfapi failover problem
| >>>             on replica bricks
| >>>             |
| >>>             | Roman,
| >>>             |      The file went into split-brain. I think we should
| >>>             do these tests
| >>>             | with 3.5.2. Where monitoring the heals is easier. Let
| >>>             me also come up
| >>>             | with a document about how to do this testing you are
| >>>             trying to do.
| >>>             |
| >>>             | Humble/Niels,
| >>>             |      Do we have debs available for 3.5.2? In 3.5.1
| >>>             there was packaging
| >>>             | issue where /usr/bin/glfsheal is not packaged along
| >>>             with the deb. I
| >>>             | think that should be fixed now as well?
| >>>             |
| >>>             Pranith,
| >>>
| >>>             The 3.5.2 packages for debian is not available yet. We
| >>>             are co-ordinating internally to get it processed.
| >>>             I will update the list once its available.
| >>>
| >>>             --Humble
| >>>             |
| >>>             | On 08/06/2014 11:52 AM, Roman wrote:
| >>>             | > good morning,
| >>>             | >
| >>>             | > root@stor1:~# getfattr -d -m. -e hex
| >>>             | > /exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | > getfattr: Removing leading '/' from absolute path names
| >>>             | > # file:
| >>>             exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >
| >>>             trusted.afr.HA-fast-150G-PVE1-client-0=0x000000000000000000000000
| >>>             | >
| >>>             trusted.afr.HA-fast-150G-PVE1-client-1=0x000001320000000000000000
| >>>             | > trusted.gfid=0x23c79523075a4158bea38078da570449
| >>>             | >
| >>>             | > getfattr: Removing leading '/' from absolute path names
| >>>             | > # file:
| >>>             exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >
| >>>             trusted.afr.HA-fast-150G-PVE1-client-0=0x000000040000000000000000
| >>>             | >
| >>>             trusted.afr.HA-fast-150G-PVE1-client-1=0x000000000000000000000000
| >>>             | > trusted.gfid=0x23c79523075a4158bea38078da570449
| >>>             | >
| >>>             | >
| >>>             | >
| >>>             | > 2014-08-06 9:20 GMT+03:00 Pranith Kumar Karampuri
| >>>             <pkarampu@xxxxxxxxxx <mailto:pkarampu@xxxxxxxxxx>
| >>>             | > <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >
| >>>             | >
| >>>             | >     On 08/06/2014 11:30 AM, Roman wrote:
| >>>             | >>     Also, this time files are not the same!
| >>>             | >>
| >>>             | >>     root@stor1:~# md5sum
| >>>             | >>  /exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >>  32411360c53116b96a059f17306caeda
| >>>             | >> /exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >>
| >>>             | >>     root@stor2:~# md5sum
| >>>             | >>  /exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >>  65b8a6031bcb6f5fb3a11cb1e8b1c9c9
| >>>             | >> /exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >     What is the getfattr output?
| >>>             | >
| >>>             | >     Pranith
| >>>             | >
| >>>             | >>
| >>>             | >>
| >>>             | >>     2014-08-05 16:33 GMT+03:00 Roman
| >>>             <romeo.r@xxxxxxxxx <mailto:romeo.r@xxxxxxxxx>
| >>>             | >>     <mailto:romeo.r@xxxxxxxxx
| >>>             <mailto:romeo.r@xxxxxxxxx>>>:
| >>>             | >>
| >>>             | >>         Nope, it is not working. But this time it
| >>>             went a bit other way
| >>>             | >>
| >>>             | >>  root@gluster-client:~# dmesg
| >>>             | >>         Segmentation fault
| >>>             | >>
| >>>             | >>
| >>>             | >>         I was not able even to start the VM after I
| >>>             done the tests
| >>>             | >>
| >>>             | >>         Could not read qcow2 header: Operation not
| >>>             permitted
| >>>             | >>
| >>>             | >>         And it seems, it never starts to sync files
| >>>             after first
| >>>             | >>         disconnect. VM survives first disconnect,
| >>>             but not second (I
| >>>             | >>         waited around 30 minutes). Also, I've
| >>>             | >>         got network.ping-timeout: 2 in volume
| >>>             settings, but logs
| >>>             | >>         react on first disconnect around 30
| >>>             seconds. Second was
| >>>             | >>         faster, 2 seconds.
| >>>             | >>
| >>>             | >>         Reaction was different also:
| >>>             | >>
| >>>             | >>         slower one:
| >>>             | >>         [2014-08-05 13:26:19.558435] W
| >>>             [socket.c:514:__socket_rwv]
| >>>             | >>         0-glusterfs: readv failed (Connection timed
| >>>             out)
| >>>             | >>         [2014-08-05 13:26:19.558485] W
| >>>             | >>  [socket.c:1962:__socket_proto_state_machine]
| >>>             0-glusterfs:
| >>>             | >>         reading from socket failed. Error
| >>>             (Connection timed out),
| >>>             | >>         peer (10.250.0.1:24007
| >>>             <http://10.250.0.1:24007> <http://10.250.0.1:24007>)
| >>>             | >>         [2014-08-05 13:26:21.281426] W
| >>>             [socket.c:514:__socket_rwv]
| >>>             | >>  0-HA-fast-150G-PVE1-client-0: readv failed
| >>>             (Connection timed out)
| >>>             | >>         [2014-08-05 13:26:21.281474] W
| >>>             | >>  [socket.c:1962:__socket_proto_state_machine]
| >>>             | >>  0-HA-fast-150G-PVE1-client-0: reading from socket
| >>>             failed.
| >>>             | >>         Error (Connection timed out), peer
| >>>             (10.250.0.1:49153 <http://10.250.0.1:49153>
| >>>             | >>         <http://10.250.0.1:49153>)
| >>>             | >>         [2014-08-05 13:26:21.281507] I
| >>>             | >>  [client.c:2098:client_rpc_notify]
| >>>             | >>  0-HA-fast-150G-PVE1-client-0: disconnected
| >>>             | >>
| >>>             | >>         the fast one:
| >>>             | >>         2014-08-05 12:52:44.607389] C
| >>>             | >>  [client-handshake.c:127:rpc_client_ping_timer_expired]
| >>>             | >>  0-HA-fast-150G-PVE1-client-1: server
| >>>             10.250.0.2:49153 <http://10.250.0.2:49153>
| >>>             | >>         <http://10.250.0.2:49153> has not responded
| >>>             in the last 2
| >>>             | >>         seconds, disconnecting.
| >>>             | >>         [2014-08-05 12:52:44.607491] W
| >>>             [socket.c:514:__socket_rwv]
| >>>             | >>  0-HA-fast-150G-PVE1-client-1: readv failed (No
| >>>             data available)
| >>>             | >>         [2014-08-05 12:52:44.607585] E
| >>>             | >>  [rpc-clnt.c:368:saved_frames_unwind]
| >>>             | >>
| >>>              (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xf8)
| >>>             | >>  [0x7fcb1b4b0558]
| >>>             | >>
| >>>              (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3)
| >>>             | >>  [0x7fcb1b4aea63]
| >>>             | >>
| >>>              (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)
| >>>             | >>  [0x7fcb1b4ae97e]))) 0-HA-fast-150G-PVE1-client-1:
| >>>             forced
| >>>             | >>         unwinding frame type(GlusterFS 3.3)
| >>>             op(LOOKUP(27)) called at
| >>>             | >>         2014-08-05 12:52:42.463881 (xid=0x381883x)
| >>>             | >>         [2014-08-05 12:52:44.607604] W
| >>>             | >>  [client-rpc-fops.c:2624:client3_3_lookup_cbk]
| >>>             | >>  0-HA-fast-150G-PVE1-client-1: remote operation failed:
| >>>             | >>         Transport endpoint is not connected. Path: /
| >>>             | >>  (00000000-0000-0000-0000-000000000001)
| >>>             | >>         [2014-08-05 12:52:44.607736] E
| >>>             | >>  [rpc-clnt.c:368:saved_frames_unwind]
| >>>             | >>
| >>>              (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xf8)
| >>>             | >>  [0x7fcb1b4b0558]
| >>>             | >>
| >>>              (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3)
| >>>             | >>  [0x7fcb1b4aea63]
| >>>             | >>
| >>>              (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)
| >>>             | >>  [0x7fcb1b4ae97e]))) 0-HA-fast-150G-PVE1-client-1:
| >>>             forced
| >>>             | >>         unwinding frame type(GlusterFS Handshake)
| >>>             op(PING(3)) called
| >>>             | >>         at 2014-08-05 12:52:42.463891 (xid=0x381884x)
| >>>             | >>         [2014-08-05 12:52:44.607753] W
| >>>             | >>  [client-handshake.c:276:client_ping_cbk]
| >>>             | >>  0-HA-fast-150G-PVE1-client-1: timer must have expired
| >>>             | >>         [2014-08-05 12:52:44.607776] I
| >>>             | >>  [client.c:2098:client_rpc_notify]
| >>>             | >>  0-HA-fast-150G-PVE1-client-1: disconnected
| >>>             | >>
| >>>             | >>
| >>>             | >>
| >>>             | >>         I've got SSD disks (just for an info).
| >>>             | >>         Should I go and give a try for 3.5.2?
| >>>             | >>
| >>>             | >>
| >>>             | >>
| >>>             | >>         2014-08-05 13:06 GMT+03:00 Pranith Kumar
| >>>             Karampuri
| >>>             | >>         <pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx> <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>
| >>>             | >>             reply along with gluster-users please
| >>>             :-). May be you are
| >>>             | >>             hitting 'reply' instead of 'reply all'?
| >>>             | >>
| >>>             | >>             Pranith
| >>>             | >>
| >>>             | >>             On 08/05/2014 03:35 PM, Roman wrote:
| >>>             | >>>             To make sure and clean, I've created
| >>>             another VM with raw
| >>>             | >>>             format and goint to repeat those
| >>>             steps. So now I've got
| >>>             | >>>             two VM-s one with qcow2 format and
| >>>             other with raw
| >>>             | >>>             format. I will send another e-mail
| >>>             shortly.
| >>>             | >>>
| >>>             | >>>
| >>>             | >>>  2014-08-05 13:01 GMT+03:00 Pranith Kumar Karampuri
| >>>             | >>>             <pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx> <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>>
| >>>             | >>>
| >>>             | >>>  On 08/05/2014 03:07 PM, Roman wrote:
| >>>             | >>>>    really, seems like the same file
| >>>             | >>>>
| >>>             | >>>>    stor1:
| >>>             | >>>>  a951641c5230472929836f9fcede6b04
| >>>             | >>>>
| >>>             /exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >>>>
| >>>             | >>>>    stor2:
| >>>             | >>>>  a951641c5230472929836f9fcede6b04
| >>>             | >>>>
| >>>             /exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >>>>
| >>>             | >>>>
| >>>             | >>>>    one thing I've seen from logs, that somehow
| >>>             proxmox
| >>>             | >>>>    VE is connecting with wrong version to servers?
| >>>             | >>>>    [2014-08-05 09:23:45.218550] I
| >>>             | >>>>
| >>>              [client-handshake.c:1659:select_server_supported_programs]
| >>>             | >>>>    0-HA-fast-150G-PVE1-client-0: Using Program
| >>>             | >>>>    GlusterFS 3.3, Num (1298437), Version (330)
| >>>             | >>>  It is the rpc (over the network data structures)
| >>>             | >>>  version, which is not changed at all from 3.3 so
| >>>             | >>>  thats not a problem. So what is the conclusion? Is
| >>>             | >>>  your test case working now or not?
| >>>             | >>>
| >>>             | >>>  Pranith
| >>>             | >>>
| >>>             | >>>>    but if I issue:
| >>>             | >>>>    root@pve1:~# glusterfs -V
| >>>             | >>>>    glusterfs 3.4.4 built on Jun 28 2014 03:44:57
| >>>             | >>>>    seems ok.
| >>>             | >>>>
| >>>             | >>>>    server  use 3.4.4 meanwhile
| >>>             | >>>>    [2014-08-05 09:23:45.117875] I
| >>>             | >>>>  [server-handshake.c:567:server_setvolume]
| >>>             | >>>>    0-HA-fast-150G-PVE1-server: accepted client from
| >>>             | >>>>
| >>>              stor1-9004-2014/08/05-09:23:45:93538-HA-fast-150G-PVE1-client-1-0
| >>>             | >>>>    (version: 3.4.4)
| >>>             | >>>>    [2014-08-05 09:23:49.103035] I
| >>>             | >>>>  [server-handshake.c:567:server_setvolume]
| >>>             | >>>>    0-HA-fast-150G-PVE1-server: accepted client from
| >>>             | >>>>
| >>>              stor1-8998-2014/08/05-09:23:45:89883-HA-fast-150G-PVE1-client-0-0
| >>>             | >>>>    (version: 3.4.4)
| >>>             | >>>>
| >>>             | >>>>    if this could be the reason, of course.
| >>>             | >>>>    I did restart the Proxmox VE yesterday (just
| >>>             for an
| >>>             | >>>>    information)
| >>>             | >>>>
| >>>             | >>>>
| >>>             | >>>>
| >>>             | >>>>
| >>>             | >>>>
| >>>             | >>>>    2014-08-05 12:30 GMT+03:00 Pranith Kumar Karampuri
| >>>             | >>>>  <pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx> <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>>>
| >>>             | >>>>
| >>>             | >>>>        On 08/05/2014 02:33 PM, Roman wrote:
| >>>             | >>>>>            Waited long enough for now, still
| >>>             different
| >>>             | >>>>>            sizes and no logs about healing :(
| >>>             | >>>>>
| >>>             | >>>>>            stor1
| >>>             | >>>>>            # file:
| >>>             | >>>>>
| >>>              exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >>>>>
| >>>              trusted.afr.HA-fast-150G-PVE1-client-0=0x000000000000000000000000
| >>>             | >>>>>
| >>>              trusted.afr.HA-fast-150G-PVE1-client-1=0x000000000000000000000000
| >>>             | >>>>>  trusted.gfid=0xf10ad81b58484bcd9b385a36a207f921
| >>>             | >>>>>
| >>>             | >>>>>            root@stor1:~# du -sh
| >>>             | >>>>>  /exports/fast-test/150G/images/127/
| >>>             | >>>>>            1.2G /exports/fast-test/150G/images/127/
| >>>             | >>>>>
| >>>             | >>>>>
| >>>             | >>>>>            stor2
| >>>             | >>>>>            # file:
| >>>             | >>>>>
| >>>              exports/fast-test/150G/images/127/vm-127-disk-1.qcow2
| >>>             | >>>>>
| >>>              trusted.afr.HA-fast-150G-PVE1-client-0=0x000000000000000000000000
| >>>             | >>>>>
| >>>              trusted.afr.HA-fast-150G-PVE1-client-1=0x000000000000000000000000
| >>>             | >>>>>  trusted.gfid=0xf10ad81b58484bcd9b385a36a207f921
| >>>             | >>>>>
| >>>             | >>>>>
| >>>             | >>>>>            root@stor2:~# du -sh
| >>>             | >>>>>  /exports/fast-test/150G/images/127/
| >>>             | >>>>>            1.4G /exports/fast-test/150G/images/127/
| >>>             | >>>>        According to the changelogs, the file doesn't
| >>>             | >>>>        need any healing. Could you stop the
| >>>             operations
| >>>             | >>>>        on the VMs and take md5sum on both these
| >>>             machines?
| >>>             | >>>>
| >>>             | >>>>        Pranith
| >>>             | >>>>
| >>>             | >>>>>
| >>>             | >>>>>
| >>>             | >>>>>
| >>>             | >>>>>
| >>>             | >>>>>            2014-08-05 11:49 GMT+03:00 Pranith Kumar
| >>>             | >>>>>            Karampuri <pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>
| >>>             | >>>>>        <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>>>>
| >>>             | >>>>>
| >>>             | >>>>>                On 08/05/2014 02:06 PM, Roman wrote:
| >>>             | >>>>>>                    Well, it seems like it
| >>>             doesn't see the
| >>>             | >>>>>>                    changes were made to the
| >>>             volume ? I
| >>>             | >>>>>>                    created two files 200 and
| >>>             100 MB (from
| >>>             | >>>>>>                    /dev/zero) after I
| >>>             disconnected the first
| >>>             | >>>>>>                    brick. Then connected it
| >>>             back and got
| >>>             | >>>>>>                    these logs:
| >>>             | >>>>>>
| >>>             | >>>>>>                    [2014-08-05 08:30:37.830150] I
| >>>             | >>>>>>  [glusterfsd-mgmt.c:1584:mgmt_getspec_cbk]
| >>>             | >>>>>>                    0-glusterfs: No change in
| >>>             volfile, continuing
| >>>             | >>>>>>                    [2014-08-05 08:30:37.830207] I
| >>>             | >>>>>>  [rpc-clnt.c:1676:rpc_clnt_reconfig]
| >>>             | >>>>>>  0-HA-fast-150G-PVE1-client-0: changing
| >>>             | >>>>>>                    port to 49153 (from 0)
| >>>             | >>>>>>                    [2014-08-05 08:30:37.830239] W
| >>>             | >>>>>>  [socket.c:514:__socket_rwv]
| >>>             | >>>>>>  0-HA-fast-150G-PVE1-client-0: readv
| >>>             | >>>>>>                    failed (No data available)
| >>>             | >>>>>>                    [2014-08-05 08:30:37.831024] I
| >>>             | >>>>>>
| >>>              [client-handshake.c:1659:select_server_supported_programs]
| >>>             | >>>>>>  0-HA-fast-150G-PVE1-client-0: Using
| >>>             | >>>>>>                    Program GlusterFS 3.3, Num
| >>>             (1298437),
| >>>             | >>>>>>                    Version (330)
| >>>             | >>>>>>                    [2014-08-05 08:30:37.831375] I
| >>>             | >>>>>>  [client-handshake.c:1456:client_setvolume_cbk]
| >>>             | >>>>>>  0-HA-fast-150G-PVE1-client-0: Connected
| >>>             | >>>>>>                    to 10.250.0.1:49153
| >>>             <http://10.250.0.1:49153>
| >>>             | >>>>>>                <http://10.250.0.1:49153>,
| >>>             attached to
| >>>             | >>>>>>                      remote volume
| >>>             '/exports/fast-test/150G'.
| >>>             | >>>>>>                    [2014-08-05 08:30:37.831394] I
| >>>             | >>>>>>  [client-handshake.c:1468:client_setvolume_cbk]
| >>>             | >>>>>>  0-HA-fast-150G-PVE1-client-0: Server and
| >>>             | >>>>>>                    Client lk-version numbers
| >>>             are not same,
| >>>             | >>>>>>                    reopening the fds
| >>>             | >>>>>>                    [2014-08-05 08:30:37.831566] I
| >>>             | >>>>>>  [client-handshake.c:450:client_set_lk_version_cbk]
| >>>             | >>>>>>  0-HA-fast-150G-PVE1-client-0: Server lk
| >>>             | >>>>>>                    version = 1
| >>>             | >>>>>>
| >>>             | >>>>>>
| >>>             | >>>>>>                    [2014-08-05 08:30:37.830150] I
| >>>             | >>>>>>  [glusterfsd-mgmt.c:1584:mgmt_getspec_cbk]
| >>>             | >>>>>>                    0-glusterfs: No change in
| >>>             volfile, continuing
| >>>             | >>>>>>                    this line seems weird to me tbh.
| >>>             | >>>>>>                    I do not see any traffic on
| >>>             switch
| >>>             | >>>>>>                    interfaces between gluster
| >>>             servers, which
| >>>             | >>>>>>                    means, there is no syncing
| >>>             between them.
| >>>             | >>>>>>                    I tried to ls -l the files
| >>>             on the client
| >>>             | >>>>>>                    and servers to trigger the
| >>>             healing, but
| >>>             | >>>>>>                    seems like no success.
| >>>             Should I wait more?
| >>>             | >>>>>                Yes, it should take around 10-15
| >>>             minutes.
| >>>             | >>>>>                Could you provide 'getfattr -d
| >>>             -m. -e hex
| >>>             | >>>>>  <file-on-brick>' on both the bricks.
| >>>             | >>>>>
| >>>             | >>>>>                Pranith
| >>>             | >>>>>
| >>>             | >>>>>>
| >>>             | >>>>>>
| >>>             | >>>>>>                    2014-08-05 11:25 GMT+03:00
| >>>             Pranith Kumar
| >>>             | >>>>>>                    Karampuri
| >>>             <pkarampu@xxxxxxxxxx <mailto:pkarampu@xxxxxxxxxx>
| >>>             | >>>>>>                <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>>>>>
| >>>             | >>>>>>
| >>>             | >>>>>>                        On 08/05/2014 01:10 PM,
| >>>             Roman wrote:
| >>>             | >>>>>>>                            Ahha! For some
| >>>             reason I was not able
| >>>             | >>>>>>>                            to start the VM
| >>>             anymore, Proxmox VE
| >>>             | >>>>>>>                            told me, that it is
| >>>             not able to read
| >>>             | >>>>>>>                            the qcow2 header
| >>>             due to permission
| >>>             | >>>>>>>                            is denied for some
| >>>             reason. So I just
| >>>             | >>>>>>>  deleted that file and created a new
| >>>             | >>>>>>>                            VM. And the nex
| >>>             message I've got was
| >>>             | >>>>>>>                            this:
| >>>             | >>>>>>                        Seems like these are the
| >>>             messages
| >>>             | >>>>>>                        where you took down the
| >>>             bricks before
| >>>             | >>>>>>  self-heal. Could you restart the run
| >>>             | >>>>>>                        waiting for self-heals
| >>>             to complete
| >>>             | >>>>>>                        before taking down the
| >>>             next brick?
| >>>             | >>>>>>
| >>>             | >>>>>>                        Pranith
| >>>             | >>>>>>
| >>>             | >>>>>>>
| >>>             | >>>>>>>
| >>>             | >>>>>>>  [2014-08-05 07:31:25.663412] E
| >>>             | >>>>>>>
| >>>              [afr-self-heal-common.c:197:afr_sh_print_split_brain_log]
| >>>             | >>>>>>>  0-HA-fast-150G-PVE1-replicate-0:
| >>>             | >>>>>>>  Unable to self-heal contents of
| >>>             | >>>>>>>  '/images/124/vm-124-disk-1.qcow2'
| >>>             | >>>>>>>  (possible split-brain). Please
| >>>             | >>>>>>>  delete the file from all but the
| >>>             | >>>>>>>  preferred subvolume.- Pending
| >>>             | >>>>>>>  matrix:  [ [ 0 60 ] [ 11 0 ] ]
| >>>             | >>>>>>>  [2014-08-05 07:31:25.663955] E
| >>>             | >>>>>>>
| >>>              [afr-self-heal-common.c:2262:afr_self_heal_completion_cbk]
| >>>             | >>>>>>>  0-HA-fast-150G-PVE1-replicate-0:
| >>>             | >>>>>>>  background  data self-heal failed on
| >>>             | >>>>>>>  /images/124/vm-124-disk-1.qcow2
| >>>             | >>>>>>>
| >>>             | >>>>>>>
| >>>             | >>>>>>>
| >>>             | >>>>>>>  2014-08-05 10:13 GMT+03:00 Pranith
| >>>             | >>>>>>>                            Kumar Karampuri
| >>>             <pkarampu@xxxxxxxxxx <mailto:pkarampu@xxxxxxxxxx>
| >>>             | >>>>>>>                      
| >>>              <mailto:pkarampu@xxxxxxxxxx <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>>>>>>
| >>>             | >>>>>>>                                I just
| >>>             responded to your earlier
| >>>             | >>>>>>>  mail about how the log looks.
| >>>             | >>>>>>>  The log comes on the mount's logfile
| >>>             | >>>>>>>
| >>>             | >>>>>>>  Pranith
| >>>             | >>>>>>>
| >>>             | >>>>>>>  On 08/05/2014 12:41 PM, Roman wrote:
| >>>             | >>>>>>>>  Ok, so I've waited enough, I
| >>>             | >>>>>>>>  think. Had no any traffic on
| >>>             | >>>>>>>>  switch ports between servers.
| >>>             | >>>>>>>>  Could not find any suitable log
| >>>             | >>>>>>>>  message about completed
| >>>             | >>>>>>>>  self-heal (waited about 30
| >>>             | >>>>>>>>  minutes). Plugged out the other
| >>>             | >>>>>>>>  server's UTP cable this time
| >>>             | >>>>>>>>  and got in the same situation:
| >>>             | >>>>>>>>  root@gluster-test1:~# cat
| >>>             | >>>>>>>>  /var/log/dmesg
| >>>             | >>>>>>>>  -bash: /bin/cat: Input/output error
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>  brick logs:
| >>>             | >>>>>>>>  [2014-08-05 07:09:03.005474] I
| >>>             | >>>>>>>>  [server.c:762:server_rpc_notify]
| >>>             | >>>>>>>>  0-HA-fast-150G-PVE1-server:
| >>>             | >>>>>>>>  disconnecting connectionfrom
| >>>             | >>>>>>>>
| >>>              pve1-27649-2014/08/04-13:27:54:720789-HA-fast-150G-PVE1-client-0-0
| >>>             | >>>>>>>>  [2014-08-05 07:09:03.005530] I
| >>>             | >>>>>>>>  [server-helpers.c:729:server_connection_put]
| >>>             | >>>>>>>>  0-HA-fast-150G-PVE1-server:
| >>>             | >>>>>>>>  Shutting down connection
| >>>             | >>>>>>>>
| >>>              pve1-27649-2014/08/04-13:27:54:720789-HA-fast-150G-PVE1-client-0-0
| >>>             | >>>>>>>>  [2014-08-05 07:09:03.005560] I
| >>>             | >>>>>>>>  [server-helpers.c:463:do_fd_cleanup]
| >>>             | >>>>>>>>  0-HA-fast-150G-PVE1-server: fd
| >>>             | >>>>>>>>  cleanup on
| >>>             | >>>>>>>>  /images/124/vm-124-disk-1.qcow2
| >>>             | >>>>>>>>  [2014-08-05 07:09:03.005797] I
| >>>             | >>>>>>>>  [server-helpers.c:617:server_connection_destroy]
| >>>             | >>>>>>>>  0-HA-fast-150G-PVE1-server:
| >>>             | >>>>>>>>  destroyed connection of
| >>>             | >>>>>>>>
| >>>              pve1-27649-2014/08/04-13:27:54:720789-HA-fast-150G-PVE1-client-0-0
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>  2014-08-05 9:53 GMT+03:00
| >>>             | >>>>>>>>  Pranith Kumar Karampuri
| >>>             | >>>>>>>>  <pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>
| >>>             | >>>>>>>>  <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>    Do you think it is possible
| >>>             | >>>>>>>>    for you to do these tests
| >>>             | >>>>>>>>    on the latest version
| >>>             | >>>>>>>>    3.5.2? 'gluster volume heal
| >>>             | >>>>>>>>    <volname> info' would give
| >>>             | >>>>>>>>    you that information in
| >>>             | >>>>>>>>    versions > 3.5.1.
| >>>             | >>>>>>>>    Otherwise you will have to
| >>>             | >>>>>>>>    check it from either the
| >>>             | >>>>>>>>    logs, there will be
| >>>             | >>>>>>>>    self-heal completed message
| >>>             | >>>>>>>>    on the mount logs (or) by
| >>>             | >>>>>>>>    observing 'getfattr -d -m.
| >>>             | >>>>>>>>    -e hex <image-file-on-bricks>'
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>    Pranith
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>    On 08/05/2014 12:09 PM,
| >>>             | >>>>>>>>    Roman wrote:
| >>>             | >>>>>>>>>    Ok, I understand. I will
| >>>             | >>>>>>>>>    try this shortly.
| >>>             | >>>>>>>>>    How can I be sure, that
| >>>             | >>>>>>>>>    healing process is done,
| >>>             | >>>>>>>>>    if I am not able to see
| >>>             | >>>>>>>>>    its status?
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>    2014-08-05 9:30 GMT+03:00
| >>>             | >>>>>>>>>    Pranith Kumar Karampuri
| >>>             | >>>>>>>>>    <pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>
| >>>             | >>>>>>>>>  <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>        Mounts will do the
| >>>             | >>>>>>>>>        healing, not the
| >>>             | >>>>>>>>>        self-heal-daemon. The
| >>>             | >>>>>>>>>        problem I feel is that
| >>>             | >>>>>>>>>        whichever process does
| >>>             | >>>>>>>>>        the healing has the
| >>>             | >>>>>>>>>        latest information
| >>>             | >>>>>>>>>        about the good bricks
| >>>             | >>>>>>>>>        in this usecase. Since
| >>>             | >>>>>>>>>        for VM usecase, mounts
| >>>             | >>>>>>>>>        should have the latest
| >>>             | >>>>>>>>>        information, we should
| >>>             | >>>>>>>>>        let the mounts do the
| >>>             | >>>>>>>>>        healing. If the mount
| >>>             | >>>>>>>>>        accesses the VM image
| >>>             | >>>>>>>>>        either by someone
| >>>             | >>>>>>>>>        doing operations
| >>>             | >>>>>>>>>        inside the VM or
| >>>             | >>>>>>>>>        explicit stat on the
| >>>             | >>>>>>>>>        file it should do the
| >>>             | >>>>>>>>>        healing.
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>        Pranith.
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>        On 08/05/2014 10:39
| >>>             | >>>>>>>>>        AM, Roman wrote:
| >>>             | >>>>>>>>>>        Hmmm, you told me to
| >>>             | >>>>>>>>>>        turn it off. Did I
| >>>             | >>>>>>>>>>        understood something
| >>>             | >>>>>>>>>>        wrong? After I issued
| >>>             | >>>>>>>>>>        the command you've
| >>>             | >>>>>>>>>>        sent me, I was not
| >>>             | >>>>>>>>>>        able to watch the
| >>>             | >>>>>>>>>>        healing process, it
| >>>             | >>>>>>>>>>        said, it won't be
| >>>             | >>>>>>>>>>        healed, becouse its
| >>>             | >>>>>>>>>>        turned off.
| >>>             | >>>>>>>>>>
| >>>             | >>>>>>>>>>
| >>>             | >>>>>>>>>>        2014-08-05 5:39
| >>>             | >>>>>>>>>>        GMT+03:00 Pranith
| >>>             | >>>>>>>>>>        Kumar Karampuri
| >>>             | >>>>>>>>>>        <pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>
| >>>             | >>>>>>>>>>    <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>>>>>>>>>
| >>>             | >>>>>>>>>>            You didn't
| >>>             | >>>>>>>>>>            mention anything
| >>>             | >>>>>>>>>>            about
| >>>             | >>>>>>>>>>            self-healing. Did
| >>>             | >>>>>>>>>>            you wait until
| >>>             | >>>>>>>>>>            the self-heal is
| >>>             | >>>>>>>>>>            complete?
| >>>             | >>>>>>>>>>
| >>>             | >>>>>>>>>>            Pranith
| >>>             | >>>>>>>>>>
| >>>             | >>>>>>>>>>            On 08/04/2014
| >>>             | >>>>>>>>>>            05:49 PM, Roman
| >>>             | >>>>>>>>>>            wrote:
| >>>             | >>>>>>>>>>>            Hi!
| >>>             | >>>>>>>>>>>            Result is pretty
| >>>             | >>>>>>>>>>>            same. I set the
| >>>             | >>>>>>>>>>>            switch port down
| >>>             | >>>>>>>>>>>            for 1st server,
| >>>             | >>>>>>>>>>>            it was ok. Then
| >>>             | >>>>>>>>>>>            set it up back
| >>>             | >>>>>>>>>>>            and set other
| >>>             | >>>>>>>>>>>            server's port
| >>>             | >>>>>>>>>>>            off. and it
| >>>             | >>>>>>>>>>>            triggered IO
| >>>             | >>>>>>>>>>>            error on two
| >>>             | >>>>>>>>>>>            virtual
| >>>             | >>>>>>>>>>>            machines: one
| >>>             | >>>>>>>>>>>            with local root
| >>>             | >>>>>>>>>>>            FS but network
| >>>             | >>>>>>>>>>>            mounted storage.
| >>>             | >>>>>>>>>>>            and other with
| >>>             | >>>>>>>>>>>            network root FS.
| >>>             | >>>>>>>>>>>            1st gave an
| >>>             | >>>>>>>>>>>            error on copying
| >>>             | >>>>>>>>>>>            to or from the
| >>>             | >>>>>>>>>>>            mounted network
| >>>             | >>>>>>>>>>>            disk, other just
| >>>             | >>>>>>>>>>>            gave me an error
| >>>             | >>>>>>>>>>>            for even reading
| >>>             | >>>>>>>>>>>            log.files.
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>            cat:
| >>>             | >>>>>>>>>>>  /var/log/alternatives.log:
| >>>             | >>>>>>>>>>>            Input/output error
| >>>             | >>>>>>>>>>>            then I reset the
| >>>             | >>>>>>>>>>>            kvm VM and it
| >>>             | >>>>>>>>>>>            said me, there
| >>>             | >>>>>>>>>>>            is no boot
| >>>             | >>>>>>>>>>>            device. Next I
| >>>             | >>>>>>>>>>>            virtually
| >>>             | >>>>>>>>>>>            powered it off
| >>>             | >>>>>>>>>>>            and then back on
| >>>             | >>>>>>>>>>>            and it has booted.
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>            By the way, did
| >>>             | >>>>>>>>>>>            I have to
| >>>             | >>>>>>>>>>>            start/stop volume?
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>            >> Could you do
| >>>             | >>>>>>>>>>>            the following
| >>>             | >>>>>>>>>>>            and test it again?
| >>>             | >>>>>>>>>>>            >> gluster volume
| >>>             | >>>>>>>>>>>            set <volname>
| >>>             | >>>>>>>>>>>  cluster.self-heal-daemon
| >>>             | >>>>>>>>>>>            off
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>            >>Pranith
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>            2014-08-04 14:10
| >>>             | >>>>>>>>>>>            GMT+03:00
| >>>             | >>>>>>>>>>>            Pranith Kumar
| >>>             | >>>>>>>>>>>            Karampuri
| >>>             | >>>>>>>>>>>            <pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>
| >>>             | >>>>>>>>>>>        <mailto:pkarampu@xxxxxxxxxx
| >>>             <mailto:pkarampu@xxxxxxxxxx>>>:
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>                On
| >>>             | >>>>>>>>>>>                08/04/2014
| >>>             | >>>>>>>>>>>                03:33 PM,
| >>>             | >>>>>>>>>>>                Roman wrote:
| >>>             | >>>>>>>>>>>>                Hello!
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>                Facing the
| >>>             | >>>>>>>>>>>>                same
| >>>             | >>>>>>>>>>>>                problem as
| >>>             | >>>>>>>>>>>>                mentioned
| >>>             | >>>>>>>>>>>>                here:
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>
| >>>             http://supercolony.gluster.org/pipermail/gluster-users/2014-April/039959.html
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>                my set up
| >>>             | >>>>>>>>>>>>                is up and
| >>>             | >>>>>>>>>>>>                running, so
| >>>             | >>>>>>>>>>>>                i'm ready
| >>>             | >>>>>>>>>>>>                to help you
| >>>             | >>>>>>>>>>>>                back with
| >>>             | >>>>>>>>>>>>                feedback.
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>                setup:
| >>>             | >>>>>>>>>>>>                proxmox
| >>>             | >>>>>>>>>>>>                server as
| >>>             | >>>>>>>>>>>>                client
| >>>             | >>>>>>>>>>>>                2 gluster
| >>>             | >>>>>>>>>>>>                physical
| >>>             | >>>>>>>>>>>>                 servers
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>                server side
| >>>             | >>>>>>>>>>>>                and client
| >>>             | >>>>>>>>>>>>                side both
| >>>             | >>>>>>>>>>>>                running atm
| >>>             | >>>>>>>>>>>>                3.4.4
| >>>             | >>>>>>>>>>>>                glusterfs
| >>>             | >>>>>>>>>>>>                from
| >>>             | >>>>>>>>>>>>                gluster repo.
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>                the problem is:
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>                1. craeted
| >>>             | >>>>>>>>>>>>                replica bricks.
| >>>             | >>>>>>>>>>>>                2. mounted
| >>>             | >>>>>>>>>>>>                in proxmox
| >>>             | >>>>>>>>>>>>                (tried both
| >>>             | >>>>>>>>>>>>                promox
| >>>             | >>>>>>>>>>>>                ways: via
| >>>             | >>>>>>>>>>>>                GUI and
| >>>             | >>>>>>>>>>>>                fstab (with
| >>>             | >>>>>>>>>>>>                backup
| >>>             | >>>>>>>>>>>>                volume
| >>>             | >>>>>>>>>>>>                line), btw
| >>>             | >>>>>>>>>>>>                while
| >>>             | >>>>>>>>>>>>                mounting
| >>>             | >>>>>>>>>>>>                via fstab
| >>>             | >>>>>>>>>>>>                I'm unable
| >>>             | >>>>>>>>>>>>                to launch a
| >>>             | >>>>>>>>>>>>                VM without
| >>>             | >>>>>>>>>>>>                cache,
| >>>             | >>>>>>>>>>>>                meanwhile
| >>>             | >>>>>>>>>>>>                direct-io-mode
| >>>             | >>>>>>>>>>>>                is enabled
| >>>             | >>>>>>>>>>>>                in fstab line)
| >>>             | >>>>>>>>>>>>                3. installed VM
| >>>             | >>>>>>>>>>>>                4. bring
| >>>             | >>>>>>>>>>>>                one volume
| >>>             | >>>>>>>>>>>>                down - ok
| >>>             | >>>>>>>>>>>>                5. bringing
| >>>             | >>>>>>>>>>>>                up, waiting
| >>>             | >>>>>>>>>>>>                for sync is
| >>>             | >>>>>>>>>>>>                done.
| >>>             | >>>>>>>>>>>>                6. bring
| >>>             | >>>>>>>>>>>>                other
| >>>             | >>>>>>>>>>>>                volume down
| >>>             | >>>>>>>>>>>>                - getting
| >>>             | >>>>>>>>>>>>                IO errors
| >>>             | >>>>>>>>>>>>                on VM guest
| >>>             | >>>>>>>>>>>>                and not
| >>>             | >>>>>>>>>>>>                able to
| >>>             | >>>>>>>>>>>>                restore the
| >>>             | >>>>>>>>>>>>                VM after I
| >>>             | >>>>>>>>>>>>                reset the
| >>>             | >>>>>>>>>>>>                VM via
| >>>             | >>>>>>>>>>>>                host. It
| >>>             | >>>>>>>>>>>>                says (no
| >>>             | >>>>>>>>>>>>                bootable
| >>>             | >>>>>>>>>>>>                media).
| >>>             | >>>>>>>>>>>>                After I
| >>>             | >>>>>>>>>>>>                shut it
| >>>             | >>>>>>>>>>>>                down
| >>>             | >>>>>>>>>>>>                (forced)
| >>>             | >>>>>>>>>>>>                and bring
| >>>             | >>>>>>>>>>>>                back up, it
| >>>             | >>>>>>>>>>>>                boots.
| >>>             | >>>>>>>>>>>                Could you do
| >>>             | >>>>>>>>>>>                the
| >>>             | >>>>>>>>>>>                following
| >>>             | >>>>>>>>>>>                and test it
| >>>             | >>>>>>>>>>>                again?
| >>>             | >>>>>>>>>>>                gluster
| >>>             | >>>>>>>>>>>                volume set
| >>>             | >>>>>>>>>>>                <volname>
| >>>             | >>>>>>>>>>>  cluster.self-heal-daemon
| >>>             | >>>>>>>>>>>                off
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>                Pranith
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>                Need help.
| >>>             | >>>>>>>>>>>>                Tried
| >>>             | >>>>>>>>>>>>                3.4.3, 3.4.4.
| >>>             | >>>>>>>>>>>>                Still
| >>>             | >>>>>>>>>>>>                missing
| >>>             | >>>>>>>>>>>>                pkg-s for
| >>>             | >>>>>>>>>>>>                3.4.5 for
| >>>             | >>>>>>>>>>>>                debian and
| >>>             | >>>>>>>>>>>>                3.5.2
| >>>             | >>>>>>>>>>>>                (3.5.1
| >>>             | >>>>>>>>>>>>                always
| >>>             | >>>>>>>>>>>>                gives a
| >>>             | >>>>>>>>>>>>                healing
| >>>             | >>>>>>>>>>>>                error for
| >>>             | >>>>>>>>>>>>                some reason)
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>                --
| >>>             | >>>>>>>>>>>>                Best regards,
| >>>             | >>>>>>>>>>>>                Roman.
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>
| >>>             | >>>>>>>>>>>>
| >>>              _______________________________________________
| >>>             | >>>>>>>>>>>>                Gluster-users
| >>>             | >>>>>>>>>>>>                mailing list
| >>>             | >>>>>>>>>>>> Gluster-users@xxxxxxxxxxx
| >>>             <mailto:Gluster-users@xxxxxxxxxxx>
| >>>             | >>>>>>>>>>>>          
| >>>              <mailto:Gluster-users@xxxxxxxxxxx
| >>>             <mailto:Gluster-users@xxxxxxxxxxx>>
| >>>             | >>>>>>>>>>>>
| >>>             http://supercolony.gluster.org/mailman/listinfo/gluster-users
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>
| >>>             | >>>>>>>>>>>            --
| >>>             | >>>>>>>>>>>            Best regards,
| >>>             | >>>>>>>>>>>            Roman.
| >>>             | >>>>>>>>>>
| >>>             | >>>>>>>>>>
| >>>             | >>>>>>>>>>
| >>>             | >>>>>>>>>>
| >>>             | >>>>>>>>>>        --
| >>>             | >>>>>>>>>>        Best regards,
| >>>             | >>>>>>>>>>        Roman.
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>
| >>>             | >>>>>>>>>    --
| >>>             | >>>>>>>>>    Best regards,
| >>>             | >>>>>>>>>    Roman.
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>
| >>>             | >>>>>>>>  --
| >>>             | >>>>>>>>  Best regards,
| >>>             | >>>>>>>>  Roman.
| >>>             | >>>>>>>
| >>>             | >>>>>>>
| >>>             | >>>>>>>
| >>>             | >>>>>>>
| >>>             | >>>>>>>                            --
| >>>             | >>>>>>>                            Best regards,
| >>>             | >>>>>>>  Roman.
| >>>             | >>>>>>
| >>>             | >>>>>>
| >>>             | >>>>>>
| >>>             | >>>>>>
| >>>             | >>>>>>                    --
| >>>             | >>>>>>                    Best regards,
| >>>             | >>>>>>                    Roman.
| >>>             | >>>>>
| >>>             | >>>>>
| >>>             | >>>>>
| >>>             | >>>>>
| >>>             | >>>>>            --
| >>>             | >>>>>            Best regards,
| >>>             | >>>>>            Roman.
| >>>             | >>>>
| >>>             | >>>>
| >>>             | >>>>
| >>>             | >>>>
| >>>             | >>>>    --
| >>>             | >>>>    Best regards,
| >>>             | >>>>    Roman.
| >>>             | >>>
| >>>             | >>>
| >>>             | >>>
| >>>             | >>>
| >>>             | >>>             --
| >>>             | >>>             Best regards,
| >>>             | >>>  Roman.
| >>>             | >>
| >>>             | >>
| >>>             | >>
| >>>             | >>
| >>>             | >>         --
| >>>             | >>         Best regards,
| >>>             | >>         Roman.
| >>>             | >>
| >>>             | >>
| >>>             | >>
| >>>             | >>
| >>>             | >>     --
| >>>             | >>     Best regards,
| >>>             | >>     Roman.
| >>>             | >
| >>>             | >
| >>>             | >
| >>>             | >
| >>>             | > --
| >>>             | > Best regards,
| >>>             | > Roman.
| >>>             |
| >>>             |
| >>>
| >>>
| >>>
| >>>
| >>>         --
| >>>         Best regards,
| >>>         Roman.
| >>
| >>
| >>
| >>
| >>     --
| >>     Best regards,
| >>     Roman.
| >
| >
| >
| >
| > --
| > Best regards,
| > Roman.
| 
| 
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux