Hi Niels,
On 28.04.2016 15:44, Niels de Vos wrote:
On Thu, Apr 28, 2016 at 02:43:01PM +0200, Xavier Hernandez wrote:Hi, I've seen what seems a bug in the communications layer. The first sign is an "XDR decoding failed" error in the logs. This happens with Gluster 3.7.11 accessed through Ganesha and gfapi. The volume is a distributed-disperse 4*(4+2). I'm able to reproduce the problem easily doing the following test: iozone -t2 -s10g -r1024k -i0 -w -F/iozone{1..2}.dat echo 3 >/proc/sys/vm/drop_caches iozone -t2 -s10g -r1024k -i1 -w -F/iozone{1..2}.dat The error happens soon after starting the read test.Do you know if this only happens on disperse volumes, or also with others?
I have only seen the problem with EC. Not sure if it's because I do most of my tests with EC or because other volumes do not manifest this problem. However I've only seen this when I started testing Ganesha. I have never seen the problem with FUSE.
I think the reason is that FUSE is slower than Ganesha (little more than 120 MB/s vs 400 MB/s) and the combination of events needed to cause this problem would be much more unlikely to happen on FUSE. Since EC also talks to many bricks simultaneously (6 in this case), maybe this makes it more sensible to communications problems compared to a replicated volume.
If you have captured a network trace, could you provide it to me? You can use 'editcap -s0 ...' to copy only the relevant packets. But, I dont have an issue to download a few GB either if that is easier for you.As can be seen in the data below, client3_3_readv_cbk() is processing an iovec of 116 bytes, however it should be of 154 bytes (the buffer in memory really seems to contain 154 bytes). The data on the network seems ok (at least I haven't been able to identify any problem), so this must be a processing error on the client side. The last field in cut buffer of the sequentialized data corresponds to the length of the xdata field: 0x26. So at least 38 more byte should be present. My guess is that some corner case is hit reading fragmented network packets due to a high load. Debug information: Breakpoint 1, client3_3_readv_cbk (req=0x7f540e64106c, iov=0x7f540e6410ac, count=, myframe=0x7f54259a4d54) at client-rpc-fops.c:3021 3021 gf_msg (this->name, GF_LOG_ERROR, EINVAL, (gdb) print *iov $1 = {iov_base = 0x7f53e994e018, iov_len = 116} (gdb) x/116xb 0x7f53e994e018 0x7f53e994e018: 0x00 0x00 0x80 0x00 0x00 0x00 0x00 0x00 0x7f53e994e020: 0xa8 0xbf 0xa3 0xe0 0x5f 0x48 0x4c 0x1eHmm, I'm not sure how this is layed out in memory. 0x80 would be one of first bytes in RPC payload, it signals 'last record' for the RPC procedure, and we only send one record anyway. The four bytes combined like (0x80 0x.. 0x.. 0x..) should be (0x80 | rpc-record-size). Reading this in Wireshark from a .pcap.gz is much easier :)
The RPC header is already decoded here. The 116 bytes are only the content of the readv answer, as decoded by xdr_gfs3_read_rsp() function.
This means that the first 4 bytes are the op_ret, next 4 are op_errno, followed by an encoded iatt (with gfid and ino number as the first fields). Then a size field and the length of a stream of bytes (corresponding to the encoded xdata).
I've a network capture. I can upload it if you want, but I think this is what you are trying to see:
(gdb) f 1
#1 0x00007fa88cb5bab0 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7fa870413e30, pollin=pollin@entry=0x7fa86400efe0) at rpc-clnt.c:764
764 req->cbkfn (req, req->rsp, req->rspcnt, saved_frame->frame);
(gdb) print *pollin
$1 = {vector = {{iov_base = 0x7fa7ef580000, iov_len = 140}, {iov_base = 0x7fa7ef460000, iov_len = 32808}, {iov_base = 0x0, iov_len = 0} <repeats 14 times>}, count = 2,
vectored = 1 '\001', private = 0x7fa86c00ec60, iobref = 0x7fa86400e560, hdr_iobuf = 0x7fa868022ee0, is_reply = 1 '\001'}
(gdb) x/140xb 0x7fa7ef580000
0x7fa7ef580000: 0x00 0x00 0x89 0x68 0x00 0x00 0x00 0x01
0x7fa7ef580008: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x7fa7ef580010: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x7fa7ef580018: 0x00 0x00 0x80 0x00 0x00 0x00 0x00 0x00
0x7fa7ef580020: 0xc3 0x8e 0x35 0xf0 0x31 0xa1 0x45 0x01
0x7fa7ef580028: 0x8a 0x21 0x06 0x4b 0x08 0x4c 0x59 0xdf
0x7fa7ef580030: 0x8a 0x21 0x06 0x4b 0x08 0x4c 0x59 0xdf
0x7fa7ef580038: 0x00 0x00 0x00 0x00 0x00 0x00 0x08 0x00
0x7fa7ef580040: 0x00 0x00 0x81 0xa0 0x00 0x00 0x00 0x01
0x7fa7ef580048: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x7fa7ef580050: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x7fa7ef580058: 0x00 0x00 0x00 0x00 0xa0 0x00 0x00 0x00
0x7fa7ef580060: 0x00 0x00 0x10 0x00 0x00 0x00 0x00 0x00
0x7fa7ef580068: 0x00 0x50 0x00 0x00 0x57 0x22 0x33 0x1f
0x7fa7ef580070: 0x24 0x0c 0xc2 0x58 0x57 0x22 0x33 0xa1
0x7fa7ef580078: 0x34 0x11 0x3b 0x54 0x57 0x22 0x33 0xa1
0x7fa7ef580080: 0x34 0x20 0x7d 0x7a 0x00 0x00 0x80 0x00
0x7fa7ef580088: 0x00 0x00 0x00 0x26
(gdb) f 3
#3 0x00007fa88cb57853 in rpc_transport_notify (this=this@entry=0x7fa870423ab0, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data="" at rpc-transport.c:546
546 ret = this->notify (this, this->mydata, event, data);
(gdb) print *this
$3 = {ops = 0x7fa87ff76740 <tops>, listener = 0x0, private = 0x7fa870424750, xl_private = 0x0, xl = 0x7fa87000b5a0, mydata = 0x7fa870413e60, lock = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, refcount = 2,
outstanding_rpc_count = 0, ctx = 0x2688740, options = 0x7fa88788529c, name = 0x7fa8704245f0 "test-client-2", dnscache = 0x7fa870472b90, drc_client = 0x0, buf = 0x0,
init = 0x7fa87fd6b760 <init>, fini = 0x7fa87fd6b690 <fini>, reconfigure = 0x7fa87fd6b290 <reconfigure>, notify = 0x7fa88cb5bbb0 <rpc_clnt_notify>, notify_data = 0x0, peerinfo = {
sockaddr = {ss_family = 2, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>}, sockaddr_len = 16, identifier = "193.109.174.232:49152", '\000' <repeats 86 times>,
max_op_version = 0, min_op_version = 0, volname = '\000' <repeats 1023 times>}, myinfo = {sockaddr = {ss_family = 2, __ss_align = 0,
__ss_padding = '\000' <repeats 111 times>}, sockaddr_len = 16, identifier = "193.109.174.230:65205", '\000' <repeats 86 times>, max_op_version = 0, min_op_version = 0,
volname = '\000' <repeats 1023 times>}, total_bytes_read = 390116824, total_bytes_write = 5374554184, list = {next = 0x7fa870424568, prev = 0x7fa870424568}, bind_insecure = 1,
dl_handle = 0x26efad0, ssl_name = 0x0, clnt_options = 0x0}
The first 4 bytes of the dump correspond to the XID of the answer (0x8968). We can see that the answer comes from 193.109.174.232:49152. The output of the following command is attached:
# tshark -r tcpdump.pcap -n -V -Y "rpc.xid == 0x8968 and ip.addr == 193.109.174.232 and tcp.port == 49152"
We can see that the answer is received in two TCP segments: one of 116 bytes and the other of 32836 bytes. Probably this is the source of the problem.
Don't hesitate to file a bug for this, and include the exact error messages you get. If possible with network capture, but if you can not share that publicly, please get it directly to me.
I'll file a bug with all this information.
Thanks,
Xavi
Thanks, Niels0x7f53e994e028: 0x80 0xa3 0x8a 0xd8 0x9d 0xa1 0x1c 0x75 0x7f53e994e030: 0x80 0xa3 0x8a 0xd8 0x9d 0xa1 0x1c 0x75 0x7f53e994e038: 0x00 0x00 0x00 0x00 0x00 0x00 0x08 0x00 0x7f53e994e040: 0x00 0x00 0x81 0xa0 0x00 0x00 0x00 0x01 0x7f53e994e048: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x7f53e994e050: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x7f53e994e058: 0x00 0x00 0x00 0x00 0xa0 0x00 0x00 0x00 0x7f53e994e060: 0x00 0x00 0x10 0x00 0x00 0x00 0x00 0x00 0x7f53e994e068: 0x00 0x50 0x00 0x00 0x57 0x22 0x04 0x1f 0x7f53e994e070: 0x25 0x38 0x92 0x91 0x57 0x22 0x04 0xb3 0x7f53e994e078: 0x03 0x53 0x1b 0x13 0x57 0x22 0x04 0xb3 0x7f53e994e080: 0x06 0xf5 0xe1 0x99 0x00 0x00 0x80 0x00 0x7f53e994e088: 0x00 0x00 0x00 0x26 (gdb) x/154xb 0x7f53e994e018 0x7f53e994e018: 0x00 0x00 0x80 0x00 0x00 0x00 0x00 0x00 0x7f53e994e020: 0xa8 0xbf 0xa3 0xe0 0x5f 0x48 0x4c 0x1e 0x7f53e994e028: 0x80 0xa3 0x8a 0xd8 0x9d 0xa1 0x1c 0x75 0x7f53e994e030: 0x80 0xa3 0x8a 0xd8 0x9d 0xa1 0x1c 0x75 0x7f53e994e038: 0x00 0x00 0x00 0x00 0x00 0x00 0x08 0x00 0x7f53e994e040: 0x00 0x00 0x81 0xa0 0x00 0x00 0x00 0x01 0x7f53e994e048: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x7f53e994e050: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x7f53e994e058: 0x00 0x00 0x00 0x00 0xa0 0x00 0x00 0x00 0x7f53e994e060: 0x00 0x00 0x10 0x00 0x00 0x00 0x00 0x00 0x7f53e994e068: 0x00 0x50 0x00 0x00 0x57 0x22 0x04 0x1f 0x7f53e994e070: 0x25 0x38 0x92 0x91 0x57 0x22 0x04 0xb3 0x7f53e994e078: 0x03 0x53 0x1b 0x13 0x57 0x22 0x04 0xb3 0x7f53e994e080: 0x06 0xf5 0xe1 0x99 0x00 0x00 0x80 0x00 0x7f53e994e088: 0x00 0x00 0x00 0x26 0x00 0x00 0x00 0x01 0x7f53e994e090: 0x00 0x00 0x00 0x17 0x00 0x00 0x00 0x02 0x7f53e994e098: 0x67 0x6c 0x75 0x73 0x74 0x65 0x72 0x66 0x7f53e994e0a0: 0x73 0x2e 0x69 0x6e 0x6f 0x64 0x65 0x6c 0x7f53e994e0a8: 0x6b 0x2d 0x63 0x6f 0x75 0x6e 0x74 0x00 0x7f53e994e0b0: 0x31 0x00 (gdb) bt #0 client3_3_readv_cbk (req=0x7f540e64106c, iov=0x7f540e6410ac, count=, myframe=0x7f54259a4d54) at client-rpc-fops.c:3021 #1 0x00007f542a677ab0 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7f54101cdef0, pollin=pollin@entry=0x7f54000091f0) at rpc-clnt.c:764 #2 0x00007f542a677d6f in rpc_clnt_notify (trans=, mydata=0x7f54101cdf20, event=, data="" at rpc-clnt.c:925 #3 0x00007f542a673853 in rpc_transport_notify (this=this@entry=0x7f54101ddb70, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data="" at rpc-transport.c:546 #4 0x00007f541d881666 in socket_event_poll_in (this=this@entry=0x7f54101ddb70) at socket.c:2237 #5 0x00007f541d8842c4 in socket_event_handler (fd=fd@entry=30, idx=idx@entry=20, data="" poll_in=1, poll_out=0, poll_err=0) at socket.c:2350 #6 0x00007f542a90aa4a in event_dispatch_epoll_handler (event=0x7f540effc540, event_pool=0xd3d9a0) at event-epoll.c:575 #7 event_dispatch_epoll_worker (data="" at event-epoll.c:678 #8 0x00007f542d0fbdc5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f542c7bc28d in clone () from /lib64/libc.so.6 Xavi _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxxhttp://www.gluster.org/mailman/listinfo/gluster-devel
Frame 333872: 234 bytes on wire (1872 bits), 234 bytes captured (1872 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Apr 28, 2016 18:00:47.703751000 Hora de verano romance
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1461859247.703751000 seconds
[Time delta from previous captured frame: 0.000002000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 5.863075000 seconds]
Frame Number: 333872
Frame Length: 234 bytes (1872 bits)
Capture Length: 234 bytes (1872 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:tcp:rpc:glusterfs]
Ethernet II, Src: 00:1b:21:91:aa:17, Dst: 00:1b:21:91:aa:30
Destination: 00:1b:21:91:aa:30
Address: 00:1b:21:91:aa:30
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: 00:1b:21:91:aa:17
Address: 00:1b:21:91:aa:17
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 193.109.174.230, Dst: 193.109.174.232
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 220
Identification: 0x686a (26730)
Flags: 0x02 (Don't Fragment)
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (6)
Header checksum: 0xf107 [validation disabled]
[Good: False]
[Bad: False]
Source: 193.109.174.230
Destination: 193.109.174.232
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 65205 (65205), Dst Port: 49152 (49152), Seq: 1957929, Ack: 383408129, Len: 168
Source Port: 65205
Destination Port: 49152
[Stream index: 11]
[TCP Segment Len: 168]
Sequence number: 1957929 (relative sequence number)
[Next sequence number: 1958097 (relative sequence number)]
Acknowledgment number: 383408129 (relative ack number)
Header Length: 32 bytes
Flags: 0x018 (PSH, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 1... = Push: Set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: *******AP***]
Window size value: 16517
[Calculated window size: 16517]
[Window size scaling factor: -1 (unknown)]
Checksum: 0xe178 [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
Timestamps: TSval 192258002, TSecr 192225003
Kind: Time Stamp Option (8)
Length: 10
Timestamp value: 192258002
Timestamp echo reply: 192225003
[SEQ/ACK analysis]
[Bytes in flight: 168]
Remote Procedure Call, Type:Call XID:0x00008968
Fragment header: Last fragment, 164 bytes
1... .... .... .... .... .... .... .... = Last Fragment: Yes
.000 0000 0000 0000 0000 0000 1010 0100 = Fragment Length: 164
XID: 0x00008968 (35176)
Message Type: Call (0)
RPC Version: 2
Program: GlusterFS (1298437)
Program Version: 330
Procedure: READ (12)
Credentials
Flavor: AUTH_GLUSTERFS (390039)
Length: 24
PID: 23555
UID: 0
GID: 0
Auxiliary GIDs (0)
Lock Owner: 035c0000
Verifier
Flavor: AUTH_NULL (0)
Length: 0
GlusterFS
[Program Version: 330]
[GlusterFS: READ (12)]
GFID: c38e35f0-31a1-4501-8a21-064b084c59df
File Descriptor: 3
Offset: 340426752
Size: 32768
Flags: 0, O_RDONLY
.... .... .... .... .... .... .... ...0 = O_WRONLY: Not set
.... .... .... .... .... .... .... ..0. = O_RDWR: Not set
.... .... .... .... .... .... .0.. .... = O_CREAT: Not set
.... .... .... .... .... .... 0... .... = O_EXCL: Not set
.... .... .... .... .... ...0 .... .... = O_NOCTTY: Not set
.... .... .... .... .... ..0. .... .... = O_TRUNC: Not set
.... .... .... .... .... .0.. .... .... = O_APPEND: Not set
.... .... .... .... .... 0... .... .... = O_NONBLOCK: Not set
.... .... .... .... .... 0... .... .... = O_NDELAY: Not set
.... .... .... .... ...0 .... .... .... = O_SYNC: Not set
.... .... .... .... ..0. .... .... .... = O_ASYNC: Not set
.... .... .... .... .0.. .... .... .... = O_DIRECT: Not set
.... .... .... .... 0... .... .... .... = O_LARGEFILE: Not set
.... .... .... ...0 .... .... .... .... = O_DIRECTORY: Not set
.... .... .... ..0. .... .... .... .... = O_NOFOLLOW: Not set
.... .... .... .0.. .... .... .... .... = O_NOATIME: Not set
.... .... .... 0... .... .... .... .... = O_CLOEXEC: Not set
0000 0000 0000 .... .... .... ..00 00.. = Unused: Not set
0000 0000 0000 0000 0000 0000 0000 0000 = O_RDONLY: Set
.... .... .... .... .... .... .... ..00 = O_ACCMODE: Not set
Dict, contains 1 item
[Size: 56 (56 bytes inc. RPC-roundup)]
Items: 1
glusterfs.inodelk-dom-count: test-disperse-0
Frame 334067: 53754 bytes on wire (430032 bits), 53754 bytes captured (430032 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Apr 28, 2016 18:00:47.706023000 Hora de verano romance
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1461859247.706023000 seconds
[Time delta from previous captured frame: 0.000130000 seconds]
[Time delta from previous displayed frame: 0.002272000 seconds]
[Time since reference or first frame: 5.865347000 seconds]
Frame Number: 334067
Frame Length: 53754 bytes (430032 bits)
Capture Length: 53754 bytes (430032 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:tcp:rpc:glusterfs:data:rpc]
Ethernet II, Src: 00:1b:21:91:aa:30, Dst: 00:1b:21:91:aa:17
Destination: 00:1b:21:91:aa:17
Address: 00:1b:21:91:aa:17
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: 00:1b:21:91:aa:30
Address: 00:1b:21:91:aa:30
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 193.109.174.232, Dst: 193.109.174.230
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 53740
Identification: 0x517a (20858)
Flags: 0x02 (Don't Fragment)
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (6)
Header checksum: 0x36e7 [validation disabled]
[Good: False]
[Bad: False]
Source: 193.109.174.232
Destination: 193.109.174.230
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 49152 (49152), Dst Port: 65205 (65205), Seq: 383891321, Ack: 1959945, Len: 53688
Source Port: 49152
Destination Port: 65205
[Stream index: 11]
[TCP Segment Len: 53688]
Sequence number: 383891321 (relative sequence number)
[Next sequence number: 383945009 (relative sequence number)]
Acknowledgment number: 1959945 (relative ack number)
Header Length: 32 bytes
Flags: 0x010 (ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 0... = Push: Not set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: *******A****]
Window size value: 17518
[Calculated window size: 17518]
[Window size scaling factor: -1 (unknown)]
Checksum: 0xb289 [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
Timestamps: TSval 192225006, TSecr 192258004
Kind: Time Stamp Option (8)
Length: 10
Timestamp value: 192225006
Timestamp echo reply: 192258004
[SEQ/ACK analysis]
[Bytes in flight: 53688]
TCP segment data (32836 bytes)
TCP segment data (20852 bytes)
[2 Reassembled TCP Segments (32952 bytes): #334064(116), #334067(32836)]
[Frame: 334064, payload: 0-115 (116 bytes)]
[Frame: 334067, payload: 116-32951 (32836 bytes)]
[Segment count: 2]
[Reassembled TCP length: 32952]
[Reassembled TCP Data: 800080b40000896800000001000000000000000000000000...]
Remote Procedure Call, Type:Reply XID:0x00008968
Fragment header: Last fragment, 32948 bytes
1... .... .... .... .... .... .... .... = Last Fragment: Yes
.000 0000 0000 0000 1000 0000 1011 0100 = Fragment Length: 32948
XID: 0x00008968 (35176)
Message Type: Reply (1)
[Program: GlusterFS (1298437)]
[Program Version: 330]
[Procedure: READ (12)]
Reply State: accepted (0)
[This is a reply to a request in frame 333872]
[Time from request: 0.002272000 seconds]
Verifier
Flavor: AUTH_NULL (0)
Length: 0
Accept State: RPC executed successfully (0)
GlusterFS
[Program Version: 330]
[GlusterFS: READ (12)]
Return value: 32768
Errno: 0 (No error)
IATT
GFID: c38e35f0-31a1-4501-8a21-064b084c59df
Inode: 9953243570797042143
Device: 0x0000000000000800
Mode: 0100640, S_IRUSR, S_IWUSR, S_IRGRP
.... .... .... .... .... 0... .... .... = S_ISUID: Not set
.... .... .... .... .... .0.. .... .... = S_ISGID: Not set
.... .... .... .... .... ..0. .... .... = S_ISVTX: Not set
.... .... .... .... .... ...1 .... .... = S_IRUSR: Set
.... .... .... .... .... .... 1... .... = S_IWUSR: Set
.... .... .... .... .... .... .0.. .... = S_IXUSR: Not set
.... .... .... .... .... .... ..1. .... = S_IRGRP: Set
.... .... .... .... .... .... ...0 .... = S_IWGRP: Not set
.... .... .... .... .... .... .... 0... = S_IXGRP: Not set
.... .... .... .... .... .... .... .0.. = S_IROTH: Not set
.... .... .... .... .... .... .... ..0. = S_IWOTH: Not set
.... .... .... .... .... .... .... ...0 = S_IXOTH: Not set
Number of hard links: 1
UID: 0
GID: 0
Root device: 0x0000000000000000
Size: 2684354560
Block size: 4096
Blocks: 5242880
Time of last access: Apr 28, 2016 17:58:23.604815960 Hora de verano romance
Time of last modification: Apr 28, 2016 18:00:33.873544532 Hora de verano romance
Time of last status change: Apr 28, 2016 18:00:33.874544506 Hora de verano romance
Size: 32768
Dict, contains 1 item
[Size: 38 (40 bytes inc. RPC-roundup)]
Items: 1
glusterfs.inodelk-count: 1
[RPC-roundup bytes: 00005c5c5c5c5c5c5c5c0000000000000000000000000000...]
Data (32768 bytes)
<raw data stripped>
Data: 5c5c5c5c5c5c5c5c00000000000000000000000000000000...
[Length: 32768]
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
