RE: LOOKUP conflict => OPEN fails_

[Dongmin Yu <min@xxxxxxxxxxxxx>]

Hello,

 

I think I’m getting similar problem at 3.0.3. My configuration is 2 distributions and replication for each distribution. (2x2)

And I turned off all the performance options on both client and server to check if caching cause this issue.

 

My test were creating and deleting a same files through TCP client.

Creating and deleting itself worked fine, as the file was created and deleted on a distributed volume and its replica correctly.

But even when the file existed and readdir said the file was there, I was not able to access the file(got NOENT) once every two trials.

 

[2010-04-14 03:28:12] D [libglusterfsclient.c:2932:libgf_client_opendir] libglusterfsclient: opendir: path /t/test1201/m, status 0, errno 117

[2010-04-14 03:28:12] D [libglusterfsclient.c:4394:glusterfs_readdir] libglusterfsclient: offset 0

[2010-04-14 03:28:12] D [libglusterfsclient.c:4412:glusterfs_readdir] libglusterfsclient: new offset 8, entry .

[2010-04-14 03:28:12] D [libglusterfsclient.c:4394:glusterfs_readdir] libglusterfsclient: offset 8

[2010-04-14 03:28:12] D [libglusterfsclient.c:4412:glusterfs_readdir] libglusterfsclient: new offset 12, entry ..

[2010-04-14 03:28:12] D [libglusterfsclient.c:4394:glusterfs_readdir] libglusterfsclient: offset 12

[2010-04-14 03:28:12] D [libglusterfsclient.c:4412:glusterfs_readdir] libglusterfsclient: new offset 1024, entry myentunnel-unicode.zip

[2010-04-14 03:28:12] D [libglusterfsclient.c:4394:glusterfs_readdir] libglusterfsclient: offset 1024

[2010-04-14 03:28:12] D [libglusterfsclient.c:4400:glusterfs_readdir] libglusterfsclient: readdir failed: Success

[2010-04-14 03:28:12] D [libglusterfsclient.c:4971:glusterfs_stat] libglusterfsclient: path /hs2/t/test1201/m/myentunnel-unicode.zip

[2010-04-14 03:28:12] D [libglusterfsclient.c:1643:_libgf_vmp_search_entry] libglusterfsclient: VMP Search: path /hs2/t/test1201/m/myentunnel-unicode.zip, type: LongestPrefix

[2010-04-14 03:28:12] D [libglusterfsclient.c:1730:libgf_vmp_search_entry] libglusterfsclient: VMP Entry found: path :/hs2/t/test1201/m/m entunnel-unicode.zip vmp: /hs2/

[2010-04-14 03:28:12] D [libglusterfsclient.c:898:libgf_resolve_path_light] libglusterfsclient: Path: /t/test1201/m/myentunnel-unicode.zip, Resolved Path: /t/test1201/m/myentunnel-unicode.zip

[2010-04-14 03:28:12] D [libglusterfsclient.c:4892:__glusterfs_stat] libglusterfsclient: path /t/test1201/m/myentunnel-unicode.zip, op: 2

[2010-04-14 03:28:12] D [libglusterfsclient-dentry.c:390:libgf_client_path_lookup] libglusterfsclient: resolved path(/t/test1201/m/myentunnel-unicode.zip) to 1466437/1462232

[2010-04-14 03:28:12] D [dht-common.c:976:dht_attr_cbk] distribute: subvolume replicate01 returned -1 (No such file or directory)

[2010-04-14 03:28:12] D [libglusterfsclient.c:4822:libgf_client_stat] libglusterfsclient: path /t/test1201/m/myentunnel-unicode.zip, status -1, errno 2

 

Thanks

 

DongMin Yu

 

HOSTWAY IDC Corp. / R&D Principal Researcher

TEL. +822 2105 6037

FAX. +822 2105 6019

CELL. +8216 2086 1357

EMAIL: min.yu@xxxxxxxxxxxxxxx

Website: http://www.hostway.com

 

NOTICE:  This email and any file transmitted are confidential and/or legally privileged and intended only for the person(s) directly addressed.  If you are not the intended recipient, any use, copying, transmission, distribution, or other forms of dissemination is strictly prohibited.  If you have received this email in error, please notify the sender immediately and permanently delete the email and files, if any.

From: gluster-devel-bounces+min=hostway.co.kr@xxxxxxxxxx [mailto:gluster-devel-bounces+min=hostway.co.kr@xxxxxxxxxx] On Behalf Of Raghavendra G
Sent: Thursday, February 11, 2010 2:49 AM
To: =SMTP:fredrik.widlund@xxxxxxxxxx
Cc: =SMTP:gluster-devel@xxxxxxxxxx
Subject: Re: LOOKUP conflict => OPEN fails_

 

Sorry, No need of logfiles. I saw this mail first and then other mails, hence the confusion :).

On Wed, Feb 10, 2010 at 8:56 PM, Raghavendra G <raghavendra@xxxxxxxxxxx> wrote:

Hi Fredrik,

Can you please send complete log files of client and server?

regards,

On Mon, Feb 8, 2010 at 6:02 PM, Fredrik Widlund <fredrik.widlund@xxxxxxxxxx> wrote:

 

Hi,

 

I’m running a simple AFR setup, thouch currently with only one backend, and 2 tcp clients. Version is 3.0.0 from jan 20.

 

Basically one client is writing a large number of files, continuously, and the other client is reading.

 

I have a growing problem with lookup “conflicts”, resulting in files being listed in directories but where reads are returning “-1 (No such file…”.

 

Restarting the client does not solve the conflict, but restarting the server does and the files becomes available again.

 

The filesystem is a 5TB XFS hw raid-5 with around 150k files.

 

Debug trace of client:

[2010-02-08 13:39:29] N [trace.c:148:trace_open_cbk] replicated: 3073: (op_ret=0, op_errno=117, *fd=0x129a430)

[2010-02-08 13:39:37] N [trace.c:1837:trace_open] replicated: 3094: (loc {path=/download/90910/live/webb1/webb1/Layer3/prog_index.m3u8, ino=5042185}, flags=32768, fd=0x1296fc0, wbflags=0)

[2010-02-08 13:39:37] N [trace.c:148:trace_open_cbk] replicated: 3094: (op_ret=-1, op_errno=2, *fd=0x1296fc0)

[2010-02-08 13:39:37] W [fuse-bridge.c:858:fuse_fd_cbk] glusterfs-fuse: 3094: OPEN() /download/90910/live/webb1/webb1/Layer3/prog_index.m3u8 => -1 (No such file or directory)

[2010-02-08 13:39:38] N [trace.c:1837:trace_open] replicated: 3100: (loc {path=/download/90910/live/webb1/webb1/Layer4/prog_index.m3u8, ino=5013773}, flags=32768, fd=0x1296fc0, wbflags=0)

[2010-02-08 13:39:38] N [trace.c:148:trace_open_cbk] replicated: 3100: (op_ret=0, op_errno=117, *fd=0x1296fc0)

[2010-02-08 13:39:38] N [trace.c:1837:trace_open] replicated: 3106: (loc {path=/download/90910/live/webb1/webb1/Layer4/Period1/segment277.ts, ino=5050371}, flags=32768, fd=0x129a430, wbflags=0)

[…]

 

And server:

[2010-02-08 13:39:09] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e43f3

[2010-02-08 13:39:09] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e440b

[2010-02-08 13:39:17] D [server-protocol.c:2037:server_open_cbk] server: 1719: OPEN (null) (0) ==> -1 (No such file or directory)

[2010-02-08 13:39:18] D [server-protocol.c:2037:server_open_cbk] server: 1724: OPEN (null) (0) ==> -1 (No such file or directory)

[2010-02-08 13:39:28] D [server-resolve.c:238:resolve_path_deep] store0: RESOLVE OPEN() seeking deep resolution of /download/90910/live/webb1/webb1/Layer3/prog_index.m3u8

[2010-02-08 13:39:28] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e43db

[2010-02-08 13:39:28] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e43f3

[2010-02-08 13:39:28] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e440b

[2010-02-08 13:39:28] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e43db

[…]

 

Kind regards,

Fredrik Widlund

 

 

 

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel

--
Raghavendra G

--
Raghavendra G