Hi everyone, Hope anyone can shed some light on some strange behavior we've just having with Gluster. Last night, while using gluster3.3b2, we came up with some errors, after running tests for several days. We're running the test in a striped 4 disk volume: # gluster volume info data Volume Name: data Type: Stripe Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: 192.168.19.103:/gluster/enclusterdata_1 Brick2: 192.168.19.103:/gluster/enclusterdata_2 Brick3: 192.168.19.103:/gluster/enclusterdata_3 Brick4: 192.168.19.103:/gluster/enclusterdata_4 When creating a lot of files, some of the files started to have erratic behavior. They are in the file system (ls commands below) but we are not being able to read some of the files (file ending with 22413, in the command below), 9 in total to be exact. Both files were created programatically, under the same circumstances at nearly the same time. The directory has currently 12668 files inside. # ls -la fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4 -rw-r--r--. 1 1000 1000 41759951 Mar 21 20:34 fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4 # ls -la fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22412-22412.mp4 -rw-r--r--. 1 1000 1000 40994275 Mar 21 20:34 fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22412-22412.mp4 # file fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4 fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4: writable, regular file, no read permission # file fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22412-22412.mp4 fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22412-22412.mp4: ISO Media, MPEG v4 system, 3GPP JVT AVC When we checked the logs of each brick, all of the bricks had these errors for the 9 files in question: [2012-03-21 20:34:36.309525] E [posix-helpers.c:585:posix_handle_pair] 0-data-posix: /gluster/enclusterdata_4/encluster/01c79d00-683a-11e1-a105-003048bfe717/sujwork/fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4: key:trusted.data-stripe-0.stripe-size error:File exists gluster-enclusterdata_4.log:[2012-03-21 20:34:36.309580] E [posix.c:1766:posix_create] 0-data-posix: setting xattrs on /encluster/01c79d00-683a-11e1-a105-003048bfe717/sujwork/fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4 failed (File exists) gluster-enclusterdata_4.log:[2012-03-22 08:15:09.205467] I [server3_1-fops.c:1493:server_stat_cbk] 0-data-server: 176411: STAT /encluster/01c79d00-683a-11e1-a105-003048bfe717/sujwork/fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4 (0) ==> -1 (No such file or directory) gluster-enclusterdata_4.log:[2012-03-22 08:15:29.287542] I [server3_1-fops.c:1493:server_stat_cbk] 0-data-server: 184724: STAT /encluster/01c79d00-683a-11e1-a105-003048bfe717/sujwork/fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4 (0) ==> -1 (No such file or directory) Whenever we try to read the file, the "No such file or directory" error crops up in the logs. Any idea what could be going wrong? In total we have 9 files that show this erratic behavior, all with similar information on the brick logs. Thanks in advance, Daniel
