Yes, I just used a text editor i.e. vi to test this out, but my real application is surveillance camera system. The video was stored in GlusterFS by FUSE and Samba.
Because of the surveillance camera system 7*24 working mode, it doesn't have to cache the files. So I added O_DIRECT flag in GlusterFS client protocol open() and create(). According my test, I wrote a video file into GlusterFS in windows, then I didn't find any cache in GlusterFS node, and I read a video file in windows, it also wasn't cached in GlusterFS node, but some frames in the video file(at the last) was lost, just like my vi test.
IIRC the page-alighed may be the reason of this question. How to solve this problem? Or in GlusterFS 3.6, 3.7, is there the option to enable io-direct mode to avoid the video cache in GlusterFS node?
At 2016-07-11 18:51:02, "Krutika Dhananjay" <kdhananj@xxxxxxxxxx> wrote:
-KrutikaAlso, what are the offsets and sizes of the writes on this file by this application in the strace output?What's the application you are running? Sounds like you're using a text editor like vi(m) to test this out?Do you have the strace output of the running application that confirms that it is open()ing the file with O_DIRECT?
Is the application opening the files with O_DIRECT?On Mon, Jul 11, 2016 at 2:44 PM, Keiviw <keiviw@xxxxxxx> wrote:I have checked the page-aligned, i.e. the file was larger than one page, a part of the file(one page size) was saved successfully, and the rest(more than one page but less than two pages) was lost.
At 2016-07-11 12:53:32, "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> wrote:
Is it possible to share the test you are running? As per your volume, o-direct is not enabled on your volume, i.e. the file shouldn't be opened with o-direct but as per the logs it is giving Invalid Argument as if there is something wrong with the arguments when we do o-direct write with wrong size. so I would like to test out why exactly is it giving this problem. Please note that for o-direct write to succeed, both offset and size should be page-aligned, something like multiple of 512 is one way to check it.On Sun, Jul 10, 2016 at 5:19 PM, Keiviw <keiviw@xxxxxxx> wrote:My volume info:Volume Name: testType: DistributeVolume ID: 9294b122-d81e-4b12-9b5c-46e89ee0e40bStatus: StartedNumber of Bricks: 2Transport-type: tcpBricks:Brick1: compute2:/home/brick1Brick2: compute2:/home/brick2Options Reconfigured:performance.flush-behind: offstorage.linux-aio: offMy brick logs(I have cleaned up the history log):[2016-07-10 11:42:50.577683] E [posix.c:2128:posix_writev] 0-test-posix: write failed: offset 0, Invalid argument[2016-07-10 11:42:50.577735] I [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569840: WRITEV 5 (526a3118-9994-429e-afc0-4aa063606bde) ==> -1 (Invalid argument)[2016-07-10 11:42:54.583038] E [posix.c:2128:posix_writev] 0-test-posix: write failed: offset 0, Invalid argument[2016-07-10 11:42:54.583080] I [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569870: WRITEV 5 (c3d28f34-8f43-446d-8d0b-80841ae8ec5b) ==> -1 (Invalid argument)My mnt-test-.logs:[2016-07-10 11:42:50.577816] W [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote operation failed: Invalid argument[2016-07-10 11:42:50.578508] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12398282: FSYNC() ERR => -1 (Invalid argument)[2016-07-10 11:42:54.583156] W [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote operation failed: Invalid argument[2016-07-10 11:42:54.583762] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12398317: FSYNC() ERR => -1 (Invalid argument)
在 2016-07-10 19:18:18，"Krutika Dhananjay" <kdhananj@xxxxxxxxxx> 写道：
To me it looks like a case of a flush triggering a write() that was cached by write-behind and because the write bufferdid not meet the page alignment requirement with o-direct write, it was failed with EINVAL and the trigger fop - i.e., flush() was failed with the 'Invalid argument' error code.Could you attach the brick logs as well, so that we can confirm the theory?-KrutikaOn Sat, Jul 9, 2016 at 9:31 PM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:Pranith/Krutika,Your inputs please, IIRC we'd need to turn on some o_direct option here?
On Saturday 9 July 2016, Keiviw <keiviw@xxxxxxx> wrote:The errors also occured in GlusterFS 3.6.7，I just add the O_DIRECT flag in client protocol open() and create()! How to explain and solve the problem?发自 网易邮箱大师On 07/09/2016 17:58, Atin Mukherjee wrote:Any specific reason of using 3.3 given that its really quite old? We are at 3.6, 3.7 & 3.8 supportability matrix now.
On Saturday 9 July 2016, Keiviw <keiviw@xxxxxxx> wrote:hi,I have installed GlusterFS 3.3.0, and now I get Fsync failures when saving files with the O_DIRECT flag in open() and create().1, I tried to save a flie in vi and got this error:"test" E667: Fsync failed2, I see this in the logs:[2016-07-07 14:20:10.325400] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 102: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:20:13.930384] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 137: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:20:51.199448] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 174: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 14:21:32.804738] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 206: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 14:21:43.702146] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 276: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:21:51.296809] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 314: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:21:54.062687] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 349: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:22:54.678960] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 429: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:24:35.546980] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 505: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:24:48.696888] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 538: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:24:55.824752] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 575: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:25:04.364629] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 612: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:27:08.579052] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 713: FSYNC() ERR => -1 (Invalid argument)[2016-07-07 14:46:44.635169] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 859: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 14:46:55.892041] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 989: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 14:51:45.904233] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 148747: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 14:54:45.603557] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 148986: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 14:59:18.818972] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 292106: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 15:48:32.708207] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 6753592: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 16:24:10.476228] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 10733703: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 16:30:42.388220] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 11497698: FLUSH() ERR => -1 (Invalid argument)[2016-07-07 16:44:18.247511] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12037185: FLUSH() ERR => -1 (Invalid argument)[2016-07-08 01:00:55.131141] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12181783: FLUSH() ERR => -1 (Invalid argument)[2016-07-08 06:54:13.418607] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12397060: FLUSH() ERR => -1 (Invalid argument)
--
Atin
Sent from iPhone
--
Atin
Sent from iPhone
--Pranith
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
