Hi,
In my case, block size of underlying filesystem is 4K.
So, I tested again using bs=768K (multiples of 12KB).
Your advice worked!
10gnode1:disp on /mnt/gluster type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
root@node1:~/gluster_ec# dd if=/dev/zero of=/mnt/gluster/test.io.1 bs=768k count=1024
1024+0 records in
1024+0 records out
805306368 bytes (805 MB) copied, 9.5404 s, 84.4 MB/s
root@node1:~/gluster_ec#
Thank you for your advice.
Regards.
2015-09-20 6:54 GMT+09:00 Xavier Hernandez <xhernandez@xxxxxxxxxx>:
Hi,
what filesystem are you using on the bricks and what block size is it using ?
Direct I/O requires strict alignment of buffer offsets and block sizes. The disperse volume writes fragments multiples of 512 bytes in the bricks. If the underlying filesystem requires an alignment different than 512 bytes, it could cause the problem you are seeing.
For example if the required alignment of the underlying filesystem is 4KB, you will need to write to the disperse volume in multiples of 12KB (you are using 3 data bricks).
Xavi
El 18/09/2015 a las 3:00, 이태화 escribió:
Hi, I have tested Disperse volume.
so I perform a test using "dd"
I'm using 4 nodes. and nodes are connected through 10g network.
*Volume creation & mount command:*
root@node1:~# gluster vol create disp_vol disperse 4 redundancy 1
10gnode1:/node/dispv 10gnode2:/node/dispv 10gnode3:/node/dispv
10gnode4:/node/dispv
This configuration is not optimal on most workloads. Do you want to use
it ? (y/n) y
volume create: disp_vol: success: please start the volume to access data
root@node1:~# gluster vol start disp_vol
volume start: disp_vol: success
root@node1:~# mount -t glusterfs 10gnode1:/disp_vol /mnt/gluster
*Test command & error message:*
root@node1:~# dd if=/dev/zero of=/mnt/gluster/test.io.1 oflag=direct
bs=1024k count=1024
dd: error writing ‘/mnt/gluster/test.io.1’: Invalid argument
dd: closing output file ‘/mnt/gluster/test.io.1’: Invalid argument
To recognize why, I saw logs. but I didn't know reasons.
*Warning messages in /var/log/glusterfs/mnt-gluster.log*_______________________________________________
[2015-09-18 00:39:38.460320] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-2: remote
operation failed [Invalid argument]
[2015-09-18 00:39:38.460320] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-1: remote
operation failed [Invalid argument]
[2015-09-18 00:39:38.464081] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-3: remote
operation failed [Invalid argument]
[2015-09-18 00:39:38.464297] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-0: remote
operation failed [Invalid argument]
[2015-09-18 00:39:38.464459] W [fuse-bridge.c:2240:fuse_writev_cbk]
0-glusterfs-fuse: 18: WRITE => -1 (Invalid argument)
[2015-09-18 00:39:38.464643] W [fuse-bridge.c:2240:fuse_writev_cbk]
0-glusterfs-fuse: 19: WRITE => -1 (Invalid argument)
[2015-09-18 00:39:38.464801] W [fuse-bridge.c:2240:fuse_writev_cbk]
0-glusterfs-fuse: 20: WRITE => -1 (Invalid argument)
[2015-09-18 00:39:38.464958] W [fuse-bridge.c:2240:fuse_writev_cbk]
0-glusterfs-fuse: 21: WRITE => -1 (Invalid argument)
[2015-09-18 00:39:38.465114] W [fuse-bridge.c:2240:fuse_writev_cbk]
0-glusterfs-fuse: 22: WRITE => -1 (Invalid argument)
[2015-09-18 00:39:38.465278] W [fuse-bridge.c:2240:fuse_writev_cbk]
0-glusterfs-fuse: 23: WRITE => -1 (Invalid argument)
[2015-09-18 00:39:38.465433] W [fuse-bridge.c:2240:fuse_writev_cbk]
0-glusterfs-fuse: 24: WRITE => -1 (Invalid argument)
[2015-09-18 00:39:38.465585] W [fuse-bridge.c:2240:fuse_writev_cbk]
0-glusterfs-fuse: 25: WRITE => -1 (Invalid argument)
[2015-09-18 00:39:38.465529] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-0: remote
operation failed [Invalid argument]
[2015-09-18 00:39:38.465863] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-1: remote
operation failed [Invalid argument]
[2015-09-18 00:39:38.465890] W [fuse-bridge.c:690:fuse_truncate_cbk]
0-glusterfs-fuse: 26: FTRUNCATE() ERR => -1 (Invalid argument)
[2015-09-18 00:39:38.466192] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-2: remote
operation failed [Invalid argument]
[2015-09-18 00:39:38.466353] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-3: remote
operation failed [Invalid argument]
[2015-09-18 00:39:38.467464] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-0: remote
operation failed [Invalid argument]
[2015-09-18 00:39:38.467848] W [MSGID: 114031]
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-1: remote
operation failed [Invalid argument]
---- repeated ----
[2015-09-18 00:39:38.473542] W [fuse-bridge.c:1230:fuse_err_cbk]
0-glusterfs-fuse: 27: FLUSH() ERR => -1 (Invalid argument)
I need your help. anything about this is fine.
If you need some environment informations, I'm ready to inform that.
Thanks
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
