I suspect this is a bug with 'Transport: rdma' part. We have called out for de-scoping that feature as we are lacking experts in that domain right now. Recommend you to use IPoIB option, and use tcp/socket transport type (which is default). That should mostly fix all the issues.
-Amar
On Thu, Jan 24, 2019 at 5:31 AM Jim Kinney <jim.kinney@xxxxxxxxx> wrote:
_______________________________________________That really sounds like a bug with the sharding. I'm not using sharding on my setup and files are writeable (vim) with 2 bytes and no errors occur.Perhaps the small size is cached until it's large enough to trigger a writeOn Wed, 2019-01-23 at 21:46 -0200, Lindolfo Meira wrote:Also I noticed that any subsequent write (after the first write with 340bytes or more), regardless the size, will work as expected.Lindolfo Meira, MScDiretor Geral, Centro Nacional de SupercomputaçãoUniversidade Federal do Rio Grande do Sul+55 (51) 3308-3139On Wed, 23 Jan 2019, Lindolfo Meira wrote:Just checked: when the write is >= 340 bytes, everything works assupposed. If the write is smaller, the error takes place. And when itdoes, nothing is logged on the server. The client, however, logs thefollowing:[2019-01-23 23:28:54.554664] W [MSGID: 103046][rdma.c:3502:gf_rdma_decode_header] 0-rpc-transport/rdma: received a msgof type RDMA_ERROR[2019-01-23 23:28:54.554728] W [MSGID: 103046][rdma.c:3939:gf_rdma_process_recv] 0-rpc-transport/rdma: peer(172.24.1.6:49152), couldn't encode or decode the msg properly or writechunks were not provided for replies that were bigger thanRDMA_INLINE_THRESHOLD (2048)[2019-01-23 23:28:54.554765] W [MSGID: 114031][client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-gfs-client-5: remoteoperation failed [Transport endpoint is not connected][2019-01-23 23:28:54.554850] W [fuse-bridge.c:1436:fuse_err_cbk]0-glusterfs-fuse: 1723199: FLUSH() ERR => -1 (Transport endpoint is notconnected)Lindolfo Meira, MScDiretor Geral, Centro Nacional de SupercomputaçãoUniversidade Federal do Rio Grande do Sul+55 (51) 3308-3139On Wed, 23 Jan 2019, Lindolfo Meira wrote:Hi Jim. Thanks for taking the time.Sorry I didn't express myself properly. It's not a simple matter ofpermissions. Users can write to the volume alright. It's when vim and nanoare used, or when small file writes are performed (by cat or echo), thatit doesn't work. The file is updated with the write in the server, but itshows up as empty in the client.I guess it has something to do with the size of the write, because I ran atest writing to a file one byte at a time, and it never showed up ashaving any content in the client (although in the server it kept growingaccordingly).I should point out that I'm using a sharded volume. But when I was testinga striped volume, it also happened. Output of "gluster volume info"follows bellow:Volume Name: gfsType: DistributeVolume ID: b5ef065f-1ba2-481f-8108-e8f6d2d3f036Status: StartedSnapshot Count: 0Number of Bricks: 6Transport-type: rdmaBricks:Brick1: pfs01-ib:/mnt/dataBrick2: pfs02-ib:/mnt/dataBrick3: pfs03-ib:/mnt/dataBrick4: pfs04-ib:/mnt/dataBrick5: pfs05-ib:/mnt/dataBrick6: pfs06-ib:/mnt/dataOptions Reconfigured:nfs.disable: onfeatures.shard: onLindolfo Meira, MScDiretor Geral, Centro Nacional de SupercomputaçãoUniversidade Federal do Rio Grande do Sul+55 (51) 3308-3139On Wed, 23 Jan 2019, Jim Kinney wrote:Check permissions on the mount. I have multiple dozens of systemsmounting 18 "exports" using fuse and it works for multiple userread/write based on user access permissions to the mount point space./home is mounted for 150+ users plus another dozen+ lab storage spaces.I do manage user access with freeIPA across all systems to keep thingsconsistent.On Wed, 2019-01-23 at 19:31 -0200, Lindolfo Meira wrote:Am I missing something here? A mere write operation, using vim ornano, cannot be performed on a gluster volume mounted over fuse! Whatgives?Lindolfo Meira, MScDiretor Geral, Centro Nacional deSupercomputaçãoUniversidade Federal do Rio Grande do Sul+55 (51)3308-3139_______________________________________________Gluster-usersmailinglistGluster-users@xxxxxxxxxxxhttps://lists.gluster.org/mailman/listinfo/gluster-users--James P. Kinney IIIEvery time you stop a school, you will have to build a jail. What yougain at one end you lose at the other. It's like feeding a dog on hisown tail. It won't fatten the dog.- Speech 11/23/1900 Mark Twainhttp://heretothereideas.blogspot.com/--James P. Kinney III Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark Twain http://heretothereideas.blogspot.com/
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
Amar Tumballi (amarts)
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
