Hello list, We are having critical failures under load of CentOS7 glusterfs 5.3 with our servers losing their local mount point with the issue - "Transport endpoint is not connected" Not sure if it is related but the logs are full of the following message. [2019-03-18 14:00:02.656876] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler We operate multiple separate glusterfs distributed clusters of about 6-8 nodes. Our 2 biggest, separate, and most I/O active glusterfs clusters are both having the issues. We are trying to use glusterfs as a unified file system for pureftpd backup services for a VPS service. We have a relatively small backup window of the weekend where all our servers backup at the same time. When backups start early on Saturday it causes a sustained massive amount of FTP file upload I/O for around 48 hours with all the compressed backup files being uploaded. For our london 8 node cluster for example there is about 45 TB of uploads in ~48 hours currently. We do have some other smaller issues with directory listing under this load too but, it has been working for a couple years since 3.x until we've updated recently and randomly now all servers are losing their glusterfs mount with the "Transport endpoint is not connected" issue. Our glusterfs servers are all mostly the same with small variations. Mostly they are supermicro E3 cpu, 16 gb ram, LSI raid10 hdd (with and without bbu). Drive arrays vary between 4-16 sata3 hdd drives each node depending on if the servers are older or newer. Firmware is kept up-to-date as well as running the latest LSI compiled driver. the newer 16 drive backup servers have 2 x 1Gbit LACP teamed interfaces also. [root@lonbaknode3 ~]# uname -r 3.10.0-957.5.1.el7.x86_64 [root@lonbaknode3 ~]# rpm -qa |grep gluster centos-release-gluster5-1.0-1.el7.centos.noarch glusterfs-libs-5.3-2.el7.x86_64 glusterfs-api-5.3-2.el7.x86_64 glusterfs-5.3-2.el7.x86_64 glusterfs-cli-5.3-2.el7.x86_64 glusterfs-client-xlators-5.3-2.el7.x86_64 glusterfs-server-5.3-2.el7.x86_64 glusterfs-fuse-5.3-2.el7.x86_64 [root@lonbaknode3 ~]# [root@lonbaknode3 ~]# gluster volume info all Volume Name: volbackups Type: Distribute Volume ID: 32bf4fe9-5450-49f8-b6aa-05471d3bdffa Status: Started Snapshot Count: 0 Number of Bricks: 8 Transport-type: tcp Bricks: Brick1: lonbaknode3.domain.net:/lvbackups/brick Brick2: lonbaknode4.domain.net:/lvbackups/brick Brick3: lonbaknode5.domain.net:/lvbackups/brick Brick4: lonbaknode6.domain.net:/lvbackups/brick Brick5: lonbaknode7.domain.net:/lvbackups/brick Brick6: lonbaknode8.domain.net:/lvbackups/brick Brick7: lonbaknode9.domain.net:/lvbackups/brick Brick8: lonbaknode10.domain.net:/lvbackups/brick Options Reconfigured: transport.address-family: inet nfs.disable: on cluster.min-free-disk: 1% performance.cache-size: 8GB performance.cache-max-file-size: 128MB diagnostics.brick-log-level: WARNING diagnostics.brick-sys-log-level: WARNING client.event-threads: 3 performance.client-io-threads: on performance.io-thread-count: 24 network.inode-lru-limit: 1048576 performance.parallel-readdir: on performance.cache-invalidation: on performance.md-cache-timeout: 600 features.cache-invalidation: on features.cache-invalidation-timeout: 600 [root@lonbaknode3 ~]# Mount output shows the following: lonbaknode3.domain.net:/volbackups on /home/volbackups type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) If you notice anything in our volume or mount settings above missing or otherwise bad feel free to let us know. Still learning this glusterfs. I tried searching for any recommended performance settings but, it's not always clear which setting is most applicable or beneficial to our workload. I have just found this post that looks like it is the same issues. https://lists.gluster.org/pipermail/gluster-users/2019-March/035958.html We have not yet tried the suggestion of "performance.write-behind: off" but, we will do so if that is recommended. Could someone knowledgeable advise anything for these issues? If any more information is needed do let us know. |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
