On 7/14/18, Raghavendra Gowdappa <rgowdapp@xxxxxxxxxx> wrote: > On Fri, Jul 13, 2018 at 5:00 AM, Pat Haley <phaley@xxxxxxx> wrote: > >> >> Hi Raghavendra, >> >> We were wondering if you have had a chance to look at this again, and if >> so, did you have any further suggestions? >> > > Sorry Pat. Too much of work :). I'll be working on a patch today to make > sure read-ahead doesn't flush its cache due to fstats making this behavior > optional. You can try this patch and let us know about results. > Will let you know when patch is ready. Can you apply following two patches, build and remount your client and redo sequential read tests? https://review.gluster.org/#/q/status:open+project:glusterfs+branch:master+topic:ref-1455872 You can keep performance.open-behind on during your tests. Let us know the results. > > >> Thanks >> >> Pat >> >> >> >> On 07/06/2018 01:27 AM, Raghavendra Gowdappa wrote: >> >> >> >> On Fri, Jul 6, 2018 at 5:29 AM, Pat Haley <phaley@xxxxxxx> wrote: >> >>> >>> Hi Raghavendra, >>> >>> Our technician may have some time to look at this issue tomorrow. Are >>> there any tests that you'd like to see? >>> >> >> Sorry. I've been busy with other things and was away from work for couple >> of days. It'll take me another 2 days to work on this issue again. So, >> most >> likely you'll have an update on this next week. >> >> >>> Thanks >>> >>> Pat >>> >>> >>> >>> On 06/29/2018 11:25 PM, Raghavendra Gowdappa wrote: >>> >>> >>> >>> On Fri, Jun 29, 2018 at 10:38 PM, Pat Haley <phaley@xxxxxxx> wrote: >>> >>>> >>>> Hi Raghavendra, >>>> >>>> We ran the tests (write tests) and I copied the log files for both the >>>> server and the client to http://mseas.mit.edu/download/ >>>> phaley/GlusterUsers/2018/Jun29/ . Is there any additional trace >>>> information you need? (If so, where should I look for it?) >>>> >>> >>> Nothing for now. I can see from logs that workaround is not helping. >>> fstat requests are not absorbed by md-cache and read-ahead is witnessing >>> them and flushing its read-ahead cache. I am investigating more on >>> md-cache >>> (It also seems to be invalidating inodes quite frequently which actually >>> might be the root cause of seeing so many fstat requests from kernel). >>> Will >>> post when I find anything relevant. >>> >>> >>>> Also the volume information you requested >>>> >>>> [root@mseas-data2 ~]# gluster volume info data-volume >>>> >>>> Volume Name: data-volume >>>> Type: Distribute >>>> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 >>>> Status: Started >>>> Number of Bricks: 2 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: mseas-data2:/mnt/brick1 >>>> Brick2: mseas-data2:/mnt/brick2 >>>> Options Reconfigured: >>>> diagnostics.client-log-level: TRACE >>>> network.inode-lru-limit: 50000 >>>> performance.md-cache-timeout: 60 >>>> performance.open-behind: off >>>> disperse.eager-lock: off >>>> auth.allow: * >>>> server.allow-insecure: on >>>> nfs.exports-auth-enable: on >>>> diagnostics.brick-sys-log-level: WARNING >>>> performance.readdir-ahead: on >>>> nfs.disable: on >>>> nfs.export-volumes: off >>>> [root@mseas-data2 ~]# >>>> >>>> >>>> On 06/29/2018 12:28 PM, Raghavendra Gowdappa wrote: >>>> >>>> >>>> >>>> On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <phaley@xxxxxxx> wrote: >>>> >>>>> >>>>> Hi Raghavendra, >>>>> >>>>> Our technician was able to try the manual setting today. He found >>>>> that >>>>> our upper limit for performance.md-cache-timeout was 60 not 600, so he >>>>> used that value, along with the network.inode-lru-limit=50000. >>>>> >>>>> The result was another small (~1%) increase in speed. Does this >>>>> suggest some addition tests/changes we could try? >>>>> >>>> >>>> Can you set gluster option diagnostics.client-log-level to TRACE and >>>> run sequential read tests again (with md-cache-timeout value of 60)? >>>> >>>> #gluster volume set <volname> diagnostics.client-log-level TRACE >>>> >>>> Also are you sure that open-behind was turned off? Can you give the >>>> output of, >>>> >>>> # gluster volume info <volname> >>>> >>>> >>>>> Thanks >>>>> >>>>> Pat >>>>> >>>>> >>>>> >>>>> >>>>> On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote: >>>>> >>>>> >>>>> >>>>> On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley@xxxxxxx> wrote: >>>>> >>>>>> >>>>>> Hi Raghavendra, >>>>>> >>>>>> Setting the performance.write-behind off had a small improvement on >>>>>> the write speed (~3%), >>>>>> >>>>>> We were unable to turn on "group metadata-cache". When we try get >>>>>> errors like >>>>>> >>>>>> # gluster volume set data-volume group metadata-cache >>>>>> '/var/lib/glusterd/groups/metadata-cache' file format not valid. >>>>>> >>>>>> Was metadata-cache available for gluster 3.7.11? We ask because the >>>>>> release notes for 3.11 mentions “Feature for metadata-caching/small >>>>>> file >>>>>> performance is production ready.” (https://gluster.readthedocs.i >>>>>> o/en/latest/release-notes/3.11.0/). >>>>>> >>>>>> Do any of these results suggest anything? If not, what further tests >>>>>> would be useful? >>>>>> >>>>> >>>>> Group metadata-cache is just a bunch of options one sets on a volume. >>>>> So, You can set them manually using gluster cli. Following are the >>>>> options >>>>> and their values: >>>>> >>>>> performance.md-cache-timeout=600 >>>>> network.inode-lru-limit=50000 >>>>> >>>>> >>>>> >>>>>> Thanks >>>>>> >>>>>> Pat >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley <phaley@xxxxxxx> wrote: >>>>>> >>>>>>> >>>>>>> Hi Raghavendra, >>>>>>> >>>>>>> Thanks for the suggestions. Our technician will be in on Monday. >>>>>>> We'll test then and let you know the results. >>>>>>> >>>>>>> One question I have, is the "group metadata-cache" option supposed >>>>>>> to >>>>>>> directly impact the performance or is it to help collect data? If >>>>>>> the >>>>>>> latter, where will the data be located? >>>>>>> >>>>>> >>>>>> It impacts performance. >>>>>> >>>>>> >>>>>>> Thanks again. >>>>>>> >>>>>>> Pat >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa < >>>>>>> rgowdapp@xxxxxxxxxx> wrote: >>>>>>> >>>>>>>> For the case of writes to glusterfs mount, >>>>>>>> >>>>>>>> I saw in earlier conversations that there are too many lookups, but >>>>>>>> small number of writes. Since writes cached in write-behind would >>>>>>>> invalidate metadata cache, lookups won't be absorbed by md-cache. I >>>>>>>> am >>>>>>>> wondering what would results look like if we turn off >>>>>>>> performance.write-behind. >>>>>>>> >>>>>>>> @Pat, >>>>>>>> >>>>>>>> Can you set, >>>>>>>> >>>>>>>> # gluster volume set <volname> performance.write-behind off >>>>>>>> >>>>>>> >>>>>>> Please turn on "group metadata-cache" for write tests too. >>>>>>> >>>>>>> >>>>>>>> and redo the tests writing to glusterfs mount? Let us know about >>>>>>>> the >>>>>>>> results you see. >>>>>>>> >>>>>>>> regards, >>>>>>>> Raghavendra >>>>>>>> >>>>>>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa < >>>>>>>> rgowdapp@xxxxxxxxxx> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa < >>>>>>>>> rgowdapp@xxxxxxxxxx> wrote: >>>>>>>>> >>>>>>>>>> For the case of reading from Glusterfs mount, read-ahead should >>>>>>>>>> help. However, we've known issues with read-ahead[1][2]. To work >>>>>>>>>> around >>>>>>>>>> these, can you try with, >>>>>>>>>> >>>>>>>>>> 1. Turn off performance.open-behind >>>>>>>>>> #gluster volume set <volname> performance.open-behind off >>>>>>>>>> >>>>>>>>>> 2. enable group meta metadata-cache >>>>>>>>>> # gluster volume set <volname> group metadata-cache >>>>>>>>>> >>>>>>>>> >>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508 >>>>>>>>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489 >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley <phaley@xxxxxxx> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> We were recently revisiting our problems with the slowness of >>>>>>>>>>> gluster writes (http://lists.gluster.org/pipe >>>>>>>>>>> rmail/gluster-users/2017-April/030529.html). Specifically we >>>>>>>>>>> were testing the suggestions in a recent post ( >>>>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2018-March >>>>>>>>>>> /033699.html). The first two suggestions (specifying a >>>>>>>>>>> negative-timeout in the mount settings or adding >>>>>>>>>>> rpc-auth-allow-insecure to >>>>>>>>>>> glusterd.vol) did not improve our performance, while setting >>>>>>>>>>> "disperse.eager-lock off" provided a tiny (5%) speed-up. >>>>>>>>>>> >>>>>>>>>>> Some of the various tests we have tried earlier can be seen in >>>>>>>>>>> the links below. Do any of the above observations suggest what >>>>>>>>>>> we could >>>>>>>>>>> try next to either improve the speed or debug the issue? Thanks >>>>>>>>>>> >>>>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/0 >>>>>>>>>>> 31565.html >>>>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/03 >>>>>>>>>>> 0937.html >>>>>>>>>>> >>>>>>>>>>> Pat >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>>>>>>>>> Pat Haley Email: phaley@xxxxxxx >>>>>>>>>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>>>>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>>>>>>>>> MIT, Room 5-213 >>>>>>>>>>> http://web.mit.edu/phaley/www/ >>>>>>>>>>> 77 Massachusetts Avenue >>>>>>>>>>> Cambridge, MA 02139-4301 >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>> Gluster-users@xxxxxxxxxxx >>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>>>>> Pat Haley Email: phaley@xxxxxxx >>>>>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>>>>>> 77 Massachusetts Avenue >>>>>>> Cambridge, MA 02139-4301 >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users@xxxxxxxxxxx >>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>>>> Pat Haley Email: phaley@xxxxxxx >>>>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>>>>> 77 Massachusetts Avenue >>>>>> Cambridge, MA 02139-4301 >>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>>> Pat Haley Email: phaley@xxxxxxx >>>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>>>> 77 Massachusetts Avenue >>>>> Cambridge, MA 02139-4301 >>>>> >>>>> >>>> >>>> -- >>>> >>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>> Pat Haley Email: phaley@xxxxxxx >>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>>> 77 Massachusetts Avenue >>>> Cambridge, MA 02139-4301 >>>> >>>> >>> >>> -- >>> >>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>> Pat Haley Email: phaley@xxxxxxx >>> Center for Ocean Engineering Phone: (617) 253-6824 >>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>> 77 Massachusetts Avenue >>> Cambridge, MA 02139-4301 >>> >>> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: phaley@xxxxxxx >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
