On Thu, 11 Oct 2007 13:06:39 -0400, Bill Davidsen wrote:
> Andrew Clayton wrote:
> > On Fri, 5 Oct 2007 16:56:03 -0400, John Stoffel wrote:
> >
> > >> Can you start a 'vmstat 1' in one window, then start whatever
> > >> you do
> >> to get crappy performance. That would be interesting to see.
> >> >
> > In trying to find something simple that can show the problem I'm
> > seeing. I think I may have found the culprit.
> >
> > Just testing on my machine at home, I made this simple program.
> >
> > /* fslattest.c */
> >
> > #define _GNU_SOURCE
> >
> > #include <stdio.h>
> > #include <stdlib.h>
> > #include <unistd.h>
> > #include <sys/stat.h>
> > #include <sys/types.h>
> > #include <fcntl.h>
> > #include <string.h>
> >
> >
> > int main(int argc, char *argv[])
> > {
> > char file[255];
> >
> > if (argc < 2) {
> > printf("Usage: fslattest file\n");
> > exit(1);
> > }
> >
> > strncpy(file, argv[1], 254);
> > printf("Opening %s\n", file);
> >
> > while (1) {
> > int testfd = open(file, >
> > O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600); close(testfd);
> > unlink(file);
> > sleep(1);
> > }
> >
> > exit(0);
> > }
> >
> >
> > If I run this program under strace in my home directory (XFS file
> > system on a (new) disk (no raid involved) all to its own.like
> >
> > $ strace -T -e open ./fslattest test
> >
> > It doesn't looks too bad.
> >
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <0.005043> open("test",
> > O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000212>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <0.016844>
> >
> > If I then start up a dd in the same place.
> >
> > $ dd if=/dev/zero of=bigfile bs=1M count=500
> >
> > Then I see the problem I'm seeing at work.
> >
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <2.000348> open("test",
> > O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <1.594441>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <2.224636> open("test",
> > O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <1.074615>
> >
> > Doing the same on my other disk which is Ext3 and contains the root
> > fs, it doesn't ever stutter
> >
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <0.015423> open("test",
> > O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000092>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <0.000093> open("test",
> > O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000088>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <0.000103> open("test",
> > O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000096>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <0.000094> open("test",
> > O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000114>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <0.000091> open("test",
> > O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000274>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3
> > <0.000107>
> >
> >
> > Somewhere in there was the dd, but you can't tell.
> >
> > I've found if I mount the XFS filesystem with nobarrier, the
> > latency is reduced to about 0.5 seconds with occasional spikes > 1
> > second.
> >
> > When doing this on the raid array.
> >
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <0.009164>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <0.000071>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <0.002667>
> >
> > dd kicks in
> >
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <11.580238>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <3.222294>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <0.888863>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <4.297978>
> >
> > dd finishes >
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <0.000199>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <0.013413>
> > open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 <0.025134>
> >
> >
> > I guess I should take this to the XFS folks.
>
> Try mounting the filesystem "noatime" and see if that's part of the
> problem.
Yeah, it's mounted noatime. Looks like I tracked this down to an XFS
regression.
http://marc.info/?l=linux-fsdevel&m=119211228609886&w=2
Cheers,
Andrew
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
