On Mon, Feb 16, 2009 at 7:09 PM, Gordan Bobic <gordan@xxxxxxxxxx> wrote: > >> When a large multi user, multi file, multi thread simulation of a >> total file output of 18GB is run, I plot the output of vmstat 1 and >> see a definite pattern with is very periodic. The bo values start at >> around 200MB, then drop down to 0 in most cases for a few seconds, >> then spike to ~700MB/s then eases back down to 200, 150 and back down >> to 0. It looks very much like a cacheing issue to me. These numbers >> are almost identical on the FC switches. > > How are your test files distributed across directories, and what is your > ratio of reads to writes? Are you mounting with noatime,nodiratime,noquota? > What is your clustering network connection? The tests are from a single node in the cluster, using a single filesystem/mount point. I am doing writes only. I mount using noatime,nodiratime. When I tried noquota, I got an error for invalid option. > > If all your files are in the same directory (or a small number of > subdirectories) and the access is distributed across all the nodes, then I > have to say that you may well be out of luck and what you are seeing is > normal. Bouncing directory locks between the nodes on each access will > introduce enough latency to kill the performance. Also remember that no two > nodes can have a lock on the same file at the same time, and for file > creation/deletion, that means a directory lock, which in turn means only one > file creation/deletion per directory at any one time. I'll try it with a single file and check the difference. Thanks. -Corey > > I can well believe the 900MB/s figure if you are just reading back one big > file from multiple nodes. But the performance will fall off a cliff on > random I/O involving writes. > > Gordan > > -- > Linux-cluster mailing list > Linux-cluster@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
