OK, so the previous good results are indeed too good to be true. Here's a more reasonable evaluation http://www.cs.princeton.edu/~wdong/gluster/large.gif where I enlarged the number of images created by 10x so everything no longer fit in main memory. Still look good to me. Wei Dong wrote: > I think it is fuse that causes the slowness. I ran all experiments > with booster enabled and here's the new figure: > http://www.cs.princeton.edu/~wdong/gluster/summary-booster.gif . The > numbers are MUCH better than NFS in most cases except for the local > setting, which is not practically interesting. The interesting thing > is that all of a sudden, the deleting rate drop by 4-10 times -- > though I don't really care about file deletion. > > I must say that I'm totally satisfied by the results. > > - Wei > > > Wei Dong wrote: >> Hi All, >> >> I complained about the low file creation rate with the glusterfs on >> my cluster weeks ago and Avati suggested I started with a small >> number of nodes. I finally get sometime to seriously benchmark >> glusterfs with Bonnie++ today and the results confirms that glusterfs >> is indeed slow in terms of file creating. My application is to store >> a large number of ~200KB image files. I use the following bonnie++ >> command for evaluation (create 10K files of 200KiB each scattered >> under 100 directories): >> >> bonnie++ -d . -s 0 -n 10:200000:200000:100 >> >> Since sequential I/O is not that interesting to me, I only keep the >> random I/O results. >> >> My hardware configuration is 2xquadcore Xeon E5430 2.66GHz, 16GB >> memory, 4 x Seagate 1500GiB 7200RPM hard drive. The machines are >> connected with gigabit ethernet. >> >> I ran several GlusterFS configurations, each named as N-R-T, where N >> is the number of replicated volumes aggregated, R is the number of >> replications and T is number of server side I/O thread. I use one >> machine to serve one volume so there are NxR servers and one separate >> client running for each experiment. On the client side, the server >> volumes are first replicated and then aggregated -- even with 1-1-2 >> configuration, the single volume is wrapped by a replicate and a >> distribute translator. To show the overhead of those translators, I >> also run a "simple" configuration which is 1-1-2 without the extra >> replicate & distribute translators, and a "local" configuration which >> is "simple" with client & server running on the same machine. These >> configurations are compared to "nfs" and "nfs-local", which is NFS >> with server and client on the same machine. The GlusterFS volume >> file templates are attached to the email. >> >> The result is at >> http://www.cs.princeton.edu/~wdong/gluster/summary.gif . The >> bars/numbers shown are operations/second, so the larger the better. >> >> Following are the messages shown by the figure: >> 1. GlusterFS is doing a exceptionally good job on deleting files, >> but creates and reads files much slower than both NFS. >> 2. At least for one node server configuration, network doesn't >> affects the file creation rate and does affects file read rate. >> 3. The extra dummy replicate & distribute translators lowers file >> creation rate by almost half. 4. Replication doesn't hurt >> performance a lot. >> 5. I'm running only single-threaded benchmark, so it's hard to say >> about scalability, but adding more servers does helps a little bit >> even in single-threaded setting. >> >> Note that my results are not really that different from >> http://gluster.com/community/documentation/index.php/GlusterFS_2.0_I/O_Benchmark_Results, >> where the single node configuration file create rate is about 30/second. >> >> I see no reason why GlusterFS has to be that slower than NFS in file >> creation in single node configuration. I'm wondering if someone here >> can help me figure out what's wrong in my configuration or what's >> wrong in the GlusterFS implementation. >> >> - Wei >> >> Server volume: >> >> volume posix >> type storage/posix >> option directory /state/partition1/wdong/gluster >> end-volume >> >> volume lock >> type features/locks >> subvolumes posix >> end-volume >> >> volume brick >> type performance/io-threads >> option thread-count 2 >> subvolumes lock >> end-volume >> >> volume server >> type protocol/server >> option transport-type tcp >> option auth.addr.brick.allow 192.168.99.* >> option transport.socket.listen-port 6999 >> subvolumes brick >> end-volume >> >> >> Client volume >> >> volume brick-0-0 >> type protocol/client >> option transport-type tcp >> option remote-host c8-0-0 >> option remote-port 6999 >> option remote-subvolume brick >> end-volume >> >> volume brick-0-1 ... >> >> volume rep-0 >> type cluster/replicate >> subvolumes brick-0-0 brick-0-1 ... >> >> ... >> volume union >> type cluster/distribute >> subvolumes rep-0 rep-1 rep-2 rep-3 rep-4 rep-5 rep-6 rep-7 >> end-volume >> >> volume client >> type performance/write-behind >> option cache-size 32MB >> option flush-behind on >> subvolumes union >> end-volume >> >> >> For those who are interested enough to see the real configuration >> files, I have all the configuration files and server/client logs >> uploaded to http://www.cs.princeton.edu/~wdong/gluster/run.tar.gz . >> > >
