|
Hi guys, (I'm new to this, so pardon me if my shenanigans turns out to be a waste of your time.) I have been experimenting with Gluster by copying and deleting large numbers of files of all sizes. What I found was that when deleting a large number of small files, the deletion process seems to take a good chunk of my time -- in some cases it seemed to take a significant percentage of the time that it took to copy the files to the cluster to begin with. I'm guessing that the reason is a combination of find and rm -fr processing files serially and having to wait on the packets to travel back and forth over the network. But with a clustering filesystem, the bottleneck is processing files serially and waiting for network packets when you don't have to. So I decided to try an experiment. Instead of using /bin/rm to delete files serially, I wrote my own quick-and-dirty recursive rm (and recursive ls) that uses pthreads (listed as "cluster-rm" and "cluster-ls" in the table below): Methods: 1) This was done on a Linux system. I suspect that Linux (or any modern OS) caches filesystem information. For example, after setting up a directory, when running rm -fr on that directory, the time for rm to complete is lessened if I first run find on the same directory. So to avoid this caching effect, each command was run on it's own test directory. (I.e. find was never run on the same directory as rm -fr or cluster-rm.) This approach seemed to prevent inconsistencies resulting from any caching behavior, resulting in run times that were more consistent. 2) Each test directory contained the exact same data for each of the four commands tested (find, cluster-ls, rm, cluster-rm) for each test run. 3) All commands were run on a client machine and not one of the cluster nodes. Results:
Conclusions: 1) Once I had a threaded version of rm, a threaded version of ls was easy to make, so I included it in the tests (listed above as cluster-ls). Performance looked spiffy up until the 1.1TB range, when cluster-ls started taking more time than find. Right now I can't explain why. 1.1TB takes a long time to set up and process (about a day for each set of four commands), it could be that regular nightly backups might be interfering with performance. If that's the case, then it calls into question the usefulness of my threaded approach. Also, naturally the output from cluster-ls is out of order, so grep and sed would most likely be used in conjunction with something like that, and I haven't yet time-tested 'cluster-ls | some-other-command' against using plain old find by itself. 2) Results from cluster-rm look pretty good to me across the board. Again, performance seems to fall off in the 1.1TB tests, and the reasons are not clear to me at this time, but performance is still half that of rm -fr. Run times fluctuate more than in the previous tests, but I suppose that's to be expected. But since performance does drop, it makes me wonder how well this approach scales on larger sets of data. 3) My threaded cluster-rm/ls commands are not clever. While traversing directories, any subdirectories found would result in a new thread to process it, up until some hard-coded limit is reached (for the above results, 100 threads were used). After the thread count limit is reached, directories are processed using plain old recursion until a thread exits, freeing up a thread to process another subdirectory. Further Research: A) I would like to test further with larger data sets. B) I would like to implement a smarter algorithm for determining how many threads to use to maximize performance. Rather than a hard-coded maximum, a better approach might be to use some metric for measuring number of inodes processed per second, and use that to determine the effectiveness of adding more threads until a local maxima is reached. C) How do these numbers change if the commands are run on one of the cluster nodes instead of a client? I have some ideas of smarter things to try, but I am at best an inexperienced (if enthusiastic) dabbler in the programming arts. A professional would likely do a much better job. But if this data looks at all interesting or useful, then maybe there would be a call for a handful of cluster-specific filesystem tools? Michael Peek |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
