On 05/27/2011 04:31 PM, Joe Landman wrote: > On 05/27/2011 07:12 AM, Jon Tegner wrote: >> A general question, suppose I have a parallel application, using mpi, >> where really fast access to the file system is critical. >> >> Would it be stupid to consider a ram disk based setup? Say a 36 port QDR > > Ram disks won't work directly, due to lack of locking in tmpfs. You > could create a tmpfs, then create a file that fills this up, then a > loopback device pointing to that file, then build a file system atop > that, and mount it. And then mount gluster atop that. > > Needless to say, all these layers significantly decrease performance > and introduce inefficiencies. > >> infiniband with half of the ports connected to computational nodes and >> the other half to gluster nodes? > > There may be other options, but the options are not going to be > cheap/inexpensive. How fast, and by fast do you mean bandwidth and/or > latency (e.g. streaming bandwidth or random IOPs)? What does your IO > profile look like? > > You can get nodes that stream 4.6+ GB/s read, and 3.6+ GB/s writes for > single readers/writers to single files. For MPI jobs with single > readers/writers, this is good. For very large IO jobs where you need > 10's of GB/s, you probably need a more specific design to your problem. > > Regards, > > Joe > > Thanks! Would you say that the inefficiencies related to the ram disk would remove all advantages of using ram instead of hard drives (or could it still be worth a try)? As for speed, I would think that latency would be the most critical - but I don't really know, it its the code of a colleague of mine (I was trying to come up with a replacement of his SMP-machine which is getting old). Regards, /jon
