On Sat, 17 May 2008, Luke McGregor wrote:
ok so basicly a file lookup works by broadcasting to the network and anyone who has the file replies, the location is then cached locally and the next time the file is needed no broadcast is needed as it is in local cache? correct?
The file's _location_ gets cached, from how I understand what Avati said. Not the file itself. It avoids the look-up, but doesn't avoid the transfer.
The reason we dont really want to go with caching as a final solution is that it wont reduce toward an optimal solution over time, ideally files that are commonly used should no longer be located in nodes that dont use them, instead they should be located locally. Well thats the theory anyway, if this isnt the case i think it may still be useful doing the work to prove the point that it doesnt provide any great benefit.
It would depend on your workload, but I think it WOULD give considerable benefit. You'd just need to "cache" the file to the local store, rather than caching it's location. The file in the local store is not really "cached", it's copied, so next time another node asks for it, the node that copied it to itself would respond to the request.
This would need a change in the handling of how a file is locked/accessed. A lock would need to be made on all the nodes that could respond with the file. AFR doesn't quite do this, the primary node (first one on the AFR list) is the one that acts as the lock server. If this could be overcome, then you'd have most of the components you need.
Im a little worried that there might be a sticking point with the current lookup scheme if there are multiple copies however. Im not quite sure how to get around consistancy if you want to guarentee that every write accesses the most recent copy only. I can see this as a serious problem. Im not too sure how to get around it but i will have a think about it.
I think there is an upcoming fix in 1.3.10 that enures that O_APPEND writes are atomic in AFR. You would need something similar, with implicit locking across the cluster.
I personally think that they would get a better performance benefit by breaking the file down into small pieces and spreading them over the network to get better read performance from the network as there are more hosts doing small amounts of disk IO each, i suppose this is similar to your stripe?
That sounds exactly the same as the stripe, but if your network I/O is faster than your disk I/O, then why bother caching locally in the first place? The main assumption in the first place was that a file that is closer could be read faster, hence why local migration.
However the acedemics in the department all seem very sold on the migration idea. personally i come from a RAID/SAN background.
File migration idea is more academically sexy. You have a self-optimizing file system, which is a very cool idea, and a lot more redundant than RAID/SAN. With file migration + minimum redundancy specification, you could lose entire nodes transparently. SAN cannot have that level of redundancy. SAN also becomes the bottleneck, whereas the GlusterFS solution with migrating redundant files would be much, much more scaleable.
Gordan
