On 11/04/2009 08:20 PM, Evan Jones wrote:
The hail project seems very interesting. Two quick questions: a) Was Yahoo Zookeeper considered instead of reimplementing these ideas as CLD? Was it discarded for some reason?
Yes, I think it departed too much from a desirable filesystem-like API. Using lock/unlock/trylock for ordered contention resolution, leader election, garbage collection, and similar tasks is a convenient model. CLD will probably be more Chubby-like than Zookeeper.
At the time I started CLD, it looked like Zookeeper was a project on life support, Yahoo was bleeding money, and the move to Apache looked like a dump-n-run project with no promising future. AT THE TIME it seemed like something to avoid; but does still continue to have a life and receive attention, so this assessment was probably too conservative and premature.
b) Has there been any consideration for re-using an existing serialization format (XDR, Thrift, Protocol Buffers, whatever)?
In what context should these be considered? We are certainly open to using open, documented protocols -- one big reason why Amazon S3 (HTTP REST) and NFSv4 were chosen as production examples for the tabled and nfs4d projects, respectively.
chunkd's protocol is custom because I'm not aware of a highly efficient binary network storage protocol that fits its niche. It sits somewhere in between nbd and iSCSI+{SBC|OSD}, technology-wise. chunkd provides storage at a low level; data is already serialized by the time it reaches chunkd.
I'm just interested in why existing infrastructure may be inappropriate for Hail. Part of my motivation is that I hope to see some "consolidation" in this space: I would like there to be a small number (1 or 2) of various distributed systems infrastructure, so we might start to see some real "compatibility." Right now it seems as though everyone decides to build absolutely everything from scratch.
Right now cloud computing is a hotbed of activity, a very exciting time. I would rather see -more- interesting projects, than less, at this juncture. Open source cloud computing is really just in its infancy, even though its been deployed in closed source production for years.
Jeff -- To unsubscribe from this list: send the line "unsubscribe hail-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
