On 06/07/2010 09:21 AM, Jeff Darcy wrote:
On 06/04/2010 11:27 PM, Pete Zaitcev wrote:
About the focus, ZK is just like CLD from a certain angle (it has
the good old files and provides a set of un-posixy operations
on them: watches, uniques, "ephemerals"), but it's also entirely
unlike CLD (e.g. no locks in the protocol). CLD's model is that
clients are daemons, each of which reads a few of its files, maybe
locks one or two at boot, and then nothing happens except keepalives.
Zookeeper's model... honestly I don't know what it is because it's
never explained concisely, but the docs that I saw seem to imply
huge numbers of clients all doing random ops all the time on the
same files, enough to cause a herd concerns. It looks like Yahoo
may be using Zookeeper as a lease manager or something. Crazy.
According to the Chubby paper (available at e.g.
http://labs.google.com/papers/chubby.html) Chubby itself is mostly used
as a naming service. They even say explicitly that few clients hold
locks (page 11, left column, under "use and behavior").
I heard people say they cribbed from the same Chubby paper, but
it's bollocks. It's absolutely nothing like what Chubby implies.
No locks for one thing.
I guess it could be argued that locks are the essential feature of
Chubby - both the name of the service and the title of the paper refer
to it - but I think your dismissal of ZK and Chubby as unrelated is
itself bollocks. Watches and ephemerals within a hierarchical namespace
of small pseudo-files are also central to how Chubby is actually used
within Google, even without locking. ZK copied those features, along
with many of the underlying algorithms and protocols. CLD copied a
slightly different set. There's still a lot of overlap, even if the one
feature that has been overused in Hail is absent (by design) in ZK.
I think you're overselling that angle a bit. Google discourages use of
Chubby as a strict publish-subscribe mechanism, so watches and
ephemerals aren't the bread-and-butter of Chubby necessarily. A lot of
the supposed commonality comes simply from the attribute of being a
centralized respository of data for autonomous cloud systems -- a
shared, highly reliable filesystem -- not any particular attribute
related to watches or ephemerals.
So I do not see
an upside for us to switch at this point, and I have better things
to do than learning Zookeeper ropes for weeks.
That's a fair enough evaluation. I think I'm the one who might have
suggested investigating ZK, but the reasons for that are largely
community-related and not technical. If the technical barriers are too
high, then it's probably not worth the bother. That does, however,
leave the door open for someone else to provide more scalable solutions
for the same problems by using more appropriate primitives. If the goal
of Hail is to provide building blocks for people to use in constructing
their own higher-level service, then it seems to me that the building
blocks should be of a familiar shape and the adoption of ZK is about
100x that of CLD for similar purposes.
Not sure what you mean by familiar shape? ZK design is too haphazard,
and developers IMO have an easier time grasping CLD's fundamental
FS-like API. The implementation is also 3x or more in terms of size,
compared to CLD.
I'd rather get the basics right first, then pursue a popularity contest :)
Jeff
--
To unsubscribe from this list: send the line "unsubscribe hail-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
