Re: Zookeeper instead of CLD in Hail

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Well, Zookeeper is written in Java. Presumably it requires you to put
a JVM on each node. Unfortunately the JVM has kind of a large memory
footprint. That's great if your software is written in Java. If your
company didn't go that route, ZK doesn't seem like such a great
option.

On Tue, Jun 8, 2010 at 8:07 AM, Jeff Darcy <jdarcy@xxxxxxxxxx> wrote:
> On 06/07/2010 10:32 PM, Jeff Garzik wrote:
>> I think you're overselling that angle a bit.  Google discourages use of
>> Chubby as a strict publish-subscribe mechanism, so watches and
>> ephemerals aren't the bread-and-butter of Chubby necessarily.

The Chubby paper specifically calls out using Chubby for publish /
subscribe as "abusive behavior." But ZooKeeper "is used at Yahoo! as
the coordination and failure recovery service for Yahoo! Message
Broker, which is a highly scalable publish-subscribe system" according
to hadoop.apache.org.

Although they might be equivalent in some theoretical computer science
sense, I get the impression that the two systems are very different
beasts...

>> A lot of
>> the supposed commonality comes simply from the attribute of being a
>> centralized respository of data for autonomous cloud systems -- a
>> shared, highly reliable filesystem -- not any particular attribute
>> related to watches or ephemerals.
>
> Nonetheless, those are features both share, and many might argue that
> they're preferable to locks.  Locking is a fundamentally lousy way to
> build scalable and reliable distributed systems, as has been well known
> for more than a decade.

I think *fine-grained* locking is a fundamentally lousy way to build
distributed systems. I haven't heard anyone argue that coarse-grained
locking is bad.
Have you read any interesting papers or books about this topic?

> If somebody who already had ZK running could point
> chunkd/tabled at that instead of having to download/install/configure
> CLD (and fight with their IT folks about the DNS dependency), then they
> might actually be more likely to use chunkd/tabled instead of something
> even more broken.  Some few of them might even start to contribute.
> Surely that would be a good thing.  If self-imposed technical obstacles
> still mean it's too much effort then fine, but I don't think we need
> more NIH syndrome.

It would be cool to have support another backend, I guess.

However... It seems to me that once you make the decision to use
ZooKeeper and Java, Walrus or Hadoop is probably a more practical
choice for the upper layers.
I've never deployed either though, so I could be wrong.

Colin McCabe
--
To unsubscribe from this list: send the line "unsubscribe hail-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [Fedora Clound]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux