On Fri, Aug 26, 2011 at 12:10 PM, Tommi Virtanen
<tommi.virtanen@xxxxxxxxxxxxx> wrote:
> On Fri, Aug 26, 2011 at 11:51, Christian Brunner <chb@xxxxxx> wrote:
>> + if (!buf) {
>> + len = ofs-lastofs;
>> + tempbuf = (byte *) malloc(len);
>> + if (!tempbuf)
>> + return -ENOMEM;
>> + hashbuf = tempbuf;
>> + }
>> + Hash->Update((const byte *) hashbuf, len);
>
> That'll still try to allocate 100GB of RAM for a 100GB hole. It needs
> to loop through big holes in smaller chunks, feeding them to the hash
> e.g. 8kB at a time. And at that point you might as well just use read
> and not read_iterate, that'll do the memsetting etc for you, and then
> you can use a static buffer and avoid malloc/free every round. There's
> no shortcut to be had from "skipping" holes when you need to feed
> bytes to a hash function.
Well, when using read_iterate you avoid reading extra data over the
network when the object (chunk) exists (and is sparse). We can
probably have some optimization here, and only allocate and memset a
buffer once for the case where len == objsize and reuse it later.
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
