Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx> wrote:
>
> On Mon, 28 May 2007, Shawn O. Pearce wrote:
> >
> > What if we use a "slow" storage by "refs/notes/$objname/$notename",
>
> This _really_ won't scale. Even if the notes don't exist, just doing the
> lookup (which will fail for most commits) will be horribly slow, and will
> populate the dentry cache with negative entries.
Yes.
I think you missed what I was trying to say. We *definately* do not
want to probe the OS and ask "do you have $objname1? $objname2?"
for exactly the reason you just stated. (Though you being some
sort of kernel guru means you know a hellva lot more about that
dentry cache thing than I do!)
What I meant was something more like:
DIR *d = opendir("refs/notes");
if (d) {
... notes can exist in both packfiles and "loose" ...
only_packed_notes = 0;
while (readdir(d)) {
... the entry name here is the name of an object ...
... stuff into a hash table, we can come back later ...
... for the subdirectory contents if we need it ...
}
closedir(d);
... also examine "packed-refs", in case any are there ...
} else {
only_packed_notes = 1;
... we *only* have notes in packfiles, if we have any at all...
}
Then looking up a note is just a probe into our in-memory hash
(if only_packed_notes is false) and a probe into the packfile(s) to
find the notes for the object. Not very expensive if the packfiles
have the reverse obj->tag mappings indexed within them.
> To get good filesystem performance, you have to
> - have reasonable hit-rates (and looking it up for each commit is _not_
> going to do that)
> - not have millions of objects.
Which is why we:
a) allow these things to migrate into packed-refs, because
getting into there is a hellva lot cheaper than getting into
a packfile;
b) move them into a packfile when we repack loose objects,
because then we have really good access.
c) take them out of packed-refs once they are into a packfile,
and get them out of the loose refs/notes directory as early as
possible.
--
Shawn.
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
