On 10/30/07, Jeff King <peff@xxxxxxxx> wrote: > On Tue, Oct 30, 2007 at 08:38:24AM -0700, Linus Torvalds wrote: > > > > with the old and new code. Pairs like Documentation/git-add-script.txt > > > -> Documentation/git-add.txt are not found, because the file is composed > > > almost entirely of boilerplate. > > > > Ok, that does imply to me that we cannot just drop boilerplate text, > > because the fact is, lots of files contain boilerplate, but people still > > think they are "similar". > > Well, the problem is that instead of just "dropping" boilerplate text, > we fail to count it as a similarity, but it still counts towards the > file size. It may be that just dropping it totally is the right thing > (in which case those renames _will_ turn up, because they will be filled > with identical non-boilerplate goodness). Right, in the demo I make an extra pass after the inverted indexing step to prune the index -- which means eliminating the common lines *entirely* from the index (so they don't get attributed to a random file) *and* decrementing all the file sizes by 1. That way the similarity scores shouldn't get skewed. And as you mentioned we could bump the threshold from 1 to some other small integer. Intuitively I guess you could say it is common to copy a file to 2 places or 3 places, and you don't want all the lines to get thrown out because of that. But usually you don't copy a file to 10 or 50 places. Andy - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
