Re: Question: .idx without .pack causes performance issues?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Jul 21, 2015 at 4:37 PM, Doug Kelly <dougk.ff7@xxxxxxxxx> wrote:
> On Tue, Jul 21, 2015 at 3:48 PM, Junio C Hamano <gitster@xxxxxxxxx> wrote:
>> Junio C Hamano <gitster@xxxxxxxxx> writes:
>>
>>> While I still think that it is more important to prevent such a
>>> situation from occurring in the first place, ignoring .idx that lack
>>> corresponding .pack should be fairly simple, perhaps like this.
>>> ...
>>
>> Sorry for the noise, but this patch is worthless.  We already have
>> an equivalent test in add_packed_git() that is called from this same
>> place.
>
> And a few extra updates from me: we found that this appears to occur
> even after update to 1.9.5, and setting core.fscache on 2.4.6 has no
> appreciable impact on the time it takes to run "git fetch", either.
> Our thought was antivirus (or something else?) might have the file
> open when git attempts to unlink the .idx, but perhaps it's something
> else, too?  In one case, we had ~560 orphaned .idx files, but 150
> seems sufficient to slow a fetch operation for a few minutes until it
> actually begins transferring objects.
>
> The "git gc" approach to cleaning up the mess is certainly looking
> more and more attractive... :)

Here's a change to prune.c that at least addresses the issue by removing
.idx files without an associated pack, but it's by no means pretty.  If anyone
has any feedback before I turn this into a formal patch, it's more than welcome!

diff --git a/builtin/prune.c b/builtin/prune.c
index 10b03d3..8a60282 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "commit.h"
 #include "diff.h"
+#include "dir.h"
 #include "revision.h"
 #include "builtin.h"
 #include "reachable.h"
@@ -85,15 +86,31 @@ static void remove_temporary_files(const char *path)
 {
        DIR *dir;
        struct dirent *de;
+       struct strbuf idx, pack;

        dir = opendir(path);
        if (!dir) {
                fprintf(stderr, "Unable to open directory %s\n", path);
                return;
        }
-       while ((de = readdir(dir)) != NULL)
+       while ((de = readdir(dir)) != NULL) {
                if (starts_with(de->d_name, "tmp_"))
                        prune_tmp_file(mkpath("%s/%s", path, de->d_name));
+               if (ends_with(de->d_name, ".idx")) {
+                       strbuf_init(&idx, 0);
+                       strbuf_init(&pack, 0);
+                       strbuf_addstr(&idx, de->d_name);
+                       strbuf_addbuf(&pack, &idx);
+                       if (strbuf_strip_suffix(&pack, ".idx")) {
+                               strbuf_addstr(&pack, ".pack");
+                               if (!file_exists(mkpath("%s/%s", path,
pack.buf)))
+                                       prune_tmp_file(mkpath("%s/%s",
path, idx.buf));
+                       }
+                       strbuf_release(&idx);
+                       strbuf_release(&pack);
+               }
+
+       }
        closedir(dir);
 }

--
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]