[PATCH 0/8] Speed up cache loading time

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, May 9, 2014 at 5:27 PM, Duy Nguyen <pclouds@xxxxxxxxx> wrote:
> The below patch implements such a daemon to cache the index. It takes
> 91ms and 377ms to load a 25MB index with and without the daemon. I use
> share memory instead of pipe, but the format is still "on disk" not
> "in memory" for simplicity. I think we're good even without in memory
> format.

Here is a better version (on top of split-index). I duplicated webkit
index 8 times to get its size to 199MB (version 2), close to what
Facebook tried last time [1]. read_cache() on index v2, v4, with the
daemon caching v2 and v4 respectively is 2994.861ms (199MB index
file), 2245.113ms (118MB) and 663.399ms and 880.935ms. The best number
is 4.5 times better the worst.

That is clocked at 800 MHz. A repository at this size deserves a
better CPU. At 2.5 GHz we spend 183.228ms on loading the index. A
reasonable number to me. If we scale other parts of git-status as well
as this, we should be able to make "git status" within 1 or 2 seconds.

The tested index does not have fully populated cache-tree so real
world numbers could be a bit higher.

[1] http://thread.gmane.org/gmane.comp.version-control.git/189776/focus=190156

Nguyễn Thái Ngọc Duy (8):
  read-cache: allow to keep mmap'd memory after reading
  unix-socket: stub impl. for platforms with no unix socket support
  daemonize: set a flag before exiting the main process
  Add read-cache--daemon for caching index and related stuff
  read-cache: try index data from shared memory
  read-cache--daemon: do not read index from shared memory
  read-cache: skip verifying trailing SHA-1 on cached index
  read-cache: inform the daemon that the index has been updated

 .gitignore                                     |   1 +
 Documentation/config.txt                       |   4 +
 Documentation/git-read-cache--daemon.txt (new) |  27 ++++
 Makefile                                       |   8 +
 builtin/gc.c                                   |   2 +-
 cache.h                                        |   7 +-
 config.c                                       |  12 ++
 config.mak.uname                               |   1 +
 daemon.c                                       |   2 +-
 environment.c                                  |   1 +
 read-cache--daemon.c (new)                     | 208 +++++++++++++++++++++++++
 read-cache.c                                   | 116 +++++++++++++-
 setup.c                                        |   4 +-
 submodule.c                                    |   1 +
 unix-socket.h                                  |  18 +++
 wrapper.c                                      |  14 ++
 16 files changed, 414 insertions(+), 12 deletions(-)
 create mode 100644 Documentation/git-read-cache--daemon.txt
 create mode 100644 read-cache--daemon.c

-- 
1.9.1.346.ga2b5940

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]