On 11/21/06, Shawn Pearce <spearce@xxxxxxxxxxx> wrote:
Of course this only looks at a single blob object and does not take into account the tree and commit overheads for a given revision, but it does give a really good idea of what is going on.
I have some numbers that also includes the other object types. They are based on running a set of scripts in 5 different repositories. First, each repository has been both packed and unpacked with respect to the different object types to show the compression level and disk-space saving. The main results are compression level for the different object types and a test to see which pack sizes provide "optimal" packing. I will not post the numbers here. They are available in http://jonas.nitro.dk/tmp/stats.pdf for those interested. The following is my "analysis" of the numbers. It can be seen that the blob objects generally control the overall packing properties of the repository, especially when it comes to the compression level. Generally, there are more tree objects than blob objects, which can be due to the fact that both the ELinks and Linux kernel tree are structured into many subdirectories containing few files. The Tig repository is exceptional in that it has only few files and one tree object per revision, which has the effect of reducing the tree object compression level. At 83% on average, tree objects compress very well and in the general case better than blob objects. As expected, the randomness of the content of both commit and tag objects results in a very poor packing performance of only 2%. In terms of disk-space usage between packed and unpacked object stores, it is obvious that the overhead of many small object files is unavoidable. Next, is the examination of how different pack sizes affect the compression level. This also includes looking at how the size of the index file varies. The columns on per-object sizes are of interest. The per-object sizes can be compared with the average object size of each project to get a rough idea of the compression level. In some of the tables, rows are missing because not all repositories contain enough objects with respect to the pack sizes being examined. The data show that for minimal index files, the packs need to contain more than 2500 objects. The 24 bytes per-object for the optimal case includes 20-bytes for the object SHA1, and thus cannot be expected to become lower. For commit objects, the numbers show that there is almost no difference between small and big packs. The same is assumed to be the case for tag objects. For tree objects, it becomes very clear that in repositories with structured source trees, the tree objects compress much better for big packs, whereas a project, such as Git, does not save much after pack files reach a size of 250 objects. Across all repositories, bigger packs always leads to better compression and fewer bytes per object, but only for blob objects. In conclusion, the heuristic of packing based on object type is very good. Neither commit nor tag objects compress very well when packed. While both tree and blob objects compress well, tree objects do not require bigger packs to provide better compression. The pack index files do not require many objects to become optimal. It should also be noted from looking at the columns containing numbers about the minimum and maximum pack sizes that they can vary a lot compared to the average size. -- Jonas Fonseca - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html