This adds the reftable library, and hooks it up as a ref backend. Includes testing support, to test: make -C t/ GIT_TEST_REFTABLE=1 Summary 18964 tests pass 2871 tests fail Some issues: * worktree support looks broken * submodules (eg. t1013) * many tests inspect .git/logs/ directly. v14 * Fixes for Windows (thanks, Dscho!) * Detect file/directory conflicts. * Plug some memory leaks. * Fixes for autocompaction Han-Wen Nienhuys (10): refs.h: clarify reflog iteration order Iterate over the "refs/" namespace in for_each_[raw]ref refs: document how ref_iterator_advance_fn should handle symrefs Add .gitattributes for the reftable/ directory reftable: define version 2 of the spec to accomodate SHA256 reftable: clarify how empty tables should be written Add reftable library Reftable support for git-core Add some reftable testing infrastructure t: use update-ref and show-ref to reading/writing refs Johannes Schindelin (1): vcxproj: adjust for the reftable changes Jonathan Nieder (1): reftable: file format documentation Documentation/Makefile | 1 + Documentation/technical/reftable.txt | 1080 +++++++++++++++ .../technical/repository-version.txt | 7 + Makefile | 27 +- builtin/clone.c | 3 +- builtin/init-db.c | 56 +- cache.h | 6 +- config.mak.uname | 2 +- contrib/buildsystems/Generators/Vcxproj.pm | 11 +- refs.c | 33 +- refs.h | 8 +- refs/refs-internal.h | 6 + refs/reftable-backend.c | 1045 ++++++++++++++ reftable/.gitattributes | 1 + reftable/LICENSE | 31 + reftable/README.md | 11 + reftable/VERSION | 5 + reftable/basics.c | 215 +++ reftable/basics.h | 53 + reftable/block.c | 436 ++++++ reftable/block.h | 126 ++ reftable/blocksource.h | 22 + reftable/constants.h | 21 + reftable/file.c | 99 ++ reftable/iter.c | 240 ++++ reftable/iter.h | 63 + reftable/merged.c | 327 +++++ reftable/merged.h | 38 + reftable/pq.c | 115 ++ reftable/pq.h | 34 + reftable/reader.c | 758 ++++++++++ reftable/reader.h | 68 + reftable/record.c | 1131 +++++++++++++++ reftable/record.h | 121 ++ reftable/refname.c | 215 +++ reftable/refname.h | 39 + reftable/reftable.c | 91 ++ reftable/reftable.h | 564 ++++++++ reftable/slice.c | 225 +++ reftable/slice.h | 76 + reftable/stack.c | 1229 +++++++++++++++++ reftable/stack.h | 45 + reftable/system.h | 54 + reftable/tree.c | 67 + reftable/tree.h | 34 + reftable/update.sh | 24 + reftable/writer.c | 661 +++++++++ reftable/writer.h | 60 + reftable/zlib-compat.c | 92 ++ repository.c | 2 + repository.h | 3 + setup.c | 12 +- t/t0002-gitfile.sh | 2 +- t/t0031-reftable.sh | 109 ++ t/t1400-update-ref.sh | 32 +- t/t1506-rev-parse-diagnosis.sh | 2 +- t/t3210-pack-refs.sh | 6 + t/t6050-replace.sh | 2 +- t/t9020-remote-svn.sh | 4 +- t/test-lib.sh | 5 + 60 files changed, 9798 insertions(+), 57 deletions(-) create mode 100644 Documentation/technical/reftable.txt create mode 100644 refs/reftable-backend.c create mode 100644 reftable/.gitattributes create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/constants.h create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/refname.c create mode 100644 reftable/refname.h create mode 100644 reftable/reftable.c create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100755 reftable/update.sh create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c create mode 100755 t/t0031-reftable.sh base-commit: b34789c0b0d3b137f0bb516b417bd8d75e0cb306 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-539%2Fhanwen%2Freftable-v11 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-539/hanwen/reftable-v11 Pull-Request: https://github.com/gitgitgadget/git/pull/539 Range-diff vs v10: 1: d3d8ed2032c = 1: dfa5fd74f85 refs.h: clarify reflog iteration order 2: 45fd65f72e0 = 2: 340c5c415e1 Iterate over the "refs/" namespace in for_each_[raw]ref 3: bc89bcd9c8c < -: ----------- create .git/refs in files-backend.c 4: fd67cdff0cd = 3: 6553285043b refs: document how ref_iterator_advance_fn should handle symrefs 5: 5373828f854 = 4: 7dc47c7756f Add .gitattributes for the reftable/ directory 6: 8eeed29ebbf = 5: 06fcb49e903 reftable: file format documentation 7: 5ebc272e23d = 6: 093fa74a3d0 reftable: define version 2 of the spec to accomodate SHA256 8: 95aae041613 = 7: 6d9031372ce reftable: clarify how empty tables should be written 9: 59209a5ad39 ! 8: 6ee6c44752c Add reftable library @@ reftable/README.md (new) ## reftable/VERSION (new) ## @@ -+commit a6d6b31bea256044621d789df64a8159647f21bc ++commit 3a486f79abcd17e88e3bca62f43188a7c8b80ff3 +Author: Han-Wen Nienhuys <hanwen@xxxxxxxxxx> -+Date: Mon Apr 27 20:46:21 2020 +0200 ++Date: Mon May 4 19:20:21 2020 +0200 + -+ C: prefix error codes with REFTABLE_ ++ C: include "cache.h" in git-core sleep_millisec() ## reftable/basics.c (new) ## @@ @@ reftable/basics.c (new) +{ + out[0] = (byte)((i >> 16) & 0xff); + out[1] = (byte)((i >> 8) & 0xff); -+ out[2] = (byte)((i)&0xff); ++ out[2] = (byte)(i & 0xff); +} + +uint32_t get_be24(byte *in) @@ reftable/basics.c (new) +void put_be16(uint8_t *out, uint16_t i) +{ + out[0] = (uint8_t)((i >> 8) & 0xff); -+ out[1] = (uint8_t)((i)&0xff); ++ out[1] = (uint8_t)(i & 0xff); +} + -+int binsearch(int sz, int (*f)(int k, void *args), void *args) ++int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args) +{ -+ int lo = 0; -+ int hi = sz; ++ size_t lo = 0; ++ size_t hi = sz; + + /* invariant: (hi == sz) || f(hi) == true + (lo == 0 && f(0) == true) || fi(lo) == false + */ + while (hi - lo > 1) { -+ int mid = lo + (hi - lo) / 2; ++ size_t mid = lo + (hi - lo) / 2; + + int val = f(mid, args); + if (val) { @@ reftable/basics.c (new) + +const char *reftable_error_str(int err) +{ ++ static char buf[250]; + switch (err) { + case REFTABLE_IO_ERROR: + return "I/O error"; @@ reftable/basics.c (new) + return "misuse of the reftable API"; + case REFTABLE_ZLIB_ERROR: + return "zlib failure"; ++ case REFTABLE_NAME_CONFLICT: ++ return "file/directory conflict"; ++ case REFTABLE_REFNAME_ERROR: ++ return "invalid refname"; + case -1: + return "general error"; + default: -+ return "unknown error code"; ++ snprintf(buf, sizeof(buf), "unknown error code %d", err); ++ return buf; + } +} + -+int reftable_error_to_errno(int err) { ++int reftable_error_to_errno(int err) ++{ + switch (err) { + case REFTABLE_IO_ERROR: + return EIO; @@ reftable/basics.c (new) + } +} + -+ +void *(*reftable_malloc_ptr)(size_t sz) = &malloc; +void *(*reftable_realloc_ptr)(void *, size_t) = &realloc; +void (*reftable_free_ptr)(void *) = &free; @@ reftable/basics.h (new) + find smallest index i in [0, sz) at which f(i) is true, assuming + that f is ascending. Return sz if f(i) is false for all indices. +*/ -+int binsearch(int sz, int (*f)(int k, void *args), void *args); ++int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args); + +/* + Frees a NULL terminated array of malloced strings. The array itself is also @@ reftable/block.c (new) + memcpy(w->buf + block_header_skip, compressed.buf, + dest_len); + w->next = dest_len + block_header_skip; ++ slice_clear(&compressed); + break; + } + } @@ reftable/block.c (new) + if (typ == BLOCK_TYPE_LOG) { + struct slice uncompressed = { 0 }; + int block_header_skip = 4 + header_off; -+ uLongf dst_len = sz - block_header_skip; ++ uLongf dst_len = sz - block_header_skip; /* total size of dest ++ buffer. */ + uLongf src_len = block->len - block_header_skip; + ++ /* Log blocks specify the *uncompressed* size in their header. ++ */ + slice_resize(&uncompressed, sz); ++ ++ /* Copy over the block header verbatim. It's not compressed. */ + memcpy(uncompressed.buf, block->data, block_header_skip); + ++ /* Uncompress */ + if (Z_OK != uncompress_return_consumed( + uncompressed.buf + block_header_skip, + &dst_len, block->data + block_header_skip, @@ reftable/block.c (new) + return REFTABLE_ZLIB_ERROR; + } + ++ if (dst_len + block_header_skip != sz) { ++ return REFTABLE_FORMAT_ERROR; ++ } ++ ++ /* We're done with the input data. */ + block_source_return_block(block->source, block); + block->data = uncompressed.buf; -+ block->len = dst_len; /* XXX: 4 bytes missing? */ ++ block->len = sz; + block->source = malloc_block_source(); + full_block_size = src_len + block_header_skip; + } else if (full_block_size == 0) { @@ reftable/block.c (new) + struct block_reader *r; +}; + -+static int restart_key_less(int idx, void *args) ++static int restart_key_less(size_t idx, void *args) +{ + struct restart_find_args *a = (struct restart_find_args *)args; + uint32_t off = block_reader_restart_offset(a->r, idx); @@ reftable/block.c (new) + slice_copy(&dest->last_key, src->last_key); +} + -+/* return < 0 for error, 0 for OK, > 0 for EOF. */ +int block_iter_next(struct block_iter *it, struct record rec) +{ + if (it->next_off >= it->br->block_len) { @@ reftable/block.h (new) +int block_reader_first_key(struct block_reader *br, struct slice *key); + +void block_iter_copy_from(struct block_iter *dest, struct block_iter *src); ++ ++/* return < 0 for error, 0 for OK, > 0 for EOF. */ +int block_iter_next(struct block_iter *it, struct record rec); + +/* Seek to `want` with in the block pointed to by `it` */ @@ reftable/blocksource.h (new) + +#endif - ## reftable/bytes.c (new) ## - - ## reftable/config.h (new) ## -@@ -+/* empty */ - ## reftable/constants.h (new) ## @@ +/* @@ reftable/constants.h (new) + +#endif - ## reftable/dump.c (new) ## -@@ -+/* -+Copyright 2020 Google LLC -+ -+Use of this source code is governed by a BSD-style -+license that can be found in the LICENSE file or at -+https://developers.google.com/open-source/licenses/bsd -+*/ -+ -+#include "system.h" -+ -+#include "reftable.h" -+ -+static int dump_table(const char *tablename) -+{ -+ struct block_source src = { 0 }; -+ int err = block_source_from_file(&src, tablename); -+ if (err < 0) { -+ return err; -+ } -+ -+ struct reader *r = NULL; -+ err = new_reader(&r, src, tablename); -+ if (err < 0) { -+ return err; -+ } -+ -+ { -+ struct iterator it = { 0 }; -+ err = reader_seek_ref(r, &it, ""); -+ if (err < 0) { -+ return err; -+ } -+ -+ struct ref_record ref = { 0 }; -+ while (1) { -+ err = iterator_next_ref(it, &ref); -+ if (err > 0) { -+ break; -+ } -+ if (err < 0) { -+ return err; -+ } -+ ref_record_print(&ref, 20); -+ } -+ iterator_destroy(&it); -+ ref_record_clear(&ref); -+ } -+ -+ { -+ struct iterator it = { 0 }; -+ err = reader_seek_log(r, &it, ""); -+ if (err < 0) { -+ return err; -+ } -+ struct log_record log = { 0 }; -+ while (1) { -+ err = iterator_next_log(it, &log); -+ if (err > 0) { -+ break; -+ } -+ if (err < 0) { -+ return err; -+ } -+ log_record_print(&log, 20); -+ } -+ iterator_destroy(&it); -+ log_record_clear(&log); -+ } -+ return 0; -+} -+ -+int main(int argc, char *argv[]) -+{ -+ int opt; -+ const char *table = NULL; -+ while ((opt = getopt(argc, argv, "t:")) != -1) { -+ switch (opt) { -+ case 't': -+ table = xstrdup(optarg); -+ break; -+ case '?': -+ printf("usage: %s [-table tablefile]\n", argv[0]); -+ return 2; -+ break; -+ } -+ } -+ -+ if (table != NULL) { -+ int err = dump_table(table); -+ if (err < 0) { -+ fprintf(stderr, "%s: %s: %s\n", argv[0], table, -+ error_str(err)); -+ return 1; -+ } -+ } -+ return 0; -+} - ## reftable/file.c (new) ## @@ +/* @@ reftable/file.c (new) + p->size = st.st_size; + p->fd = fd; + ++ assert(bs->ops == NULL); + bs->ops = &file_vtable; + bs->arg = p; + } + return 0; +} + -+int reftable_fd_write(void *arg, byte *data, int sz) ++int reftable_fd_write(void *arg, byte *data, size_t sz) +{ + int *fdp = (int *)arg; + return write(*fdp, data, sz); @@ reftable/iter.c (new) + +void iterator_set_empty(struct reftable_iterator *it) +{ ++ assert(it->ops == NULL); + it->iter_arg = NULL; + it->ops = &empty_vtable; +} @@ reftable/iter.c (new) + (struct filtering_ref_iterator *)iter_arg; + struct reftable_ref_record *ref = + (struct reftable_ref_record *)rec.data; -+ ++ int err = 0; + while (true) { -+ int err = reftable_iterator_next_ref(fri->it, ref); ++ err = reftable_iterator_next_ref(fri->it, ref); + if (err != 0) { -+ return err; ++ break; + } + + if (fri->double_check) { + struct reftable_iterator it = { 0 }; + -+ int err = reftable_reader_seek_ref(fri->r, &it, -+ ref->ref_name); ++ err = reftable_reader_seek_ref(fri->r, &it, ++ ref->ref_name); + if (err == 0) { + err = reftable_iterator_next_ref(it, ref); + } @@ reftable/iter.c (new) + reftable_iterator_destroy(&it); + + if (err < 0) { -+ return err; ++ break; + } + + if (err > 0) { @@ reftable/iter.c (new) + return 0; + } + } ++ ++ reftable_ref_record_clear(ref); ++ return err; +} + +struct reftable_iterator_vtable filtering_ref_iterator_vtable = { @@ reftable/iter.c (new) +void iterator_from_filtering_ref_iterator(struct reftable_iterator *it, + struct filtering_ref_iterator *fri) +{ ++ assert(it->ops == NULL); + it->iter_arg = fri; + it->ops = &filtering_ref_iterator_vtable; +} @@ reftable/iter.c (new) +void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it, + struct indexed_table_ref_iter *itr) +{ ++ assert(it->ops == NULL); + it->iter_arg = itr; + it->ops = &indexed_table_ref_iter_vtable; +} @@ reftable/iter.h (new) + +void iterator_set_empty(struct reftable_iterator *it); +int iterator_next(struct reftable_iterator it, struct record rec); ++ ++/* Returns true for a zeroed out iterator, such as the one returned from ++ iterator_destroy. */ +bool iterator_is_null(struct reftable_iterator it); + +/* iterator that produces only ref records that point to `oid` */ @@ reftable/merged.c (new) + reftable_free(mi->stack); +} + -+static int merged_iter_advance_subiter(struct merged_iter *mi, int idx) ++static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx) +{ + if (iterator_is_null(mi->stack[idx])) { + return 0; @@ reftable/merged.c (new) +static void iterator_from_merged_iter(struct reftable_iterator *it, + struct merged_iter *mi) +{ ++ assert(it->ops == NULL); + it->iter_arg = mi; + it->ops = &merged_iter_vtable; +} @@ reftable/merged.c (new) + return mt->min; +} + -+static int merged_table_seek_record(struct reftable_merged_table *mt, -+ struct reftable_iterator *it, -+ struct record rec) ++int merged_table_seek_record(struct reftable_merged_table *mt, ++ struct reftable_iterator *it, struct record rec) +{ + struct reftable_iterator *iters = reftable_calloc( + sizeof(struct reftable_iterator) * mt->stack_len); @@ reftable/merged.h (new) +}; + +void merged_table_clear(struct reftable_merged_table *mt); ++int merged_table_seek_record(struct reftable_merged_table *mt, ++ struct reftable_iterator *it, struct record rec); + +#endif @@ reftable/reader.c (new) +static void iterator_from_table_iter(struct reftable_iterator *it, + struct table_iter *ti) +{ ++ assert(it->ops == NULL); + it->iter_arg = ti; + it->ops = &table_iter_vtable; +} @@ reftable/reader.c (new) + if (err == 0) { + *p = rd; + } else { ++ block_source_close(&src); + reftable_free(rd); + } + return err; @@ reftable/reader.c (new) + struct record got_rec = { 0 }; + int err = 0; + ++ /* Look through the reverse index. */ + record_from_obj(&want_rec, &want); -+ + err = reader_seek(r, &oit, want_rec); + if (err != 0) { -+ return err; ++ goto exit; + } + ++ /* read out the obj_record */ + record_from_obj(&got_rec, &got); + err = iterator_next(oit, got_rec); -+ reftable_iterator_destroy(&oit); + if (err < 0) { -+ return err; ++ goto exit; + } + + if (err > 0 || + memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) { ++ /* didn't find it; return empty iterator */ + iterator_set_empty(it); -+ return 0; ++ err = 0; ++ goto exit; + } + + { @@ reftable/reader.c (new) + hash_size(r->hash_id), + got.offsets, got.offset_len); + if (err < 0) { -+ record_clear(got_rec); -+ return err; ++ goto exit; + } + got.offsets = NULL; -+ record_clear(got_rec); -+ + iterator_from_indexed_table_ref_iter(it, itr); + } + -+ return 0; ++exit: ++ reftable_iterator_destroy(&oit); ++ record_clear(got_rec); ++ return err; +} + +static int reftable_reader_refs_for_unindexed(struct reftable_reader *r, @@ reftable/record.c (new) + memcpy(r->message, dest.buf, dest.len); + r->message[dest.len] = 0; + ++ slice_clear(&dest); + return start.len - in.len; + +error: @@ reftable/record.c (new) + +struct record new_record(byte typ) +{ -+ struct record rec; ++ struct record rec = { NULL }; + switch (typ) { + case BLOCK_TYPE_REF: { + struct reftable_ref_record *r = @@ reftable/record.c (new) + +void record_clear(struct record rec) +{ -+ return rec.ops->clear(rec.data); ++ rec.ops->clear(rec.data); +} + +bool record_is_deletion(struct record rec) @@ reftable/record.c (new) + +void record_from_ref(struct record *rec, struct reftable_ref_record *ref_rec) +{ ++ assert(rec->ops == NULL); + rec->data = ref_rec; + rec->ops = &reftable_ref_record_vtable; +} + +void record_from_obj(struct record *rec, struct obj_record *obj_rec) +{ ++ assert(rec->ops == NULL); + rec->data = obj_rec; + rec->ops = &obj_record_vtable; +} + +void record_from_index(struct record *rec, struct index_record *index_rec) +{ ++ assert(rec->ops == NULL); + rec->data = index_rec; + rec->ops = &index_record_vtable; +} + +void record_from_log(struct record *rec, struct reftable_log_record *log_rec) +{ ++ assert(rec->ops == NULL); + rec->data = log_rec; + rec->ops = &reftable_log_record_vtable; +} @@ reftable/record.h (new) + +#endif + ## reftable/refname.c (new) ## +@@ ++/* ++ Copyright 2020 Google LLC ++ ++ Use of this source code is governed by a BSD-style ++ license that can be found in the LICENSE file or at ++ https://developers.google.com/open-source/licenses/bsd ++*/ ++ ++#include "system.h" ++#include "reftable.h" ++#include "basics.h" ++#include "refname.h" ++#include "slice.h" ++ ++struct find_arg { ++ char **names; ++ const char *want; ++}; ++ ++static int find_name(size_t k, void *arg) ++{ ++ struct find_arg *f_arg = (struct find_arg *)arg; ++ ++ return strcmp(f_arg->names[k], f_arg->want) >= 0; ++} ++ ++int modification_has_ref(struct modification *mod, const char *name) ++{ ++ struct reftable_ref_record ref = { 0 }; ++ int err = 0; ++ ++ if (mod->add_len > 0) { ++ struct find_arg arg = { ++ .names = mod->add, ++ .want = name, ++ }; ++ int idx = binsearch(mod->add_len, find_name, &arg); ++ if (idx < mod->add_len && !strcmp(mod->add[idx], name)) { ++ return 0; ++ } ++ } ++ ++ if (mod->del_len > 0) { ++ struct find_arg arg = { ++ .names = mod->del, ++ .want = name, ++ }; ++ int idx = binsearch(mod->del_len, find_name, &arg); ++ if (idx < mod->del_len && !strcmp(mod->del[idx], name)) { ++ return 1; ++ } ++ } ++ ++ err = reftable_table_read_ref(mod->tab, name, &ref); ++ reftable_ref_record_clear(&ref); ++ return err; ++} ++ ++static void modification_clear(struct modification *mod) ++{ ++ FREE_AND_NULL(mod->add); ++ FREE_AND_NULL(mod->del); ++ mod->add_len = 0; ++ mod->del_len = 0; ++} ++ ++int modification_has_ref_with_prefix(struct modification *mod, ++ const char *prefix) ++{ ++ struct reftable_iterator it = { NULL }; ++ struct reftable_ref_record ref = { NULL }; ++ int err = 0; ++ ++ if (mod->add_len > 0) { ++ struct find_arg arg = { ++ .names = mod->add, ++ .want = prefix, ++ }; ++ int idx = binsearch(mod->add_len, find_name, &arg); ++ if (idx < mod->add_len && ++ !strncmp(prefix, mod->add[idx], strlen(prefix))) { ++ goto exit; ++ } ++ } ++ ++ err = reftable_table_seek_ref(mod->tab, &it, prefix); ++ if (err) { ++ goto exit; ++ } ++ ++ while (true) { ++ err = reftable_iterator_next_ref(it, &ref); ++ if (err) { ++ goto exit; ++ } ++ ++ if (mod->del_len > 0) { ++ struct find_arg arg = { ++ .names = mod->del, ++ .want = ref.ref_name, ++ }; ++ int idx = binsearch(mod->del_len, find_name, &arg); ++ if (idx < mod->del_len && ++ !strcmp(ref.ref_name, mod->del[idx])) { ++ continue; ++ } ++ } ++ ++ if (strncmp(ref.ref_name, prefix, strlen(prefix))) { ++ err = 1; ++ goto exit; ++ } ++ err = 0; ++ goto exit; ++ } ++ ++exit: ++ reftable_ref_record_clear(&ref); ++ reftable_iterator_destroy(&it); ++ return err; ++} ++ ++int validate_ref_name(const char *name) ++{ ++ while (true) { ++ char *next = strchr(name, '/'); ++ if (!*name) { ++ return REFTABLE_REFNAME_ERROR; ++ } ++ if (!next) { ++ return 0; ++ } ++ if (next - name == 0 || (next - name == 1 && *name == '.') || ++ (next - name == 2 && name[0] == '.' && name[1] == '.')) ++ return REFTABLE_REFNAME_ERROR; ++ name = next + 1; ++ } ++ return 0; ++} ++ ++int validate_ref_record_addition(struct reftable_table tab, ++ struct reftable_ref_record *recs, size_t sz) ++{ ++ struct modification mod = { ++ .tab = tab, ++ .add = reftable_calloc(sizeof(char *) * sz), ++ .del = reftable_calloc(sizeof(char *) * sz), ++ }; ++ int i = 0; ++ int err = 0; ++ for (; i < sz; i++) { ++ if (reftable_ref_record_is_deletion(&recs[i])) { ++ mod.del[mod.del_len++] = recs[i].ref_name; ++ } else { ++ mod.add[mod.add_len++] = recs[i].ref_name; ++ } ++ } ++ ++ err = modification_validate(&mod); ++ modification_clear(&mod); ++ return err; ++} ++ ++static void slice_trim_component(struct slice *sl) ++{ ++ while (sl->len > 0) { ++ bool is_slash = (sl->buf[sl->len - 1] == '/'); ++ sl->len--; ++ if (is_slash) ++ break; ++ } ++} ++ ++int modification_validate(struct modification *mod) ++{ ++ struct slice slashed = { 0 }; ++ int err = 0; ++ int i = 0; ++ for (; i < mod->add_len; i++) { ++ err = validate_ref_name(mod->add[i]); ++ if (err) { ++ goto exit; ++ } ++ slice_set_string(&slashed, mod->add[i]); ++ slice_append_string(&slashed, "/"); ++ ++ err = modification_has_ref_with_prefix( ++ mod, slice_as_string(&slashed)); ++ if (err == 0) { ++ err = REFTABLE_NAME_CONFLICT; ++ goto exit; ++ } ++ if (err < 0) { ++ goto exit; ++ } ++ ++ slice_set_string(&slashed, mod->add[i]); ++ while (slashed.len) { ++ slice_trim_component(&slashed); ++ err = modification_has_ref(mod, ++ slice_as_string(&slashed)); ++ if (err == 0) { ++ err = REFTABLE_NAME_CONFLICT; ++ goto exit; ++ } ++ if (err < 0) { ++ goto exit; ++ } ++ } ++ } ++ err = 0; ++exit: ++ slice_clear(&slashed); ++ return err; ++} + + ## reftable/refname.h (new) ## +@@ ++/* ++ Copyright 2020 Google LLC ++ ++ Use of this source code is governed by a BSD-style ++ license that can be found in the LICENSE file or at ++ https://developers.google.com/open-source/licenses/bsd ++*/ ++#ifndef REFNAME_H ++#define REFNAME_H ++ ++struct modification { ++ struct reftable_table tab; ++ ++ char **add; ++ size_t add_len; ++ ++ char **del; ++ size_t del_len; ++}; ++ ++// -1 = error, 0 = found, 1 = not found ++int modification_has_ref(struct modification *mod, const char *name); ++ ++// -1 = error, 0 = found, 1 = not found. ++int modification_has_ref_with_prefix(struct modification *mod, ++ const char *prefix); ++ ++// 0 = OK. ++int validate_ref_name(const char *name); ++ ++int validate_ref_record_addition(struct reftable_table tab, ++ struct reftable_ref_record *recs, size_t sz); ++ ++int modification_validate(struct modification *mod); ++ ++/* illegal name, or dir/file conflict */ ++#define REFTABLE_REFNAME_ERROR -9 ++ ++#endif + + ## reftable/reftable.c (new) ## +@@ ++/* ++Copyright 2020 Google LLC ++ ++Use of this source code is governed by a BSD-style ++license that can be found in the LICENSE file or at ++https://developers.google.com/open-source/licenses/bsd ++*/ ++ ++#include "reftable.h" ++#include "record.h" ++#include "reader.h" ++#include "merged.h" ++ ++struct reftable_table_vtable { ++ int (*seek)(void *tab, struct reftable_iterator *it, struct record); ++}; ++ ++static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it, ++ struct record rec) ++{ ++ return reader_seek((struct reftable_reader *)tab, it, rec); ++} ++ ++static struct reftable_table_vtable reader_vtable = { ++ .seek = reftable_reader_seek_void, ++}; ++ ++static int reftable_merged_table_seek_void(void *tab, ++ struct reftable_iterator *it, ++ struct record rec) ++{ ++ return merged_table_seek_record((struct reftable_merged_table *)tab, it, ++ rec); ++} ++ ++static struct reftable_table_vtable merged_table_vtable = { ++ .seek = reftable_merged_table_seek_void, ++}; ++ ++int reftable_table_seek_ref(struct reftable_table tab, ++ struct reftable_iterator *it, const char *name) ++{ ++ struct reftable_ref_record ref = { ++ .ref_name = (char *)name, ++ }; ++ struct record rec = { 0 }; ++ record_from_ref(&rec, &ref); ++ return tab.ops->seek(tab.table_arg, it, rec); ++} ++ ++void reftable_table_from_reader(struct reftable_table *tab, ++ struct reftable_reader *reader) ++{ ++ assert(tab->ops == NULL); ++ tab->ops = &reader_vtable; ++ tab->table_arg = reader; ++} ++ ++void reftable_table_from_merged_table(struct reftable_table *tab, ++ struct reftable_merged_table *merged) ++{ ++ assert(tab->ops == NULL); ++ tab->ops = &merged_table_vtable; ++ tab->table_arg = merged; ++} ++ ++int reftable_table_read_ref(struct reftable_table tab, const char *name, ++ struct reftable_ref_record *ref) ++{ ++ struct reftable_iterator it = { 0 }; ++ int err = reftable_table_seek_ref(tab, &it, name); ++ if (err) { ++ goto exit; ++ } ++ ++ err = reftable_iterator_next_ref(it, ref); ++ if (err) { ++ goto exit; ++ } ++ ++ if (strcmp(ref->ref_name, name) || ++ reftable_ref_record_is_deletion(ref)) { ++ reftable_ref_record_clear(ref); ++ err = 1; ++ goto exit; ++ } ++ ++exit: ++ reftable_iterator_destroy(&it); ++ return err; ++} + ## reftable/reftable.h (new) ## @@ +/* @@ reftable/reftable.h (new) + + /* Wrote a table without blocks. */ + REFTABLE_EMPTY_TABLE_ERROR = -8, ++ ++ /* Dir/file conflict. */ ++ REFTABLE_NAME_CONFLICT = -9, ++ ++ /* Illegal ref name. */ ++ REFTABLE_REFNAME_ERROR = -10, +}; + +/* convert the numeric error code to a string. The string should not be @@ reftable/reftable.h (new) + * Defaults to SHA1 if unset + */ + uint32_t hash_id; ++ ++ /* boolean: do not check ref names for validity or dir/file conflicts. ++ */ ++ int skip_name_check; +}; + +/* reftable_block_stats holds statistics for a single block type */ @@ reftable/reftable.h (new) + +/* reftable_new_writer creates a new writer */ +struct reftable_writer * -+reftable_new_writer(int (*writer_func)(void *, uint8_t *, int), ++reftable_new_writer(int (*writer_func)(void *, uint8_t *, size_t), + void *writer_arg, struct reftable_write_options *opts); + +/* write to a file descriptor. fdp should be an int* pointing to the fd. */ -+int reftable_fd_write(void *fdp, uint8_t *data, int size); ++int reftable_fd_write(void *fdp, uint8_t *data, size_t size); + +/* Set the range of update indices for the records we will add. When + writing a table into a stack, the min should be at least @@ reftable/reftable.h (new) +struct reftable_reader; + +/* reftable_new_reader opens a reftable for reading. If successful, returns 0 -+ * code and sets pp. The name is used for creating a -+ * stack. Typically, it is the basename of the file. ++ * code and sets pp. The name is used for creating a stack. Typically, it is the ++ * basename of the file. The block source `src` is owned by the reader, and is ++ * closed on calling reftable_reader_destroy(). + */ +int reftable_new_reader(struct reftable_reader **pp, -+ struct reftable_block_source, const char *name); ++ struct reftable_block_source src, const char *name); + +/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted + in the table. To seek to the start of the table, use name = "". @@ reftable/reftable.h (new) +void reftable_merged_table_free(struct reftable_merged_table *m); + +/**************************************************************** ++ Generic tables ++ ++ A unified API for reading tables, either merged tables, or single readers. ++ ****************************************************************/ ++ ++struct reftable_table { ++ struct reftable_table_vtable *ops; ++ void *table_arg; ++}; ++ ++int reftable_table_seek_ref(struct reftable_table tab, ++ struct reftable_iterator *it, const char *name); ++ ++void reftable_table_from_reader(struct reftable_table *tab, ++ struct reftable_reader *reader); ++void reftable_table_from_merged_table(struct reftable_table *tab, ++ struct reftable_merged_table *table); ++ ++/* convenience function to read a single ref. Returns < 0 for error, 0 ++ for success, and 1 if ref not found. */ ++int reftable_table_read_ref(struct reftable_table tab, const char *name, ++ struct reftable_ref_record *ref); ++ ++/**************************************************************** + Mutable ref database + + The stack presents an interface to a mutable sequence of reftables. @@ reftable/reftable.h (new) +/* statistics on past compactions. */ +struct reftable_compaction_stats { + uint64_t bytes; /* total number of bytes written */ ++ uint64_t entries_written; /* total number of entries written, including ++ failures. */ + int attempts; /* how often we tried to compact */ + int failures; /* failures happen on concurrent updates */ +}; @@ reftable/slice.c (new) + } +} + -+int slice_write(struct slice *b, byte *data, int sz) ++int slice_write(struct slice *b, byte *data, size_t sz) +{ + if (b->len + sz > b->cap) { + int newcap = 2 * b->cap + 1; @@ reftable/slice.c (new) + return sz; +} + -+int slice_write_void(void *b, byte *data, int sz) ++int slice_write_void(void *b, byte *data, size_t sz) +{ + return slice_write((struct slice *)b, data, sz); +} @@ reftable/slice.c (new) +void block_source_from_slice(struct reftable_block_source *bs, + struct slice *buf) +{ ++ assert(bs->ops == NULL); + bs->ops = &slice_vtable; + bs->arg = buf; +} @@ reftable/slice.h (new) +int slice_compare(struct slice a, struct slice b); + +/* Append `data` to the `dest` slice. */ -+int slice_write(struct slice *dest, byte *data, int sz); ++int slice_write(struct slice *dest, byte *data, size_t sz); + +/* Append `add` to `dest. */ +void slice_append(struct slice *dest, struct slice add); + +/* Like slice_write, but suitable for passing to reftable_new_writer + */ -+int slice_write_void(void *b, byte *data, int sz); ++int slice_write_void(void *b, byte *data, size_t sz); + +/* Find the longest shared prefix size of `a` and `b` */ +int common_prefix_size(struct slice a, struct slice b); @@ reftable/stack.c (new) +#include "system.h" +#include "merged.h" +#include "reader.h" ++#include "refname.h" +#include "reftable.h" +#include "writer.h" + @@ reftable/stack.c (new) +{ + struct reftable_stack *p = + reftable_calloc(sizeof(struct reftable_stack)); -+ struct slice list_file_name = {}; ++ struct slice list_file_name = { 0 }; + int err = 0; + + if (config.hash_id == 0) { @@ reftable/stack.c (new) + } + } + } ++ +exit: + slice_clear(&table_path); + { @@ reftable/stack.c (new) + free_names(names); + free_names(names_after); + -+ delay = delay + (delay * rand()) / RAND_MAX + 100; -+ usleep(delay); ++ delay = delay + (delay * rand()) / RAND_MAX + 1; ++ sleep_millisec(delay); + } + + return 0; @@ reftable/stack.c (new) + int err = stack_try_add(st, write, arg); + if (err < 0) { + if (err == REFTABLE_LOCK_ERROR) { -+ // Ignore error return, we want to propagate -+ // REFTABLE_LOCK_ERROR. ++ /* Ignore error return, we want to propagate ++ REFTABLE_LOCK_ERROR. ++ */ + reftable_stack_reload(st); + } + return err; @@ reftable/stack.c (new) +void reftable_addition_close(struct reftable_addition *add) +{ + int i = 0; -+ struct slice nm = {}; ++ struct slice nm = { 0 }; + for (i = 0; i < add->new_tables_len; i++) { + slice_set_string(&nm, add->stack->list_file); + slice_append_string(&nm, "/"); + slice_append_string(&nm, add->new_tables[i]); + unlink(slice_as_string(&nm)); -+ + reftable_free(add->new_tables[i]); + add->new_tables[i] = NULL; + } @@ reftable/stack.c (new) + + free_names(add->names); + add->names = NULL; ++ slice_clear(&nm); +} + +int reftable_addition_commit(struct reftable_addition *add) @@ reftable/stack.c (new) + struct reftable_writer *wr = NULL; + int err = 0; + int tab_fd = 0; -+ uint64_t next_update_index = 0; + + slice_resize(&next_name, 0); -+ format_name(&next_name, next_update_index, next_update_index); ++ format_name(&next_name, add->next_update_index, add->next_update_index); + + slice_set_string(&temp_tab_file_name, add->stack->reftable_dir); + slice_append_string(&temp_tab_file_name, "/"); @@ reftable/stack.c (new) + goto exit; + } + -+ if (wr->min_update_index < next_update_index) { ++ err = stack_check_addition(add->stack, ++ slice_as_string(&temp_tab_file_name)); ++ if (err < 0) { ++ goto exit; ++ } ++ ++ if (wr->min_update_index < add->next_update_index) { + err = REFTABLE_API_ERROR; + goto exit; + } @@ reftable/stack.c (new) + struct reftable_ref_record ref = { 0 }; + struct reftable_log_record log = { 0 }; + ++ uint64_t entries = 0; ++ + int i = 0, j = 0; + for (i = first, j = 0; i <= last; i++) { + struct reftable_reader *t = st->merged->stack[i]; @@ reftable/stack.c (new) + if (err < 0) { + break; + } ++ entries++; + } ++ reftable_iterator_destroy(&it); + + err = reftable_merged_table_seek_log(mt, &it, ""); + if (err < 0) { @@ reftable/stack.c (new) + continue; + } + -+ /* XXX collect stats? */ -+ + if (config != NULL && config->time > 0 && + log.time < config->time) { + continue; @@ reftable/stack.c (new) + if (err < 0) { + break; + } ++ entries++; + } + +exit: @@ reftable/stack.c (new) + reftable_merged_table_free(mt); + } + reftable_ref_record_clear(&ref); -+ ++ reftable_log_record_clear(&log); ++ st->stats.entries_written += entries; + return err; +} + @@ reftable/stack.c (new) + } + goto exit; + } ++ /* Don't want to write to the lock for now. */ ++ close(lock_file_fd); ++ lock_file_fd = 0; ++ + have_lock = true; + err = stack_uptodate(st); + if (err != 0) { @@ reftable/stack.c (new) + } + have_lock = false; + ++ /* Reload the stack before deleting. On windows, we can only delete the ++ files after we closed them. ++ */ ++ err = reftable_stack_reload_maybe_reuse(st, first < last); ++ + { + char **p = delete_on_success; + while (*p) { @@ reftable/stack.c (new) + } + } + -+ err = reftable_stack_reload_maybe_reuse(st, first < last); +exit: + free_names(delete_on_success); + { @@ reftable/stack.c (new) +int fastlog2(uint64_t sz) +{ + int l = 0; -+ assert(sz > 0); ++ if (sz == 0) { ++ return 0; ++ } + for (; sz; sz /= 2) { + l++; + } @@ reftable/stack.c (new) + int next = 0; + struct segment cur = { 0 }; + int i = 0; ++ ++ if (n == 0) { ++ *seglen = 0; ++ return segs; ++ } + for (i = 0; i < n; i++) { + int log = fastlog2(sizes[i]); + if (cur.log != log && cur.bytes > 0) { @@ reftable/stack.c (new) + cur.end = i + 1; + cur.bytes += sizes[i]; + } -+ if (next > 0) { -+ segs[next++] = cur; -+ } ++ segs[next++] = cur; + *seglen = next; + return segs; +} @@ reftable/stack.c (new) + uint64_t *sizes = + reftable_calloc(sizeof(uint64_t) * st->merged->stack_len); + int version = (st->config.hash_id == SHA1_ID) ? 1 : 2; -+ int overhead = footer_size(version) + header_size(version) - 1; ++ int overhead = header_size(version) - 1; + int i = 0; + for (i = 0; i < st->merged->stack_len; i++) { + sizes[i] = st->merged->stack[i]->size - overhead; @@ reftable/stack.c (new) +int reftable_stack_read_ref(struct reftable_stack *st, const char *refname, + struct reftable_ref_record *ref) +{ ++ struct reftable_table tab = { NULL }; ++ reftable_table_from_merged_table(&tab, reftable_stack_merged_table(st)); ++ return reftable_table_read_ref(tab, refname, ref); ++} ++ ++int reftable_stack_read_log(struct reftable_stack *st, const char *refname, ++ struct reftable_log_record *log) ++{ + struct reftable_iterator it = { 0 }; + struct reftable_merged_table *mt = reftable_stack_merged_table(st); -+ int err = reftable_merged_table_seek_ref(mt, &it, refname); ++ int err = reftable_merged_table_seek_log(mt, &it, refname); + if (err) { + goto exit; + } + -+ err = reftable_iterator_next_ref(it, ref); ++ err = reftable_iterator_next_log(it, log); + if (err) { + goto exit; + } + -+ if (strcmp(ref->ref_name, refname) || -+ reftable_ref_record_is_deletion(ref)) { ++ if (strcmp(log->ref_name, refname) || ++ reftable_log_record_is_deletion(log)) { + err = 1; + goto exit; + } + +exit: ++ if (err) { ++ reftable_log_record_clear(log); ++ } + reftable_iterator_destroy(&it); + return err; +} + -+int reftable_stack_read_log(struct reftable_stack *st, const char *refname, -+ struct reftable_log_record *log) ++int stack_check_addition(struct reftable_stack *st, const char *new_tab_name) +{ -+ struct reftable_iterator it = { 0 }; -+ struct reftable_merged_table *mt = reftable_stack_merged_table(st); -+ int err = reftable_merged_table_seek_log(mt, &it, refname); -+ if (err) { ++ int err = 0; ++ struct reftable_block_source src = { 0 }; ++ struct reftable_reader *rd = NULL; ++ struct reftable_table tab = { NULL }; ++ struct reftable_ref_record *refs = NULL; ++ struct reftable_iterator it = { NULL }; ++ int cap = 0; ++ int len = 0; ++ int i = 0; ++ ++ if (st->config.skip_name_check) { ++ return 0; ++ } ++ ++ err = reftable_block_source_from_file(&src, new_tab_name); ++ if (err < 0) { + goto exit; + } + -+ err = reftable_iterator_next_log(it, log); -+ if (err) { ++ err = reftable_new_reader(&rd, src, new_tab_name); ++ if (err < 0) { + goto exit; + } + -+ if (strcmp(log->ref_name, refname) || -+ reftable_log_record_is_deletion(log)) { -+ err = 1; ++ err = reftable_reader_seek_ref(rd, &it, ""); ++ if (err > 0) { ++ err = 0; + goto exit; + } ++ if (err < 0) { ++ goto exit; ++ } ++ ++ while (true) { ++ struct reftable_ref_record ref = { 0 }; ++ err = reftable_iterator_next_ref(it, &ref); ++ if (err > 0) { ++ break; ++ } ++ if (err < 0) { ++ goto exit; ++ } ++ ++ if (len >= cap) { ++ cap = 2 * cap + 1; ++ refs = reftable_realloc(refs, cap * sizeof(refs[0])); ++ } ++ ++ refs[len++] = ref; ++ } ++ ++ reftable_table_from_merged_table(&tab, reftable_stack_merged_table(st)); ++ ++ err = validate_ref_record_addition(tab, refs, len); ++ ++ for (i = 0; i < len; i++) { ++ reftable_ref_record_clear(&refs[i]); ++ } + +exit: ++ free(refs); + reftable_iterator_destroy(&it); ++ reftable_reader_free(rd); + return err; +} @@ reftable/stack.h (new) + int first, int last, + struct reftable_log_expiry_config *config); +int fastlog2(uint64_t sz); ++int stack_check_addition(struct reftable_stack *st, const char *new_tab_name); + +struct segment { + int start, end; @@ reftable/system.h (new) +#if 1 /* REFTABLE_IN_GITCORE */ + +#include "git-compat-util.h" ++#include "cache.h" +#include <zlib.h> + +#else @@ reftable/update.sh (new) +set -eu + +# Override this to import from somewhere else, say "../reftable". -+SRC=${SRC:-origin} BRANCH=${BRANCH:-origin/master} ++SRC=${SRC:-origin} ++BRANCH=${BRANCH:-master} + -+((git --git-dir reftable-repo/.git fetch ${SRC} && cd reftable-repo && git checkout ${BRANCH} ) || ++((git --git-dir reftable-repo/.git fetch ${SRC} ${BRANCH}:import && cd reftable-repo && git checkout -f $(git rev-parse import) ) || + git clone https://github.com/google/reftable reftable-repo) + +cp reftable-repo/c/*.[ch] reftable/ +cp reftable-repo/c/include/*.[ch] reftable/ +cp reftable-repo/LICENSE reftable/ -+git --git-dir reftable-repo/.git show --no-patch origin/master \ ++git --git-dir reftable-repo/.git show --no-patch HEAD \ + > reftable/VERSION + +mv reftable/system.h reftable/system.h~ @@ reftable/writer.c (new) +} + +struct reftable_writer * -+reftable_new_writer(int (*writer_func)(void *, byte *, int), void *writer_arg, -+ struct reftable_write_options *opts) ++reftable_new_writer(int (*writer_func)(void *, byte *, size_t), ++ void *writer_arg, struct reftable_write_options *opts) +{ + struct reftable_writer *wp = + reftable_calloc(sizeof(struct reftable_writer)); @@ reftable/writer.c (new) + } + w->pending_padding = 0; + if (empty_table) { -+ // Empty tables need a header anyway. ++ /* Empty tables need a header anyway. */ + byte header[28]; + int n = writer_write_header(w, header); + err = padded_write(w, header, n, 0); @@ reftable/writer.h (new) +#include "tree.h" + +struct reftable_writer { -+ int (*write)(void *, byte *, int); ++ int (*write)(void *, byte *, size_t); + void *write_arg; + int pending_padding; + struct slice last_key; 10: be2371cd6e4 ! 9: d731e1669b7 Reftable support for git-core @@ Commit message * Resolve spots marked with XXX - * Detect and prevent directory/file conflicts in naming. - * Support worktrees (t0002-gitfile "linked repo" testcase) Example use: see t/t0031-reftable.sh @@ Makefile: TEST_SHELL_PATH = $(SHELL_PATH) VCSSVN_LIB = vcs-svn/lib.a +REFTABLE_LIB = reftable/libreftable.a + GENERATED_H += config-list.h GENERATED_H += command-list.h - -@@ Makefile: LIB_OBJS += rebase-interactive.o +@@ Makefile: LIB_OBJS += ref-filter.o LIB_OBJS += reflog-walk.o LIB_OBJS += refs.o LIB_OBJS += refs/files-backend.o @@ Makefile: THIRD_PARTY_SOURCES += compat/regex/% EXTLIBS = GIT_USER_AGENT = git/$(GIT_VERSION) -@@ Makefile: VCSSVN_OBJS += vcs-svn/fast_export.o +@@ Makefile: VCSSVN_OBJS += vcs-svn/sliding_window.o VCSSVN_OBJS += vcs-svn/svndiff.o VCSSVN_OBJS += vcs-svn/svndump.o +REFTABLE_OBJS += reftable/basics.o +REFTABLE_OBJS += reftable/block.o -+REFTABLE_OBJS += reftable/bytes.o +REFTABLE_OBJS += reftable/file.o +REFTABLE_OBJS += reftable/iter.o +REFTABLE_OBJS += reftable/merged.o +REFTABLE_OBJS += reftable/pq.o +REFTABLE_OBJS += reftable/reader.o +REFTABLE_OBJS += reftable/record.o ++REFTABLE_OBJS += reftable/refname.o ++REFTABLE_OBJS += reftable/reftable.o +REFTABLE_OBJS += reftable/slice.o +REFTABLE_OBJS += reftable/stack.o +REFTABLE_OBJS += reftable/tree.o @@ builtin/init-db.c: static int create_default_files(const char *template_path, + reinit = (!access(path, R_OK) || + readlink(path, junk, sizeof(junk) - 1) != -1); + -+ /* -+ * refs/heads is a file when using reftable. We can't reinitialize with -+ * a reftable because it will overwrite HEAD -+ */ ++ /* ++ * refs/heads is a file when using reftable. We can't reinitialize with ++ * a reftable because it will overwrite HEAD ++ */ + if (reinit && (!strcmp(fmt->ref_storage, "reftable")) == + is_directory(git_path_buf(&buf, "refs/heads"))) { + die("cannot switch ref storage format."); @@ builtin/init-db.c: static int create_default_files(const char *template_path, /* * We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. - */ -- - if (refs_init_db(&err)) - die("failed to set up refs db: %s", err.buf); - @@ builtin/init-db.c: static int create_default_files(const char *template_path, * Create the default symlink from ".git/HEAD" to the "master" * branch, if it does not exist yet. @@ builtin/init-db.c: static int create_default_files(const char *template_path, if (!reinit) { if (create_symref("HEAD", "refs/heads/master", NULL) < 0) exit(1); -- } -- -- initialize_repository_version(fmt->hash_algo); -+ } + } +- initialize_repository_version(fmt->hash_algo); + initialize_repository_version(fmt->hash_algo, fmt->ref_storage); -+ + /* Check filemode trustability */ path = git_path_buf(&buf, "config"); - filemode = TEST_FILEMODE; @@ builtin/init-db.c: static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash } @@ refs.c: struct ref_store *get_main_ref_store(struct repository *r) BUG("attempting to get main_ref_store outside of repository"); - r->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS); -+ r->refs_private = ref_store_init( -+ r->gitdir, -+ r->ref_storage_format ? r->ref_storage_format : DEFAULT_REF_STORAGE, -+ REF_STORE_ALL_CAPS); ++ r->refs_private = ref_store_init(r->gitdir, ++ r->ref_storage_format ? ++ r->ref_storage_format : ++ DEFAULT_REF_STORAGE, ++ REF_STORE_ALL_CAPS); return r->refs_private; } @@ refs/reftable-backend.c (new) + unsigned int store_flags; + + int err; -+ char *repo_dir; ++ char *repo_dir; + char *reftable_dir; + struct reftable_stack *stack; +}; @@ refs/reftable-backend.c (new) + + base_ref_store_init(ref_store, &refs_be_reftable); + refs->store_flags = store_flags; -+ refs->repo_dir = xstrdup(path); ++ refs->repo_dir = xstrdup(path); + strbuf_addf(&sb, "%s/reftable", path); + refs->reftable_dir = xstrdup(sb.buf); + strbuf_reset(&sb); @@ refs/reftable-backend.c (new) + strbuf_addf(&sb, "%s/refs/heads", refs->repo_dir); + write_file(sb.buf, "this repository uses the reftable format"); + -+ return 0; ++ return 0; +} + +struct git_reftable_iterator { @@ refs/reftable-backend.c (new) + } + + err = reftable_stack_read_ref(refs->stack, refname, &ref); -+ if (err > 0) { -+ errno = ENOENT; -+ err = -1; -+ goto exit; -+ } ++ if (err > 0) { ++ errno = ENOENT; ++ err = -1; ++ goto exit; ++ } + if (err < 0) { -+ errno = reftable_error_to_errno(err); -+ err = -1; ++ errno = reftable_error_to_errno(err); ++ err = -1; + goto exit; + } + if (ref.target != NULL) { @@ refs/reftable-backend.c (new) + } else if (ref.value != NULL) { + hashcpy(oid->hash, ref.value); + } else { -+ *type |= REF_ISBROKEN; -+ errno = EINVAL; -+ err = -1; -+ } ++ *type |= REF_ISBROKEN; ++ errno = EINVAL; ++ err = -1; ++ } +exit: + reftable_ref_record_clear(&ref); + return err; @@ refs/reftable-backend.c (new) + reftable_reflog_expire +}; - ## reftable/update.sh ## -@@ reftable/update.sh: SRC=${SRC:-origin} BRANCH=${BRANCH:-origin/master} - cp reftable-repo/c/*.[ch] reftable/ - cp reftable-repo/c/include/*.[ch] reftable/ - cp reftable-repo/LICENSE reftable/ --git --git-dir reftable-repo/.git show --no-patch origin/master \ -+git --git-dir reftable-repo/.git show --no-patch ${BRANCH} \ - > reftable/VERSION - - mv reftable/system.h reftable/system.h~ - ## repository.c ## @@ repository.c: int repo_init(struct repository *repo, if (worktree) @@ t/t0031-reftable.sh (new) + git checkout -b master +' + ++ ++test_expect_success 'dir/file conflict' ' ++ initialize && ++ test_commit file && ++ ! git branch master/forbidden ++' ++ ++ +test_expect_success 'do not clobber existing repo' ' + rm -rf .git && + git init --ref-storage=files && @@ t/t0031-reftable.sh (new) + test_cmp expect actual +' + ++ +test_done ++ -: ----------- > 10: 513d585f0f8 vcxproj: adjust for the reftable changes 11: 8af67b85e49 ! 11: 846fe29fa4b Add some reftable testing infrastructure @@ builtin/init-db.c: static const char *const init_db_usage[] = { ## refs.c ## @@ refs.c: struct ref_store *get_main_ref_store(struct repository *r) - if (!r->gitdir) - BUG("attempting to get main_ref_store outside of repository"); - -- r->refs_private = ref_store_init( -- r->gitdir, -- r->ref_storage_format ? r->ref_storage_format : DEFAULT_REF_STORAGE, -- REF_STORE_ALL_CAPS); -+ r->refs_private = ref_store_init(r->gitdir, -+ r->ref_storage_format ? -+ r->ref_storage_format : + r->refs_private = ref_store_init(r->gitdir, + r->ref_storage_format ? + r->ref_storage_format : +- DEFAULT_REF_STORAGE, + default_ref_storage(), -+ REF_STORE_ALL_CAPS); + REF_STORE_ALL_CAPS); return r->refs_private; } - @@ refs.c: struct ref_store *get_submodule_ref_store(const char *submodule) goto done; 12: b02b83a92ad = 12: a7d1d2e721c t: use update-ref and show-ref to reading/writing refs -- gitgitgadget
