From: Lars Schneider <larsxschneider@xxxxxxxxx> Hi, Patches 1-4 and 6 are preparation and helper functions. Patch 5 is the actual change. This series depends on Torsten's "convert_to_git(): safe_crlf/checksafe becomes int conv_flags" patch: https://public-inbox.org/git/20180113224931.27031-1-tboegi@xxxxxx/ Changes since v3: * I renamed the attribute from "checkout-encoding" to "working-tree-encoding" in the hope to convey better what the attribute is about. * I rebased the series to Git 2.16 and removed Torsten's patch as he posted the patch on his own. * Fix documentation wording. (Torsten) * A macro was used in a commit before it's introduction. Fixed!(Junio) Thanks, Lars RFC: https://public-inbox.org/git/BDB9B884-6D17-4BE3-A83C-F67E2AFA2B46@xxxxxxxxx/ v1: https://public-inbox.org/git/20171211155023.1405-1-lars.schneider@xxxxxxxxxxxx/ v2: https://public-inbox.org/git/20171229152222.39680-1-lars.schneider@xxxxxxxxxxxx/ v3: https://public-inbox.org/git/20180106004808.77513-1-lars.schneider@xxxxxxxxxxxx/ Base Ref: Web-Diff: https://github.com/larsxschneider/git/commit/21f4dac5ab Checkout: git fetch https://github.com/larsxschneider/git encoding-v4 && git checkout 21f4dac5ab ### Interdiff (v3-rebased-2.16..v4): diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt index 1bc03e69cb..a8dbf4be30 100644 --- a/Documentation/gitattributes.txt +++ b/Documentation/gitattributes.txt @@ -272,8 +272,8 @@ few exceptions. Even though... catch potential problems early, safety triggers. -`checkout-encoding` -^^^^^^^^^^^^^^^^^^^ +`working-tree-encoding` +^^^^^^^^^^^^^^^^^^^^^^^ Git recognizes files encoded with ASCII or one of its supersets (e.g. UTF-8 or ISO-8859-1) as text files. All other encodings are usually @@ -281,17 +281,17 @@ interpreted as binary and consequently built-in Git text processing tools (e.g. 'git diff') as well as most Git web front ends do not visualize the content. -In these cases you can teach Git the encoding of a file in the working -directory with the `checkout-encoding` attribute. If a file with this +In these cases you can tell Git the encoding of a file in the working +directory with the `working-tree-encoding` attribute. If a file with this attributes is added to Git, then Git reencodes the content from the specified encoding to UTF-8 and stores the result in its internal data structure (called "the index"). On checkout the content is encoded back to the specified encoding. -Please note that using the `checkout-encoding` attribute may have a +Please note that using the `working-tree-encoding` attribute may have a number of pitfalls: -- Git clients that do not support the `checkout-encoding` attribute +- Git clients that do not support the `working-tree-encoding` attribute will checkout the respective files UTF-8 encoded and not in the expected encoding. Consequently, these files will appear different which typically causes trouble. This is in particular the case for @@ -304,7 +304,7 @@ number of pitfalls: - Reencoding content requires resources that might slow down certain Git operations (e.g 'git checkout' or 'git add'). -Use the `checkout-encoding` attribute only if you cannot store a file in +Use the `working-tree-encoding` attribute only if you cannot store a file in UTF-8 encoding and if you want Git to be able to process the content as text. @@ -313,7 +313,7 @@ with byte order mark (BOM) and you want Git to perform automatic line ending conversion based on your platform. ------------------------ -*.txt text checkout-encoding=UTF-16 +*.txt text working-tree-encoding=UTF-16 ------------------------ Use the following attributes if your '*.txt' files are UTF-16 little @@ -321,7 +321,7 @@ endian encoded without BOM and you want Git to use Windows line endings in the working directory. ------------------------ -*.txt checkout-encoding=UTF-16LE text eol=CRLF +*.txt working-tree-encoding=UTF-16LE text eol=CRLF ------------------------ You can get a list of all available encodings on your platform with the diff --git a/convert.c b/convert.c index 8559651b3f..13fad490ce 100644 --- a/convert.c +++ b/convert.c @@ -323,7 +323,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len, const char *advise_msg = _( "You told Git to treat '%s' as %s. A byte order mark " "(BOM) is prohibited with this encoding. Either use " - "%.6s as checkout encoding or remove the BOM from the " + "%.6s as working tree encoding or remove the BOM from the " "file."); advise(advise_msg, path, enc->name, enc->name, enc->name); @@ -339,7 +339,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len, const char *advise_msg = _( "You told Git to treat '%s' as %s. A byte order mark " "(BOM) is required with this encoding. Either use " - "%sBE/%sLE as checkout encoding or add a BOM to the " + "%sBE/%sLE as working tree encoding or add a BOM to the " "file."); advise(advise_msg, path, enc->name, enc->name, enc->name); if (conv_flags & CONV_WRITE_OBJECT) @@ -1237,7 +1237,7 @@ static void convert_attrs(struct conv_attrs *ca, const char *path) if (!check) { check = attr_check_initl("crlf", "ident", "filter", - "eol", "text", "checkout-encoding", + "eol", "text", "working-tree-encoding", NULL); user_convert_tail = &user_convert; encoding_tail = &encoding; diff --git a/t/t0028-checkout-encoding.sh b/t/t0028-working-tree-encoding.sh similarity index 89% rename from t/t0028-checkout-encoding.sh rename to t/t0028-working-tree-encoding.sh index 5f1c911c07..0f36d4990a 100755 --- a/t/t0028-checkout-encoding.sh +++ b/t/t0028-working-tree-encoding.sh @@ -1,6 +1,6 @@ #!/bin/sh -test_description='checkout-encoding conversion via gitattributes' +test_description='working-tree-encoding conversion via gitattributes' . ./test-lib.sh @@ -10,7 +10,7 @@ test_expect_success 'setup test repo' ' git config core.eol lf && text="hallo there!\ncan you read me?" && - echo "*.utf16 text checkout-encoding=utf-16" >.gitattributes && + echo "*.utf16 text working-tree-encoding=utf-16" >.gitattributes && printf "$text" >test.utf8.raw && printf "$text" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw && cp test.utf16.raw test.utf16 && @@ -45,10 +45,10 @@ test_expect_success 'check prohibited UTF BOM' ' printf "\0\0\376\777\0\0\0a\0\0\0b\0\0\0c" >bebom.utf32be.raw && printf "\777\376\0\0a\0\0\0b\0\0\0c\0\0\0" >lebom.utf32le.raw && - echo "*.utf16be text checkout-encoding=utf-16be" >>.gitattributes && - echo "*.utf16le text checkout-encoding=utf-16le" >>.gitattributes && - echo "*.utf32be text checkout-encoding=utf-32be" >>.gitattributes && - echo "*.utf32le text checkout-encoding=utf-32le" >>.gitattributes && + echo "*.utf16be text working-tree-encoding=utf-16be" >>.gitattributes && + echo "*.utf16le text working-tree-encoding=utf-16le" >>.gitattributes && + echo "*.utf32be text working-tree-encoding=utf-32be" >>.gitattributes && + echo "*.utf32le text working-tree-encoding=utf-32le" >>.gitattributes && # Here we add a UTF-16 files with BOM (big-endian and little-endian) # but we tell Git to treat it as UTF-16BE/UTF-16LE. In these cases @@ -91,7 +91,7 @@ test_expect_success 'check prohibited UTF BOM' ' ' test_expect_success 'check required UTF BOM' ' - echo "*.utf32 text checkout-encoding=utf-32" >>.gitattributes && + echo "*.utf32 text working-tree-encoding=utf-32" >>.gitattributes && cp nobom.utf16be.raw nobom.utf16 && test_must_fail git add nobom.utf16 2>err.out && @@ -143,11 +143,11 @@ test_expect_success 'eol conversion for UTF-16 encoded files on checkout' ' test_expect_success 'check unsupported encodings' ' - echo "*.nothing text checkout-encoding=" >>.gitattributes && + echo "*.nothing text working-tree-encoding=" >>.gitattributes && printf "nothing" >t.nothing && git add t.nothing && - echo "*.garbage text checkout-encoding=garbage" >>.gitattributes && + echo "*.garbage text working-tree-encoding=garbage" >>.gitattributes && printf "garbage" >t.garbage && test_must_fail git add t.garbage 2>err.out && test_i18ngrep "fatal: failed to encode" err.out && @@ -161,7 +161,7 @@ test_expect_success 'error if encoding round trip is not the same during refresh BEFORE_STATE=$(git rev-parse HEAD) && # Skip the UTF-16 filter for the added file - # This simulates a Git version that has no checkoutEncoding support + # This simulates a Git version that has no working tree encoding support echo "hallo" >nonsense.utf16 && TEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) && git update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 && ### Patches Lars Schneider (6): strbuf: remove unnecessary NUL assignment in xstrdup_tolower() strbuf: add xstrdup_toupper() utf8: add function to detect prohibited UTF-16/32 BOM utf8: add function to detect a missing UTF-16/32 BOM convert: add 'working-tree-encoding' attribute convert: add tracing for 'working-tree-encoding' attribute Documentation/gitattributes.txt | 60 +++++++++++ convert.c | 218 ++++++++++++++++++++++++++++++++++++++- convert.h | 1 + sha1_file.c | 2 +- strbuf.c | 13 ++- strbuf.h | 1 + t/t0028-working-tree-encoding.sh | 198 +++++++++++++++++++++++++++++++++++ utf8.c | 37 +++++++ utf8.h | 25 +++++ 9 files changed, 552 insertions(+), 3 deletions(-) create mode 100755 t/t0028-working-tree-encoding.sh base-commit: 8a2f0888555ce46ac87452b194dec5cb66fb1417 -- 2.16.0
