On Sun, Dec 12 2021, Andrew Oakley wrote: > On Fri, 10 Dec 2021 15:30:59 +0000 > Joel Holdsworth <jholdsworth@xxxxxxxxxx> wrote: >> The motivation for this change is that there is a family of issues >> with git-p4's handling of incoming text data when it contains bytes >> which cannot be decoded into UTF-8 characters. For text files created >> in Windows, CP1252 Smart Quote Characters (0x93 and 0x94) are seen >> fairly frequently. These codes are invalid in UTF-8, so if the script >> encounters any file or file name containing them, on Python 2 the >> symbols will be corrupted, and on Python 3 the script will fail with >> an exception. > > As I've pointed out previously, peforce fails to store the encoding of > text like commit messages. With Windows perforce clients, the encoding > used seems to be based on the current code page on the client which > made the commit. If you're part of a global organisation with people > in different locales making commits then you will find that there is > not a consistent encoding for commit messages. > > Given that you don't know the encoding of the text, what's the best > thing to do with the data? Options I can see are: > > - Feed the raw bytes directly into git. The i18n.commitEncoding config > option can be set by the user if they want to attempt to decode the > commit messages in something other than UTF-8. > - Attempt to detect the encoding somehow, feed the raw bytes directly > into git and set the encoding on the commit. > - Attempt to dedect the encoding somehow and reencode everything into > UTF-8. > > Right now, if you use python2 then you get the behaviour as described > in the first of these options. It doesn't "corrupt" anything, it just > transfers the bytes from perforce into git. If you use python3 then > git-p4 is unusable because it throws exceptions trying to decode things. > > [...] > > I think that this is the problem which really needs solving. Dropping > support for python2 doesn't make the issue go away (although it might > make it slightly easier to write the code). I think that the python2 > compatibility should be maintained at least until the encoding problems > have been solved for python3. > > I previously wrote some patches which attempt to move in what I think is > the right direction, but unfortunately they never got upstreamed: > > https://lore.kernel.org/git/20210412085251.51475-1-andrew@xxxxxxxxxxxxx/ > > Your comments elsewhere that git-p4 could benifit from some clean-up > seem accurate to me, and it would be good to see that kind of change. This summary makes sense, i.e. if the original SCM doesn't have a declared or consistent encoding then having no "encoding" header etc. in git likewise makes sense, and we should be trying to handle it in our output layer. [Snipped from above]: > It's not clear to me how "attempt to detect the encoding somehow" would > work. The first option therefore seems like the best choice. This really isn't possible to do in the general case, but you can get pretty far with heuristics. The best way to do this that I'm aware of is Mozilla's character detection library: https://www-archive.mozilla.org/projects/intl/chardetinterface Here's an old (maybe/probably not up-to-date) copy of its sources that I've worked with: https://metacpan.org/release/JGMYERS/Encode-Detect-1.01/source/src?sort=[[2,1]] I.e. if you get arbitrary text the best you can do if you're going in blind is to have various character/language frequency tables to try to guess at what encoding is the most plausible, but even then you might still be wrong. I'd think for git-p4 (and git-svn, how do we handle this there?) a sensible first approximation would be to use UTF-8, and if we encounter data that doesn't conform die. Then offer the user to manually configure a "fallback" encoding. I think most real-world projects only have two of those, e.g. some old latin1 data before a UTF-8 migration. And maybe start with a "don't even try" mode, which AFAICT is what you're describing Python 2 doing. If we change the Python 3 code to do what the Python 2 code does now, will we pull the rug from under users who have only ever used the Python 3 code, and are relying on those semantics somehow?
