On Fri, 10 Dec 2021 15:30:59 +0000 Joel Holdsworth <jholdsworth@xxxxxxxxxx> wrote: > The motivation for this change is that there is a family of issues > with git-p4's handling of incoming text data when it contains bytes > which cannot be decoded into UTF-8 characters. For text files created > in Windows, CP1252 Smart Quote Characters (0x93 and 0x94) are seen > fairly frequently. These codes are invalid in UTF-8, so if the script > encounters any file or file name containing them, on Python 2 the > symbols will be corrupted, and on Python 3 the script will fail with > an exception. As I've pointed out previously, peforce fails to store the encoding of text like commit messages. With Windows perforce clients, the encoding used seems to be based on the current code page on the client which made the commit. If you're part of a global organisation with people in different locales making commits then you will find that there is not a consistent encoding for commit messages. Given that you don't know the encoding of the text, what's the best thing to do with the data? Options I can see are: - Feed the raw bytes directly into git. The i18n.commitEncoding config option can be set by the user if they want to attempt to decode the commit messages in something other than UTF-8. - Attempt to detect the encoding somehow, feed the raw bytes directly into git and set the encoding on the commit. - Attempt to dedect the encoding somehow and reencode everything into UTF-8. Right now, if you use python2 then you get the behaviour as described in the first of these options. It doesn't "corrupt" anything, it just transfers the bytes from perforce into git. If you use python3 then git-p4 is unusable because it throws exceptions trying to decode things. It's not clear to me how "attempt to detect the encoding somehow" would work. The first option therefore seems like the best choice. I think that this is the problem which really needs solving. Dropping support for python2 doesn't make the issue go away (although it might make it slightly easier to write the code). I think that the python2 compatibility should be maintained at least until the encoding problems have been solved for python3. I previously wrote some patches which attempt to move in what I think is the right direction, but unfortunately they never got upstreamed: https://lore.kernel.org/git/20210412085251.51475-1-andrew@xxxxxxxxxxxxx/ Your comments elsewhere that git-p4 could benifit from some clean-up seem accurate to me, and it would be good to see that kind of change.
