On Fri, 30 Apr 2021 11:08:57 -0700 Tzadik Vanderhoof <tzadik.vanderhoof@xxxxxxxxx> wrote: > My server is not unicode. > > These conversions are happening even with a non-Unicode perforce db. > I don't think it's the p4d code per se that is doing the conversion, > but rather an interaction between the OS and the code, which is > different under Linux vs Windows. It's not particularly obvious exactly what is happening here. The perforce command line client is written in a rather odd way - it uses the unicode (UTF-16) wWinMainCRTStartup entry point but then calls an undocumented API to get the "narrow" version of the command line. The code is here: https://swarm.workshop.perforce.com/projects/perforce_software-p4/files/2016-1/client/clientmain.cc I think that perforce will end up with the data in a code page that depends on the configuration of the machine. I don't think the exact details matter here - just that it's some semi-arbitrary encoding that isn't recorded in the commit. The key thing that I'm trying to point out here is that the encoding is not necessarily consistent between different commits. The changes that you have proposed force you to pick one encoding that will be used for every commit. If it's wrong then data will be corrupted, and there is no option provided to avoid that. The only way I can see to avoid this issue is to not attempt to re-encode the data - just pass it directly to git. I think another way to solve the issue you have is the encoding header on git commits. We can pass the bytes through git-p4 unmodified, but mark the commit message as being encoded using something that isn't UTF-8. That avoids any potentially lossy conversions when cloning the repository, but should allow the data to be displayed correctly in git. > In any event, if you look at my patch (v6 is the latest... > https://lore.kernel.org/git/20210429073905.837-1-tzadik.vanderhoof@xxxxxxxxx/ > ), you will see I have written tests that pass under both Linux and > Windows. (If you want to run them yourself, you need to base my patch > off of "master", not "seen"). The tests make clear what the > different behavior is and also show that p4d is not set to Unicode > (since the tests do not change the default setting). I don't think the tests are doing anything interesting on Linux - you stick valid UTF-8 in, and valid UTF-8 data comes out. I suspect the tests will fail on Windows if the relevant code page is set to a value that you're not expecting. For the purposes of writing tests that work the same everywhere we can use `p4 submit -i`. The data written on stdin isn't reencoded, even on Windows. I can rework my test to use `p4 submit -i` on windows. It should be fairly simple to write another change which allows the encoding to be set on commits created by git-p4. Does that seem like a reasonable way forward? I think it gets us: - sensible handling for repositories with mixed encodings - sensible handling for repositories with known encodings - tests that work the same on Linux and Windows
