Thanks for your answers and recommendations! I totally agree that https interception is a bad idea, but we can't get away from it. We have been living with this problem for about 1.5 years. It gets worse every month. We have reached the boiling point and want to solve the problem without turning off https interception. > You are more likely to be able to setup a local git mirror, within your security perimeter and enforce a review (think peer review software) of outbound pushes on a non-MITM connection between those two points only. Security management are much more likely to approve this bypassing your security appliance. We have a local mirror(artifactory) inside the security perimeter, but it's impossible for us to download tons of terabytes from GitHub... > Yes, that's because you're tampering with the data. The output you're > getting clearly indicates something is modifying the data. TLS normally > protects the data from accidental as well as intentional errors, so > there's no situation in which this could be an accident but for your > proxy. > Git will work in this case if and only if your proxy does the things I > told you, which your proxy doesn't. It isn't a transparent proxy, since > that by definition requires that requests and responses are not > modified. > This is a non-supported use case for Git. That being said - you are going to have to trace out the communications. This is going to start with GIT_TRACE = true GIT_CURL_VERBOSE = 1 on your command. > You are going to need to invest in the time and knowledge to identify why your Cisco WSA is modifying data. Think it is. We think this is a bug on Cisco WSA that in some cases traffic may change. Let's look at the decrypted traffic. > I do appreciate you mentioning the proxy you're using so we can include > it as a known broken proxy in future versions of the Git FAQ. Hmmm, is there already such a list in the Git FAQ? > Git sends data that is compressed but not using a normal compressed > archive format. Thus, if you do anything that inspects the data to see > if it is "malicious" or "inappropriate," your technology will likely > flag data that just happens to have a byte sequence that collides with > something that you think is bad. For example, if you flag the text > "sex" because you think it is inappropriate, then the probably is about > 1 in 2 ^ 24 that sequence will appear in the stream and you will break the > protocol, since compressed data often appears random. Good idea, it might be. Thank you, we will look in this direction. ср, 14 апр. 2021 г. в 15:29, Jason Pyeron <jpyeron@xxxxxxxx>: > > > From: brian m. Carlson > > Sent: Wednesday, April 14, 2021 7:49 AM > > > > On 2021-04-14 at 09:35:48, Vitaly VS wrote: > > > Thank you for the fast response. > > > > > > About our network environment > > > We are Cisco WSA(Servers SW, ASA, ISR) that proxies http/https > > > traffic. Client requests a website, network device redirects traffic > > > to WSA using WCCPv2, then WSA proxies the request to Cisco ASA > > > Firewall and internet. > > > > > > Yes, that our transparent proxy is not completely transparent because > > > HTTPS Interception. > > > If network guys turning off HTTPS Interception for github.com "git > > > clone" work well through the transparent proxy... > > > > Yes, that's because you're tampering with the data. The output you're > > getting clearly indicates something is modifying the data. TLS normally > > protects the data from accidental as well as intentional errors, so > > there's no situation in which this could be an accident but for your > > proxy. > > > > Git will work in this case if and only if your proxy does the things I > > told you, which your proxy doesn't. It isn't a transparent proxy, since > > that by definition requires that requests and responses are not > > modified. > > > > I do appreciate you mentioning the proxy you're using so we can include > > it as a known broken proxy in future versions of the Git FAQ. > > > > This is a non-supported use case for Git. That being said - you are going to have to trace out the communications. This is going to start with GIT_TRACE=true GIT_CURL_VERBOSE=1 on your command. > > You are going to need to invest in the time and knowledge to identify why your Cisco WSA is modifying data. > > > > Disabled https interception for github is a security issue for > > > us(corporate risks, code leak, etc). That's why I asked about can the > > > git client working with https interception. > > > > Many major companies manage to avoid these risks without introducing > > security holes into their network and breaking common applications that > > speak standard protocols by avoiding using TLS-intercepting proxies. In > > fact, I've worked at a company which was very diligent about these > > matters and had strict policies on them, and in no situation did we > > intercept TLS traffic. > > > > > Proxy didn't alter any of the contents of the stream(that says to me > > > our SecOps), but I've not received decrypted traffic yet to be sure. > > > HTTPS traffic caching but we are also disabled this feature for github. > > > > > > Common downloads with curl or browser from the same sources from > > > github or gitlab working well. > > > > Git sends data that is compressed but not using a normal compressed > > archive format. Thus, if you do anything that inspects the data to see > > if it is "malicious" or "inappropriate," your technology will likely > > flag data that just happens to have a byte sequence that collides with > > something that you think is bad. For example, if you flag the text > > "sex" because you think it is inappropriate, then the probably is about > > 1 in 2^24 that sequence will appear in the stream and you will break the > > protocol, since compressed data often appears random. > > > > This is, I suspect, why Git tends to break in situations where other > > programs do not. > > > > If you want Git to work reliably, you can never modify the data of the > > stream, no matter what. You also can't buffer the stream (for > > example, try to turn chunked encoding to non-chunked). This is > > something you're going to have to accept; bargaining isn't going to work > > here, no matter how much you want it to. > > > > I can't force you to listen to me here, but I strongly recommend that if > > you don't, you clearly communicate to your users using Git what you're > > doing and that you know this will break Git so other parties don't have > > to. I'm sure that the support teams for GitHub and GitLab will tell you > > that it's your proxy and that you have to remove or disable it just as I > > am here. > > > > > Brian, really thank you for pdf but we haven't Client-end TLS > > > interception on our clients. > > > > Here are some articles covering hardware middleboxes as well: > > > > https://blog.cloudflare.com/monsters-in-the-middleboxes/ > > https://jhalderm.com/pub/papers/interception-ndss17.pdf > > > > I should point out that this problem is so pervasive that TLS 1.3 > > includes intentional countermeasures against some of the worst practices > > of TLS middleboxes. I'm certain that most of the TLS working group > > would prevent TLS middleboxes from working at all if they could find a > > way to do so, and many of those people are at the vanguard of Internet > > security. > > While I agree, I work in environments where I do not have the luxury to change their minds. As a result I sometimes have to develop (very expensive use of time) test suites to tease out the secret sauce of security appliances. Sometimes we are fortunate that there is enough money to get the vendor to care and share. > > You are more likely to be able to setup a local git mirror, within your security perimeter and enforce a review (think peer review software) of outbound pushes on a non-MITM connection between those two points only. Security management are much more likely to approve this bypassing your security appliance. > > [I set the reply to header, don’t email me directly I am on the list] > > -- > Jason Pyeron | Architect > PD Inc | > 10 w 24th St | > Baltimore, MD | > > .mil: jason.j.pyeron.ctr@xxxxxxxx > .com: jpyeron@xxxxxxxx > tel : 202-741-9397 > > > >
