Re: [BUG?] git http connection reuse

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Feb 15, 2014, at 20:05, Jeff King wrote:
I've noticed that git does not reuse http connections when fetching, and
I'm trying to figure out why. It seems related to our use of two curl
features:

 1. If we use the curl_multi interface (even though we are doing the
    requests sequentially), we do not get reuse.

Take a look at the curl_multi_setopt page [1]. I think these explain the curl_multi issue:

CURLMOPT_PIPELINING

Pass a long set to 1 to enable or 0 to disable. Enabling pipelining on a multi handle will make it attempt to perform HTTP Pipelining as far as possible for transfers using this handle. This means that if you add a second request that can use an already existing connection, the second request will be "piped" on the same connection rather than being executed in parallel. (Added in 7.16.0)


CURLMOPT_MAX_TOTAL_CONNECTIONS

Pass a long. The set number will be used as the maximum amount of simultaneously open connections in total. For each new session, libcurl will open a new connection up to the limit set by CURLMOPT_MAX_TOTAL_CONNECTIONS. When the limit is reached, the sessions will be pending until there are available connections. If CURLMOPT_PIPELINING is 1, libcurl will try to pipeline if the host is capable of it.

The default value is 0, which means that there is no limit. However, for backwards compatibility, setting it to 0 when CURLMOPT_PIPELINING is 1 will not be treated as unlimited. Instead it will open only 1 connection and try to pipeline on it.

(Added in 7.30.0)

If pipelining is off (the default) and total connections is not 1 it sounds to me from the description above that the requests will be executed on separate connections until the maximum number of connections is in use and then there might be some reuse. That might not be what's actually going on with multi, but that's my guess and I think setting CURLMOPT_PIPELINING to to 1 and then also setting CURLMOPT_MAX_TOTAL_CONNECTIONS to a non-zero value might be what you want.

 2. If we set CURLOPT_HTTPAUTH to CURLAUTH_ANY, we do not get reuse.
[snip]
My curl version is 7.35.0, if that makes a difference.

And that is explained by this from the CHANGES file:

Daniel Stenberg (7 Jan 2014)
- ConnectionExists: fix NTLM check for new connection

  When the requested authentication bitmask includes NTLM, we cannot
  re-use a connection for another username/password as we then risk
  re-using NTLM (connection-based auth).

This has the unfortunate downside that if you include NTLM as a possible auth, you cannot re-use connections for other usernames/passwords even
  if NTLM doesn't end up the auth type used.

  Reported-by: Paras S
  Patched-by: Paras S
  Bug: http://curl.haxx.se/mail/lib-2014-01/0046.html

Looks like you're just lucky as that above change first appears in 7.35.0. But it seems there are some patches for older versions so they might be affected as well [2].

Adjusting your test program to leave the first connection set to CURLAUTH_ANY, but then setting the second one to CURLAUTH_ANY with the two NTLM bits turned off (CURLAUTH_NTLM and CURLAUTH_NTLM_WB) results in the connection being reused for me. :)

With the older cURL versions I already had installed, the second connection is always reused. I didn't see the non-reuse with your sample code until I fetched 7.35.0.

--Kyle

[1] http://curl.haxx.se/libcurl/c/curl_multi_setopt.html
[2] http://curl.haxx.se/docs/adv_20140129.html

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]