In our network we are behind a proxy that I don't have access to. In order to speed up deployments and development I am trying to set up a caching squid proxy for yum and maven repositories.
Naturally, this proxy needs to be configured to use our company's global proxy as parent.
I have successfully set it up to the point where it works when e.g. downloading files using wget. However when using it with an actual maven build, the build hangs when trying to download pom or jar files.
After having increased the log level I found out that my squid does not use the parent proxy in such cases, and tries to connect to the internet which is not possible since we can only connect through the global proxy.
Naturally, this proxy needs to be configured to use our company's global proxy as parent.
I have successfully set it up to the point where it works when e.g. downloading files using wget. However when using it with an actual maven build, the build hangs when trying to download pom or jar files.
After having increased the log level I found out that my squid does not use the parent proxy in such cases, and tries to connect to the internet which is not possible since we can only connect through the global proxy.
A closer look at the logs revealed that maven issued HEAD instead of GET requests in my case. I could hence reproduce the problem without maven using this command line:
In this example we would normally get a 404 (maven tries out a configured list of servers to find a particular resource), however the same problem applies with existing resources.
Here's a snippet of what I find in the log after such an unsuccessful request:
In contrast, when I issue the above command without '-I', I get a different log output:
--
---------- MDietze@xxxxxxxxx --/-- martin@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ----
------------- / http://herbert.the-little-red-haired-girl.org / -------------
curl -I http://repo.springsource.org/snapshot/org/springframework/boot/spring-boot-starter-parent/1.2.2.RELEASE/spring-boot-starter-parent-1.2.2.RELEASE.pom
In this example we would normally get a 404 (maven tries out a configured list of servers to find a particular resource), however the same problem applies with existing resources.
To me it seems like my squid does not understand it needs to use the global proxy for HEAD requests as well as for GET. But I could not find any reference to this particular problem anywhere in the web.
I've appended all information that seems relevant below. Now I would really like to know: what am I doing wrong?
Cheers,
Martin
Appendix: system information, log messages and configuration.
I am using squid 3.1.23 on an Oracle Linux 6.7 system (an RHEL6 variant). I have reproduced the same problem on an Oracle Linux 7.1 system with squid 3.5.3 with basically the same configuration.
Here's a snippet of what I find in the log after such an unsuccessful request:
2015/09/15 12:29:06.364| peerSelectFoo: 'HEAD repo.springsource.org'
2015/09/15 12:29:06.364| peerSelectFoo: direct = DIRECT_MAYBE
2015/09/15 12:29:06.364| peerSelectIcpPing: http://repo.springsource.org/snapshot/org/springframework/boot/spring-boot-starter-parent/1.2.2.RELEASE/spring-boot-starter-parent-1.2.2.RELEASE.pom
2015/09/15 12:29:06.364| peerAddFwdServer: adding DIRECT DIRECT
2015/09/15 12:29:06.364| peerSelectCallback: http://repo.springsource.org/snapshot/org/springframework/boot/spring-boot-starter-parent/1.2.2.RELEASE/spring-boot-starter-parent-1.2.2.RELEASE.pom
2015/09/15 12:29:06.364| cbdataReferenceValid: 0x7fa2dbf5f3c8
2015/09/15 12:29:06.364| cbdataUnlock: 0x7fa2dbf5f3c8=1
2015/09/15 12:29:06.364| fwdStartComplete: http://repo.springsource.org/snapshot/org/springframework/boot/spring-boot-starter-parent/1.2.2.RELEASE/spring-boot-starter-parent-1.2.2.RELEASE.pom
2015/09/15 12:29:06.364| fwdConnectStart: http://repo.springsource.org/snapshot/org/springframework/boot/spring-boot-starter-parent/1.2.2.RELEASE/spring-boot-starter-parent-1.2.2.RELEASE.pom
2015/09/15 12:29:06.364| PconnPool::key(repo.springsource.org,80,(no domain),[::]is {repo.springsource.org:80}
In contrast, when I issue the above command without '-I', I get a different log output:
2015/09/15 12:32:54.348| peerSelectFoo: 'GET repo.springsource.org'
2015/09/15 12:32:54.348| peerSelectFoo: direct = DIRECT_MAYBE
2015/09/15 12:32:54.348| peerDigestLookup: peer proxy.local.lan
2015/09/15 12:32:54.348| peerDigestLookup: gone!
2015/09/15 12:32:54.348| neighborsDigestSelect: choices: 0 (0)
2015/09/15 12:32:54.348| peerNoteDigestLookup: peer <none>, lookup: LOOKUP_NONE
2015/09/15 12:32:54.348| peerSelectIcpPing: http://repo.springsource.org/snapshot/org/springframework/boot/spring-boot-starter-parent/1.2.2.RELEASE/spring-boot-starter-parent-1.2.2.RELEASE.pom
2015/09/15 12:32:54.348| neighborsCount: 0
2015/09/15 12:32:54.348| peerSelectIcpPing: counted 0 neighbors
2015/09/15 12:32:54.348| peerGetSomeParent: GET repo.springsource.org
2015/09/15 12:32:54.348| neighbors.cc(339) getRoundRobinParent: returning NULL
2015/09/15 12:32:54.348| getWeightedRoundRobinParent: returning NULL
2015/09/15 12:32:54.348| neighborUp: UP (no-query): proxy.local.lan (172.16.8.250:3130)
2015/09/15 12:32:54.348| neighborUp: UP (no-query): proxy.local.lan (172.16.8.250:3130)
2015/09/15 12:32:54.348| getFirstUpParent: returning proxy.local.lan
2015/09/15 12:32:54.348| peerSelect: FIRST_UP_PARENT/proxy.local.lan
2015/09/15 12:32:54.348| peerAddFwdServer: adding proxy.local.lan FIRST_UP_PARENT
2015/09/15 12:32:54.348| cbdataLock: 0x7fa2dbca0d58=1
2015/09/15 12:32:54.348| peerAddFwdServer: adding DIRECT DIRECT
2015/09/15 12:32:54.348| peerSelectCallback: http://repo.springsource.org/snapshot/org/springframework/boot/spring-boot-starter-parent/1.2.2.RELEASE/spring-boot-starter-parent-1.2.2.RELEASE.pom
2015/09/15 12:32:54.348| cbdataReferenceValid: 0x7fa2dbf5f3c8
2015/09/15 12:32:54.348| cbdataUnlock: 0x7fa2dbf5f3c8=1
2015/09/15 12:32:54.348| fwdStartComplete: http://repo.springsource.org/snapshot/org/springframework/boot/spring-boot-starter-parent/1.2.2.RELEASE/spring-boot-starter-parent-1.2.2.RELEASE.pom
As we see, in the second example the parent proxy is used, while in the first it is not (and hence trying to connect repo.springsource.org fails).
Here is what I changed to the default configuration:
#acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
#acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
[...]
cache_dir ufs /var/cache/squid 1000 16 256
[...]
cache_mem 64 MB
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
cache_effective_user squid
cache_effective_group squid
emulate_httpd_log on
debug_options ALL,10
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
cache_peer proxy.dermalog.hh parent 3128 3130 no-query no-digest no-netdb-exchange
prefer_direct off
--
---------- MDietze@xxxxxxxxx --/-- martin@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ----
------------- / http://herbert.the-little-red-haired-girl.org / -------------
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
