I'm currently in the process of testing some software installs behind a proxy and ran into something I don't quite understand. While running behind a Squid proxy, apt-key calls were failing to process key requests. The same requests run fine directly connected to the internet. >From a machine directly connected to the internet: /usr/bin/apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 765C5E49F87CBDE0 Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /tmp/tmp.5JakWGN817 --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 765C5E49F87CBDE0 gpg: requesting key F87CBDE0 from hkp server keyserver.ubuntu.com gpg: key F87CBDE0: "RCB Build <rcb-deploy@xxxxxxxxxxxxxxxxxxx>" not changed gpg: Total number processed: 1 gpg: unchanged: 1 >From a machine with proxy only access, http_proxy set in /etc/environment & sudoers env_keep: /usr/bin/apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 765C5E49F87CBDE0 Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /tmp/tmp.9YzGDcyc3K --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 765C5E49F87CBDE0 gpg: requesting key F87CBDE0 from hkp server keyserver.ubuntu.com gpgkeys: key 765C5E49F87CBDE0 not found on keyserver gpg: no valid OpenPGP data found. gpg: Total number processed: 0 And in the access.log on the proxy: 1365271816.885 294 10.10.10.20 TCP_MISS/403 3827 GET http://keyserver.ubuntu.com/pks/lookup?op=get&options=mr&search=0xF87CBDE0 - DIRECT/91.189.89.49 text/html I've further distilled this down to a simple curl statement against keyserver.ubuntu.com, removing apt-key, gpg from the picture: curl -v http://keyserver.ubuntu.com ...output here is the squid access denied error message page.... To be clear, at this point, this same server behind squid CAN access other resources on the internet: curl -v ubuntu.com * About to connect() to proxy 10.10.10.10 port 3128 (#0)* Trying 10.10.10.10... connected > GET HTTP://ubuntu.com HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: ubuntu.com > Accept: */*> Proxy-Connection: Keep-Alive> * HTTP 1.0, assume close after body< HTTP/1.0 301 Moved Permanently < Date: Sat, 06 Apr 2013 18:41:26 GMT < Server: Apache/2.2.14 (Ubuntu) < Location: http://www.ubuntu.com/ < Content-Length: 306 < Content-Type: text/html; charset=iso-8859-1 < X-Cache-Lookup: MISS from localhost:3128 < Via: 1.0 localhost (squid/3.1.19) * HTTP/1.0 connection set to keep alive! < Connection: keep-alive ...output here is the rest of the / page... And in the proxy access.log: 1365272030.197 161 10.10.10.20 TCP_MISS/403 3814 GET http://keyserver.ubuntu.com/ - DIRECT/91.189.90.55 text/html Now, on a machine with direct access to the internet: curl -v http://keyserver.ubuntu.com * About to connect() to keyserver.ubuntu.com port 80 (#0) * Trying 91.189.89.49... connected > GET / HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: keyserver.ubuntu.com > Accept: */* > * HTTP 1.0, assume close after body < HTTP/1.0 200 OK < Date: Sat, 06 Apr 2013 18:16:49 GMT < Server: sks_www/1.1.4 < Cache-Control: no-cache < Pragma: no-cache < Expires: 0 < Content-Length: 1417 < Content-Type: text/html; charset=UTF-8 < X-Cache: MISS from localhost < X-Cache-Lookup: MISS from localhost:11371 < Via: 1.0 localhost (squid/3.1.19) * HTTP/1.0 connection set to keep alive! < Connection: keep-alive ...remaining content of the actual / page on the keyserver... Just for fun, if connect through a local TinyProxy instead of Squid, it works: curl -v http://keyserver.ubuntu.com * About to connect() to proxy 10.10.10.10 port 8888 (#0)* Trying 10.10.10.10... connected > GET http://keyserver.ubuntu.com HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: keyserver.ubuntu.com > Accept: */*> Proxy-Connection: Keep-Alive> * HTTP 1.0, assume close after body< HTTP/1.0 200 OK < Via: 1.0 localhost (squid/3.1.19), 1.1 tinyproxy (tinyproxy/1.8.3) < Content-Length: 1417 < Expires: 0 < Server: sks_www/1.1.4 < Date: Sat, 06 Apr 2013 18:33:46 GMT < X-Cache-Lookup: MISS from localhost:11371 < X-Cache: MISS from localhost < Cache-Control: no-cache < Pragma: no-cache < Content-Type: text/html; charset=UTF-8 ...remaining content of the actual / page on the keyserver... I'm not an http/proxy guru. What I think is happening is that the local Squid proxy receives the X-Cache MISS from upstream, and simply halts the response as if it were a peer. Can someone shed some insight on this? I always thought that a reverse proxy should never expose it's X-Cache headers (probably for this very reason). Is this a local Squid configuration issue? Is this a keyserver.ubuntu.com reverse proxy config issue? Is this something else? Thank you so much for the help. -=Chris
