Hi all,
We run a farm of about 20 squid proxy servers (3.2.0.8) behind an SLB
which do authentication and access control. We're having a problem with
iTunes updates of IOS devices since the release of iTunes 11.
When a user attempts to update an IOS device (iPhone / iPad, etc) with
iTunes 11, the IOS is successfully downloaded, and extracted, but when
iTunes goes to verify it with Apple, the process fails, with an error
that the software update server could not be contacted.
After using a single proxy server (bypassing the SLB) to allow packet
capturing, we found out that iTunes was POSTing to gs-nwk.apple.com
(via Squid) and Squid was sending back a 502 Bad Gateway to iTunes, with
an ERR_READ_ERROR 104.
Further analysis determined that iTunes was POSTing approx 5000 bytes
of config to Apple, Squid was successfully receiving and acknowledging
the packets containing the POST, but when it sent the first packet of
the POST to apple, it got nothing back, and after sending the second
packet, it received a RESET from Apple.
This is what we're seeing received by the proxy from iTunes on the
client side (after the handshake, receiving the POST):
16:52:05.714245 IP (tos 0x5c, ttl 51, id 34532, offset 0, flags [DF],
proto TCP (6), length 282) itunes-client.37910 > proxy-20.webcache: P,
cksum 0x814a (correct), 1:231(230) ack 1 win 8208 <nop,nop,timestamp
1079299838 813961821>
16:52:05.714265 IP (tos 0x0, ttl 64, id 12905, offset 0, flags [DF],
proto TCP (6), length 52) proxy-20.webcache > itunes-client.37910: .,
cksum 0xff79 (correct), 1:1(0) ack 231 win 54 <nop,nop,timestamp
813961823 1079299838>
16:52:05.714712 IP (tos 0x5c, ttl 51, id 47080, offset 0, flags [DF],
proto TCP (6), length 1420) itunes-client.37910 > proxy-20.webcache: .,
cksum 0xa622 (correct), 231:1599(1368) ack 1 win 8208 <nop,nop,timestamp
1079299838 813961821>
16:52:05.714730 IP (tos 0x0, ttl 64, id 12906, offset 0, flags [DF],
proto TCP (6), length 52) proxy-20.webcache > itunes-client.37910: .,
cksum 0xfa0c (correct), 1:1(0) ack 1599 win 75 <nop,nop,timestamp
813961823 1079299838>
16:52:05.714736 IP (tos 0x5c, ttl 51, id 41438, offset 0, flags [DF],
proto TCP (6), length 144) itunes-client.37910 > proxy-20.webcache: P,
cksum 0xaf3b (correct), 1599:1691(92) ack 1 win 8208 <nop,nop,timestamp
1079299838 813961821>
16:52:05.714741 IP (tos 0x0, ttl 64, id 12907, offset 0, flags [DF],
proto TCP (6), length 52) proxy-20.webcache > itunes-client.37910: .,
cksum 0xf9b0 (correct), 1:1(0) ack 1691 win 75 <nop,nop,timestamp
813961823 1079299838>
16:52:05.714774 IP (tos 0x5c, ttl 51, id 64957, offset 0, flags [DF],
proto TCP (6), length 1420) itunes-client.37910 > proxy-20.webcache: .,
cksum 0xb67a (correct), 1691:3059(1368) ack 1 win 8208
<nop,nop,timestamp 1079299838 813961821>
16:52:05.714786 IP (tos 0x0, ttl 64, id 12908, offset 0, flags [DF],
proto TCP (6), length 52) proxy-20.webcache > itunes-client.37910: .,
cksum 0xf442 (correct), 1:1(0) ack 3059 win 97 <nop,nop,timestamp
813961823 1079299838>
16:52:05.714793 IP (tos 0x5c, ttl 51, id 29642, offset 0, flags [DF],
proto TCP (6), length 144) itunes-client.37910 > proxy-20.webcache: P,
cksum 0x6d18 (correct), 3059:3151(92) ack 1 win 8208 <nop,nop,timestamp
1079299838 813961821>
16:52:05.714799 IP (tos 0x0, ttl 64, id 12909, offset 0, flags [DF],
proto TCP (6), length 52) proxy-20.webcache > itunes-client.37910: .,
cksum 0xf3e6 (correct), 1:1(0) ack 3151 win 97 <nop,nop,timestamp
813961823 1079299838>
<snip>
This is what we're seeing sent by the proxy to Apple (after the
handshake, sending the POST):
16:52:05.879081 IP (tos 0x0, ttl 64, id 29262, offset 0, flags [DF],
proto TCP (6), length 339) proxy-20.41812 > gs-nwk.apple.com.www: P,
cksum 0x7859 (incorrect (-> 0xda55), 1:300(299) ack 1 win 5840
16:52:05.879102 IP (tos 0x0, ttl 64, id 29263, offset 0, flags [DF],
proto TCP (6), length 1400) proxy-20.41812 > gs-nwk.apple.com.www: .,
cksum 0x7c7e (incorrect (-> 0x671a), 300:1660(1360) ack 1 win 5840
16:52:06.042751 IP (tos 0x28, ttl 237, id 36800, offset 0, flags [DF],
proto TCP (6), length 40) gs-nwk.apple.com.www > proxy-20.41812: R,
cksum 0x660b (correct), 4169648848:4169648848(0) win 8201
Should we be concerned about the incorrect checksums on the outbound
connection from Squid?
Any idea why the second packet squid receives is 1420 bytes, yet it
only sends 1400 bytes to Apple?
Any thoughts on why we're getting a RESET back from Apple instead of an
ack on the first packet containing POST data?
Any ideas on what we can do to fix this, and get iTunes IOS device
updates working through squid?
I can provide the full packet captures if required.
Thanks,
Prk.
