Re: Rewriting HTTP to HTTPS for generic package proxy

[Alex Rousskov <rousskov@xxxxxxxxxxxxxxxxxxxxxxx>]

On 2024-07-09 18:25, Fiehe, Christoph wrote:

> I hope that somebody has an idea, what I am doing wrong. 

AFAICT from the debugging log, it is your parent proxy that returns an 
ERR_SECURE_CONNECT_FAIL error page in response to a seemingly valid 
"HEAD https://..."; request. Can you ask their admin to investigate? You 
may also recommend that they upgrade from Squid v4 that has many known 
security vulnerabiities.

If parent is uncooperative, you can try to reproduce the problem by 
temporary installing your own parent Squid instance and configuring your 
child Squid to use that instead.

HTH,

Alex.
P.S. Unlike Amos, I do not see serious conceptual problems with 
rewriting request target scheme (as a temporary compatibility measure). 
It may not always work, for various reasons, but it does not necessarily 
make things worse (and may make things better).




I try to build a generic package proxy with Squid and need the feature 
to rewrite (not redirect) a HTTP request to a package repository 
transparently to a HTTPS-based package source. I was able to get Jesred 
working and defined the following rewrite rule:
> regex ^http:\/\/download\.docker\.com(.*)$ https://download.docker.com\1
>
> I had to use a parent upstream proxy. In my test case the rule gets applied successfully:
>
> 1720558404.106 10.2.59.102/molecule-ubuntu-jammy.lx.mycompany.de http://download.docker.com/linux/ubuntu/dists/jammy/InRelease[http://download.docker.com/linux/ubuntu/dists/jammy/InRelease] https://download.docker.com/linux/ubuntu/dists/jammy/InRelease 2
>
> I have validated that the returned URL is correct and that the resource is accessible via my upstream proxy.
>
> But at the very end, the client receives a 503 error code. I have set "debug_options ALL,3" and this gives the log:
>
> [...]
> 2024/07/09 23:35:40.115 kid1| 11,2| client_side.cc(1333) parseHttpRequest: HTTP Client REQUEST:
> ---------
> HEAD http://download.docker.com/linux/ubuntu/dists/jammy/InRelease[http://download.docker.com/linux/ubuntu/dists/jammy/InRelease] HTTP/1.1
> Host: download.docker.com
> User-Agent: curl/7.81.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
>
> ----------
> 2024/07/09 23:35:40.115 kid1| 33,3| client_side.cc(1364) parseHttpRequest: complete request received. prefix_sz = 174, request-line-size=77, mime-header-size=97, mime header block:
> Host: download.docker.com
> User-Agent: curl/7.81.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
>
> ----------
> 2024/07/09 23:35:40.115 kid1| 87,3| clientStream.cc(139) clientStreamInsertHead: clientStreamInsertHead: Inserted node 0x5c3ba4154308 with data 0x5c3ba4152950 after head
> 2024/07/09 23:35:40.115 kid1| 5,3| comm.cc(599) commSetConnTimeout: conn9 local=10.2.59.103:8000 remote=10.2.59.102:56466 FD 15 flags=1 timeout 86400
> 2024/07/09 23:35:40.115 kid1| 33,3| client_side.cc(1767) add: 0x5c3ba41518e0*3 to 0/0
> 2024/07/09 23:35:40.115 kid1| 33,3| Pipeline.cc(24) add: Pipeline 0x5c3ba41501f0 add request 1 0x5c3ba41518e0*4
> 2024/07/09 23:35:40.115 kid1| 23,3| Uri.cc(446) parse: Split URL 'http://download.docker.com/linux/ubuntu/dists/jammy/InRelease'[http://download.docker.com/linux/ubuntu/dists/jammy/InRelease'] into proto='http', host='download.docker.com', port='80', path='/linux/ubuntu/dists/jammy/InRelease'
> 2024/07/09 23:35:40.115 kid1| 14,3| Address.cc(389) lookupHostIP: Given Non-IP 'download.docker.com': Name or service not known
> 2024/07/09 23:35:40.115 kid1| 33,3| client_side.cc(702) clientSetKeepaliveFlag: http_ver = HTTP/1.1
> 2024/07/09 23:35:40.115 kid1| 33,3| client_side.cc(703) clientSetKeepaliveFlag: method = HEAD
> 2024/07/09 23:35:40.115 kid1| 85,3| client_side_request.cc(122) ClientRequestContext: ClientRequestContext constructed, this=0x5c3ba4154e78
> 2024/07/09 23:35:40.115 kid1| 83,3| client_side_request.cc(1708) doCallouts: Doing calloutContext->hostHeaderVerify()
> 2024/07/09 23:35:40.115 kid1| 85,3| client_side_request.cc(606) hostHeaderVerify: validate host=download.docker.com, port=0, portStr=NULL
> 2024/07/09 23:35:40.115 kid1| 85,3| client_side_request.cc(620) hostHeaderVerify: validate skipped.
> 2024/07/09 23:35:40.115 kid1| 83,3| client_side_request.cc(1715) doCallouts: Doing calloutContext->clientAccessCheck()
> 2024/07/09 23:35:40.115 kid1| 28,3| Checklist.cc(69) preCheck: 0x5c3ba41552d8 checking slow rules
> 2024/07/09 23:35:40.115 kid1| 28,3| Ip.cc(538) match: aclIpMatchIp: '10.2.59.102:56466' found
> 2024/07/09 23:35:40.115 kid1| 28,3| Acl.cc(175) matches: checked: all = 1
> 2024/07/09 23:35:40.115 kid1| 28,3| Acl.cc(175) matches: checked: http_access#1 = 1
> 2024/07/09 23:35:40.115 kid1| 28,3| Acl.cc(175) matches: checked: http_access = 1
> 2024/07/09 23:35:40.115 kid1| 28,3| Checklist.cc(62) markFinished: 0x5c3ba41552d8 answer ALLOWED for match
> 2024/07/09 23:35:40.115 kid1| 28,3| Checklist.cc(162) checkCallback: ACLChecklist::checkCallback: 0x5c3ba41552d8 answer=ALLOWED
> 2024/07/09 23:35:40.115 kid1| 85,2| client_side_request.cc(714) clientAccessCheckDone: The request HEAD http://download.docker.com/linux/ubuntu/dists/jammy/InRelease[http://download.docker.com/linux/ubuntu/dists/jammy/InRelease] is ALLOWED; last ACL checked: all
> 2024/07/09 23:35:40.115 kid1| 83,3| AccessCheck.cc(42) Start: adaptation off, skipping
> 2024/07/09 23:35:40.115 kid1| 83,3| client_side_request.cc(1735) doCallouts: Doing calloutContext->clientRedirectStart()
> 2024/07/09 23:35:40.115 kid1| 78,3| dns_internal.cc(1836) idnsPTRLookup: idnsPTRLookup: buf is 42 bytes for 10.2.59.102, id = 0x8d95
> 2024/07/09 23:35:40.115 kid1| 50,3| comm.cc(927) comm_udp_sendto: comm_udp_sendto: Attempt to send UDP packet to 127.0.0.53:53 using FD 11 using Port 54280
> 2024/07/09 23:35:40 kid1| Starting new redirector helpers...
> current master transaction: master54
> 2024/07/09 23:35:40 kid1| helperOpenServers: Starting 1/3 'jesred' processes
> current master transaction: master54
> 2024/07/09 23:35:40.115 kid1| 51,3| fd.cc(168) fd_open: fd_open() FD 17 IPC UNIX STREAM Parent
> 2024/07/09 23:35:40.115 kid1| 51,3| fd.cc(168) fd_open: fd_open() FD 19 IPC UNIX STREAM Parent
> 2024/07/09 23:35:40.115 kid1| 54,3| ipc.cc(212) ipcCreate: ipcCreate: prfd FD 17
> 2024/07/09 23:35:40.115 kid1| 54,3| ipc.cc(213) ipcCreate: ipcCreate: pwfd FD 17
> 2024/07/09 23:35:40.115 kid1| 54,3| ipc.cc(214) ipcCreate: ipcCreate: crfd FD 19
> 2024/07/09 23:35:40.115 kid1| 54,3| ipc.cc(215) ipcCreate: ipcCreate: cwfd FD 19
> 2024/07/09 23:35:40.116 kid1| 5,3| comm.cc(850) _comm_close: start closing FD 19 by ipc.cc:271
> 2024/07/09 23:35:40.116 kid1| 5,3| comm.cc(586) commUnsetFdTimeout: Remove timeout for FD 19
> 2024/07/09 23:35:40.116 kid1| 21,3| tools.cc(561) leave_suid: leave_suid: PID 503746 called
> 2024/07/09 23:35:40.116 kid1| 21,3| tools.cc(651) no_suid: no_suid: PID 503746 giving up root privileges forever
> 2024/07/09 23:35:40.116 kid1| 5,3| comm.cc(586) commUnsetFdTimeout: Remove timeout for FD 17
> 2024/07/09 23:35:40.117 kid1| 84,3| helper.cc(1310) GetFirstAvailable: GetFirstAvailable: Least-loaded helper is fully loaded!
> 2024/07/09 23:35:40.117 kid1| 51,3| fd.cc(93) fd_close: fd_close FD 19 IPC UNIX STREAM Parent
> 2024/07/09 23:35:40.117 kid1| 78,3| dns_internal.cc(1319) idnsRead: idnsRead: starting with FD 11
> 2024/07/09 23:35:40.117 kid1| 78,3| dns_internal.cc(1365) idnsRead: idnsRead: FD 11: received 92 bytes from 127.0.0.53:53
> 2024/07/09 23:35:40.117 kid1| 78,3| dns_internal.cc(1172) idnsGrokReply: idnsGrokReply: QID 0x8d95, 1 answers
> 2024/07/09 23:35:40.117 kid1| 35,3| fqdncache.cc(336) fqdncacheParse: fqdncacheParse: 1 answers for '10.2.59.102'
> 2024/07/09 23:35:40.117 kid1| 5,3| IoCallback.cc(112) finish: called for conn11 local=[::] remote=[::] FD 17 flags=1 (0, 0)
> 2024/07/09 23:35:40.125 kid1| 5,3| Read.cc(148) HandleRead: FD 17, size 32767, retval 80, errno 0
> 2024/07/09 23:35:40.125 kid1| 5,3| IoCallback.cc(112) finish: called for conn10 local=[::] remote=[::] FD 17 flags=1 (0, 0)
> 2024/07/09 23:35:40.125 kid1| 84,3| helper.cc(1022) helperHandleRead: helperHandleRead: end of reply found
> 2024/07/09 23:35:40.125 kid1| 84,3| Reply.cc(41) finalize: Parsing helper buffer
> 2024/07/09 23:35:40.125 kid1| 84,3| Reply.cc(59) finalize: Buff length is larger than 2
> 2024/07/09 23:35:40.125 kid1| 84,3| Reply.cc(63) finalize: helper Result = OK
> 2024/07/09 23:35:40.125 kid1| 23,3| Uri.cc(446) parse: Split URL 'https://download.docker.com/linux/ubuntu/dists/jammy/InRelease'[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease'] into proto='https', host='download.docker.com', port='443', path='/linux/ubuntu/dists/jammy/InRelease'
> 2024/07/09 23:35:40.125 kid1| 14,3| Address.cc(389) lookupHostIP: Given Non-IP 'download.docker.com': Name or service not known
> 2024/07/09 23:35:40.125 kid1| 61,2| client_side_request.cc(1235) clientRedirectDone: URL-rewriter diverts URL from http://download.docker.com/linux/ubuntu/dists/jammy/InRelease[http://download.docker.com/linux/ubuntu/dists/jammy/InRelease] to https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.125 kid1| 83,3| client_side_request.cc(1743) doCallouts: Doing calloutContext->clientAccessCheck2()
> 2024/07/09 23:35:40.125 kid1| 85,2| client_side_request.cc(692) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
> 2024/07/09 23:35:40.125 kid1| 85,2| client_side_request.cc(714) clientAccessCheckDone: The request HEAD https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease] is ALLOWED; last ACL checked: all
> 2024/07/09 23:35:40.126 kid1| 83,3| client_side_request.cc(1761) doCallouts: Doing clientInterpretRequestHeaders()
> 2024/07/09 23:35:40.126 kid1| 83,3| client_side_request.cc(1770) doCallouts: Doing calloutContext->checkNoCache()
> 2024/07/09 23:35:40.126 kid1| 28,3| Checklist.cc(69) preCheck: 0x5c3ba41552d8 checking slow rules
> 2024/07/09 23:35:40.126 kid1| 28,3| RegexData.cc(50) match: checking 'https://download.docker.com/linux/ubuntu/dists/jammy/InRelease'[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease']
> 2024/07/09 23:35:40.126 kid1| 28,3| Acl.cc(175) matches: checked: no_cache = 0
> 2024/07/09 23:35:40.126 kid1| 28,3| Acl.cc(175) matches: checked: cache#1 = 0
> 2024/07/09 23:35:40.126 kid1| 28,3| Ip.cc(538) match: aclIpMatchIp: '10.2.59.102:56466' found
> 2024/07/09 23:35:40.126 kid1| 28,3| Acl.cc(175) matches: checked: all = 1
> 2024/07/09 23:35:40.126 kid1| 28,3| Acl.cc(175) matches: checked: cache#2 = 1
> 2024/07/09 23:35:40.126 kid1| 28,3| Acl.cc(175) matches: checked: cache = 1
> 2024/07/09 23:35:40.126 kid1| 28,3| Checklist.cc(62) markFinished: 0x5c3ba41552d8 answer ALLOWED for match
> 2024/07/09 23:35:40.126 kid1| 28,3| Checklist.cc(162) checkCallback: ACLChecklist::checkCallback: 0x5c3ba41552d8 answer=ALLOWED
> 2024/07/09 23:35:40.126 kid1| 85,3| client_side_request.cc(116) ~ClientRequestContext: ClientRequestContext destructed, this=0x5c3ba4154e78
> 2024/07/09 23:35:40.126 kid1| 83,3| client_side_request.cc(1855) doCallouts: calling processRequest()
> 2024/07/09 23:35:40.126 kid1| 87,3| clientStream.cc(178) clientStreamRead: clientStreamRead: Calling 1 with cbdata 0x5c3ba4153e70 from node 0x5c3ba4154308
> 2024/07/09 23:35:40.126 kid1| 73,3| HttpRequest.cc(742) storeId: sent back effectiveRequestUrl: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 20,3| Controller.cc(429) peek: DE850794EBC405A27A7718F51795E32A
> 2024/07/09 23:35:40.126 kid1| 73,3| HttpRequest.cc(742) storeId: sent back effectiveRequestUrl: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 20,3| Controller.cc(429) peek: D3522EE27FB0ED7004DD594AF7674667
> 2024/07/09 23:35:40.126 kid1| 85,3| client_side_reply.cc(1523) identifyFoundObject: StoreEntry is NULL - MISS
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(730) storeCreatePureEntry: storeCreateEntry: 'https://download.docker.com/linux/ubuntu/dists/jammy/InRelease'[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease']
> 2024/07/09 23:35:40.126 kid1| 20,3| MemObject.cc(99) MemObject: MemObject constructed, this=0x5c3ba416ef10
> 2024/07/09 23:35:40.126 kid1| 88,3| MemObject.cc(82) setUris: 0x5c3ba416ef10 storeId: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(434) lock: storeCreateEntry locked key [null_store_key] e:=V/0x5c3ba416ee90*1
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(536) setPrivateKey: 00 e:=V/0x5c3ba416ee90*1
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(412) hashInsert: StoreEntry::hashInsert: Inserting Entry e:=IV/0x5c3ba416ee90*1 key '020000000000000061AF070001000000'
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(434) lock: store_client locked key 020000000000000061AF070001000000 e:=IV/0x5c3ba416ee90*2
> 2024/07/09 23:35:40.126 kid1| 90,3| store_client.cc(243) copy: store_client::copy: 020000000000000061AF070001000000, from 0, for length 4096, cb 1, cbdata 0x5c3ba4152dd8
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(434) lock: store_client::copy locked key 020000000000000061AF070001000000 e:=IV/0x5c3ba416ee90*3
> 2024/07/09 23:35:40.126 kid1| 90,3| store_client.cc(343) storeClientCopy2: storeClientCopy2: 020000000000000061AF070001000000
> 2024/07/09 23:35:40.126 kid1| 90,3| store_client.cc(390) doCopy: store_client::doCopy: Waiting for more
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(457) unlock: store_client::copy unlocking key 020000000000000061AF070001000000 e:=IV/0x5c3ba416ee90*3
> 2024/07/09 23:35:40.126 kid1| 17,3| FwdState.cc(373) Start: 'https://download.docker.com/linux/ubuntu/dists/jammy/InRelease'[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease']
> 2024/07/09 23:35:40.126 kid1| 17,2| FwdState.cc(133) FwdState: Forwarding client request conn9 local=10.2.59.103:8000 remote=10.2.59.102:56466 FD 15 flags=1, url=https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(434) lock: FwdState locked key 020000000000000061AF070001000000 e:=IV/0x5c3ba416ee90*3
> 2024/07/09 23:35:40.126 kid1| 17,3| FwdState.cc(140) FwdState: FwdState constructed, this=0x5c3ba416fa18
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(309) peerSelect: e:=IV/0x5c3ba416ee90*3 https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(434) lock: peerSelect locked key 020000000000000061AF070001000000 e:=IV/0x5c3ba416ee90*4
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(612) selectMore: HEAD download.docker.com
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(626) selectMore: direct = DIRECT_UNKNOWN (never_direct to be checked)
> 2024/07/09 23:35:40.126 kid1| 28,3| Checklist.cc(69) preCheck: 0x5c3ba4170638 checking slow rules
> 2024/07/09 23:35:40.126 kid1| 28,3| Ip.cc(538) match: aclIpMatchIp: '10.2.59.102:56466' found
> 2024/07/09 23:35:40.126 kid1| 28,3| Acl.cc(175) matches: checked: all = 1
> 2024/07/09 23:35:40.126 kid1| 28,3| Acl.cc(175) matches: checked: never_direct#1 = 1
> 2024/07/09 23:35:40.126 kid1| 28,3| Acl.cc(175) matches: checked: never_direct = 1
> 2024/07/09 23:35:40.126 kid1| 28,3| Checklist.cc(62) markFinished: 0x5c3ba4170638 answer ALLOWED for match
> 2024/07/09 23:35:40.126 kid1| 28,3| Checklist.cc(162) checkCallback: ACLChecklist::checkCallback: 0x5c3ba4170638 answer=ALLOWED
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(345) checkNeverDirectDone: ALLOWED
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(351) checkNeverDirectDone: direct = DIRECT_NO (never_direct allow)
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(612) selectMore: HEAD download.docker.com
> 2024/07/09 23:35:40.126 kid1| 14,3| ipcache.cc(732) ipcache_gethostbyname: ipcache_gethostbyname: 'download.docker.com', flags=0
> 2024/07/09 23:35:40.126 kid1| 14,3| Address.cc(389) lookupHostIP: Given Non-IP 'download.docker.com': Name or service not known
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(286) peerSelectIcpPing: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 15,3| neighbors.cc(283) neighborsCount: neighborsCount: 0
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(297) peerSelectIcpPing: counted 0 neighbors
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(833) selectSomeParent: HEAD download.docker.com
> 2024/07/09 23:35:40.126 kid1| 15,3| neighbors.cc(350) getRoundRobinParent: returning [nil]
> 2024/07/09 23:35:40.126 kid1| 15,3| neighbors.cc(403) getWeightedRoundRobinParent: returning [nil]
> 2024/07/09 23:35:40.126 kid1| 15,3| neighbors.cc(309) getFirstUpParent: returning 212.89.128.96
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(1102) addSelection: adding FIRSTUP_PARENT/212.89.128.96
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(1095) addSelection: skipping ANY_OLD_PARENT/212.89.128.96; have FIRSTUP_PARENT/212.89.128.96
> 2024/07/09 23:35:40.126 kid1| 15,3| neighbors.cc(493) getDefaultParent: returning 212.89.128.96
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(1095) addSelection: skipping DEFAULT_PARENT/212.89.128.96; have FIRSTUP_PARENT/212.89.128.96
> 2024/07/09 23:35:40.126 kid1| 44,2| peer_select.cc(460) resolveSelected: Find IP destination for: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease'[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease'] via 212.89.128.96
> 2024/07/09 23:35:40.126 kid1| 44,2| peer_select.cc(1174) handlePath: PeerSelector1 found conn12 local=0.0.0.0 remote=212.89.128.96:3128 FIRSTUP_PARENT flags=1, destination #1 for https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 44,2| peer_select.cc(1180) handlePath: always_direct = DENIED
> 2024/07/09 23:35:40.126 kid1| 44,2| peer_select.cc(1181) handlePath: never_direct = ALLOWED
> 2024/07/09 23:35:40.126 kid1| 44,2| peer_select.cc(1182) handlePath: timedout = 0
> 2024/07/09 23:35:40.126 kid1| 17,3| FwdState.cc(610) noteDestination: conn12 local=0.0.0.0 remote=212.89.128.96:3128 FIRSTUP_PARENT flags=1
> 2024/07/09 23:35:40.126 kid1| 17,3| FwdState.cc(1124) connectStart: 1+ paths to https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 44,2| peer_select.cc(479) resolveSelected: PeerSelector1 found all 1 destinations for https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 44,2| peer_select.cc(480) resolveSelected: always_direct = DENIED
> 2024/07/09 23:35:40.126 kid1| 44,2| peer_select.cc(481) resolveSelected: never_direct = ALLOWED
> 2024/07/09 23:35:40.126 kid1| 44,2| peer_select.cc(482) resolveSelected: timedout = 0
> 2024/07/09 23:35:40.126 kid1| 44,3| peer_select.cc(241) ~PeerSelector: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.126 kid1| 20,3| store.cc(457) unlock: peerSelect unlocking key 020000000000000061AF070001000000 e:=p2IV/0x5c3ba416ee90*4
> 2024/07/09 23:35:40.126 kid1| 48,3| pconn.cc(474) popStored: lookup for key {212.89.128.96:3128} failed.
> 2024/07/09 23:35:40.126 kid1| 17,3| FwdState.cc(1568) GetMarkingsToServer: from 0.0.0.0 tos 0 netfilter mark 0
> 2024/07/09 23:35:40.126 kid1| 5,3| ConnOpener.cc(42) ConnOpener: will connect to conn14 local=0.0.0.0 remote=212.89.128.96:3128 FIRSTUP_PARENT flags=1 with 30 timeout
> 2024/07/09 23:35:40.126 kid1| 50,3| comm.cc(378) comm_openex: comm_openex: Attempt open socket for: 0.0.0.0
> 2024/07/09 23:35:40.126 kid1| 50,3| comm.cc(420) comm_openex: comm_openex: Opened socket conn15 local=0.0.0.0 remote=[::] FD 19 flags=1 : family=2, type=1, protocol=6
> 2024/07/09 23:35:40.126 kid1| 51,3| fd.cc(168) fd_open: fd_open() FD 19
> 2024/07/09 23:35:40.126 kid1| 5,3| ConnOpener.cc(312) createFd: conn14 local=0.0.0.0 remote=212.89.128.96:3128 FIRSTUP_PARENT flags=1 will timeout in 30
> 2024/07/09 23:35:40.127 kid1| 17,3| FwdState.cc(1197) dispatch: conn9 local=10.2.59.103:8000 remote=10.2.59.102:56466 FD 15 flags=1: Fetching HEAD https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.127 kid1| 14,3| Address.cc(389) lookupHostIP: Given Non-IP 'download.docker.com': Name or service not known
> 2024/07/09 23:35:40.127 kid1| 78,3| dns_internal.cc(1793) idnsALookup: idnsALookup: buf is 37 bytes for download.docker.com, id = 0xe779
> 2024/07/09 23:35:40.127 kid1| 50,3| comm.cc(927) comm_udp_sendto: comm_udp_sendto: Attempt to send UDP packet to 127.0.0.53:53 using FD 11 using Port 54280
> 2024/07/09 23:35:40.127 kid1| 78,3| dns_internal.cc(1729) idnsSendSlaveAAAAQuery: buf is 37 bytes for download.docker.com, id = 0x8aee
> 2024/07/09 23:35:40.127 kid1| 50,3| comm.cc(927) comm_udp_sendto: comm_udp_sendto: Attempt to send UDP packet to 127.0.0.53:53 using FD 11 using Port 54280
> 2024/07/09 23:35:40.127 kid1| 11,3| http.cc(2516) httpStart: HEAD https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.127 kid1| 20,3| store.cc(434) lock: Client locked key 020000000000000061AF070001000000 e:=p2IV/0x5c3ba416ee90*4
> 2024/07/09 23:35:40.127 kid1| 5,3| comm.cc(599) commSetConnTimeout: conn14 local=10.2.59.103:39370 remote=212.89.128.96:3128 FIRSTUP_PARENT FD 19 flags=1 timeout 86400
> 2024/07/09 23:35:40.127 kid1| 22,3| refresh.cc(636) getMaxAge: getMaxAge: 'https://download.docker.com/linux/ubuntu/dists/jammy/InRelease'[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease']
> 2024/07/09 23:35:40.127 kid1| 11,2| http.cc(2472) sendRequest: HTTP Server conn14 local=10.2.59.103:39370 remote=212.89.128.96:3128 FIRSTUP_PARENT FD 19 flags=1
> 2024/07/09 23:35:40.127 kid1| 11,2| http.cc(2473) sendRequest: HTTP Server REQUEST:
> ---------
> HEAD https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease] HTTP/1.1
> Host: download.docker.com
> User-Agent: curl/7.81.0
> Accept: */*
> Via: 1.1 pkg-proxy (squid/6.6)
> X-Forwarded-For: 10.2.59.102
> Cache-Control: max-age=0
> Connection: keep-alive
>
>
> ----------
> 2024/07/09 23:35:40.127 kid1| 5,3| IoCallback.cc(112) finish: called for conn14 local=10.2.59.103:39370 remote=212.89.128.96:3128 FIRSTUP_PARENT FD 19 flags=1 (0, 0)
> 2024/07/09 23:35:40.127 kid1| 5,3| comm.cc(599) commSetConnTimeout: conn14 local=10.2.59.103:39370 remote=212.89.128.96:3128 FIRSTUP_PARENT FD 19 flags=1 timeout 900
> 2024/07/09 23:35:40.137 kid1| 78,3| dns_internal.cc(1319) idnsRead: idnsRead: starting with FD 11
> 2024/07/09 23:35:40.137 kid1| 78,3| dns_internal.cc(1365) idnsRead: idnsRead: FD 11: received 304 bytes from 127.0.0.53:53
> 2024/07/09 23:35:40.137 kid1| 78,3| dns_internal.cc(1172) idnsGrokReply: idnsGrokReply: QID 0x8aee, 9 answers
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(480) ipcacheParse: 9 answers for download.docker.com
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #1 [2600:9000:2490:6c00:3:db06:4200:93a1]
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #2 [2600:9000:2490:a600:3:db06:4200:93a1]
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #3 [2600:9000:2490:9c00:3:db06:4200:93a1]
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #4 [2600:9000:2490:6000:3:db06:4200:93a1]
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #5 [2600:9000:2490:c00:3:db06:4200:93a1]
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #6 [2600:9000:2490:5200:3:db06:4200:93a1]
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #7 [2600:9000:2490:9a00:3:db06:4200:93a1]
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #8 [2600:9000:2490:2c00:3:db06:4200:93a1]
> 2024/07/09 23:35:40.137 kid1| 78,3| dns_internal.cc(1319) idnsRead: idnsRead: starting with FD 11
> 2024/07/09 23:35:40.137 kid1| 78,3| dns_internal.cc(1365) idnsRead: idnsRead: FD 11: received 144 bytes from 127.0.0.53:53
> 2024/07/09 23:35:40.137 kid1| 78,3| dns_internal.cc(1172) idnsGrokReply: idnsGrokReply: QID 0xe779, 5 answers
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(480) ipcacheParse: 5 answers for download.docker.com
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #9 108.138.7.33
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #10 108.138.7.18
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #11 108.138.7.88
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(535) addGood: download.docker.com #12 108.138.7.48
> 2024/07/09 23:35:40.137 kid1| 14,3| ipcache.cc(586) ipcacheHandleReply: done with download.docker.com: [2600:9000:2490:6c00:3:db06:4200:93a1] #1/12-0
> 2024/07/09 23:35:40.137 kid1| 38,3| net_db.cc(337) netdbSendPing: netdbSendPing: pinging download.docker.com
> 2024/07/09 23:35:40.137 kid1| 37,2| IcmpSquid.cc(88) SendEcho: to [2600:9000:2490:6c00:3:db06:4200:93a1], opcode 3, len 19
> 2024/07/09 23:35:40.137 pinger| 42,2| IcmpPinger.cc(198) Recv: Pass [2600:9000:2490:6c00:3:db06:4200:93a1] off to ICMPv6 module.
> 2024/07/09 23:35:40 pinger| SendEcho ERROR: sending to ICMPv6 packet to [2600:9000:2490:6c00:3:db06:4200:93a1]: (101) Network is unreachable
> 2024/07/09 23:35:40.138 pinger| 42,2| Icmp.cc(90) Log: pingerLog: 1720560940.138021 [2600:9000:2490:6c00:3:db06:4200:93a1] 0
> 2024/07/09 23:35:40.323 kid1| 5,3| IoCallback.cc(112) finish: called for conn14 local=10.2.59.103:39370 remote=212.89.128.96:3128 FIRSTUP_PARENT FD 19 flags=1 (0, 0)
> 2024/07/09 23:35:40.324 kid1| 5,3| Read.cc(93) ReadNow: conn14 local=10.2.59.103:39370 remote=212.89.128.96:3128 FIRSTUP_PARENT FD 19 flags=1, size 65536, retval 348, errno 0
> 2024/07/09 23:35:40.324 kid1| 11,3| http.cc(649) processReplyHeader: processReplyHeader: key '020000000000000061AF070001000000'
> 2024/07/09 23:35:40.324 kid1| 11,2| http.cc(696) processReplyHeader: HTTP Server conn14 local=10.2.59.103:39370 remote=212.89.128.96:3128 FIRSTUP_PARENT FD 19 flags=1
> 2024/07/09 23:35:40.324 kid1| 11,2| http.cc(697) processReplyHeader: HTTP Server RESPONSE:
> ---------
> HTTP/1.1 503 Service Unavailable
> Server: squid/4.10
> Mime-Version: 1.0
> Date: Tue, 09 Jul 2024 21:35:40 GMT
> Content-Type: text/html;charset=utf-8
> Content-Length: 3879
> X-Squid-Error: ERR_SECURE_CONNECT_FAIL 71
> X-Cache: MISS from proxy-srv2
> X-Cache-Lookup: MISS from proxy-srv2:3128
> Via: 1.1 proxy-srv2 (squid/4.10)
> Connection: keep-alive
>
> ----------
> 2024/07/09 23:35:40.324 kid1| 83,3| AccessCheck.cc(42) Start: adaptation off, skipping
> 2024/07/09 23:35:40.324 kid1| 20,3| store.cc(1693) replaceHttpReply: StoreEntry::replaceHttpReply: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease[https://download.docker.com/linux/ubuntu/dists/jammy/InRelease]
> 2024/07/09 23:35:40.324 kid1| 11,3| http.cc(949) haveParsedReplyHeaders: HTTP CODE: 503
>
> Has anybody an idea what I can do to solve the issue?
>
> This is my configuration borrowed from squid-deb-proxy:
>
> # this file contains private networks (10.0.0.0/8, 172.16.0.0/12,
> # 192.168.0.0/16) by default, you can add/remove additional allowed
> # source networks in it to customize it for your setup
> acl src_networks src "/etc/squid/acl/src-networks.acl"
>
> # this file contains the archive mirrors by default,
> # if you use a different mirror, add it there
> acl to_archive_mirrors dstdomain "/etc/squid/acl/archive-mirrors.acl"
>
> # Disable Cache for defined domains
> acl no_cache url_regex "/etc/squid/acl/no-cache.acl"
>
> # this contains the package blacklist
> acl blockedpkgs urlpath_regex "/etc/squid/pkg-blacklist-regexp.acl"
>
> # default to a different port than stock squid
> http_port 8000
>
> # -------------------------------------------------
> # settings below probably do not need customization
>
> # user visible name
> visible_hostname pkg-proxy
>
> # we need a big cache, some debs are huge
> maximum_object_size 512 MB
>
> # use a different dir than stock squid and default to 40G
> cache_dir aufs /var/cache/squid 40000 16 256
>
> cache_peer 212.89.128.96 parent 3128 0 no-query default
> never_direct allow all
>
> # use different logs
> cache_access_log /var/log/squid/access.log
> cache_log /var/log/squid/cache.log
> cache_store_log /var/log/squid/store.log
>
> # tweaks to speed things up
> cache_mem 200 MB
> maximum_object_size_in_memory 10240 KB
>
> # pid
> pid_filename /var/run/squid.pid
>
> # refresh pattern for debs and udebs
> refresh_pattern deb$ 129600 100% 129600
> refresh_pattern udeb$ 129600 100% 129600
> refresh_pattern tar.gz$ 129600 100% 129600
> refresh_pattern tar.xz$ 129600 100% 129600
> refresh_pattern tar.bz2$ 129600 100% 129600
>
> # always refresh Packages and Release files
> refresh_pattern \/(Packages|Sources)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims
> refresh_pattern \/Release(|\.gpg)$ 0 0% 0 refresh-ims
> refresh_pattern \/InRelease$ 0 0% 0 refresh-ims
> refresh_pattern \/(Translation-.*)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims
>
> # handle meta-release and changelogs.ubuntu.com special
> # (fine to have this on debian too)
> refresh_pattern changelogs.ubuntu.com\/.* 0 1% 1
>
> # only allow connects to ports for http, https
> acl SSL_ports port 443 563
> acl Safe_ports port 80
> acl Safe_ports port 443 563
>
> # only allow ports we trust
> http_access deny !Safe_ports
>
> # do not allow to download from the pkg blacklist
> http_access deny blockedpkgs
>
> # allow access only to official archive mirrors
> # uncomment the third and fouth line to permit any unlisted domain
> http_access deny !to_archive_mirrors
>
> # allow access from our network and localhost
> http_access allow src_networks
>
> # And finally deny all other access to this proxy
> http_access deny all
>
> # don't cache domains not listed in the mirrors file
> # uncomment the third and fourth line to cache any unlisted domains
> cache deny no_cache
>
> # And finally cache everything else
> cache allow all
>
> url_rewrite_children 3 startup=0 idle=1 concurrency=1
> url_rewrite_program /usr/lib/squid/jesred
>
> debug_options ALL,3
>
> Thanks a lot.
>
> Regards,
> Christoph
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> https://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
https://lists.squid-cache.org/listinfo/squid-users