On 24/10/2016 6:28 a.m., garryd@xxxxxxxxx wrote: > On 2016-10-23 18:31, Amos Jeffries wrote: >> On 23/10/2016 2:32 a.m., garryd wrote: >>> Since I started use Squid, it's configuration always RFC compliant by >>> default, _but_ there were always knobs for users to make it HTTP >>> violent. It was in hands of users to decide how to handle a web >>> resource. Now it is not always possible, and the topic is an evidence. >>> For example, in terms of this topic, users can't violate this RFC >>> statement [1]: >>> >>> A Vary field value of "*" signals that anything about the request >>> might play a role in selecting the response representation, possibly >>> including elements outside the message syntax (e.g., the client's >>> network address). A recipient will not be able to determine whether >>> this response is appropriate for a later request without forwarding >>> the request to the origin server. A proxy MUST NOT generate a Vary >>> field with a "*" value. >>> >>> [1] https://tools.ietf.org/html/rfc7231#section-7.1.4 >> >> >> Please name the option in any version of Squid which allowed Squid to >> cache those "Vary: *" responses. >> >> No such option ever existed. For the 20+ years Vary has existed Squid >> has behaved in the same way it does today. For all that time you did not >> notice these responses. > > You are absolutely right, but there were not such abuse vector in the > past (at least in my practice). There were tools provided by devs to > admins to protect against trending abuse cases. What trend? There is exactly one mentioned URL that I'm aware of, the Chrome browser download URL. I've posted two reasons why Chrome uses the Vary:* header. Just opinions of mine, but formed after actual discussions with the Chrome developers some years back. [I very much dislike writing this. But you seem to have been sucked in and deserve to know the history.] All the fuss that is going on AFAICS was started by Yuri. His comment history here and in bugzilla, and in private responses range from non-compromising "cache everything no matter what - do what I say, now!" (repeatedy in unrelated bugzilla reports), "f*ck the RFCs and anyone following them, just store everything I dont care about what happens" (this mornings post), to personal attacks against anyone who mentions the previous stance might have problems (all the "Squid developers believe/say/..." comments - none of which match what the team we have actually said to him or believe). There is one other email address which changes its name occasionally and posts almost exactly the same words as Yuri's. So it looks to me as Yuri and some sock puppets performing a campaign to spread lies and FUD about Squid and hurt the people doing work on it. Not exactly a good way to get people to do things for free. But it seems to have worked on getting you and a few others now doing the coding part for him at no cost, and I have now wasted time responding to you and thinking of a solution for it that might get accepted for merge. This particular topic is not the first to have such behaviour by Yuri. There have been other things where someone made a mistake (overlooked something) and all hell full of insults broke loose at them. And several other cases where missing features in Squid did not get instant obedience to quite blunt and insulting demands. Followed by weeks of insults until the bug was fixed by other people - then suddenly polite Yuri comes back overnight. As a developer, I personally decided not to write the requested code. Not in the way demanded. This seems to have upset Yuri who has taken to insulting me and the rest of the dev team as a whole. I'm not sure if he is trolling to intentionally cause the above mentioned effects, or really in need of medical assistance to deal with work related stress. [/history] > So, the question arised, > what changed in Squid development policy? In policy: Nothing I'm aware of in the past 10 years. What changed on the Internet? a new bunch of RFCs came out, the server and clients Squid talks to all got updated to follow those documents more closely. What changed in Squid? the dev team have been slowly adding the new abilities to Squid. One by one, its only ~90% (maybe less) compliant withe the MUST conditions, not even close to that on the SHOULDs, MAYs, and implied processing abilities. What do you think should happen to Squid when all the software it talks to speaks and expects what the RFCs say they should expect from recipients/Squid ? Consider the extreme this leads towards: You could easily write a piece of software that took HTTP requests and sent back random data that looked vaguely like HTTP responses. But how useless that is when used as a proxy. So much for ignoring the HTTP specs. > Why there is no configuration > option like 'ignore_vary [acl]', so highly demanded by many users in the > list? AFAIK nobody has ever even proposed adding one until these past few weeks. A proposals to ignore "Vary: *" has come up every few years, but when the proposer was made aware of the server expectations as stated in the RFC 2616 they went away, no attempt to go any further. So I assume that means it wasn't really a problem for them, not a serious one. In RFC 2616 (or older) compliant web there is no benefit to caching these objects. The revalidate clause did not exist so the server can be expected to always produce a new one. So what is gained by caching it? a waste of space other HIT's could have used better. Negative benefits really, not even zero gain. The middle sentence of that part of RFC 7231 has only existed for the past few years. Even I had overlooked it until today. Sorry, but I have been concentrating on getting Cache-Control going properly (no-cache and side effects are still a hot topic 4 years after it went in). Vary is much broken in other serious ways, '*' storage seems a minor issue to me. Also be aware that Squid is still in the process of moving from HTTP/1.0 behaviour to HTTP/1.1 revalidation. We (Eduard, Alex and myself) have not magically made everything work perfectly in one release. This is one of probably many cases where revalidation is potentially usable - just that nobody has coded it yet. Or cases which were previously forbidden and HTTPbis discussions (recent 7-ish years) shown that other vendors are widely okay with it so its been allowed now by the 723x RFC series. As maintainer its my responsibility to point out the issues to anyone even proposing a violation gets added. To make sure they are aware of the expectations the servers and clients may have of Squid. All violation proposals get the same treatment (heck all patches get this same treatment too, I'm just a little more lenient on compliance increasing patches): - make the proposer aware of the problems they will cause, ** my posts in reply to this and other threads. - mention any alternatives (where available), ** others have mentioned eCAP/ICAP as better suited to such violations. That is exactly what those interfaces are designed for. ** re-reading the section you quoted above, the middle sentence adds a possibility I overlooked earlier. You now have a potential direction to go without even doing any violation. The audit process shows its worth right there :-). - and let them decide if it is worth the trouble to go forward. ** so far mostly whats happened is a lot of complaints that "Squid dev team" are not providing things for free, on demand, to service a very abusive and extremist person. Or just outright abuse. Seems like the individual who started the discussions does not care about getting it to happen. You care more, and as you say below ... > Personally, I'm no affected by the Vary abuse, but I suppose there > will be increasing number of abuse cases in the future. Me neither (for seeing the problem in my sysadmin roles), so for both of us and many others it seems (to me) not to be a major problem. Or at least not worth facing the consequences the proposed change risks causing. We can guess at what the future holds. But until we get there we really dont know. My view of things includes years of app developer queries sent to the IETF HTTPbis WG. There is an increasing number of applications that *need* proxies to follow the spec requirements. I am not seeing them much in the wild yet, but those applications are definitely around and breaking them could lead people to a lot of trouble. FWIW: The current spec for HTTP/1.1 is extremely tollerant. There are a very few places where it comes to an absolute inviolate rule. Those are cases which have very clear use-case mentioned in the spec and problems being avoided mentioned as well. For example the quoted section above about Vary. I overlooked that middle sentance that has been added since 2616. That creates the leeway we need to cache the object within compliance. We are thusly allowed to treat "Vary:*" as another of the must-revalidate caching cases, just like Cc:private or Authenticated responses. (From the use-cases that have been mentioned in IETF WG about this header, and cases people have been advised to use it I very much doubt any server will respond with 304 to such revalidation. But we might get lucky, or future servers might do so.) NP: all previous discussions and proposals of Vary:* handling came up when 2616 was the spec to follow, and Squid did not do 1.1 much anyway - so using revalidation was not even on the horizon so to speak. Anyhow, back to me rant ;-P The current specs were written by representatives standing for *all* the major browsers, *all* the major web servers, *all* the major language/library frameworks, almost all the major command-line tools, and many application developers as well. So it is not written in isolation by some few "ivory tower types" Yuri would have you believe (yes some specs are - HTTP is not, or at least those types are outnumbered for HTTP). The RFC 723x set is explicitly a collection of details about how the current HTTP applications actually found around the world *do* talk to each other today and how best to exchange messages so the processing operations are understood at both ends of the network connection. The HTTP RFC's are effectively a description of current best practice. Violating is only useful if you are fixing some broken application your particular proxy has to deal with. All other cases are just causing inefficiency and varying amounts of nasty depending on the violation. This latter point is part of why we require a clear use-case before accepting violation patches - the need for it should be well thought out. People who choose to violate that type of spec without a good reason are just being stupid. The three persons (well, 2 use aliased email addresses - I suspect are actually one or two persons from the same region of the world) providing the most noise about Vary keep demanding it be provided for free. > One of your > answers confirmed my assumption regarding the question: > >> - there is a very high risk of copy-and-paste sysadmin spreading the >> problems without realising what they are doing. Particularly since those >> proposing it are so vocal about how great it *seems* for them. > When someone else does the coding I act as both auditor and maintainer. The policies checked during audit do not go into good/bad judgements - just: code style, quality (no identifiable bugs) and a clear statement of the need behind the addition (for the merge repo comment to guide future maintenance of that code). As maintainer I merge patches myself only if I'm happy to take on code maintenance for them, or supplier explicitly takes it on themselves (ie helper authors). Sometimes I get patches up to scratch in audit then let others merge, or advise they just be used as custom patches. In the latter case we at least know that patch is not full of obvious bug behaviours. But note so far *nobody* has submitted patches to audit about this. It has all just been a rather heated discussion in various unrelated bug reports and some threads in this users list. Amos _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
