On 24/10/2016 6:28 a.m., garryd@xxxxxxxxx wrote:
On 2016-10-23 18:31, Amos Jeffries wrote:
On 23/10/2016 2:32 a.m., garryd wrote:
Since I started use Squid, it's configuration always RFC compliant by
default, _but_ there were always knobs for users to make it HTTP
violent. It was in hands of users to decide how to handle a web
resource. Now it is not always possible, and the topic is an evidence.
For example, in terms of this topic, users can't violate this RFC
statement [1]:
A Vary field value of "*" signals that anything about the request
might play a role in selecting the response representation, possibly
including elements outside the message syntax (e.g., the client's
network address). A recipient will not be able to determine whether
this response is appropriate for a later request without forwarding
the request to the origin server. A proxy MUST NOT generate a Vary
field with a "*" value.
[1] https://tools.ietf.org/html/rfc7231#section-7.1.4
Please name the option in any version of Squid which allowed Squid to
cache those "Vary: *" responses.
No such option ever existed. For the 20+ years Vary has existed Squid
has behaved in the same way it does today. For all that time you did not
notice these responses.
You are absolutely right, but there were not such abuse vector in the
past (at least in my practice). There were tools provided by devs to
admins to protect against trending abuse cases.
What trend? There is exactly one mentioned URL that I'm aware of, the
Chrome browser download URL. I've posted two reasons why Chrome uses the
Vary:* header. Just opinions of mine, but formed after actual
discussions with the Chrome developers some years back.
[I very much dislike writing this. But you seem to have been sucked in
and deserve to know the history.]
All the fuss that is going on AFAICS was started by Yuri. His comment
history here and in bugzilla, and in private responses range from
non-compromising "cache everything no matter what - do what I say, now!"
(repeatedy in unrelated bugzilla reports), "f*ck the RFCs and anyone
following them, just store everything I dont care about what happens"
(this mornings post), to personal attacks against anyone who mentions
the previous stance might have problems (all the "Squid developers
believe/say/..." comments - none of which match what the team we have
actually said to him or believe).
There is one other email address which changes its name occasionally and
posts almost exactly the same words as Yuri's. So it looks to me as Yuri
and some sock puppets performing a campaign to spread lies and FUD about
Squid and hurt the people doing work on it.
Not exactly a good way to get people to do things for free. But it seems
to have worked on getting you and a few others now doing the coding part
for him at no cost, and I have now wasted time responding to you and
thinking of a solution for it that might get accepted for merge.
This particular topic is not the first to have such behaviour by Yuri.
There have been other things where someone made a mistake (overlooked
something) and all hell full of insults broke loose at them. And several
other cases where missing features in Squid did not get instant
obedience to quite blunt and insulting demands. Followed by weeks of
insults until the bug was fixed by other people - then suddenly polite
Yuri comes back overnight.
As a developer, I personally decided not to write the requested code.
Not in the way demanded. This seems to have upset Yuri who has taken to
insulting me and the rest of the dev team as a whole. I'm not sure if he
is trolling to intentionally cause the above mentioned effects, or
really in need of medical assistance to deal with work related stress.
[/history]
So, the question arised,
what changed in Squid development policy?
In policy: Nothing I'm aware of in the past 10 years.
What changed on the Internet? a new bunch of RFCs came out, the server
and clients Squid talks to all got updated to follow those documents
more closely.
What changed in Squid? the dev team have been slowly adding the new
abilities to Squid. One by one, its only ~90% (maybe less) compliant
withe the MUST conditions, not even close to that on the SHOULDs, MAYs,
and implied processing abilities.
What do you think should happen to Squid when all the software it talks
to speaks and expects what the RFCs say they should expect from
recipients/Squid ?
Consider the extreme this leads towards: You could easily write a piece
of software that took HTTP requests and sent back random data that
looked vaguely like HTTP responses. But how useless that is when used as
a proxy. So much for ignoring the HTTP specs.
Why there is no configuration
option like 'ignore_vary [acl]', so highly demanded by many users in the
list?
AFAIK nobody has ever even proposed adding one until these past few weeks.
A proposals to ignore "Vary: *" has come up every few years, but when
the proposer was made aware of the server expectations as stated in the
RFC 2616 they went away, no attempt to go any further. So I assume that
means it wasn't really a problem for them, not a serious one.
In RFC 2616 (or older) compliant web there is no benefit to caching
these objects. The revalidate clause did not exist so the server can be
expected to always produce a new one.
So what is gained by caching it? a waste of space other HIT's could have
used better. Negative benefits really, not even zero gain.
The middle sentence of that part of RFC 7231 has only existed for the
past few years. Even I had overlooked it until today. Sorry, but I have
been concentrating on getting Cache-Control going properly (no-cache and
side effects are still a hot topic 4 years after it went in). Vary is
much broken in other serious ways, '*' storage seems a minor issue to me.
Also be aware that Squid is still in the process of moving from HTTP/1.0
behaviour to HTTP/1.1 revalidation. We (Eduard, Alex and myself) have
not magically made everything work perfectly in one release. This is one
of probably many cases where revalidation is potentially usable - just
that nobody has coded it yet. Or cases which were previously forbidden
and HTTPbis discussions (recent 7-ish years) shown that other vendors
are widely okay with it so its been allowed now by the 723x RFC series.
As maintainer its my responsibility to point out the issues to anyone
even proposing a violation gets added. To make sure they are aware of
the expectations the servers and clients may have of Squid.
All violation proposals get the same treatment (heck all patches get
this same treatment too, I'm just a little more lenient on compliance
increasing patches):
- make the proposer aware of the problems they will cause,
** my posts in reply to this and other threads.
- mention any alternatives (where available),
** others have mentioned eCAP/ICAP as better suited to such violations.
That is exactly what those interfaces are designed for.
** re-reading the section you quoted above, the middle sentence adds a
possibility I overlooked earlier. You now have a potential direction to
go without even doing any violation. The audit process shows its worth
right there :-).
- and let them decide if it is worth the trouble to go forward.
** so far mostly whats happened is a lot of complaints that "Squid dev
team" are not providing things for free, on demand, to service a very
abusive and extremist person. Or just outright abuse. Seems like the
individual who started the discussions does not care about getting it to
happen. You care more, and as you say below ...
Personally, I'm no affected by the Vary abuse, but I suppose there
will be increasing number of abuse cases in the future.
Me neither (for seeing the problem in my sysadmin roles), so for both of
us and many others it seems (to me) not to be a major problem. Or at
least not worth facing the consequences the proposed change risks causing.
We can guess at what the future holds. But until we get there we really
dont know.
My view of things includes years of app developer queries sent to the
IETF HTTPbis WG. There is an increasing number of applications that
*need* proxies to follow the spec requirements. I am not seeing them
much in the wild yet, but those applications are definitely around and
breaking them could lead people to a lot of trouble.
FWIW: The current spec for HTTP/1.1 is extremely tollerant. There are a
very few places where it comes to an absolute inviolate rule. Those are
cases which have very clear use-case mentioned in the spec and problems
being avoided mentioned as well.
For example the quoted section above about Vary. I overlooked that
middle sentance that has been added since 2616. That creates the leeway
we need to cache the object within compliance. We are thusly allowed to
treat "Vary:*" as another of the must-revalidate caching cases, just
like Cc:private or Authenticated responses.
(From the use-cases that have been mentioned in IETF WG about this
header, and cases people have been advised to use it I very much doubt
any server will respond with 304 to such revalidation. But we might get
lucky, or future servers might do so.)
NP: all previous discussions and proposals of Vary:* handling came up
when 2616 was the spec to follow, and Squid did not do 1.1 much anyway -
so using revalidation was not even on the horizon so to speak.
Anyhow, back to me rant ;-P
The current specs were written by representatives standing for *all* the
major browsers, *all* the major web servers, *all* the major
language/library frameworks, almost all the major command-line tools,
and many application developers as well.
So it is not written in isolation by some few "ivory tower types" Yuri
would have you believe (yes some specs are - HTTP is not, or at least
those types are outnumbered for HTTP). The RFC 723x set is explicitly a
collection of details about how the current HTTP applications actually
found around the world *do* talk to each other today and how best to
exchange messages so the processing operations are understood at both
ends of the network connection.
The HTTP RFC's are effectively a description of current best practice.
Violating is only useful if you are fixing some broken application your
particular proxy has to deal with. All other cases are just causing
inefficiency and varying amounts of nasty depending on the violation.
This latter point is part of why we require a clear use-case before
accepting violation patches - the need for it should be well thought
out. People who choose to violate that type of spec without a good
reason are just being stupid.
The three persons (well, 2 use aliased email addresses - I suspect are
actually one or two persons from the same region of the world) providing
the most noise about Vary keep demanding it be provided for free.
One of your
answers confirmed my assumption regarding the question:
- there is a very high risk of copy-and-paste sysadmin spreading the
problems without realising what they are doing. Particularly since those
proposing it are so vocal about how great it *seems* for them.
When someone else does the coding I act as both auditor and maintainer.
The policies checked during audit do not go into good/bad judgements -
just: code style, quality (no identifiable bugs) and a clear statement
of the need behind the addition (for the merge repo comment to guide
future maintenance of that code).
As maintainer I merge patches myself only if I'm happy to take on code
maintenance for them, or supplier explicitly takes it on themselves (ie
helper authors). Sometimes I get patches up to scratch in audit then let
others merge, or advise they just be used as custom patches. In the
latter case we at least know that patch is not full of obvious bug
behaviours.
But note so far *nobody* has submitted patches to audit about this. It
has all just been a rather heated discussion in various unrelated bug
reports and some threads in this users list.
Amos
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
