On Tue, Mar 16, 2004 at 09:22:28AM -0600, mw-list-yum@xxxxxx wrote:
> On Mon, Mar 15, 2004 at 03:08:48PM -0700, Michael Stenner wrote:
> > The issue is not that publicfile is violating the rfc. As far as I
> > can tell, it isn't. It is, however, behaving in a way that's
> > different from most other servers. We were using that common (but
> > not rfc-mandated) behavior to deal with a complex issue.
>
> Just for clarification, it seems the rfc, in fact, _forbids_ the
> http/1.1 server to send content-length if transfer-encoding is sent.
> And it also says that if a client sees both transfer-encoding and
> content-length, it must ignore content-length.
I agree. What makes publicfile different is that it seems to use
chunked encoding all the time. There's no rule against this, but most
servers only use chunked encoding when they "need to". The chunked
encoding is intended largely to deal with dynamic data for which the
content-length is not known at the time the transaction begins.
Othwerwise, it would not be possible to do keepalive because the
client would have no way to locate message boundaries. For most
static files with an obvious (from stat) file size, most servers send
the file with "identity" encoding and provide a content-length.
Again, the issue is not that publicfile is wrong. It's just a little
different from how most servers behave. Neither it nor the other
servers are incorrect. This is not a protocol-violation issue. The
issue in yum is above the protocol layer.
> > using google with site:lists.dulug.duke.edu does a pretty decent job.
> >
> > yum content-length site:lists.dulug.duke.edu
> >
> > that will get you started.
>
> Well, I was hoping for some explanation what problem the
> content-length check is supposed to solve, but I have not been able to
> find the thread. Is it that you are trying to do something
> independently from the protocol?
In the next paragraph, you say you commented out this check.
Immediately above that code, there is this comment:
# this is a cute little hack - if there isn't a "Content-Length"
# header then its probably something generated dynamically, such
# as php, cgi, a directory listing, or an error message. It is
# probably not what we want.
Yes. The problem is that we're trying to make things work across
file:// ftp:// and http://. urllib2 (and worse, urllib) provides only
modest uniformity for these protocols, and so it was very difficult to
tell when something was wrong. urllib2 and urlgrabber can deal with
chunked encoding just fine. It's the error-checking in yum is where
things get caught.
For example, downloading a directory listing is AN ERROR in yum.
There is absolutely no circumstance when it is correct behavior.
Unfortunately, it's not an error in either ftp or http, so they both
report happily that everything worked just fine. This leaves yum with
a handful of data that it's told is good. It will then determine that
the data is "corrupted" or somesuch thing and lead the user down a
very wrong path.
The behavior we were taking advantage of is this: MOST servers (ftp
and http both - and this is crucial) provide content-length when
sending a normal static file (what we want) and do not provide
content-length when sending dynamic data (directory listings,
http error codes, etc).
The behavior for your server is an unfortunate side-effect, but it's
done so that yum will work better for the majority of users. Sure,
there are brute force ways around this, but there are also time
constraints and code-tidiness issues to be considered.
This is all largely moot, though, because we agree that this is a very
undesirable way to deal with it and intend to change it.
> For example, I just commented out the three lines in urlgrabber.py,
> and I can now update my system (and do many other things) with yum.
> Can you show me an example when this change in yum will bite me?
Described above. And the thing is, I don't mind it biting YOU,
because even if it does bite you here and there, it's an improvement!
Yum didn't work at all before! The problem is that it will bite a lot
of other people, too.
> > Also check out the thread entitled YUM+FTP
>
> Funny thread, but does not explain the complex isssue you mention
> above. Hope you are not "seriously" subscribing to the idea that the
> norm is the set of features the most popular servers are
> providing. wu-ftpd, proftpd, BIND or sendmail may not bring fond
> memories to all sysadms.
You might be getting a tad off-topic here. I never used the word
"norm" or "normal". I used the word "common". Besides, these other
servers are ALSO completeley within the RFC. different != wrong.
It's not a right/wrong issue. It's an issue of making yum work for
as many people as much of the time as possible.
Now, because (as I've said repeatedly) we plan to change this
behavior, I just really do not see the point in dumping any more time
into this conversation.
-Michael
--
Michael D. Stenner mstenner@xxxxxxxxxxxxxxx
ECE Department, the University of Arizona 520-626-1619
1230 E. Speedway Blvd., Tucson, AZ 85721-0104 ECE 524G
