On Sat, Mar 1, 2014 at 10:33 PM, Sam Gleske <sam.mxracer@xxxxxxxxx> wrote:
FYI most of the requests will only be HTTP method HEAD - http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.4
It only requests headers and not the response body so it would actually be less of an impact than most crawlers when doing the actual link testing. For the crawling itself the response body is received so that it can find other links.
SAM
@gimp-dev, @gimp-web
My crawler has a stable release now. I'd like to request permission to crawl www.gimp.org. It will be restricted to that domain and have request delays implemented to minimize impact.
FYI most of the requests will only be HTTP method HEAD - http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.4
It only requests headers and not the response body so it would actually be less of an impact than most crawlers when doing the actual link testing. For the crawling itself the response body is received so that it can find other links.
SAM
_______________________________________________ gegl-developer-list mailing list List address: gegl-developer-list@xxxxxxxxx List membership: https://mail.gnome.org/mailman/listinfo/gegl-developer-list
