Anyone else have anything on this or should I move this to FBR for tomorrow?
On Mon, 18 Mar 2019 at 11:34, Kevin Fenzi <kevin@xxxxxxxxx> wrote:
>
> On 3/18/19 7:43 AM, Stephen John Smoogen wrote:
> > So in looking at a problem with greenwave getting hit with various
> > web-bots I found that very few of our configured robots.txt for sites
> > work because we normally proxypass everything to the backend. This
> > causes extra work on each backend needing to either add this to their
> > codebase (if they are a flask containers) and could be slow for other
> > reasons. However this will change all robots.conf for all websites so
> > I don't want to look at a FBR without discussion.
> >
> > diff --git a/roles/httpd/website/templates/robots.conf
> > b/roles/httpd/website/templates/robots.conf
> > index f442128..a59eb68 100644
> > --- a/roles/httpd/website/templates/robots.conf
> > +++ b/roles/httpd/website/templates/robots.conf
> > @@ -1 +1,7 @@
> > +## Make sure that we don't skip this because we proxy pass it to a
> > +## slow backend
> > +<Location "/robots.txt">
> > + ProxyPass !
> > +</Location>
> > +
> > Alias /robots.txt /srv/web/{{site_name}}-robots.txt
>
> +1 here.
>
> kevin
>
>
> _______________________________________________
> infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
> To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
--
Stephen J Smoogen.
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
