+100 from me.
On 16 April 2015 at 14:23, Patrick Uiterwijk <puiterwijk@xxxxxxxxxx> wrote:
+1 from me, though you might need to add a <Directory /var/www/html> with allow from all.
----- Original Message -----
> ok. Next thing... I notice that some queries hitting particularly hard
> are from spiders.
>
> Back in the puppet days, koji disallowed all spidering.
>
> I never have seen useful search results from it.
>
> So, I'd like to block all the spiders and see if that helps out load
> issues.
>
> robots.txt would have:
>
> User-agent: *
> Disallow: /
>
> +1s?
>
> kevin
> --
> diff --git a/roles/koji_hub/files/kojiweb.conf
> b/roles/koji_hub/files/kojiweb.conf index 86abd2e..8334274 100644
> --- a/roles/koji_hub/files/kojiweb.conf
> +++ b/roles/koji_hub/files/kojiweb.conf
> @@ -6,6 +6,8 @@ KeepAlive On
> Alias /koji "/usr/share/koji-web/scripts/wsgi_publisher.py"
> #(configuration goes in /etc/kojiweb/web.conf)
>
> +Alias /robots.txt /var/www/html/robots.txt
> +
> <Directory "/usr/share/koji-web/scripts/">
> Options ExecCGI
> SetHandler wsgi-script
> diff --git a/roles/koji_hub/tasks/main.yml
> b/roles/koji_hub/tasks/main.yml index 0b6cd82..a5fc795 100644
> --- a/roles/koji_hub/tasks/main.yml
> +++ b/roles/koji_hub/tasks/main.yml
> @@ -171,6 +171,13 @@
> notify: restart httpd
> when: env != "staging"
>
> +- name: koji robots.txt config
> + copy: src="" dest=/var/www/html/robots.txt
> + tags:
> + - config
> + - koji_hub
> + notify: restart httpd
> +
> - name: kojira log dir
> file: dest=/var/log/kojira owner=root group=root mode=0750
> state=directory tags:
>
> _______________________________________________
> infrastructure mailing list
> infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
> https://admin.fedoraproject.org/mailman/listinfo/infrastructure
_______________________________________________
infrastructure mailing list
infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Stephen J Smoogen.
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
