Freeze Break request: Add custom robots.txt for Koschei

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

Koschei has issues due to increased robots activity. In particular the
slow "affected-by" query seems to be problematic.
This causes user-visible downtime from time to time and I was unable
to fix it by simply scaling up the application.
More details: https://pagure.io/fedora-infrastructure/issue/12180

The issue causes significant service degradation to our users.
Therefore I think it's justified to add custom robots.txt for Koschei
before the freeze is lifted.
I would like to apply the following ansible.git patch:

diff --git a/roles/httpd/website/templates/robots/koschei.fedoraproject.org-robots.txt
b/roles/httpd/website/templates/robots/koschei.fedoraproject.org-robots.txt
new file mode 100644
index 0000000000..9a36280d39
--- /dev/null
+++ b/roles/httpd/website/templates/robots/koschei.fedoraproject.org-robots.txt
@@ -0,0 +1,5 @@
+User-agent: *
+Crawl-delay: 2
+Disallow: /login
+Disallow: /bugreport
+Disallow: /affected-by

And deploy the changes to proxies by running:

# ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t robots

+1s to this FBR?

--
Mikolaj Izdebski

-- 
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Index of Archives]     [Fedora Development]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux