For some reason, yahoo's bot has been hitting our taskotron instance really hard all of a sudden - slowing it down enough to trigger nagios alerts twice this week. There is a robots.txt served by the buildmaster instance but since that isn't at the webroot in our setup, bots may not read it. I'd like to add a robots.txt to the public facing taskotron instances so that we can avoid this in the future. Rendered content of the robots.txt is at the end of this email. I've already deployed the file to taskotron-dev which can be seen at: http://taskotron-dev.fedoraproject.org/robots.txt Proposed content would be at: http://taskotron.stg.fedoraproject.org/robots.txt http://taskotron.fedoraproject.org/robots.txt Tim User-agent: * Disallow: /taskmaster/waterfall Disallow: /taskmaster/builders Disallow: /taskmaster/changes Disallow: /taskmaster/buildslaves Disallow: /taskmaster/schedulers Disallow: /taskmaster/one_line_per_build Disallow: /taskmaster/grid Disallow: /taskmaster/tgrid Disallow: /taskmaster/json
Attachment:
pgpxIqHNyZ11c.pgp
Description: OpenPGP digital signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
