On Wed, Aug 05, 2015 at 11:53:49AM -0600, Tim Flink wrote: > For some reason, yahoo's bot has been hitting our taskotron instance > really hard all of a sudden - slowing it down enough to trigger nagios > alerts twice this week. > > There is a robots.txt served by the buildmaster instance but since that > isn't at the webroot in our setup, bots may not read it. > > I'd like to add a robots.txt to the public facing taskotron instances > so that we can avoid this in the future. Rendered content of the > robots.txt is at the end of this email. > > I've already deployed the file to taskotron-dev which can be seen at: > http://taskotron-dev.fedoraproject.org/robots.txt > > Proposed content would be at: > http://taskotron.stg.fedoraproject.org/robots.txt > http://taskotron.fedoraproject.org/robots.txt > > Tim > > > User-agent: * > Disallow: /taskmaster/waterfall > Disallow: /taskmaster/builders > Disallow: /taskmaster/changes > Disallow: /taskmaster/buildslaves > Disallow: /taskmaster/schedulers > Disallow: /taskmaster/one_line_per_build > Disallow: /taskmaster/grid > Disallow: /taskmaster/tgrid > Disallow: /taskmaster/json +1 for me Pierre
Attachment:
pgp0t7iOabewC.pgp
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
