Re: Looking for some apache config help to block evil spiders

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



2009/10/10 Steven W. Orr <steveo@xxxxxxxxxxx>:
> On 10/10/09 14:37, quoth Steven W. Orr:
>> RewriteCond %{HTTP_USER_AGENT}  ^Baiduspider.* [OR]
>> RewriteCond %{HTTP_USER_AGENT}        ^msnbot.* [OR]
>> RewriteCond %{HTTP_USER_AGENT}        ^NaverBot.* [OR]
>> RewriteCond %{HTTP_USER_AGENT}        ^Sogou-Test-Spider.*
>> RewriteCond %{HTTP_USER_AGENT}        ^Mozilla/4.0.*
>> RewriteCond %{HTTP_USER_AGENT}        ^T-Mobile Dash.*
>> RewriteRule .* - [F,L]

>> The goal is to see the spiders bouncing off.

> On 10/10/09 14:55, quoth Sharpe, Sam J:
>> Are you actually missing the [OR] at the end of the 4th and 5th
>> RewriteCond lines, or is that a mispaste...
>
> Yes, thanks, I missed that, but that isn't the problem. The problem is that I
> want to be able to see what gets rejected in the log files.

Your rule didn't match anything, because there are mutually exclusive
options ANDed  - that was my point.

You can't have a user_agent that starts with Mozilla AND Sogou - it
has to be one or the other, so you would have never seen anything in
the logs.

Without access to ALL your rewrite rules, I can't tell you whether
lines such as:

>> 72.30.65.61 - - [10/Oct/2009:14:28:24 --0400] \
>> [vdom.syslang.net/sid#b7298ed0][rid#b6b488e8/initial] (1) pass through /d1/fn

are hits on the match set you posted above, or hits on another rewrite
rule, but I don't see any evidence that it's the Spider matching rule
that is generating those lines either.

You might also try upping RewriteLogLevel to something higher than 1
to see more detail...

--
Sam

-- 
fedora-list mailing list
fedora-list@xxxxxxxxxx
To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list
Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
[Index of Archives]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [Fedora Magazine]     [Fedora News]     [Fedora Summer Coding]     [Fedora Laptop]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Maintainers]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Legacy]     [Fedora Desktop]     [Fedora Fonts]     [ATA RAID]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [SSH]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Centos]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Tux]     [Yosemite News]     [Gnome Users]     [KDE Users]     [Fedora Art]     [Fedora Docs]     [Asterisk PBX]     [Fedora Sparc]     [Fedora Universal Network Connector]     [Libvirt Users]     [Fedora ARM]

  Powered by Linux