Bacchi how i use "robost.txt" plz explain with example. Daniel...... it`s working for "wget" but still we can download from other utilities like..."DownloadStudio On Sun, Jun 29, 2008 at 11:45 PM, Joy Methew <ml4joy@xxxxxxxxx> wrote: > Bacchi > > how i use "robost.txt" plz explain with example. > > Daniel...... > > it`s working for "wget" but still we can download from other utilities > like..."DownloadStudio" > > On 6/27/08, Daniel Carrillo <daniel.carrillo@xxxxxxxxx> wrote: >> >> 2008/6/27 Joy Methew <ml4joy@xxxxxxxxx>: >> >> > hiii all.... >> > >> > we can download any site from "wget -r " options. >> > if i want to stop downloading of my site from web server how i can do >> > this??? >> >> >> You can configure Apache for refuse connections with UserAgent "wget", >> but note that wget can use any UserAgent (--user-agent option). >> >> SetEnvIfNoCase User-Agent "^wget" blacklist >> <Location /> >> ... >> your options >> ... >> Order allow,deny >> Allow from all >> Deny from env=blacklist >> </Location> >> >> BTW: robots.txt only can stop crawling from "good" crawlers, like >> google, yahoo, alexa, etc. >> >> >> -- >> redhat-list mailing list >> unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe >> https://www.redhat.com/mailman/listinfo/redhat-list >> > > -- redhat-list mailing list unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe https://www.redhat.com/mailman/listinfo/redhat-list
