Bacchi how i use "robost.txt" plz explain with example. Daniel...... it`s working for "wget" but still we can download from other utilities like..."DownloadStudio" On 6/27/08, Daniel Carrillo <daniel.carrillo@xxxxxxxxx> wrote: > > 2008/6/27 Joy Methew <ml4joy@xxxxxxxxx>: > > > hiii all.... > > > > we can download any site from "wget -r " options. > > if i want to stop downloading of my site from web server how i can do > > this??? > > > You can configure Apache for refuse connections with UserAgent "wget", > but note that wget can use any UserAgent (--user-agent option). > > SetEnvIfNoCase User-Agent "^wget" blacklist > <Location /> > ... > your options > ... > Order allow,deny > Allow from all > Deny from env=blacklist > </Location> > > BTW: robots.txt only can stop crawling from "good" crawlers, like > google, yahoo, alexa, etc. > > > -- > redhat-list mailing list > unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe > https://www.redhat.com/mailman/listinfo/redhat-list > -- redhat-list mailing list unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe https://www.redhat.com/mailman/listinfo/redhat-list
