Will the header pass with using file_get_contents , or should I be using another command, and if so, which one? Ron
<?php
header('User Agent: RonBot (http://www.example.com)');
$url = "http://www.example.com";;
$input = file_get_contents($url);
The Verse of the Day
“Encouragement from God’s Word”
http://www.TheVerseOfTheDay.info
From: Shreyas Agasthya
Sent: Thursday, November 25, 2010 4:21 AM
To: Ron Piggott
Cc: php-general@xxxxxxxxxxxxx ; ash@xxxxxxxxxxxxxxxxxxxx
Subject: Re: Fw: Spoofing user_agent
A standard HTTP Request headers is : User Agent (without the underscore).
--Shreyas
On Thu, Nov 25, 2010 at 2:36 PM, Ron Piggott <ron.piggott@xxxxxxxxxxxxxxxxxx> wrote:
Is this what you are telling me to do:
header('user_agent: RonBot (http://www.theverseoftheday.info)');
Ron
The Verse of the Day
“Encouragement from God’s Word”
http://www.TheVerseOfTheDay.info
From: ash@xxxxxxxxxxxxxxxxxxxx
Sent: Thursday, November 25, 2010 3:34 AM
To: Ron Piggott ; php-general@xxxxxxxxxxxxx
Subject: Re: Fw: Spoofing user_agent
You need to set it in the header request you make. Putting it in the script you're using as a spider with ini_set won't do anything because the Target site doesn't know anything about it.
Thanks,
Ash
http://www.ashleysheridan.co.uk
----- Reply message -----
From: "Ron Piggott" <ron.piggott@xxxxxxxxxxxxxxxxxx>
Date: Thu, Nov 25, 2010 08:25
Subject: Fw: Spoofing user_agent
To: <php-general@xxxxxxxxxxxxx>
I have wrote a script to generate a sitemap of my web site. It crawls all of the site web pages. (About 30,000)
I need help to spoof the user_agent variable so the stats program running in the background ( “AWSTATS” ) will treat the crawl as a bot, not browsing usage.
The sitemap generator is a cron job. I tried the syntax:
ini_set('user_agent', 'RonBot (http://www.theverseoftheday.info)/'/);
This didn’t work. The browsing was attributed to the dedicated IP address.
How do I get AWSTATS to access this, such as other entries under the “Robots/Spiders visitors” heading:
Unknown robot (identified by 'bot*')
I don’t mean any ill will by changing this setting. Thanks for the help.
Ron
The Verse of the Day
“Encouragement from God’s Word”
http://www.TheVerseOfTheDay.info
--
Regards,
Shreyas Agasthya
