I have a MySQL database with about a million records. I'd like to use the SQL command "order by RAND()" but my ISP won't let me: whenever the server gets spidered, Google overloads their MySQL server because of all the overhead of that command. I can't just cloak the spiders because I need them to find the random pages.
So...what I've been doing for small sets of records is to use PHP to generate a bunch of random record ID's, then I construct a long SQL statement to find all the matching records. This works, but if I want to generate a big index page to list a hundred or a thousand records, it could get pretty clunky.
Anyone have any better suggestions? :)
Not sure that this would work (since I've never done it :) but perhaps you could create a static page specifically for the web spiders. The basic plan is this:
- Create a web page with the record set (as you're already doing it)
- Save this web page to your server's cache (either roll your own or you can use Cache_lite or whatever)
- Use an 8 day expiration on this file
- Then once a week you update this cache file from a cron job
- Now the key part: you alter your robots.txt file so that the spiders will not go to the dynamically created page, but rather they follow to the cached / static page.
Oh and it this actually works for you... it would be nice to get some feedback.
-- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
