on 3/28/03 7:30 AM, Ronan Chilvers at ronan@thelittledot.com appended the following bits to my mbox: > Is this reasonably efficient on your production server ? Do you have any > problems with it ? It's pretty efficient on my production server. Since I don't index the initial table, the inserts go quickly. One possible problem is that there is a MySQL process open for every apache child. We run a quad-xeon server with 1 GB of RAM so it's not an issue for us. > Also, how are you dealing with growing table sizes ? Do you summarise out > data to sub tables ? Or do you limit the date range and periodically dump > stuff that is out of range ? Right now, I'm copying the data nightly from the "live" table to the "archive" one. MySQL tends to corrupt index files when you perform long queries (joins, grouping with conditions, etc) and other clients are trying to insert data at the same time. Without indices, the querying is pretty slow. We only keep the last month of logs in the database. The tables get huge with hundreds of thousands of records in them. We still do long term processing of standard log files with Webalizer. > Part of the reason for using Perl / PHP, etc was to allow the log data to be > filtered and sorted prior to inserting to the db. This would then allow > automatic summarising so you could maintian host tables, request tables, etc > and limit the size of the main transfer log tables. Those are good benefits. I'm sure it would be faster than working with the whole dataset. Regarding your previous post, I'd probably stick to Perl (apologies to some on this list). I know the recent CLI improvements are supposed to help a lot, but I still feel Perl plays better with the shell, especially with file IO. > My approach to the queries thus far has been to keep different 'reports' > (which essentially narrow down to individual SQL queries) in a db. A user > picks a report, the SQL is selected and run and the data is then displayed > (also playing with JPGraph to produce some nice charts). Cool. We'd love to see some of your results when they are usable. Sincerely, Paul Burney <http://paulburney.com/> <?php // the statement formerly known as prince if ($the_elevator == 'tries to bring you down') { go_crazy('punch a higher floor'); } ?> -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
