On Fri, Dec 7, 2012 at 11:09 AM, Gelen James <hahaha_30k@xxxxxxxxx> wrote:
> Hi all,
>
> Has any one used any log archiving tools? Please suggest one that's your favorite.
>
> Presently I have a project to archive raw log files on hundreds of Centos boxes to a central server and stored there for two years. The basic requirements of the project are:
>
> 1, The log files will be sent from hundreds of Centos clients with a client side agent or similar.
> 2, better there is a client side configuration file which specify log files to archive
> 3, a server side configuration file specified where to logs from host ( or host groups), and the directory structure to store log files
> 4, better there is a way to remove duplication of log files, i.e. md5 checksum, time stamp, etc.
>
> Open source or commercial are both fine. And there is no need to do the indexing or other further processing, just raw files is fine.
I've done this for specific applications with some simple shell
scripts using rsync over ssh, but it has to mesh with the way the log
files roll over and are renamed. I don't think there is a generic
tool. If you just want archived daily snapshot copies, you could use
backuppc to back up /var/log on the targets. It will compress files
and de-dup exact file content matches with hard links to a pooled
instance even where the file has been renamed (like logrotate does).
That approach doesn't get you a handy way to analyze anything over a
long time span or look at more than one file as a stream - but you
could have it working in a few hours and it will be very efficient
with disk usage.
--
Les Mikesell
lesmikesell@xxxxxxxxx
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
