First: thanks very much for spelling this out, Ilyas. This was along
the lines of what I'd been considering. You addressed a number of
concerns I had (e.g.: non-blocking output) which is really helpful.
on 08:39 Fri 25 Mar, Ilyas -- (umask00@xxxxxxxxx) wrote:
> Hi!
>
> I'm using follow method for remote logging and catch logs from many servers.
> Nginx writes logs into fifo, which created via nginx init script:
>
> cat /etc/sysconfig/nginx
> ...
> # syslog-ng support for nginx
> if [ ! -p /var/log/nginx/access.log ]; then
> /bin/rm -f /var/log/nginx/access.log
> /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log
> fi
> if [ ! -p /var/log/nginx/error.log ] ; then
> /bin/rm -f /var/log/nginx/error.log
> /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log
> fi
> /bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log
>
> Nginx just writes to fifo as to file. Nginx has nonblocking output to
> logs and if nobody read fifo nginx dont stop on logs write.
Bingo.
> From other side pipe reads syslog-ng.
> cat /etc/syslog-ng/syslog-ng.conf
> ...
> source s_nginx_20 {
> fifo ("/var/log/nginx/access.log" log_prefix("nginx-access-log: "));
> };
>
> source s_nginx_21 {
> fifo ("/var/log/nginx/error.log" log_prefix("nginx-error-log: "));
> };
> ...
> destination d_remote { tcp("remote.example.com", port(514)); };
> ...
> # nginx
> filter f_nginx_20 { match("nginx-access-log: "); };
> filter f_nginx_21 { match("nginx-error-log: "); };
> ...
> # nginx
> log { source(s_nginx_20); filter(f_nginx_20); destination(d_remote); };
> log { source(s_nginx_21); filter(f_nginx_21); destination(d_remote); };
>
Nice.
> To avoid syslog-ng problems on startup (ex. if fifo does not exists)
> used follow solution:
> cat /etc/sysconfig/syslog-ng
> ...
> # syslog-ng support for nginx
> if [ ! -p /var/log/nginx/access.log ]; then
> /bin/rm -f /var/log/nginx/access.log
> /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log
> fi
> if [ ! -p /var/log/nginx/error.log ] ; then
> /bin/rm -f /var/log/nginx/error.log
> /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log
> fi
> /bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log
>
>
>
> On remote side (remote.example.com):
> cat /etc/syslog-ng/syslog-ng.conf
> ...
> source s_net {
> udp(ip(0.0.0.0) port(514));
> tcp(ip(0.0.0.0) port(514) keep-alive(yes) max-connections(128));
> };
> ...
> filter f_nginx_20 { match("nginx-access-log: "); };
> filter f_nginx_21 { match("nginx-error-log: "); };
> ...
> destination d_nginx_20 { file("/var/log/nginx/access.log"); };
> destination d_nginx_21 { file("/var/log/nginx/error.log"); };
> ...
> log { source(s_sys); filter(f_nginx_20); destination(d_nginx_20); };
> log { source(s_sys); filter(f_nginx_21); destination(d_nginx_21); };
>
>
>
> In the same way I catch logs from 20-30 servers to 1 server, approx.
> 300GB gzipped logs per day.
Great. That also answers the scaling question. We're comfortably under
that scale for now.
Very, very helpful post, thanks again.
> On Thu, Mar 24, 2011 at 11:23 PM, Dr. Ed Morbius <dredmorbius@xxxxxxxxx> wrote:
> > I'm looking for suggestions as to a good general method of
> > remote-logging services such as nginx or anything else which doesn't
> > support syslog natively.
> >
> > I'm aware that there's an nginx patch, and we're evaluating this. It
> > may be the way we fly.
> >
> > However there are other tools which may not have a patch for which
> > remote logging would be useful. If there's a general soution (something
> > as naive as tailing local logs and firing these off on a regular basis).
> >
> > I've heard rumors of a Perl script used for apache logs.
> >
> > Also that rsyslog supports logging from local files to a remote syslog
> > server, possibly. I'm RTFMing on that.
> >
> > Thanks in advance.
> >
> > --
> > Dr. Ed Morbius, Chief Scientist / |
> > Robot Wrangler / Staff Psychologist | When you seek unlimited power
> > Krell Power Systems Unlimited | Go to Krell!
> > _______________________________________________
> > CentOS mailing list
> > CentOS@xxxxxxxxxx
> > http://lists.centos.org/mailman/listinfo/centos
> >
>
>
>
> --
> Ilyas R. Khasyanov
> Unix/Linux System Administrator
> GPG Key ID: 6EC5EB27 (Changed since 2009-05-12)
--
Dr. Ed Morbius, Chief Scientist / |
Robot Wrangler / Staff Psychologist | When you seek unlimited power
Krell Power Systems Unlimited | Go to Krell!
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
