Re: Remote-logging nginx? (or other non-syslog-enabled stuff)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



Hi!

I'm using follow method for remote logging and catch logs from many servers.
Nginx writes logs into fifo, which created via nginx init script:

cat /etc/sysconfig/nginx
...
# syslog-ng support for nginx
if [ ! -p /var/log/nginx/access.log ]; then
        /bin/rm -f /var/log/nginx/access.log
        /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log
fi
if [ ! -p /var/log/nginx/error.log ] ; then
        /bin/rm -f /var/log/nginx/error.log
        /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log
fi
/bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log

Nginx just writes to fifo as to file. Nginx has nonblocking output to
logs and if nobody read fifo nginx dont stop on logs write.

>From other side pipe reads syslog-ng.
cat /etc/syslog-ng/syslog-ng.conf
...
source s_nginx_20 {
        fifo ("/var/log/nginx/access.log" log_prefix("nginx-access-log: "));
};

source s_nginx_21 {
        fifo ("/var/log/nginx/error.log" log_prefix("nginx-error-log: "));
};
...
destination d_remote { tcp("remote.example.com", port(514)); };
...
# nginx
filter f_nginx_20 { match("nginx-access-log: "); };
filter f_nginx_21 { match("nginx-error-log: "); };
...
# nginx
log { source(s_nginx_20); filter(f_nginx_20); destination(d_remote); };
log { source(s_nginx_21); filter(f_nginx_21); destination(d_remote); };


To avoid syslog-ng problems on startup (ex. if  fifo does not exists)
used follow solution:
cat /etc/sysconfig/syslog-ng
...
# syslog-ng support for nginx
if [ ! -p /var/log/nginx/access.log ]; then
        /bin/rm -f /var/log/nginx/access.log
        /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log
fi
if [ ! -p /var/log/nginx/error.log ] ; then
        /bin/rm -f /var/log/nginx/error.log
        /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log
fi
/bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log



On remote side (remote.example.com):
cat /etc/syslog-ng/syslog-ng.conf
...
source s_net {
        udp(ip(0.0.0.0) port(514));
        tcp(ip(0.0.0.0) port(514) keep-alive(yes) max-connections(128));
};
...
filter f_nginx_20 { match("nginx-access-log: "); };
filter f_nginx_21 { match("nginx-error-log: "); };
...
destination d_nginx_20 { file("/var/log/nginx/access.log"); };
destination d_nginx_21 { file("/var/log/nginx/error.log"); };
...
log { source(s_sys); filter(f_nginx_20); destination(d_nginx_20); };
log { source(s_sys); filter(f_nginx_21); destination(d_nginx_21); };



In the same way I catch logs from 20-30 servers to 1 server, approx.
300GB gzipped logs per day.

On Thu, Mar 24, 2011 at 11:23 PM, Dr. Ed Morbius <dredmorbius@xxxxxxxxx> wrote:
> I'm looking for suggestions as to a good general method of
> remote-logging services such as nginx or anything else which doesn't
> support syslog natively.
>
> I'm aware that there's an nginx patch, and we're evaluating this.  It
> may be the way we fly.
>
> However there are other tools which may not have a patch for which
> remote logging would be useful.  If there's a general soution (something
> as naive as tailing local logs and firing these off on a regular basis).
>
> I've heard rumors of a Perl script used for apache logs.
>
> Also that rsyslog supports logging from local files to a remote syslog
> server, possibly.  I'm RTFMing on that.
>
> Thanks in advance.
>
> --
> Dr. Ed Morbius, Chief Scientist /            |
>  Robot Wrangler / Staff Psychologist        | When you seek unlimited power
> Krell Power Systems Unlimited                |                  Go to Krell!
> _______________________________________________
> CentOS mailing list
> CentOS@xxxxxxxxxx
> http://lists.centos.org/mailman/listinfo/centos
>



-- 
Ilyas R. Khasyanov
Unix/Linux System Administrator
GPG Key ID: 6EC5EB27 (Changed since 2009-05-12)
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos



[Index of Archives]     [CentOS]     [CentOS Announce]     [CentOS Development]     [CentOS ARM Devel]     [CentOS Docs]     [CentOS Virtualization]     [Carrier Grade Linux]     [Linux Media]     [Asterisk]     [DCCP]     [Netdev]     [Xorg]     [Linux USB]
  Powered by Linux