We are witnessing a potential bottleneck using piped logging. Each web server handles relative large amount of concurrent connections. It is running on CentOS 5.x, using worker MPM. We use piped log to cronolog. We are discovering when the web server reach ~6K concurrent connections (our request are small in nature), we are experiencing logging bottleneck. This is confirmed by looking at the Apache Status where the vast majority of the threads are in Logging state. This is not an issue that caused by slow disks (almost no I/O wait).
I suspect the issue here is because the pipe buffer size is fixed at 4K (8 x 512), and with many threads/process potentially writing to that, its write became blocked until cronolog get its timeslice to clear out the buffer, but it will never catch up since as soon as it is cleared, it is almost immediately filled again. We can "patch" the kernel to adjust the pipe buffer size, but I would like to avoid if possible.
We can potentially alleviate the issue slightly by specifying piped log in each of the virtual host (although all writing back to the same file through pipe to cronolog). Given we have very limited number of vhost, and different logging can not be specified more detail than vhost/system wide level, this does not provide much more capacity.
Does anyone else have similar experience? Did anyone try some of the Apache module with logrotate capability?
Note, we can not use a cronjob to rotate the logs then do a graceful restart of Apache.
Thanks.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|