This 17 patches long patch series is intended (yet again) as preliminary version for splitting large 'gitweb: File based caching layer (from git.kernel.org)' mega-patch by John 'Warthog9' Hawley aka J.H., by starting small and adding features piece by piece. This is fourth version (fourth release) of this series, and is available on http://repo.or.cz/w/git/jnareb-git.git as 'gitweb/cache-kernel-v4' branch. Earlier versions are available there as branches 'gitweb/cache-kernel', 'gitweb/cache-kernel-v2' and 'gitweb/cache-kernel-v3'. Previous version of this series was sent to git mailing list as: * [RFC PATCHv3 00/10] gitweb: Simple file based output caching Message-Id: <1266349005-15393-1-git-send-email-jnareb@xxxxxxxxx> http://thread.gmane.org/gmane.comp.version-control.git/140138 This version is based on top of c0cb4ed (git-instaweb: Configure it to work with new gitweb structure, 2010-05-28), which was merged as 'ps/gitweb-soc' into next in 5db4adf, on 2010-06-03. Next version would be rebased on top of 'next', which would include "sub run" code in 'jn/gitweb-fastcgi', which would require resolving conflict with this series. This series of patches does not include comments on individual patches, comparing them to previous version, and to original series by J.H. The next release of this series, hopefully not as RFC, would include them. The MAIN DIFFERENCE from previous version is that, inspired by splitting gitweb part of "Splitting gitweb and developing write functionalities (Integrated web client for git)" Google Summer of Code 2010 project by Pavan Kumar Sankara, the gitweb output caching code now resides in separate modules under 'lib/' subdirectory, instead of being all in 'cache.pm' file. The 'gitweb: href(..., -path_info => 0|1)' which began v3 series is already merged in, together with other patches from 'jn/gitweb-caching-prep'. The following changes are available in the git repository at: git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v4 All comments welcome! Shortlog: ~~~~~~~~~ 1. gitweb: Return or exit after done serving request This patch was generated in response to noticing the following kind of errors in /var/log/httpd/error_log: [Sun Jun 13 11:58:02 2010] gitweb.cgi: Subroutine git_atom redefined at /var/www/perl/gitweb/gitweb.cgi line 6804. when running gitweb under mod_perl (with ModPerl::Registry handler). After this patch I didn't see such errors... but I failed to reproduce this error without this patch. This patch was sent separately to git mailing list as http://thread.gmane.org/gmane.comp.version-control.git/149040 but should probably be skipped. 2. gitweb: Fix typo in hash key name in %opts in git_header_html This patch was sent separately to git mailing list and is already present in 'master' as ad709ea (gitweb: Fix typo in hash key name in %opts in git_header_html, 2010-06-13). 3. gitweb: Prepare for splitting gitweb IMPORTANT!!! This patch is the main reason this series is CC-ed to Pavan Kumar Sankara (student) and Christian Couder (mentor) of "Integrated web client for git" GSoC2010 project, which includes splitting gitweb. As you can see later in series it makes it very easy to add new gitweb modules which are to be installed. Just add GITWEB_MODULES += lib/Module/Foo.pm to gitweb/Makefile for Module::Foo. 4. gitweb/lib - Very simple file based cache 5. gitweb/lib - Stat-based cache expiration The major change is that the code is now in GitwebCache::SimpleFileCache module in gitweb/lib/GitwebCache/SimpleFileCache.pm file, instead of being in gitweb/cache.pm file. Other change is removing using File::Spec to manipulate pathnames, using faster concatenation instead. Using File::Spec do not bring us extra portability, as gitweb itself assumes '/' as path separator. It should be tiny little bit faster without File::Spec::catfile etc. 6. gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/) This is new patch, not present in previous series. Note that you have to run benchmark manually; it is not run by t9503 test. The t/t9503/benchmark_caching_interface.pl includes example benchmark results at the end of the file. 7. gitweb/lib - Simple select(FH) based output capture The major change is that the code is now in GitwebCache::Capture and GitwebCache::Capture::SelectFH modules which are in *separate* files (one file per module), instead of being all in gitweb/cache.pm 8. gitweb/lib - Alternate ways of capturing output This is new patch, not present in previous series. It includes new test (not run by t9503) and new benchmark. 9. gitweb/lib - Cache captured output (using get/set) The major change is that the code is now in GitwebCache::CacheOutput module in gitweb/lib/GitwebCache/CacheOutput.pm file. 10. gitweb: Add optional output caching "use lib __DIR__.'/lib';" is now added by earlier patch in series. It uses new infrastructure instead of 'bag of modules' cache.pm file. 11. gitweb/lib - Adaptive cache expiration time 12. gitweb/lib - Use CHI compatible (compute method) caching interface Similar to what was in previous version of this series. 13. gitweb/lib - Use locking to avoid 'cache miss stampede' problem 14. gitweb/lib - Serve stale data when waiting for filling cache 15. gitweb/lib - Regenerate (refresh) cache in background The major change is that the code is now in new (not present in previous version) GitwebCache::FileCacheWithLocking module (in its own file), which is derived from GitwebCache::SimpleFileCache. This allow to choose whether to use locking (and other features that require locking) or not. 16. gitweb: Show appropriate "Generating..." page when regenerating cache 17. gitweb: Add startup delay to activity indicator for cache No major changes, I think (besides using new infrastructure). gitweb/Makefile | 10 + gitweb/README | 62 ++++ gitweb/gitweb.perl | 333 +++++++++++++++++-- gitweb/lib/GitwebCache/CacheOutput.pm | 93 ++++++ gitweb/lib/GitwebCache/Capture.pm | 66 ++++ gitweb/lib/GitwebCache/Capture/PerlIO.pm | 79 +++++ gitweb/lib/GitwebCache/Capture/SelectFH.pm | 82 +++++ gitweb/lib/GitwebCache/Capture/TiedCapture.pm | 149 +++++++++ gitweb/lib/GitwebCache/FileCacheWithLocking.pm | 264 +++++++++++++++ gitweb/lib/GitwebCache/SimpleFileCache.pm | 417 ++++++++++++++++++++++++ t/t9500-gitweb-standalone-no-errors.sh | 19 + t/t9503-gitweb-caching.sh | 38 +++ t/t9503/benchmark_caching_interface.pl | 132 ++++++++ t/t9503/benchmark_capture_implementations.pl | 198 +++++++++++ t/t9503/test_cache_interface.pl | 409 +++++++++++++++++++++++ t/t9503/test_cache_output.pl | 66 ++++ t/t9503/test_capture_interface.pl | 76 +++++ 17 files changed, 2468 insertions(+), 25 deletions(-) create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm create mode 100644 gitweb/lib/GitwebCache/Capture.pm create mode 100644 gitweb/lib/GitwebCache/Capture/PerlIO.pm create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm create mode 100644 gitweb/lib/GitwebCache/Capture/TiedCapture.pm create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm create mode 100755 t/t9503-gitweb-caching.sh create mode 100755 t/t9503/benchmark_caching_interface.pl create mode 100755 t/t9503/benchmark_capture_implementations.pl create mode 100755 t/t9503/test_cache_interface.pl create mode 100755 t/t9503/test_cache_output.pl create mode 100755 t/t9503/test_capture_interface.pl -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html