Re: Async work progress report

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Krzysztof A. Adamski wrote:
I've created async2 branch on my github account. This branch includes
all current fixes and changes and it's cleaned up so it can be
reviewed and merged. Here's the address:
http://github.com/kadamski/func/commits/async2

Full changes in one patch file can be dowloaded from:
http://szrot.ry2n.pl/async2.patch

Change statistics:
 func/forkbomb.py                  |   25 ++++++++++------
 func/jobthing.py                  |   45 ++++++++++++++++++-----------
 func/overlord/base_command.py     |    4 ++-
 func/overlord/client.py           |    2 +-
 func/overlord/cmd_modules/call.py |
58+++++++++++++++++++++++++++++++++++- test/async_test.py | 4 +- 6 files changed, 106 insertions(+), 32 deletions(-)

shows that there is no changes that could break things other than async
calls (which are already broken now).
Changes made in this branch are described below:

I. New features:
  1. Added --async, --nopoll, --sort and --jobstatus commandline
     options, as described on my earlier post:
     https://www.redhat.com/archives/func-list/2008-May/msg00054.html

     Commits:
http://github.com/kadamski/func/commit/ea70b80d0378b21da53166f76e796692ecf20b33
http://github.com/kadamski/func/commit/9de932b2608445f100565f4ee37d383d8dd72ff9
http://github.com/kadamski/func/commit/cbb7650e359f8870cd59501bc0e9904d5a9e6ea6

II. Fixes:
  1. Bucket creation errors:
     There was a simple error in modulus arithmetic which led to
     ignoring results from first minion if the job was called on more
     than one minion. Commits:
http://github.com/kadamski/func/commit/3a66e1bfe011bc8ffc42ad8dcda672c76af8352d
http://github.com/kadamski/func/commit/3f62e1d302fb63f1f3c5c72121388c8b0dd6e784

  2. Huge memory leak:
     I've found that bsddb bindings in python 2.5 are broken and they
     leak a lot of memory. Switching to dbm fixed the problem. Commit:
http://github.com/kadamski/func/commit/bb80b6e6f10bd97b2926f447e913d8249592c32d

  3. Zombies on minion:
     After each async call, minion use fork to spawn async process.
     Parent returns to it's normal operation and newer calls waitpid()
     nor handle SIGCHLD signal. This means zombie process is created
     after each async call. I've implemented one of the possible
     solutions that use double-fork to daemonize async jobs. Side
     effect is that async jobs won't be killed when funcd is shutdown,
     instead they will run until they complete. Commit:
http://github.com/kadamski/func/commit/dee316020aaf6e70c909ea177adf2742a33e2b7b

     Another solution is to keep list of child PIDs and call unblocking
     waitpid() call from time to time to collect zombies. We have to do
     this asynchronously to the normal operation not to led to many
     zombies to be present for a long time. One solution is to do this
     in SIGALARM handler and call signal.alarm(INTERVAL) each time.
     There may be a problem with using alarm() and sleep() at the same
     time so we need to ensure no one is using sleep() in main process.
     This means we can't run any job (any code from modules) in main
     process, instead we have to call it in a fork (which could be good
     idea in general). I'll test this approach soon.

  4. Concurrency in jobthing:
     __update_status() was called just after fork in both parrent and
     child. To ensure that this update won't be called AFTER finishing
     the job but before, it was actually called twice, which is not
     needed. Instead it can be called once but before fork. Commit:
http://github.com/kadamski/func/commit/721fa16f64627fcd5b0d0b5c4f9d9e382729bfa0

  5. Handling remote errors with async calls:
     The code in jobthing wasn't prepared for any remote errors and
     throw exceptions in such case. It's fixed in:
http://github.com/kadamski/func/commit/fdabc175f9ef5ee9d1e6887409be58dd44cf661d

  6. There was --forks command line parameter for 'call' cmd_module but
     it wasn't actually used. This small patch allows us to use it:
http://github.com/kadamski/func/commit/3750980ea547c73a63cb456a85be0b14cc36c30c

III. Cleanups:
  1. Jobthing status codes:
     There was JOB_ID_FINISHED and JOB_ID_ASYNC_FINISHED which was
     actually the same. NOTE: This change breaks the API. Commit:
http://github.com/kadamski/func/commit/2d23caf2f9e96022dd49781bc00c351dcab7c10c

  2. Parameters name:
     There is batch_run() function in both jobthing.py and forkbomb.py.
     They share the comments about the parameters but somehow
     jobthing.py was using different names which i think was confusing.
     Here's the patch to fix this:
http://github.com/kadamski/func/commit/58439ecd3433434b8fa31821619fec8e5bbeda27

  3. Unneeded print statement:
     Just removes one print statement:
http://github.com/kadamski/func/commit/9acec083713038365e804464cfaf7c22a405d0a0

  4. Returning unformated values from do() function (as i believe it's
     more useful):
http://github.com/kadamski/func/commit/8e96ee4101213ccaba11c95729bf88171873d03c


As always I would like to ask you all to test and/or review the code.
All comments and/or bug reports are welcome.

_______________________________________________
Func-list mailing list
Func-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/func-list
Looks good, and thanks for the detailed summary.

I merged/pushed all of the above after briefly testing on a couple of machines, async_test.py still looks good also.

Wider testing is definitely welcome (I'd be interested to know from Steve Salesvan whether your RHEL4 issues are fixed with this), as are other comments -- we'll just base changes on what's committed from now if we need to tweak anything else.

Thanks! This will definitely be useful for things like powering things like the async AJAX calls in FuncWeb too.

--Michael



_______________________________________________
Func-list mailing list
Func-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/func-list

[Index of Archives]     [Fedora Users]     [Linux Networking]     [Fedora Legacy List]     [Fedora Desktop]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite News]     [KDE Users]

  Powered by Linux