Yesterday I said on IRC that I'm giving up.. but I couldn't. I've tried
to fix the problem with deadlocks on async jobs once again. Finally
today I've found the source of problem which I consider great success
as it took couple days!
So now I need to fix this but this seams to be another problem so let
me explain what is wrong...
If you go to func/minion/server.py, there is a FuncApiMethod class. It
is a wrapper class that calls methods (both in async and non-async
mode) and do some logging. What is most important, there is __call__()
method:
def __call__(self, *args):
self.logger.debug("(X)---------------------")
try:
rc = self.__method(*args)
except codes.FuncException, e:
self.__log_exc()
(t, v, tb) = sys.exc_info()
rc = futils.nice_exception(t,v,tb)
except:
self.__log_exc()
(t, v, tb) = sys.exc_info()
rc = futils.nice_exception(t,v,tb)
self.logger.debug("Return code for %s: %s" % (self.__name, rc))
return rc
I've found that if jobs hangs, they always hang on one of the
self.logger.debug() calls. When they are removed, all works fine.
The problem is not directly related to async calls, everyting works
fine as long as you won't call more than one job at a time. So this
definitely is concurrency problem with logger calls.
I've created test script so i can experiment on some simple script:
#!/usr/bin/python
from func import logger
import os
logger = logger.Logger().logger
class FuncApiMethod:
"""
Used to hold a reference to all of the registered functions.
"""
def __init__(self, logger, name, method):
self.logger = logger
self.__method = method
self.__name = name
def __log_exc(self):
"""
Log an exception.
"""
(t, v, tb) = sys.exc_info()
self.logger.info("Exception occured: %s" % t )
self.logger.info("Exception value: %s" % v)
self.logger.info("Exception Info:\n%s" %
string.join(traceback.format_list(traceback.extract_tb(tb))))
def __call__(self, *args):
self.logger.debug("(X--------")
try:
rc = self.__method(*args)
except codes.FuncException, e:
self.__log_exc()
(t, v, tb) = sys.exc_info()
rc = futils.nice_exception(t,v,tb)
except:
self.__log_exc()
(t, v, tb) = sys.exc_info()
rc = futils.nice_exception(t,v,tb)
self.logger.debug("Return code for %s: %s" % (self.__name, rc))
return rc
def test(i):
return i
def boom(n):
if not os.fork():
if n:
boom(n-1)
os.umask(077)
os.chdir('/')
os.setsid()
if os.fork():
os._exit(0)
print "FORK", n
for i in range(100):
m=FuncApiMethod(logger, "test-%d" % n, test)
m(i)
os._exit(0)
boom(50)
I believe this is quite simple model of what is happening in func. The
problem is.. this testing script is always working correctly.. So now
I have no idea why logger in Func has some problems.. Any ideas?
_______________________________________________
Func-list mailing list
Func-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/func-list
