Michael DeHaan wrote:
Michael DeHaan wrote:
Krzysztof A. Adamski wrote:
I've done some initial work on async calls from command lines. This
change needs testing (as i only done limited testing since i only have
two testing minions right now) and comments.
You can see the code online:
http://github.com/kadamski/func/commits/async
Or import my branch to your func git tree:
git remote add kadamski git://github.com/kadamski/func.git
git fetch kadamski
git checkout -b kadamski-async kadamski/async
[make/install/test and when you are done you can remove it:]
git branch -D kadamski-async
Or just clone it from github:
git clone git://github.com/kadamski/func.git
You have to update RPM only on overlord as nothing related to minion is
changed. After installing this version you will have four additional
options:
--async
--nopoll
--sport
--jobstatus
You can call any module in async mode with:
func '*' call --async iptables policy
results should be printed in the order they come back to overlord.
You can wait for all results and then print them sorted with:
func '*' call --async --sort iptables policy
If you don't want to wait for results you can run:
func '*' call --async --nopoll iptables policy
this will return JOB_ID which can be used to check for results:
func '*' call --jobstatus JOB_ID
Waiting for comments and bug reports..
_______________________________________________
Func-list mailing list
Func-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/func-list
FYI, I'm getting this on my machine:
[root@mdehaan func]# func "*" call --async command run /bin/echo 1
Traceback (most recent call last):
File "/usr/bin/func", line 12, in <module>
cli.parse(argv)
File "/usr/lib/python2.5/site-packages/func/overlord/command.py",
line 247, in parse
return self.subCommands[command].parse(args[1:])
File
"/usr/lib/python2.5/site-packages/func/overlord/cmd_modules/call.py",
line 72, in parse
return base_command.BaseCommand.parse(self, argv)
File "/usr/lib/python2.5/site-packages/func/overlord/command.py",
line 228, in parse
ret = self.do(args)
File
"/usr/lib/python2.5/site-packages/func/overlord/cmd_modules/call.py",
line 131, in do
results = self.overlord_obj.run(self.module, self.method,
self.method_args)
File "/usr/lib/python2.5/site-packages/func/overlord/client.py", line
311, in run
results = jobthing.batch_run(self.minions, process_server, nforks)
File "/usr/lib/python2.5/site-packages/func/jobthing.py", line 121,
in batch_run
results = forkbomb.batch_run(server, process_server, nforks)
File "/usr/lib/python2.5/site-packages/func/forkbomb.py", line 139,
in batch_run
__forkbomb(0,buckets,callback,shelf_file)
File "/usr/lib/python2.5/site-packages/func/forkbomb.py", line 113,
in __forkbomb
__with_my_bucket(mybucket,buckets,what_to_do,filename)
File "/usr/lib/python2.5/site-packages/func/forkbomb.py", line 90, in
__with_my_bucket
things_in_my_bucket = buckets[bucket_number]
KeyError: 0
I saw you made a change earlier in the git history WRT forkbomb.py,
maybe that was it.
FWIW, bucket number in forkbomb wasn't it. Still looking...
There's also some noise in "ping"'s output, which may have been
something I added earlier:
[root@mdehaan func]# func "*" ping
results 1 <type 'int'>
[ ok ... ] mdehaan.rdu.redhat.com
I only have one minion in my configuration.
_______________________________________________
Func-list mailing list
Func-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/func-list
_______________________________________________
Func-list mailing list
Func-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/func-list
Further feedback.
This appears to work for me now noting changes in git.
One thing we need to do is clean up the extra output from running
commands and make error messages more human readable to run programmers.
These are things that are not neccessarily async related but would be
good to have shown in simple text
[root@mdehaan mdehaan]# func "*" call --async command run "/bin/echo
hello world"
Traceback (most recent call last):
File "/usr/bin/func", line 12, in <module>
cli.parse(argv)
File "/usr/lib/python2.5/site-packages/func/overlord/command.py", line
247, in parse
return self.subCommands[command].parse(args[1:])
File
"/usr/lib/python2.5/site-packages/func/overlord/cmd_modules/call.py",
line 72, in parse
return base_command.BaseCommand.parse(self, argv)
File "/usr/lib/python2.5/site-packages/func/overlord/command.py", line
228, in parse
ret = self.do(args)
File
"/usr/lib/python2.5/site-packages/func/overlord/cmd_modules/call.py",
line 146, in do
(return_code, async_results) = self.overlord_obj.job_status(results)
File "/usr/lib/python2.5/site-packages/func/overlord/client.py", line
244, in job_status
return jobthing.job_status(jobid, client_class=Overlord)
File "/usr/lib/python2.5/site-packages/func/jobthing.py", line 179, in
job_status
(minion_interim_rc, minion_interim_result) = minion_result
ValueError: too many values to unpack
[root@mdehaan mdehaan]# vi /var/log/func/func.log
[root@mdehaan mdehaan]# func "*" ping
results ['REMOTE_ERROR', 'socket.error', "(111, 'Connection refused')",
' File "/usr/lib/python2.5/site-packages/func/overlord/client.py", line
286, in process_server\n retval = getattr(conn, meth)(*args[:])\n File
"/usr/lib/python2.5/xmlrpclib.py", line 1150, in __call__\n return
self.__send(self.__name, args)\n File "/usr/lib/python2.5/xmlrpclib.py",
line 1440, in __request\n verbose=self.__verbose\n File
"/usr/lib/python2.5/xmlrpclib.py", line 1186, in request\n
self.send_content(h, request_body)\n File
"/usr/lib/python2.5/xmlrpclib.py", line 1300, in send_content\n
connection.endheaders()\n File "/usr/lib/python2.5/httplib.py", line
856, in endheaders\n self._send_output()\n File
"/usr/lib/python2.5/httplib.py", line 728, in _send_output\n
self.send(msg)\n File "/usr/lib/python2.5/httplib.py", line 695, in
send\n self.connect()\n File
"/usr/lib/python2.5/site-packages/func/SSLCommon.py", line 106, in
connect\n self.sock.connect((self.host, self.port))\n File "<string>",
line 1, in connect\n'] <type 'list'>
[ FAILED ] drwily.rdu.redhat.com
results 1 <type 'int'>
[ ok ... ] mdehaan.rdu.redhat.com
[root@mdehaan mdehaan]# func "*" ping
results 1 <type 'int'>
[ ok ... ] drwily.rdu.redhat.com
results 1 <type 'int'>
[ ok ... ] mdehaan.rdu.redhat.com
The "results 1" line is also something we'd want to clean up.
Otherwise, this appears to work for me.
The other thing is that (as discussed on IRC), the --jobstatus results
probably should also be human readable.
It's probably worth looking at making Func's CLI in general more human
friendly across the board before we do too much more in terms of adding
features. It's just little stuff here and there but would go a long way
towards improving someone's first experiences when looking at it.
Otherwise, seems to work well.
I ran some tests on async using a couple of machines and the test
module, and it all works well:
[root@mdehaan mdehaan]# func "*" call --async test sleep 10
('mdehaan.rdu.redhat.com', 1212506789.5311799)
('drwily.rdu.redhat.com', 1212521141.090343)
[root@mdehaan mdehaan]# func "*" call --async test sleep 10
func "*" call --jobstatus ('drwily.rdu.redhat.com', 1212521154.206342)
('mdehaan.rdu.redhat.com', 1212506802.6591611)
[root@mdehaan mdehaan]# func "*" call --async --nopoll test sleep 10
JOB_ID: 1212506809.239054
[root@mdehaan mdehaan]# func "*" call --jobstatus 1212506809.239054
(3, {})
[root@mdehaan mdehaan]# func "*" call --jobstatus 1212506809.239054
(4,
{'drwily.rdu.redhat.com': 1212521171.090343,
'mdehaan.rdu.redhat.com': 1212506819.521163})
Again, the one concern I still have is that the output is programmer
centric across the board and for sysadmins who are not writing Python
that's probably something we can improve.
Otherwise, very cool to see this in the command line. The next probable
step is to teach module methods when they should run async by default, I
think. This is something that could leverage what Denis is doing with
the return descriptors. The command line could see if the modue surfaced
a registration function, and if so, could see in advance if a method
needed to be called async. The output would then, to the user, look
exactly the same and he would not have to remember to use the --async flag.
Thoughts?
--Michael
_______________________________________________
Func-list mailing list
Func-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/func-list
