Re: Funcweb Async and layout Changes

Michael DeHaan <mdehaan@xxxxxxxxxx> · Fri, 25 Jul 2008 14:07:10 -0400

makkalot@xxxxxxxxx wrote:
On Friday 25 July 2008 08:40:00 pm you wrote:

makkalot@xxxxxxxxx wrote:

Hi, thanks for testing it :)

I just pulled down the latest "makkalot_newlayout" branch and tried this
out.   It's very very nice, good work!

I noticed that when I did not have funcd running and click on a minion,
instead of detecting the error, it tries to show each part of the error
message as a seperate method.  You can test this by stopping funcd and
trying to click on that minion.    This should be pretty easy to fix.

I think to add a global error handling to funcweb so that kind of errors
wont be problem anymore.

Some of the things I would change are very simple string related things:
-- "Methods" at the top should be "Minions"
--  "localhost.localdoman.echo.run_float" could just be "run_float"

++

I think after this we should proably try to figure out how to make the
result formatting look a bit less like Python for users, and we also
still need to instrument the other modules so they show up in the web
user interface.

Probably firstly will instrument the remaining methods to see what kind
of outputs we have. Therefore i can make some generalization about output
formatting.

Sounds good to me.

One question I had is when I log in and click on a method, it's always
going to target all of the minions in the glob, right?   Do you think
there might be some way we could also put checks by each minion we want
to reach using a checkbox, or possibly interface with the "groups"
support in Func?    It might be nice to have a tab in FuncWeb that would
allow adding systems to groups, possibly.

System groups idea is interesting but dont think we need some some
checkbox thing,because we already have the glob button so may control
exactly which minion you want to use.

If groups show up in the glob, then that would be sufficient.

For instance, if a group is named "lab4", and you type in "lab4" and see
the minions for lab4 show up in the box, that works.     One potential
concern is what happens when you have a thousand minions.  In this case,
it might be best to make that region an iframe if you haven't done so
already, so that we could scroll it.

Now it expands, the scroll will be better, easy to fix.

So let me repeat to see if i understand the idea, we will have a new tab which 
will let us to create,delete,edit groups of minions. For example, on groups 
tab i create a new minion group called "lab4". After that on the main page if 
i enter "lab4" on the glob field should see that machines of that group and 
will be able to send commands to them. Right ?

Yes, I think that would be great.   Adrian already had added an API so 
groups work, and there is code in the overlord to manipulate them, so 
the webapp should be able to use the groups API to edit the groups.

The globbing should already make them resolve, if I recall correctly, so 
if you create a group now (with the config file) it should already show 
up in your box.  It would probably be worthwhile testing this.
I saw the new "async results" tab, but when I click on it, I get a
rather long traceback.    Is everything using async with polling now or
is it non-async?    I have included the traceback below.    Let me know
what you think and how to specify in the WebUI that something should be
async.  I think it is probably reasonable for all things to be async,
really, and just poll for them -- though wasn't sure what you were
doing.   This also ought to be rather easy to fix.

Yep i use async calls for all commands we have, except
list_minions,list_methods and others of course
That is what im doing :
	-send the async call
	-wait 3 sec
	- poll for result
		if ---> status of job is FINISHED show the result immediately
		else --> you will be notified when have some result for that command
(poll here again)

To test the current async notification you can try the
service.get_running or get_enabled methods they have some time to
complete .

Sounds good.  When do they not poll?

The javascript code polls server side on every 50 secs, if we dont have any 
RUNNING or PARTIAL jobs no polling is made so we have no extra traffic. 

As long as it's possible to start another job and not have to wait for 
the results to come back that should
be ok.

For instance, if you start a yum update and then want to go do something 
else, and then just want to go back later
and look at the results.

When you have the async results stuff fixed to tolerate job id's it 
doesn't recognize I'll check out that view.

So, other than the traceback below and the error I get when clicking on
a minion that is not running, things look pretty good.  I think I'm
interested in seeing what the async results would look like, as well as
getting the output formatting printed in a way that would make
non-Python developers have an easier time reading it, but looks good.

Comments?  Thoughts?

Everyone else on this list, I would definitely appreciate your feedback
also as this is a tool intended for you too :)

Thanks!

--Michael

Here is the traceback I got when clicking on async_results, probably
because I didn't have any async tasks (as I didn't know how to launch
one in the GUI):

Page handler: <bound method Funcweb.display_async_results of
<funcweb.controllers.Funcweb object at 0x9d8324c>> Traceback (most
recent call last):
  File
"/usr/lib/python2.5/site-packages/CherryPy-2.3.0-py2.5.egg/cherrypy/_cph
ttp tools.py", line 121, in _run self.main()
  File
"/usr/lib/python2.5/site-packages/CherryPy-2.3.0-py2.5.egg/cherrypy/_cph
ttp tools.py", line 264, in main body = page_handler(*virtual_path,
**self.params)
  File "<string>", line 3, in display_async_results
  File "/usr/lib/python2.5/site-packages/turbogears/controllers.py",
line 365, in expose *args, **kw)
  File "<string>", line 5, in run_with_transaction
  File "/usr/lib/python2.5/site-packages/turbogears/database.py", line
356, in so_rwt retval = func(*args, **kw)
  File "<string>", line 5, in _expose
  File "/usr/lib/python2.5/site-packages/turbogears/controllers.py",
line 380, in <lambda> mapping, fragment, args, kw)))
  File "/usr/lib/python2.5/site-packages/turbogears/controllers.py",
line 421, in _execute_func return _process_output(output, template,
format, content_type, mapping, fragment) File
"/usr/lib/python2.5/site-packages/turbogears/controllers.py", line 87,
in _process_output fragment=fragment)
  File "/usr/lib/python2.5/site-packages/turbogears/view/base.py", line
129, in render return engine.render(**kw)
  File "/usr/lib/python2.5/site-packages/genshi/template/plugin.py",
line 104, in render return self.transform(info,
template).render(**kwargs) File
"/usr/lib/python2.5/site-packages/genshi/core.py", line 154, in render
return encode(generator, method=method, encoding=encoding) File
"/usr/lib/python2.5/site-packages/genshi/output.py", line 45, in encode
output = u''.join(list(iterator))
  File "/usr/lib/python2.5/site-packages/genshi/output.py", line 369, in
__call__ for kind, data, pos in stream:
  File "/usr/lib/python2.5/site-packages/genshi/output.py", line 618, in
__call__ for kind, data, pos in stream:
  File "/usr/lib/python2.5/site-packages/genshi/output.py", line 679, in
__call__ for kind, data, pos in chain(stream, [(None, None, None)]):
  File "/usr/lib/python2.5/site-packages/genshi/output.py", line 459, in
__call__ for ev in stream:
  File "/usr/lib/python2.5/site-packages/genshi/core.py", line 212, in
_ensure for event in stream:
  File "/usr/lib/python2.5/site-packages/genshi/template/markup.py",
line 252, in _include for event in stream:
  File "/usr/lib/python2.5/site-packages/genshi/template/markup.py",
line 322, in _match content = list(self._include(content, ctxt))
  File "/usr/lib/python2.5/site-packages/genshi/template/markup.py",
line 252, in _include for event in stream:
  File "/usr/lib/python2.5/site-packages/genshi/template/markup.py",
line 297, in _match for event in stream:
  File "/usr/lib/python2.5/site-packages/genshi/template/markup.py",
line 286, in _strip event = stream.next()
  File "/usr/lib/python2.5/site-packages/genshi/template/markup.py",
line 242, in _exec for event in stream:
  File "/usr/lib/python2.5/site-packages/genshi/template/base.py", line
421, in _eval result = data.evaluate(ctxt)
  File "/usr/lib/python2.5/site-packages/genshi/template/eval.py", line
136, in evaluate return eval(self.code, _globals, {'data': data})
  File
"/home/mdehaan/cg/func/funcweb/funcweb/templates/async_table.html", line
26, in <Expression u'job_pack[4]'> <td class="tableboxes"><p
class="intabletext">${job_pack[4]}</p></td> File
"/usr/lib/python2.5/site-packages/genshi/template/eval.py", line 281, in
lookup_item return obj[key]
IndexError: list index out of range

I didnt have any async results and couldnt reproduce that just have an
empty table. I think the problem maybe if you have some older async
results (with old job_ids) in your db so they can not be parsed according
to new ones.

This is likely, can you make it ignore those so it does not happen to
users who upgrade?  Old id's should
be cleaned out, but if you changed the job id storage format, such that
they are no longer timestamps, this may no longer be true.
We need to make sure that old job id's are removed automatically.

I run purge_old_jobs before do any async operations, but should also remove 
that old ones .

Good.
glob-module-method-time_stamp ? Another reason maybe to have old version
job_ids on minions. I think we should merge that new job_ids to master
tree to not have that kind of errors if it is ok for you of course :)

It needs to be tested, so sure.   I think you could merge all of the
above as well.

Ok

Question -- if I am using Funcweb and start a long-running task (say yum
upgrade on all systems) and want to do another task, can I do that?

You can submit as much commands as you want one after another without waiting 
previous ones to finish, it is free :)

Excellent.

--Michael

_______________________________________________
Func-list mailing list
Func-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/func-list