Re: python modul pre-import to avoid importing each time

[Rémi Cura <remi.cura@xxxxxxxxx>]

Hey,
thanks for your answer !

Yep you are right, the function I would like to test are going to be called a lot (100k times), so even 15 ms per call matters. 

I'm still a bit confused by a topic I found here : http://stackoverflow.com/questions/15023080/how-are-import-statements-in-plpython-handled

The answer gives a trick to avoid importing each time, so somehow it must be usefull.

On another internet page (can't find it anymore) somebody mentioned this module loading at server startup, one way or another, but gave no precision. It seems that the "plpy" python module get loaded by default, would'nt it be possible to hack this module to add other import inside it?

I also use PL/R (untrusted I guess) and you can create a special table to indicate which module to load at startup.

Cheers,
Rémi-C

2014-06-25 21:46 GMT+02:00 Jeff Janes <jeff.janes@xxxxxxxxx>:

On Thu, Jun 19, 2014 at 7:50 AM, Rémi Cura <remi.cura@xxxxxxxxx> wrote:
> Hey List,
>
> I use plpython with postgis and 2 python modules (numpy and shapely).
> Sadly importing such module in the plpython function is very slow (several
> hundreds of milliseconds).

Is that mostly shapely (which I don't have)?  numpy seems to be pretty
fast, like 16ms.  But that is still slow for what you want, perhaps.

>
> I also don't know if this overhead is applied each time the function is
> called in the same session.

It is not.  The overhead is once per connection, not once per call.
So using a connection pooler could be really be a help here.

> Is there a way to pre-import those modules once and for all,
> such that the python function are accelerated?

I don't think there is.  With plperl you can do this by loading the
module in plperl.on_init and by putting plperl into
shared_preload_libraries so that this happens just at server start up.
But I don't see a way to do something analogous for plpython due to
lack of plpython.on_init.  I think that is because the infrastructure
to do that is part of making a "trusted" version of the language,
which python doesn't have.  (But it could just be that no one has ever
gotten around to adding it.)

Cheers,

Jeff