On 05/09/2014 05:36 AM, Vincent de Phily wrote:
On Friday 09 May 2014 07:01:32 Tom Lane wrote:
Vincent de Phily <vincent.dephily@xxxxxxxxxxxxxxxxx> writes:
In case it changes anything, this is the uncut (but still anonimized)
function:
query = """UPDATE foo SET processing = 't' WHERE id IN
(SELECT id FROM foo WHERE processing = 'f' ORDER BY id ASC
LIMIT %d
FOR UPDATE)
RETURNING *""" % (conf_getint('DEFAULT', 'push_count', 5000),)
Well, of course this view of things exposes a relevant failure mode
you hadn't mentioned: maybe sometimes the conf_getint() call returns
something other than 5000?
True. But I've commented already that I'd be very surprised (and wouldn't know
how to begin) if that value was faulty (even though it would explain things
nicely), because
* It is parsed once at program start (using python's ConfigParser library)
What is parsed?
* It has the correct value of 5000 in most cases (as demonstrated by the
frequency of number of rows returned)
Yes, but those are not the ones of interest.
* There is no sign that I exited the loop (and therefore got the opportunity
to change the value of the query) before I start receiving overlong results.
Not sure I follow, You are passing a function as a parameter, it
would be re-run each time the query was run in:
curs.execute(query)
FYI, the psycopg2 docs recommend you not use the parameter passing
method above as it is susceptible to SQL injection:
http://initd.org/psycopg/docs/usage.html#passing-parameters-to-sql-queries
Still, I agree it's suspicious, so I'm now logging the query string whenever I
get over 5000 results (hardcoded). We'll see next time it happens.
--
Adrian Klaver
adrian.klaver@xxxxxxxxxxx
