On Thu, Mar 24, 2011 at 14:23, Merlin Moncure <mmoncure@xxxxxxxxx> wrote: > On Thu, Mar 24, 2011 at 4:17 AM, Erik Hesselink <hesselink@xxxxxxxxx> wrote: >> Hi, >> >> We're getting a deadlock in our application (a web application with a >> PostgreSQL backend) which I've traced to libpq. I've started our >> application in gdb, and when it hangs, I've inspected the backtraces. >> I've found a couple of threads I can account for (listening for new >> connections, background processes) and 77 threads waiting for a mutex >> lock: >> >> #0 0x00007ffff523d464 in __lll_lock_wait () from /lib/libpthread.so.0 >> #1 0x00007ffff52385d9 in _L_lock_953 () from /lib/libpthread.so.0 >> #2 0x00007ffff52383fb in pthread_mutex_lock () from /lib/libpthread.so.0 >> #3 0x00007ffff6160650 in ?? () from /usr/lib/libpq.so.5 >> ==> pg_lockingcallback >> #4 0x00007ffff440b791 in ?? () from /lib/libcrypto.so.0.9.8 >> #5 0x00007ffff440bcc9 in ?? () from /lib/libcrypto.so.0.9.8 >> #6 0x00007ffff47652fb in SSL_new () from /lib/libssl.so.0.9.8 >> #7 0x00007ffff61604dc in ?? () from /usr/lib/libpq.so.5 >> ==> pqsecure_open_client >> #8 0x00007ffff61525ce in PQconnectPoll () from /usr/lib/libpq.so.5 >> #9 0x00007ffff6152f5e in ?? () from /usr/lib/libpq.so.5 >> ==> connectDBComplete >> #10 0x00007ffff6153c5f in PQconnectdb () from /usr/lib/libpq.so.5 >> #11 0x0000000000f9b518 in sccR_info () >> #12 0x0000000000000000 in ?? () >> >> So it seems everything is waiting for a lock on a mutex from >> pq_lockarray (in fe-secure.c@846). Does anybody have any idea how this >> can happen? Is this something we're doing wrong (I hope so) or a bug >> in libpq? >> >> Some background: this happens only after a couple of thousand requests >> (each doing about 15 database calls), with occasional other requests >> coming in at the same time. Our server uses a Haskell binding to libpq >> (HDBC [1] and HDBC-postgresql [2]). Both client and server run on the >> same machine, running 64bit Ubuntu 10.04. The database version is >> "PostgreSQL 8.4.7 on x86_64-pc-linux-gnu, compiled by GCC gcc-4.4.real >> (Ubuntu 4.4.3-4ubuntu5) 4.4.3, 64-bit". I'm not sure how to determine >> the libpq version, but it is the most recent that comes with this >> ubuntu. The changelogs for Ubuntu suggest 8.4.7 as well. Connections >> are via TCP/IP to 127.0.0.1 with SSL turned on. The machine is under >> some CPU load when this happens. There is plenty of free memory. >> >> When I turned off SSL or connect via domain sockets, we got different >> errors that are possibly related: occasionally, the connection between >> client (our app) and server (database) is lost. On the client, we get: >> >> connectPostgreSQL: server closed the connection unexpectedly >> This probably means the server terminated abnormally >> before or while processing the request. >> >> and on the server: >> >> could not send data to client: Broken pipe >> >> There is no further context around these messages. >> >> Any help would be greatly appreciated. > > How did you initialize ssl? You are waiting inside a lock that is > getting set up inside the crypto library. Unless you are having some > type of library initialization issue, I'm suspicious the problem is > really inside libpq. Is your application multithreaded, and if so are > you properly synchronizing access to the connection object, etc? What do you mean exactly with "How did you initialize ssl"? I found [1], which I did not know about. This seems to be a very non-local problem: if one of our dependencies initializes ssl, and I use libpq as well, this will go wrong. I've done a quick look through all our dependencies, and none seem to use libcrypto or libssl. Our application is definitely multithreaded, as it is a web application. But every database transaction creates a new connection object. They are never shared between threads. The problem is very hard to reproduce. I've taken all queries that were performed when I last reproduced it, and have only those queries (and inserts/updates) running in two concurrent loops, but so far, that hasn't reproduced the problem yet. A couple of hours of running our application with a script performing requests against it can reproduce it, though. If this seems to be a problem inside libpq, should I create a bug report? I'm hesitant, as I don't have any steps to reproduce. -- Erik Hesselink http://silkapp.com [1] http://www.postgresql.org/docs/8.4/static/libpq-ssl.html#LIBPQ-SSL-INITIALIZE -- Sent via pgsql-general mailing list (pgsql-general@xxxxxxxxxxxxxx) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
