Just an update for those who may experience the same fcntl deadlock issue in Solaris in the future. We used prefork MPM initially and did not see it happening. When we started using Worker MPM, that is when we saw it happening. Looks like it is specific to the threaded model. I have updated the global mutex of Apache (AcceptMutex) from the default of fcntl to pthread and this problem is no longer occuring. Alternatively, from what I know, changing the SSLMutex itself from fcntl to pthread may also do the same job (we are making sure that there is only one set of fcntl running). This may not be an easy fix for everyone as some "other" module may use fcntl by default (and not changeable). The lucky thing for us is that fcntl is only used for the global mutex and ssl (both of which are changeable). This issue will occur only under heavy load (as heavy as the machine can sustain), and is specific to Solaris when using fcntl for locking, in a Worker MPM. We hit the tuned apache process on a T2000 server with 500 users (without ramp-up) and got only 3 deadlock ... Why is the deadlock bad? Well, as you get more and more deadlock, you can end up with Apache processes having to respawn itself continually. With Apache processes that respawns non stop and too quickly, the load will spike and the machine grinds to a halt... -----Original Message----- From: Ong, Lawrence K [mailto:Lawrence.K.Ong@xxxxxxxxxxxxxxxx] Sent: Friday, 12 December 2008 10:52 AM To: users@xxxxxxxxxxxxxxxx Subject: RE: Apache 2.2.10 - Solaris 10 - Under Load - Failed to acquire SSL session cache lock Ok, correct me if I got this wrong. This issue seems to boil down to multiple fcntl use in a threaded fashion on Solaris. If we can ensure that if we only ever use fcntl in one location, then it seems that it would not be an issue. Alternatively, we can try and ensure that we just do not use the default mutex of fcntl. Is this correct? -----Original Message----- From: Eric Covener [mailto:covener@xxxxxxxxx] Sent: Friday, 12 December 2008 10:28 AM To: users@xxxxxxxxxxxxxxxx Subject: Re: Apache 2.2.10 - Solaris 10 - Under Load - Failed to acquire SSL session cache lock On Thu, Dec 11, 2008 at 6:17 PM, Ong, Lawrence K <Lawrence.K.Ong@xxxxxxxxxxxxxxxx> wrote: > Hi All, > > We are currently getting a lot of warnings in our logs in regards to > deadlock, and I am wondering if anyone have experienced seen the following. > > Under load, we would get: > > [Fri Dec 12 08:50:23 2008] [warn] (45)Deadlock situation detected/avoided: > Failed to acquire SSL session cache lock [Fri Dec 12 08:50:39 2008] > [warn] (45)Deadlock situation detected/avoided: > Failed to acquire SSL session cache lock > > We have already set up SSLMutex and SSLSessionCache with the following: > > SSLSessionCache shm:/usr/local/apache2/logs/ssl_gcache_data(512000) > SSLMutex file:/usr/local/apache2/logs/ssl_mutex Look for a thread titled "fcntl based mutex on Solaris/EM64TLink to this message" on google. It's not resolved because there is no reasonable default (AIUI). You might be able to change the AcceptMutex and it would get you down to only 1 fcntl-based mutex. -- Eric Covener covener@xxxxxxxxx --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|