Re: [users@httpd] Apache 2 chaos

"William A. Rowe, Jr." <wrowe@xxxxxxxxxxxxx> · Tue, 18 Jul 2006 11:15:07 -0500

Ok, if Apache is truly hanging, there is a clever trick (that works on all
versions of Apache).

1) Unzip the -symbols.zip files in the root Apache2 directory tree (where
   bin\, htdocs\, modules\ etc. are usually found.)  These .pdb files should
   unpack alongside the .exe, .dll, .so binary files they represent, e.g.,
   mod_usertrack.pdb will unpack alongside mod_usertrack.so.

2) Invoke drwtsn32 and ensure you are creating a crash dump file, you are
   dumping all thread contexts, your log and crash dump paths make sense, and
   (depending on the nature of the bug) you pick an appropriate crash dump
   type.  (Full is quite large, but necessary sometimes for a programmer-type
   to load your crash dump into a debugger and begin unwinding exactly what
   has happened.  Mini is sufficient for your first pass through the process.)

3) Note that if you previously installed and then uninstalled other debugging
   software, you may need to invoke   drwtsn32 -i   in order to make Dr Watson
   your default crash dump tool.  This will replace the 'report problem to MS'
   dialogs.  Don't do this if you have a full debugger such as Visual Studio
   or windbg installed on the machine.

3) Invoke the Task Manager, Choose 'show processes from all users', and modify
   the View -> Select Columns to include *at least* the PID and Thread Count.
   You can change this just once and Task Manager should keep your preference.

4) Now, track down the errant Apache that is hanging.  The parent process has
   about three threads, we don't care about that one.  The child worker process
   we want has many more threads (a few more than you configured with the
   ThreadsPerChild directive.)  The process name is Apache (for 1.3 and 2.0)
   or httpd (for 2.2).  Make note of the child worker's PID.

5) Using the {pid} number you noted above, invoke the command

   drwtsn32 -p {pid}

Voila, you will find in your 'log file path' a drwtsn32.log file, and if you
choose to 'append to existing log file', jump through the 'App:' sections until
you find the one for the process you just killed.  Now you can provide feedback
about where 'Stack Back Trace' points to for the developer-oriented users here.

Whatever you don't don't send the whole thing to this list!!!  You will note
that many threads look identical, almost all of them polling for the next
connection, and we don't care to see those.  We want to see the ones that are
deep inside of a request at the time you kill them, and only the stack back
trace entries for those.  This can give folks a clue of where that request is
hanging, which handler module picked up the request, and what filter it might
be stuck in.

Best of luck!

Oliver Marshall wrote: