Re: How to debug 70014 and 70007 errors

[ivan Gouin <gouin.ivan@xxxxxxxxx>]

Hi,

Here's more information about my issue:

Here i will call 

user : the application  who post the request.

apache : the httpd server who receive request from the client and send them to tomcat

tomcat: the web server tomcat, a Web Service application

Versions
user : Apache CXF client 2.2.9 

apache: httpd 2.2.17 (tried a 2.2.22 too)
tomcat:  6.0.26 (jdk1.6.0_24) 

Here's the post request send by user ( * are for anonymise)

POST /***/***/ws/v2?test HTTP/1.1

Content-Type: text/xml; charset=UTF-8

SOAPAction: ""

Authorization: Basic Z3ZkMHRhb2FwcDpkbmVlc3cyYQ==

Accept: */*

User-Agent: Apache CXF 2.2.9

Content-Length: 304

Host: ***

Connection: Keep-Alive

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">************************</soap:Envelope>

Start at 11:56:57

Between 11:56:57.5357 and 11:56:58:4784: 4 request pass OK

Here's the sequence after that : (Time is of tomcat seem too be a little shifted, user and apache are on the same host)

11:56:58.4817            user : POST request

11:56:58.459722 tomcat TCP 551 50300 > 41323 [PSH, ACK] Seq=3187 Ack=3818 Win=22400 Len=485 TSval=345704305 TSecr=749474400 (HTTP 200)

11:56:58.491981 apache TCP 551 50300 > 41323 [PSH, ACK] Seq=3187 Ack=3818 Win=22400 Len=485 TSval=345704305 TSecr=749474400 (HTTP 200) 

11:56:58.492251 user HTTP/XML 594 HTTP/1.1 200 OK 

11:56:58.501457 tomcat TCP 66 41323 > 50300 [ACK] Seq=3818 Ack=3672 Win=15872 Len=0 TSval=749474449 TSecr=345704305

11:56:58.531503 apache TCP 66 41323 > 50300 [ACK] Seq=3818 Ack=3672 Win=15872 Len=0 TSval=749474449 TSecr=345704305

11:56:58.585937 user TCP 60 45501 > 80 [ACK] Seq=2893 Ack=3948 Win=49640 Len=0

11:57:03.492499 user TCP 54 80 > 45501 [FIN, ACK] Seq=3948 Ack=2893 Win=16640 Len=0

11:57:03.494663 user TCP 60 45501 > 80 [ACK] Seq=2893 Ack=3949 Win=49640 Len=0

11:57:08.694657 user TCP 335 [TCP segment of a reassembled PDU]

11:57:08.694703 user TCP 54 80 > 45501 [RST] Seq=3949 Win=0 Len=0

11:57:08.694661 user HTTP/XML 358 POST /*/*/ws/v2?test HTTP/1.1 

11:57:08.694763 user TCP 54 80 > 45501 [RST] Seq=3949 Win=0 Len=0
11:57:18.466178 tomcat TCP 66 50300 > 41323 [FIN, ACK] Seq=3672 Ack=3818 Win=22400 Len=0 TSval=345724311 TSecr=749474449

11:57:18.498510 apache TCP 66 50300 > 41323 [FIN, ACK] Seq=3672 Ack=3818 Win=22400 Len=0 TSval=345724311 TSecr=749474449

11:57:18.508720 tomcat TCP 66 41323 > 50300 [ACK] Seq=3818 Ack=3673 Win=15872 Len=0 TSval=749494456 TSecr=345724311

11:57:18.538771 apache TCP 66 41323 > 50300 [ACK] Seq=3818 Ack=3673 Win=15872 Len=0 TSval=749494456 TSecr=345724311

What make me mad is why apache send a FIN/ACK closing the communication??

Is there a time out somewhere?

This seems to happen about 5 second after the last ACK

Or  6 second the opening of the socket ( at 11:56:57.53)

thanks for yours help

On 3 August 2012 17:32, Rainer Jung <rainer.jung@xxxxxxxxxxx> wrote:

Hi Ivan,

On 03.08.2012 17:23, ivan Gouin wrote:

Here's what i see on the tcpdump:

Got a TCP connect  3-way handshake. (SYN/SYN-ACK/ACK)
Got 7 POST request who got a return code 200

After the 7 POST, got a [ FIN, ACK] from th server.
Then RST from the server

Then the 8th request who goes in time out

Is there some kind of timeout in a tcp keep alive?

Your information is a bit to short. AFAIR we have three communication nodes, client, proxy and origin server. In addition there was a timeout happening suspected.

Your info on packets doesn't contain enough about who is communicating ("return code 200", "the server") not about the timing (what time intervals are between packets.

Regards,

Rainer

On 25 July 2012 11:29, ivan Gouin <gouin.ivan@xxxxxxxxx

<mailto:gouin.ivan@xxxxxxxxx>> wrote:

    Hi rainer,
    For case 70007,  the timeout expired, in access log , i've got a 300
    second timeout
    In the same time, tomcat's access log haven't any trace of the
    corresponding request.

    For these request, response time is about 30-100ms

    Apache is Apache/2.2.17
    Tomcat is 6.0.26 (jdk1.6.0_24)

    I'm preparing a tcpdump on each side to see if i can see something
    received by tomcat .

    Ivan


    On 25 July 2012 11:02, Rainer Jung <rainer.jung@xxxxxxxxxxx

    <mailto:rainer.jung@kippdata.de>> wrote:

        On 25.07.2012 09 <tel:25.07.2012%2009>:52, ivan Gouin wrote:

            Hi,

            I've got those error in my httpd error log:

            [Wed Jul 25 08:10:55 2012] [error] (70014)End of file found:
            proxy:
            prefetch request body failed to *.*.*.*:50300 (...) from
            ..... ()
            [Wed Jul 25 00:13:18 2012] [error] (70007)The timeout
            specified has
            expired: proxy: prefetch request body failed to  to
            *.*.*.*:50300 (...)
            from ..... ()


        Maybe the Timeout has expired?


            Those error occurs with client accessing a tomcat WS through
            mod_proxy .

            Not all the requests are rejected for today, 416 out of 2194
            got one of
            these errors.

            don't really know how to proceed to debug this error.
            thanks for your help


        Add %D to your Tomcat and Apache Access Logs. It is the response
        time in milloiseconds (Tomcat) resp. microseconds (Apache). If
        the number is e.g. slightly above 60000000 for Apache and you
        had set a timeout of 60 seconds, then you know the problem is
        that the response takes to long. You can then check Tomcats
        Access Log to see how long it actually took. If it really takes
        to long in Tomcat, then take thread dumps to analyze and switch
        to the Tomcat users mailing list.

        HTH.

        Rainer

        ------------------------------__------------------------------__---------
        To unsubscribe, e-mail: users-unsubscribe@httpd.__apache.org
        <mailto:users-unsubscribe@httpd.apache.org>

        For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx

        <mailto:users-help@httpd.apache.org>




    --
    *Ivan GOUIN**
    *

    ***Mob (Suisse**)* : +41 (0)79 94107 90

    *Mail* : gouin.ivan@xxxxxxxxx <mailto:gouin.ivan@xxxxxxxxx>




--
*Ivan GOUIN**
*

***Mob (Suisse**)* : +41 (0)79 94107 90

*Mail* : gouin.ivan@xxxxxxxxx <mailto:gouin.ivan@xxxxxxxxx>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx

--
Ivan GOUIN

Mob (Suisse) : +41 (0)79 941 07 90

Mail : gouin.ivan@xxxxxxxxx