Issue while generating large documents (Apache Users)

Hi All,

We are running in to issue while generating the large documents from our Java application. The Application is deployed in WebLogic server 12.2.1.3.0 version

Apache (2.4.39) is proxying the Backend weblogic servers via WLS Plugin and an Load Balancer is sitting in front of Apache servers. It’s an clustered environment with 3 APP servers and 2 web servers.

When we are generating an larger document (invoice)which contains more than 150 + pages then the session is getting invalidate and below are the errors recorded in the logs files.

Please note that when we are generating the same document with the direct application URL we are not seeing any issue. So I suspect there is something timing out between web(WLS Plugin) and app server.

Any suggestions please. Also we recently hit the following bug for another application for which we have took the work around by disabling the mod_reqtimeout module. Tried same for this issue as well but that did not help.

Web server log :

[Thu Aug 22 01:50:49.613519 2019] [weblogic:error] [pid 15791:tid 140550548170496] [client 192.168.72.3:46983] <15791156646356926> Write to the client failed: calling URL::close at line 587 of BaseProxy.cpp, referer: https://v3.ufcwtrust.com/v3prod/app?service=external/Documents:CreateDocumentPopup&sp=SPAGE&sp=SContributions%3ATransactions&sp=l13443&sp=l7017&sp=S10184813

[Thu Aug 22 01:50:49.613641 2019] [weblogic:error] [pid 15791:tid 140550548170496] [client 192.168.72.3:46983] <15791156646356926> *******Exception type [WRITE_ERROR_TO_CLIENT] raised at line 588 of BaseProxy.cpp, referer: https://xyz/v3prod/app?service=external/Documents:CreateDocumentPopup&sp=SPAGE&sp=SContributions%3ATransactions&sp=l13443&sp=l7017&sp=S10184813

[Thu Aug 22 01:50:49.613892 2019] [weblogic:error] [pid 15791:tid 140550548170496] [client 192.168.72.3:46983] ap_proxy: trying POST /v3prod/app at backend host 10.72.72.5/7003, client 192.168.72.3/46983, total tries 1; got exception 'WRITE_ERROR_TO_CLIENT [os error=0, line 588 of BaseProxy.cpp]: '; state: writing response body to client; not failing over, referer: https://xyz/v3prod/app?service=external/Documents:CreateDocumentPopup&sp=SPAGE&sp=SContributions%3ATransactions&sp=l13443&sp=l7017&sp=S10184813

[Thu Aug 22 01:50:49.614073 2019] [weblogic:error] [pid 15791:tid 140550548170496] [client 192.168.72.3:46983] <15791156646356926> request [/v3prod/app] did NOT process successfully.................., referer: https://xyz/v3prod/app?service=external/Documents:CreateDocumentPopup&sp=SPAGE&sp=SContributions%3ATransactions&sp=l13443&sp=l7017&sp=S10184813

[Thu Aug 22 01:52:40.609230 2019] [core:warn] [pid 15788:tid 140550969862112] AH00045: child process 15874 still did not exit, sending a SIGTERM

[Thu Aug 22 01:52:42.611458 2019] [core:warn] [pid 15788:tid 140550969862112] AH00045: child process 15874 still did not exit, sending a SIGTERM

App Server Log :

**********************************************************

Exception during post-request cleanup.

Session id: kwu4gBsqTRK188S_h79LFZ-Abn2CeYya1gRJ9NpYRibKXNUa-he9!137202692!1566463499050

Client address: 192.168.72.3

Exceptions:

java.io.IOException: Broken pipe

sun.nio.ch.FileDispatcherImpl.write0(Native Method)

sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)

sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)

sun.nio.ch.IOUtil.write(IOUtil.java:65)

sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)

java.nio.channels.Channels.writeFullyImpl(Channels.java:78)

java.nio.channels.Channels.writeFully(Channels.java:98)

java.nio.channels.Channels.access$000(Channels.java:61)

java.nio.channels.Channels$1.write(Channels.java:174)

weblogic.socket.JSSEFilterImpl.writeToNetwork(JSSEFilterImpl.java:829)

weblogic.socket.JSSEFilterImpl.wrapAndWrite(JSSEFilterImpl.java:789)

weblogic.socket.JSSEFilterImpl.write(JSSEFilterImpl.java:503)

weblogic.socket.JSSESocket$JSSEOutputStream.write(JSSESocket.java:148)

weblogic.servlet.internal.ChunkOutput.writeChunkTransfer(ChunkOutput.java:625)

weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:587)

weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:471)

weblogic.servlet.internal.CharsetChunkOutput.flush(CharsetChunkOutput.java:318)

weblogic.servlet.internal.ChunkOutputWrapper.flush(ChunkOutputWrapper.java:207)

weblogic.servlet.internal.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:153)

org.apache.tapestry.request.ResponseOutputStream.forceFlush(ResponseOutputStream.java:149)

org.apache.tapestry.engine.AbstractEngine.service(AbstractEngine.java:940)

org.apache.tapestry.ApplicationServlet.doService(ApplicationServlet.java:197)

com.vitechinc.core.VitechApplicationServlet.doService(VitechApplicationServlet.java:321)

org.apache.tapestry.ApplicationServlet.doPost(ApplicationServlet.java:326)

javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)

weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)

weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)

weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)

weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)