Hi all,
I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7
install, the other is fresh Rocky 8. I installed httpd (on Centos 7
it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to
point to the same static default html file and enabled mpm event on
Centos (mpm event is default on Rocky). Then I added following options
to default config on both servers:
```
<IfModule mpm_event_module>
ThreadsPerChild 25
StartServers 3
ServerLimit 120
MinSpareThreads 75
MaxSpareThreads 3000
MaxRequestWorkers 3000
MaxConnectionsPerChild 0
</IfModule>
```
After this is done I performed ab tests with keepalive using different
Centos 7 VM in the same local network. Can you help me understand
these results? On Centos I am able to complete 1 milion requests at
1000 concurrent connections with little to no errors, however with
version 2.4.37 on Rocky I get a lot of failed requests due to length
and exceptions. Served content is static so I am assuming this is
because keepalive connections are closed by the server. I tried
various configurations of KeeAaliveTimeout, MaxKeepAliveRequests,
AsyncRequestWorkerFactor and other mpm_event options but I got no
better results. The only thing that seems to improve the stability of
keepalive connections is setting threads and servers to the moon for
example:
```
<IfModule mpm_event_module>
ThreadsPerChild 50
StartServers 120
ServerLimit 120
MinSpareThreads 6000
MaxSpareThreads 6000
MaxRequestWorkers 6000
MaxConnectionsPerChild 0
</IfModule>
```
However it stills throws more errors (~8k) than 2.4.6 on the first set
of settings. This problem occurs only when using keepalive. There are
no errors when using ab without -k option, although speed is lower. I
can replicate this issue on newest httpd build from source (2.4.57).
What is causing this difference of behavior? How can I achieve
performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing much more
resources at httpd? These errors seems to be a root cause of a problem
we have with 502 errors thrown from downstream reverse proxy server
when httpd kills keepalive connection prematurely and the proxy is
trying to use this connection.
Below results of ab tests:
Centos 7 VM:
```
ab -k -t 900 -c 1000 -n 1000000 http://centos/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.1.3.3 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests
Server Software: Apache/2.4.6
Server Hostname: 10.1.3.3
Server Port: 80
Document Path: /
Document Length: 7620 bytes
Concurrency Level: 1000
Time taken for tests: 15.285 seconds
Complete requests: 1000000
Failed requests: 67
(Connect: 0, Receive: 0, Length: 67, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 990567
Total transferred: 7919057974 bytes
HTML transferred: 7619489460 bytes
Requests per second: 65422.95 [#/sec] (mean)
Time per request: 15.285 [ms] (mean)
Time per request: 0.015 [ms] (mean, across all concurrent requests)
Transfer rate: 505945.41 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 4.7 0 1042
Processing: 3 15 16.8 13 467
Waiting: 0 14 16.2 13 433
Total: 3 15 17.7 13 1081
Percentage of the requests served within a certain time (ms)
50% 13
66% 15
75% 16
80% 17
90% 21
95% 23
98% 32
99% 44
100% 1081 (longest request)
```
Rocky 8 VM:
```
ab -k -t 900 -c 1000 -n 1000000 http://rocky/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.1.3.11 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests
Server Software: Apache/2.4.37
Server Hostname: 10.1.3.11
Server Port: 80
Document Path: /
Document Length: 7620 bytes
Concurrency Level: 1000
Time taken for tests: 19.101 seconds
Complete requests: 1000000
Failed requests: 93159
(Connect: 0, Receive: 0, Length: 85029, Exceptions: 8130)
Write errors: 0
Keep-Alive requests: 912753
Total transferred: 7248228337 bytes
HTML transferred: 6973694460 bytes
Requests per second: 52352.12 [#/sec] (mean)
Time per request: 19.101 [ms] (mean)
Time per request: 0.019 [ms] (mean, across all concurrent requests)
Transfer rate: 370566.51 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 4 54.4 0 3022
Processing: 0 15 20.6 13 1400
Waiting: 0 13 17.7 12 1400
Total: 0 19 59.0 13 3048
Percentage of the requests served within a certain time (ms)
50% 13
66% 15
75% 17
80% 19
90% 24
95% 41
98% 53
99% 106
100% 3048 (longest request)
```
---
Mateusz Kempski
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|