They are all identical VMs. We can also reproduce this on bigger
servers. I don't think this is caused by Rocky or Ubuntu config. I can
see 2 problems during my tests.
1. httpd does not add any servers when test is running. It kills
keepalive connections and logs "all workers busy or dying" but does
not add any, maybe one at the end of the test. For example if I set
StartServers to 3 it goes up to 4 at the end of the test even though
it has ServerLimit set to 120.
2. httpd seems to not register that it has free workers even when I
set StartServers to 120. There are thousands of idle threads and it
still logs "all workers busy or dying" and kills keepalive
connections. This can be worked around by setting ThreadsPerChild and
ThreadLimit much higher and lowering StartServers/ServerLimit
respectively.
For example with following settings I can easily process over 1500
concurrent connections without errors and keepalive killing:
```
<IfModule mpm_event_module>
ThreadsPerChild 200
ThreadLimit 200
StartServers 10
ServerLimit 15
MinSpareThreads 75
MaxSpareThreads 3000
MaxRequestWorkers 3000
MaxConnectionsPerChild 0
</IfModule>
```
Why does httpd behaves this way? It seems to not be intended
considering ThreadsPerChild is limited to 64 by default by
ThreadLimit. I will try to report a bug to httpd devs as it looks like
one to me.
Test results with this config:
```
ab -k -t 900 -c 1500 -n 1000000 http://rocky/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.1.3.11 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests
Server Software: Apache/2.4.37
Server Hostname: 10.1.3.11
Server Port: 80
Document Path: /
Document Length: 7620 bytes
Concurrency Level: 1500
Time taken for tests: 13.923 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 990813
Total transferred: 7919596479 bytes
HTML transferred: 7620000000 bytes
Requests per second: 71821.69 [#/sec] (mean)
Time per request: 20.885 [ms] (mean)
Time per request: 0.014 [ms] (mean, across all concurrent requests)
Transfer rate: 555467.60 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 14.1 0 1057
Processing: 0 20 33.5 16 1483
Waiting: 0 17 20.5 15 1343
Total: 0 21 36.4 16 1483
Percentage of the requests served within a certain time (ms)
50% 16
66% 20
75% 23
80% 26
90% 27
95% 31
98% 65
99% 221
100% 1483 (longest request)
```
Mateusz Kempski
Mateusz Kempski
Linux System Administrator
XTM International Ltd.
Email: mkempski@xxxxxxxxxxxx
xtm.cloud
On Wed, 24 May 2023 at 05:57, Deepak Goel <deicool@xxxxxxxxx> wrote:
>
>
>
> On Tue, May 23, 2023 at 6:19 PM Mateusz Kempski <mkempski@xxxxxxxxxxxx.invalid> wrote:
>>
>> Ubuntu 20 idle:
>> ```
>> Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
>> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s
>> ```
>> ```
>> top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01
>> Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> MiB Mem : 16006.4 total, 15259.6 free, 251.0 used, 495.8 buff/cache
>> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem
>> ```
>> Ubuntu 20 during test:
>> ```
>> Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s
>> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s
>> ```
>> ```
>> top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16
>> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si, 0.1 st
>> MiB Mem : 16006.4 total, 14672.6 free, 358.1 used, 975.7 buff/cache
>
>
> 1. The buff/cache in Ubuntu 20 is almost twice as that of Ubuntu 18. From what I have read, this should improve the performance of Ubuntu 20 over Ubuntu 18. However, the reverse is happening.
>
> 2. The Disk write in Ubuntu 20 is less than that of Ubuntu 18. This again should improve performance of Ubuntu 20 over Ubuntu 18. Again the reverse is happening. Are you writing anything to the DISK in your test?
>
> 3. Is the hardware same for all configurations (cpu, mem, disk iops, network)?
>
> 4. I wonder if we can open 'Performance Bugs' in Ubuntu & Rocky forums.
>
>
>>
>> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15353.9 avail Mem
>>
>> ```
>> Ubuntu 18 idle:
>> ```
>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
>> ```
>> ```
>> top - 12:46:31 up 1 min, 1 user, load average: 0.56, 0.30, 0.11
>> Tasks: 226 total, 1 running, 112 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> KiB Mem : 16422552 total, 15861664 free, 204228 used, 356660 buff/cache
>> KiB Swap: 4194300 total, 4194300 free, 0 used. 15936820 avail Mem
>> ```
>> Ubuntu 18 during test:
>> ```
>> top - 12:47:07 up 1 min, 1 user, load average: 0.55, 0.31, 0.12
>> Tasks: 241 total, 1 running, 127 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 11.3 us, 17.0 sy, 0.0 ni, 68.3 id, 0.0 wa, 0.0 hi, 3.3 si, 0.1 st
>> KiB Mem : 16422552 total, 15687480 free, 329456 used, 405616 buff/cache
>> KiB Swap: 4194300 total, 4194300 free, 0 used. 15809620 avail Mem
>> ```
>> ```
>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.76 M/s
>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
>> ```
>>
>>
>> Mateusz Kempski
>>
>> On Tue, 23 May 2023 at 12:13, Deepak Goel <deicool@xxxxxxxxx> wrote:
>> >
>> > can you please post top, iotop results for both apache 18/20?
>> >
>> > On Tue, 23 May 2023, 17:33 Mateusz Kempski, <mkempski@xxxxxxxxxxxx.invalid> wrote:
>> >>
>> >> On Ubuntu 20 with apache in stock settings + my mpm event config:
>> >> ```
>> >> ab -k -t 900 -c 1000 -n 1000000 http://ubuntu20/
>> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
>> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> >> Licensed to The Apache Software Foundation, http://www.apache.org/
>> >>
>> >> Benchmarking 10.1.3.3 (be patient)
>> >> Completed 100000 requests
>> >> Completed 200000 requests
>> >> Completed 300000 requests
>> >> Completed 400000 requests
>> >> Completed 500000 requests
>> >> Completed 600000 requests
>> >> Completed 700000 requests
>> >> Completed 800000 requests
>> >> Completed 900000 requests
>> >> Completed 1000000 requests
>> >> Finished 1000000 requests
>> >>
>> >>
>> >> Server Software: Apache/2.4.41
>> >> Server Hostname: 10.1.3.3
>> >> Server Port: 80
>> >>
>> >> Document Path: /
>> >> Document Length: 10918 bytes
>> >>
>> >> Concurrency Level: 1000
>> >> Time taken for tests: 19.528 seconds
>> >> Complete requests: 1000000
>> >> Failed requests: 234371
>> >> (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032)
>> >> Write errors: 0
>> >> Keep-Alive requests: 768663
>> >> Total transferred: 8630779988 bytes
>> >> HTML transferred: 8392262634 bytes
>> >> Requests per second: 51207.31 [#/sec] (mean)
>> >> Time per request: 19.528 [ms] (mean)
>> >> Time per request: 0.020 [ms] (mean, across all concurrent requests)
>> >> Transfer rate: 431600.59 [Kbytes/sec] received
>> >>
>> >> Connection Times (ms)
>> >> min mean[+/-sd] median max
>> >> Connect: 0 7 69.0 0 3022
>> >> Processing: 0 12 22.6 9 2028
>> >> Waiting: 0 6 15.8 4 2023
>> >> Total: 0 19 73.8 9 3038
>> >>
>> >> Percentage of the requests served within a certain time (ms)
>> >> 50% 9
>> >> 66% 13
>> >> 75% 16
>> >> 80% 19
>> >> 90% 30
>> >> 95% 46
>> >> 98% 56
>> >> 99% 215
>> >> 100% 3038 (longest request)
>> >> ```
>> >>
>> >> On Ubuntu 18 with apache in stock settings + my mpm event config.
>> >> There is ~15 thousands length fails on the first run after apache
>> >> restart then it stays in 100-1k range:
>> >> ```
>> >> ab -k -t 900 -c 1000 -n 1000000 http://ubuntu18/
>> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
>> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> >> Licensed to The Apache Software Foundation, http://www.apache.org/
>> >>
>> >> Benchmarking 10.1.3.3 (be patient)
>> >> Completed 100000 requests
>> >> Completed 200000 requests
>> >> Completed 300000 requests
>> >> Completed 400000 requests
>> >> Completed 500000 requests
>> >> Completed 600000 requests
>> >> Completed 700000 requests
>> >> Completed 800000 requests
>> >> Completed 900000 requests
>> >> Completed 1000000 requests
>> >> Finished 1000000 requests
>> >>
>> >>
>> >> Server Software: Apache/2.4.29
>> >> Server Hostname: 10.1.3.3
>> >> Server Port: 80
>> >>
>> >> Document Path: /
>> >> Document Length: 10918 bytes
>> >>
>> >> Concurrency Level: 1000
>> >> Time taken for tests: 17.221 seconds
>> >> Complete requests: 1000000
>> >> Failed requests: 636
>> >> (Connect: 0, Receive: 0, Length: 636, Exceptions: 0)
>> >> Write errors: 0
>> >> Keep-Alive requests: 990158
>> >> Total transferred: 11220454865 bytes
>> >> HTML transferred: 10911056152 bytes
>> >> Requests per second: 58067.59 [#/sec] (mean)
>> >> Time per request: 17.221 [ms] (mean)
>> >> Time per request: 0.017 [ms] (mean, across all concurrent requests)
>> >> Transfer rate: 636274.18 [Kbytes/sec] received
>> >>
>> >> Connection Times (ms)
>> >> min mean[+/-sd] median max
>> >> Connect: 0 0 17.4 0 1020
>> >> Processing: 0 17 11.3 17 377
>> >> Waiting: 0 8 9.0 9 368
>> >> Total: 0 17 21.6 17 1389
>> >>
>> >> Percentage of the requests served within a certain time (ms)
>> >> 50% 17
>> >> 66% 18
>> >> 75% 18
>> >> 80% 19
>> >> 90% 20
>> >> 95% 23
>> >> 98% 28
>> >> 99% 36
>> >> 100% 1389 (longest request)
>> >> ```
>> >>
>> >> Mateusz Kempski
>> >> On Mon, 22 May 2023 at 13:05, Deepak Goel <deicool@xxxxxxxxx> wrote:
>> >> >
>> >> >
>> >> >
>> >> > On Mon, 22 May 2023, 17:58 Mateusz Kempski, <mkempski@xxxxxxxxxxxx.invalid> wrote:
>> >> >>
>> >> >> @Yann Ylavic:
>> >> >> There seems to be no difference in configuration file except for some comments:
>> >> >> ```
>> >> >> diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.rocky
>> >> >> 8a
>> >> >> # See the httpd.conf(5) man page for more information on this configuration,
>> >> >> # and httpd.service(8) on using and configuring the httpd service.
>> >> >> #
>> >> >> .
>> >> >> ```
>> >> >> I will paste complete Rocky config for completeness at the end of this message.
>> >> >> EnableMMAP and EnableSendfile are set to on, differently to defaults
>> >> >> specified in comments. This is default system configuration present on
>> >> >> both Rocky 8 and Centos 7. I removed httpd and all config files and
>> >> >> installed it again to make sure.
>> >> >> ```
>> >> >> # Defaults if commented: EnableMMAP On, EnableSendfile Off
>> >> >> #
>> >> >> #EnableMMAP off
>> >> >> EnableSendfile on
>> >> >> ```
>> >> >>
>> >> >> Both systems return:
>> >> >> ```
>> >> >> ulimit -n
>> >> >> 1024
>> >> >> ```
>> >> >> There is no custom config for httpd:
>> >> >> ```
>> >> >> stat /etc/systemd/system/httpd.service.d
>> >> >> stat: cannot stat ‘/etc/systemd/system/httpd.service.d’: No such file
>> >> >> or directory
>> >> >> ```
>> >> >> systemd itself seems to not apply any limit on both systems:
>> >> >> ```
>> >> >> grep LimitNOFILE /etc/systemd/system.conf
>> >> >> #DefaultLimitNOFILE=
>> >> >> ```
>> >> >>
>> >> >> After enabling trace there is a lot of messenges about too many open
>> >> >> connections and killing keepalive connections. Example snippet:
>> >> >> ```
>> >> >> [Mon May 22 11:57:40.146451 2023] [mpm_event:debug] [pid 7710:tid
>> >> >> 139800984155904] event.c(1808): Too many open connections (73), not
>> >> >> accepting new conns in
>> >> >> this process
>> >> >> [Mon May 22 11:57:40.146455 2023] [mpm_event:trace1] [pid 7710:tid
>> >> >> 139800984155904] event.c(1811): Idle workers: 0
>> >> >> [Mon May 22 11:57:40.146457 2023] [mpm_event:trace1] [pid 7710:tid
>> >> >> 139800984155904] event.c(1574): All workers are busy or dying, will
>> >> >> close 2 keep-alive co
>> >> >> nnections
>> >> >> [Mon May 22 11:57:40.146526 2023] [mpm_event:debug] [pid 7379:tid
>> >> >> 139801403561728] event.c(492): AH00457: Accepting new connections
>> >> >> again: 50 active conns (
>> >> >> 27 lingering/0 clogged/0 suspended), 7 idle workers
>> >> >> [Mon May 22 11:57:40.146686 2023] [mpm_event:debug] [pid 7511:tid
>> >> >> 139800849938176] event.c(1808): Too many open connections (86), not
>> >> >> accepting new conns in
>> >> >> this process
>> >> >> [Mon May 22 11:57:40.146698 2023] [mpm_event:trace1] [pid 7511:tid
>> >> >> 139800849938176] event.c(1811): Idle workers: 11
>> >> >> [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7511:tid
>> >> >> 139800849938176] event.c(1574): All workers are busy or dying, will
>> >> >> close 34 keep-alive c
>> >> >> onnections
>> >> >> [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7644:tid
>> >> >> 139801336452864] event.c(1574): All workers are busy or dying, will
>> >> >> close 2 keep-alive co
>> >> >> nnections
>> >> >> [Mon May 22 11:57:40.146812 2023] [mpm_event:debug] [pid 7511:tid
>> >> >> 139800849938176] event.c(492): AH00457: Accepting new connections
>> >> >> again: 85 active conns (
>> >> >> 52 lingering/0 clogged/0 suspended), 6 idle workers
>> >> >> [Mon May 22 11:57:40.146955 2023] [mpm_event:debug] [pid 7710:tid
>> >> >> 139800984155904] event.c(492): AH00457: Accepting new connections
>> >> >> again: 73 active conns (
>> >> >> 58 lingering/0 clogged/0 suspended), 9 idle workers
>> >> >> [Mon May 22 11:57:40.148250 2023] [mpm_event:debug] [pid 7162:tid
>> >> >> 139800841545472] event.c(492): AH00457: Accepting new connections
>> >> >> again: 48 active conns (
>> >> >> 25 lingering/0 clogged/0 suspended), 2 idle workers
>> >> >> [Mon May 22 11:57:40.148562 2023] [mpm_event:debug] [pid 7162:tid
>> >> >> 139800841545472] event.c(1808): Too many open connections (48), not
>> >> >> accepting new conns in
>> >> >> this process
>> >> >> [Mon May 22 11:57:40.148573 2023] [mpm_event:trace1] [pid 7162:tid
>> >> >> 139800841545472] event.c(1811): Idle workers: 0
>> >> >> [Mon May 22 11:57:40.148921 2023] [mpm_event:debug] [pid 7710:tid
>> >> >> 139800984155904] event.c(1808): Too many open connections (70), not
>> >> >> accepting new conns in
>> >> >> this process
>> >> >> [Mon May 22 11:57:40.148930 2023] [mpm_event:trace1] [pid 7710:tid
>> >> >> 139800984155904] event.c(1811): Idle workers: 1
>> >> >> [Mon May 22 11:57:40.149594 2023] [mpm_event:debug] [pid 7511:tid
>> >> >> 139800849938176] event.c(1808): Too many open connections (69), not
>> >> >> accepting new conns in
>> >> >> this process
>> >> >> [Mon May 22 11:57:40.149603 2023] [mpm_event:trace1] [pid 7511:tid
>> >> >> 139800849938176] event.c(1811): Idle workers: 1
>> >> >> [Mon May 22 11:57:40.149630 2023] [mpm_event:debug] [pid 7710:tid
>> >> >> 139800984155904] event.c(492): AH00457: Accepting new connections
>> >> >> again: 61 active conns (
>> >> >> 38 lingering/0 clogged/0 suspended), 2 idle workers
>> >> >> [Mon May 22 11:57:40.149776 2023] [mpm_event:debug] [pid 7710:tid
>> >> >> 139800984155904] event.c(1808): Too many open connections (63), not
>> >> >> accepting new conns in
>> >> >> this process
>> >> >> [Mon May 22 11:57:40.149782 2023] [mpm_event:trace1] [pid 7710:tid
>> >> >> 139800984155904] event.c(1811): Idle workers: 0
>> >> >> [Mon May 22 11:57:40.149882 2023] [mpm_event:debug] [pid 7710:tid
>> >> >> 139800984155904] event.c(492): AH00457: Accepting new connections
>> >> >> again: 61 active conns (
>> >> >> 38 lingering/0 clogged/0 suspended), 2 idle workers
>> >> >> [Mon May 22 11:57:40.149913 2023] [mpm_event:debug] [pid 7511:tid
>> >> >> 139800849938176] event.c(492): AH00457: Accepting new connections
>> >> >> again: 63 active conns (
>> >> >> 40 lingering/0 clogged/0 suspended), 2 idle workers
>> >> >> [Mon May 22 11:57:40.150210 2023] [mpm_event:debug] [pid 7511:tid
>> >> >> 139800849938176] event.c(1808): Too many open connections (62), not
>> >> >> accepting new conns in
>> >> >> this process
>> >> >> [Mon May 22 11:57:40.150216 2023] [mpm_event:trace1] [pid 7511:tid
>> >> >> 139800849938176] event.c(1811): Idle workers: 0
>> >> >> ```
>> >> >>
>> >> >> @Deepak Goel:
>> >> >> 1. Here are result with previously mentioned KeepAlive parameters:
>> >> >> Rocky 8:
>> >> >> ```
>> >> >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/
>> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
>> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/
>> >> >>
>> >> >> Benchmarking 10.1.3.11 (be patient)
>> >> >> Completed 100000 requests
>> >> >> Completed 200000 requests
>> >> >> Completed 300000 requests
>> >> >> Completed 400000 requests
>> >> >> Completed 500000 requests
>> >> >> Completed 600000 requests
>> >> >> Completed 700000 requests
>> >> >> Completed 800000 requests
>> >> >> Completed 900000 requests
>> >> >> Completed 1000000 requests
>> >> >> Finished 1000000 requests
>> >> >>
>> >> >>
>> >> >> Server Software: Apache/2.4.37
>> >> >> Server Hostname: 10.1.3.11
>> >> >> Server Port: 80
>> >> >>
>> >> >> Document Path: /
>> >> >> Document Length: 7620 bytes
>> >> >>
>> >> >> Concurrency Level: 1000
>> >> >> Time taken for tests: 17.287 seconds
>> >> >> Complete requests: 1000000
>> >> >> Failed requests: 98218
>> >> >> (Connect: 0, Receive: 0, Length: 88397, Exceptions: 9821)
>> >> >> Write errors: 0
>> >> >> Keep-Alive requests: 911603
>> >> >> Total transferred: 7214426142 bytes
>> >> >> HTML transferred: 6946414860 bytes
>> >> >> Requests per second: 57845.81 [#/sec] (mean)
>> >> >> Time per request: 17.287 [ms] (mean)
>> >> >> Time per request: 0.017 [ms] (mean, across all concurrent requests)
>> >> >> Transfer rate: 407543.30 [Kbytes/sec] received
>> >> >>
>> >> >> Connection Times (ms)
>> >> >> min mean[+/-sd] median max
>> >> >> Connect: 0 4 59.9 0 3015
>> >> >> Processing: 0 13 14.5 13 432
>> >> >> Waiting: 0 13 14.1 13 432
>> >> >> Total: 0 17 63.2 15 3036
>> >> >>
>> >> >> Percentage of the requests served within a certain time (ms)
>> >> >> 50% 15
>> >> >> 66% 17
>> >> >> 75% 17
>> >> >> 80% 18
>> >> >> 90% 21
>> >> >> 95% 23
>> >> >> 98% 28
>> >> >> 99% 36
>> >> >> 100% 3036 (longest request)
>> >> >> ```
>> >> >> Centos 7:
>> >> >> ```
>> >> >> ab -k -t 900 -c 1000 -n 1000000 http://10.1.3.3/
>> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
>> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/
>> >> >>
>> >> >> Benchmarking 10.1.3.3 (be patient)
>> >> >> Completed 100000 requests
>> >> >> Completed 200000 requests
>> >> >> Completed 300000 requests
>> >> >> Completed 400000 requests
>> >> >> Completed 500000 requests
>> >> >> Completed 600000 requests
>> >> >> Completed 700000 requests
>> >> >> Completed 800000 requests
>> >> >> Completed 900000 requests
>> >> >> Completed 1000000 requests
>> >> >> Finished 1000000 requests
>> >> >>
>> >> >>
>> >> >> Server Software: Apache/2.4.6
>> >> >> Server Hostname: 10.1.3.3
>> >> >> Server Port: 80
>> >> >>
>> >> >> Document Path: /
>> >> >> Document Length: 7620 bytes
>> >> >>
>> >> >> Concurrency Level: 1000
>> >> >> Time taken for tests: 20.731 seconds
>> >> >> Complete requests: 1000000
>> >> >> Failed requests: 14
>> >> >> (Connect: 0, Receive: 0, Length: 14, Exceptions: 0)
>> >> >> Write errors: 0
>> >> >> Keep-Alive requests: 999986
>> >> >> Total transferred: 7913889204 bytes
>> >> >> HTML transferred: 7619893320 bytes
>> >> >> Requests per second: 48236.94 [#/sec] (mean)
>> >> >> Time per request: 20.731 [ms] (mean)
>> >> >> Time per request: 0.021 [ms] (mean, across all concurrent requests)
>> >> >> Transfer rate: 372794.72 [Kbytes/sec] received
>> >> >>
>> >> >> Connection Times (ms)
>> >> >> min mean[+/-sd] median max
>> >> >> Connect: 0 0 13.1 0 1025
>> >> >> Processing: 2 21 7.9 20 821
>> >> >> Waiting: 0 20 7.8 20 821
>> >> >> Total: 2 21 15.7 20 1244
>> >> >>
>> >> >> Percentage of the requests served within a certain time (ms)
>> >> >> 50% 20
>> >> >> 66% 22
>> >> >> 75% 23
>> >> >> 80% 24
>> >> >> 90% 25
>> >> >> 95% 27
>> >> >> 98% 29
>> >> >> 99% 30
>> >> >> 100% 1244 (longest request)
>> >> >> ```
>> >> >> 3. iotop results:
>> >> >> Centos 7 idle:
>> >> >> ```
>> >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
>> >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
>> >> >> ```
>> >> >> Rocky 8 idle:
>> >> >> ```
>> >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
>> >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
>> >> >> ```
>> >> >> Centos 7 during test:
>> >> >> ```
>> >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.56 M/s
>> >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
>> >> >> ```
>> >> >> Rocky 8 during test:
>> >> >> ```
>> >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 5.35 M/s
>> >> >
>> >> >
>> >> > The DISK WRITE is higher in Rocky as compared to Centos.
>> >> >
>> >> > Is it possible for you to test on Ubuntu 18/20 as well? ( Seems Rocky algo's are taking a performance hit )
>> >> >>
>> >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
>> >> >> ```
>> >> >> I ran top command without parameters on both systems - looks like
>> >> >> Centos defaults to KiB and Rocky to MiB.
>> >> >>
>> >> >> Complete httpd.conf:
>> >> >> ```
>> >> >> #
>> >> >> # This is the main Apache HTTP server configuration file. It contains the
>> >> >> # configuration directives that give the server its instructions.
>> >> >> # See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
>> >> >> # In particular, see
>> >> >> # <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
>> >> >> # for a discussion of each configuration directive.
>> >> >> #
>> >> >> # See the httpd.conf(5) man page for more information on this configuration,
>> >> >> # and httpd.service(8) on using and configuring the httpd service.
>> >> >> #
>> >> >> # Do NOT simply read the instructions in here without understanding
>> >> >> # what they do. They're here only as hints or reminders. If you are unsure
>> >> >> # consult the online docs. You have been warned.
>> >> >> #
>> >> >> # Configuration and logfile names: If the filenames you specify for many
>> >> >> # of the server's control files begin with "/" (or "drive:/" for Win32), the
>> >> >> # server will use that explicit path. If the filenames do *not* begin
>> >> >> # with "/", the value of ServerRoot is prepended -- so 'log/access_log'
>> >> >> # with ServerRoot set to '/www' will be interpreted by the
>> >> >> # server as '/www/log/access_log', where as '/log/access_log' will be
>> >> >> # interpreted as '/log/access_log'.
>> >> >>
>> >> >> #
>> >> >> # ServerRoot: The top of the directory tree under which the server's
>> >> >> # configuration, error, and log files are kept.
>> >> >> #
>> >> >> # Do not add a slash at the end of the directory path. If you point
>> >> >> # ServerRoot at a non-local disk, be sure to specify a local disk on the
>> >> >> # Mutex directive, if file-based mutexes are used. If you wish to share the
>> >> >> # same ServerRoot for multiple httpd daemons, you will need to change at
>> >> >> # least PidFile.
>> >> >> #
>> >> >> ServerRoot "/etc/httpd"
>> >> >>
>> >> >> #
>> >> >> # Listen: Allows you to bind Apache to specific IP addresses and/or
>> >> >> # ports, instead of the default. See also the <VirtualHost>
>> >> >> # directive.
>> >> >> #
>> >> >> # Change this to Listen on specific IP addresses as shown below to
>> >> >> # prevent Apache from glomming onto all bound IP addresses.
>> >> >> #
>> >> >> #Listen 12.34.56.78:80
>> >> >> Listen 80
>> >> >>
>> >> >> #
>> >> >> # Dynamic Shared Object (DSO) Support
>> >> >> #
>> >> >> # To be able to use the functionality of a module which was built as a DSO you
>> >> >> # have to place corresponding `LoadModule' lines at this location so the
>> >> >> # directives contained in it are actually available _before_ they are used.
>> >> >> # Statically compiled modules (those listed by `httpd -l') do not need
>> >> >> # to be loaded here.
>> >> >> #
>> >> >> # Example:
>> >> >> # LoadModule foo_module modules/mod_foo.so
>> >> >> #
>> >> >> Include conf.modules.d/*.conf
>> >> >>
>> >> >> #
>> >> >> # If you wish httpd to run as a different user or group, you must run
>> >> >> # httpd as root initially and it will switch.
>> >> >> #
>> >> >> # User/Group: The name (or #number) of the user/group to run httpd as.
>> >> >> # It is usually good practice to create a dedicated user and group for
>> >> >> # running httpd, as with most system services.
>> >> >> #
>> >> >> User apache
>> >> >> Group apache
>> >> >>
>> >> >> # 'Main' server configuration
>> >> >> #
>> >> >> # The directives in this section set up the values used by the 'main'
>> >> >> # server, which responds to any requests that aren't handled by a
>> >> >> # <VirtualHost> definition. These values also provide defaults for
>> >> >> # any <VirtualHost> containers you may define later in the file.
>> >> >> #
>> >> >> # All of these directives may appear inside <VirtualHost> containers,
>> >> >> # in which case these default settings will be overridden for the
>> >> >> # virtual host being defined.
>> >> >> #
>> >> >>
>> >> >> #
>> >> >> # ServerAdmin: Your address, where problems with the server should be
>> >> >> # e-mailed. This address appears on some server-generated pages, such
>> >> >> # as error documents. e.g. admin@xxxxxxxxxxxxxxx
>> >> >> #
>> >> >> ServerAdmin root@localhost
>> >> >>
>> >> >> #
>> >> >> # ServerName gives the name and port that the server uses to identify itself.
>> >> >> # This can often be determined automatically, but we recommend you specify
>> >> >> # it explicitly to prevent problems during startup.
>> >> >> #
>> >> >> # If your host doesn't have a registered DNS name, enter its IP address here.
>> >> >> #
>> >> >> #ServerName www.example.com:80
>> >> >>
>> >> >> #
>> >> >> # Deny access to the entirety of your server's filesystem. You must
>> >> >> # explicitly permit access to web content directories in other
>> >> >> # <Directory> blocks below.
>> >> >> #
>> >> >> <Directory />
>> >> >> AllowOverride none
>> >> >> Require all denied
>> >> >> </Directory>
>> >> >>
>> >> >> #
>> >> >> # Note that from this point forward you must specifically allow
>> >> >> # particular features to be enabled - so if something's not working as
>> >> >> # you might expect, make sure that you have specifically enabled it
>> >> >> # below.
>> >> >> #
>> >> >>
>> >> >> #
>> >> >> # DocumentRoot: The directory out of which you will serve your
>> >> >> # documents. By default, all requests are taken from this directory, but
>> >> >> # symbolic links and aliases may be used to point to other locations.
>> >> >> #
>> >> >> DocumentRoot "/var/www/html"
>> >> >>
>> >> >> #
>> >> >> # Relax access to content within /var/www.
>> >> >> #
>> >> >> <Directory "/var/www">
>> >> >> AllowOverride None
>> >> >> # Allow open access:
>> >> >> Require all granted
>> >> >> </Directory>
>> >> >>
>> >> >> # Further relax access to the default document root:
>> >> >> <Directory "/var/www/html">
>> >> >> #
>> >> >> # Possible values for the Options directive are "None", "All",
>> >> >> # or any combination of:
>> >> >> # Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
>> >> >> #
>> >> >> # Note that "MultiViews" must be named *explicitly* --- "Options All"
>> >> >> # doesn't give it to you.
>> >> >> #
>> >> >> # The Options directive is both complicated and important. Please see
>> >> >> # http://httpd.apache.org/docs/2.4/mod/core.html#options
>> >> >> # for more information.
>> >> >> #
>> >> >> Options Indexes FollowSymLinks
>> >> >>
>> >> >> #
>> >> >> # AllowOverride controls what directives may be placed in .htaccess files.
>> >> >> # It can be "All", "None", or any combination of the keywords:
>> >> >> # Options FileInfo AuthConfig Limit
>> >> >> #
>> >> >> AllowOverride None
>> >> >>
>> >> >> #
>> >> >> # Controls who can get stuff from this server.
>> >> >> #
>> >> >> Require all granted
>> >> >> </Directory>
>> >> >>
>> >> >> #
>> >> >> # DirectoryIndex: sets the file that Apache will serve if a directory
>> >> >> # is requested.
>> >> >> #
>> >> >> <IfModule dir_module>
>> >> >> DirectoryIndex index.html
>> >> >> </IfModule>
>> >> >>
>> >> >> #
>> >> >> # The following lines prevent .htaccess and .htpasswd files from being
>> >> >> # viewed by Web clients.
>> >> >> #
>> >> >> <Files ".ht*">
>> >> >> Require all denied
>> >> >> </Files>
>> >> >>
>> >> >> #
>> >> >> # ErrorLog: The location of the error log file.
>> >> >> # If you do not specify an ErrorLog directive within a <VirtualHost>
>> >> >> # container, error messages relating to that virtual host will be
>> >> >> # logged here. If you *do* define an error logfile for a <VirtualHost>
>> >> >> # container, that host's errors will be logged there and not here.
>> >> >> #
>> >> >> ErrorLog "logs/error_log"
>> >> >>
>> >> >> #
>> >> >> # LogLevel: Control the number of messages logged to the error_log.
>> >> >> # Possible values include: debug, info, notice, warn, error, crit,
>> >> >> # alert, emerg.
>> >> >> #
>> >> >> LogLevel warn
>> >> >>
>> >> >> <IfModule log_config_module>
>> >> >> #
>> >> >> # The following directives define some format nicknames for use with
>> >> >> # a CustomLog directive (see below).
>> >> >> #
>> >> >> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
>> >> >> \"%{User-Agent}i\"" combined
>> >> >> LogFormat "%h %l %u %t \"%r\" %>s %b" common
>> >> >>
>> >> >> <IfModule logio_module>
>> >> >> # You need to enable mod_logio.c to use %I and %O
>> >> >> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
>> >> >> \"%{User-Agent}i\" %I %O" combinedio
>> >> >> </IfModule>
>> >> >>
>> >> >> #
>> >> >> # The location and format of the access logfile (Common Logfile Format).
>> >> >> # If you do not define any access logfiles within a <VirtualHost>
>> >> >> # container, they will be logged here. Contrariwise, if you *do*
>> >> >> # define per-<VirtualHost> access logfiles, transactions will be
>> >> >> # logged therein and *not* in this file.
>> >> >> #
>> >> >> #CustomLog "logs/access_log" common
>> >> >>
>> >> >> #
>> >> >> # If you prefer a logfile with access, agent, and referer information
>> >> >> # (Combined Logfile Format) you can use the following directive.
>> >> >> #
>> >> >> CustomLog "logs/access_log" combined
>> >> >> </IfModule>
>> >> >>
>> >> >> <IfModule alias_module>
>> >> >> #
>> >> >> # Redirect: Allows you to tell clients about documents that used to
>> >> >> # exist in your server's namespace, but do not anymore. The client
>> >> >> # will make a new request for the document at its new location.
>> >> >> # Example:
>> >> >> # Redirect permanent /foo http://www.example.com/bar
>> >> >>
>> >> >> #
>> >> >> # Alias: Maps web paths into filesystem paths and is used to
>> >> >> # access content that does not live under the DocumentRoot.
>> >> >> # Example:
>> >> >> # Alias /webpath /full/filesystem/path
>> >> >> #
>> >> >> # If you include a trailing / on /webpath then the server will
>> >> >> # require it to be present in the URL. You will also likely
>> >> >> # need to provide a <Directory> section to allow access to
>> >> >> # the filesystem path.
>> >> >>
>> >> >> #
>> >> >> # ScriptAlias: This controls which directories contain server scripts.
>> >> >> # ScriptAliases are essentially the same as Aliases, except that
>> >> >> # documents in the target directory are treated as applications and
>> >> >> # run by the server when requested rather than as documents sent to the
>> >> >> # client. The same rules about trailing "/" apply to ScriptAlias
>> >> >> # directives as to Alias.
>> >> >> #
>> >> >> ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
>> >> >>
>> >> >> </IfModule>
>> >> >>
>> >> >> #
>> >> >> # "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
>> >> >> # CGI directory exists, if you have that configured.
>> >> >> #
>> >> >> <Directory "/var/www/cgi-bin">
>> >> >> AllowOverride None
>> >> >> Options None
>> >> >> Require all granted
>> >> >> </Directory>
>> >> >>
>> >> >> <IfModule mime_module>
>> >> >> #
>> >> >> # TypesConfig points to the file containing the list of mappings from
>> >> >> # filename extension to MIME-type.
>> >> >> #
>> >> >> TypesConfig /etc/mime.types
>> >> >>
>> >> >> #
>> >> >> # AddType allows you to add to or override the MIME configuration
>> >> >> # file specified in TypesConfig for specific file types.
>> >> >> #
>> >> >> #AddType application/x-gzip .tgz
>> >> >> #
>> >> >> # AddEncoding allows you to have certain browsers uncompress
>> >> >> # information on the fly. Note: Not all browsers support this.
>> >> >> #
>> >> >> #AddEncoding x-compress .Z
>> >> >> #AddEncoding x-gzip .gz .tgz
>> >> >> #
>> >> >> # If the AddEncoding directives above are commented-out, then you
>> >> >> # probably should define those extensions to indicate media types:
>> >> >> #
>> >> >> AddType application/x-compress .Z
>> >> >> AddType application/x-gzip .gz .tgz
>> >> >>
>> >> >> #
>> >> >> # AddHandler allows you to map certain file extensions to "handlers":
>> >> >> # actions unrelated to filetype. These can be either built into the server
>> >> >> # or added with the Action directive (see below)
>> >> >> #
>> >> >> # To use CGI scripts outside of ScriptAliased directories:
>> >> >> # (You will also need to add "ExecCGI" to the "Options" directive.)
>> >> >> #
>> >> >> #AddHandler cgi-script .cgi
>> >> >>
>> >> >> # For type maps (negotiated resources):
>> >> >> #AddHandler type-map var
>> >> >>
>> >> >> #
>> >> >> # Filters allow you to process content before it is sent to the client.
>> >> >> #
>> >> >> # To parse .shtml files for server-side includes (SSI):
>> >> >> # (You will also need to add "Includes" to the "Options" directive.)
>> >> >> #
>> >> >> AddType text/html .shtml
>> >> >> AddOutputFilter INCLUDES .shtml
>> >> >> </IfModule>
>> >> >>
>> >> >> #
>> >> >> # Specify a default charset for all content served; this enables
>> >> >> # interpretation of all content as UTF-8 by default. To use the
>> >> >> # default browser choice (ISO-8859-1), or to allow the META tags
>> >> >> # in HTML content to override this choice, comment out this
>> >> >> # directive:
>> >> >> #
>> >> >> AddDefaultCharset UTF-8
>> >> >>
>> >> >> <IfModule mime_magic_module>
>> >> >> #
>> >> >> # The mod_mime_magic module allows the server to use various hints from the
>> >> >> # contents of the file itself to determine its type. The MIMEMagicFile
>> >> >> # directive tells the module where the hint definitions are located.
>> >> >> #
>> >> >> MIMEMagicFile conf/magic
>> >> >> </IfModule>
>> >> >>
>> >> >> #
>> >> >> # Customizable error responses come in three flavors:
>> >> >> # 1) plain text 2) local redirects 3) external redirects
>> >> >> #
>> >> >> # Some examples:
>> >> >> #ErrorDocument 500 "The server made a boo boo."
>> >> >> #ErrorDocument 404 /missing.html
>> >> >> #ErrorDocument 404 "/cgi-bin/missing_handler.pl"
>> >> >> #ErrorDocument 402 http://www.example.com/subscription_info.html
>> >> >> #
>> >> >>
>> >> >> #
>> >> >> # EnableMMAP and EnableSendfile: On systems that support it,
>> >> >> # memory-mapping or the sendfile syscall may be used to deliver
>> >> >> # files. This usually improves server performance, but must
>> >> >> # be turned off when serving from networked-mounted
>> >> >> # filesystems or if support for these functions is otherwise
>> >> >> # broken on your system.
>> >> >> # Defaults if commented: EnableMMAP On, EnableSendfile Off
>> >> >> #
>> >> >> #EnableMMAP off
>> >> >> EnableSendfile on
>> >> >>
>> >> >> # Supplemental configuration
>> >> >> #
>> >> >> # Load config files in the "/etc/httpd/conf.d" directory, if any.
>> >> >> IncludeOptional conf.d/*.conf
>> >> >>
>> >> >> <IfModule mpm_event_module>
>> >> >> ThreadsPerChild 25
>> >> >> StartServers 3
>> >> >> ServerLimit 120
>> >> >> MinSpareThreads 75
>> >> >> MaxSpareThreads 3000
>> >> >> MaxRequestWorkers 3000
>> >> >> MaxConnectionsPerChild 0
>> >> >> </IfModule>
>> >> >> ```
>> >> >>
>> >> >> Mateusz Kempski
>> >> >> Linux System Administrator
>> >> >> XTM International Ltd.
>> >> >> Email: mkempski@xxxxxxxxxxxx
>> >> >> xtm.cloud
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Mon, 22 May 2023 at 11:16, Deepak Goel <deicool@xxxxxxxxx> wrote:
>> >> >> >
>> >> >> > Hi
>> >> >> >
>> >> >> > 1. Please post the test results completely. (Sorry but, saying there is no difference does not help).
>> >> >> > 2. The memory used in Rocky is comparatively higher than Centos (About 30Meg or so). Also the buff/cache is approx 10 times higher in Rocky than in Centos.
>> >> >> > 3. Please also post iotop results too.
>> >> >> >
>> >> >> > P.S: I hope the top commands is run identical for both Rocky and Centos.
>> >> >> >
>> >> >> > Deepak
>> >> >> > "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi"
>> >> >> >
>> >> >> > +91 73500 12833
>> >> >> > deicool@xxxxxxxxx
>> >> >> >
>> >> >> > Facebook: https://www.facebook.com/deicool
>> >> >> > LinkedIn: www.linkedin.com/in/deicool
>> >> >> >
>> >> >> > "Plant a Tree, Go Green"
>> >> >> >
>> >> >> > Make In India : http://www.makeinindia.com/home
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Mon, May 22, 2023 at 4:28 PM Mateusz Kempski <mkempski@xxxxxxxxxxxx.invalid> wrote:
>> >> >> >>
>> >> >> >> I tested again with settings:
>> >> >> >> ```
>> >> >> >> KeepAliveTimeout 300
>> >> >> >> MaxKeepAliveRequests 0
>> >> >> >> ```
>> >> >> >> but there was no difference in results barring normal diffs run-to-run.
>> >> >> >>
>> >> >> >> Below is the top of top from both servers when idle and during test.
>> >> >> >>
>> >> >> >> Rocky 8 no load:
>> >> >> >> ```
>> >> >> >> top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10
>> >> >> >> Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 zombie
>> >> >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> >> >> >> MiB Mem : 15824.7 total, 10682.4 free, 283.5 used, 4858.8 buff/cache
>> >> >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 avail Mem
>> >> >> >> ```
>> >> >> >> Rocky 8 during test:
>> >> >> >> ```
>> >> >> >> top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28
>> >> >> >> Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie
>> >> >> >> %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 5.4 si, 0.1 st
>> >> >> >> MiB Mem : 15824.7 total, 9863.0 free, 529.3 used, 5432.3 buff/cache
>> >> >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 avail Mem
>> >> >> >> ```
>> >> >> >> Centos 7 no load:
>> >> >> >> ```
>> >> >> >> top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00
>> >> >> >> Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie
>> >> >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> >> >> >> KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 buff/cache
>> >> >> >> KiB Swap: 0 total, 0 free, 0 used. 15720740 avail Mem
>> >> >> >> ```
>> >> >> >> Centos 7 during test:
>> >> >> >> ```
>> >> >> >> top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05
>> >> >> >> Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 zombie
>> >> >> >> %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 3.1 si, 0.1 st
>> >> >> >> KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 buff/cache
>> >> >> >> KiB Swap: 0 total, 0 free, 0 used. 15459544 avail Mem
>> >> >> >> ```
>> >> >> >>
>> >> >> >> On Mon, 22 May 2023 at 10:34, Deepak Goel <deicool@xxxxxxxxx> wrote:
>> >> >> >> >
>> >> >> >> > Hi
>> >> >> >> >
>> >> >> >> > I can see about 8000+ requests have timed out in 'Rocky'. This is mostly due to Apache, which is unable to handle the load. Is it possible to increase the parameter "KeepAliveTimeout" (and other KeepAlive parameters).
>> >> >> >> >
>> >> >> >> > Is it also possible for you to post the hardware utilisations for the 2 different servers (Centos & Rocky)?
>> >> >> >> >
>> >> >> >> > Deepak
>> >> >> >> > "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi"
>> >> >> >> >
>> >> >> >> > +91 73500 12833
>> >> >> >> > deicool@xxxxxxxxx
>> >> >> >> >
>> >> >> >> > Facebook: https://www.facebook.com/deicool
>> >> >> >> > LinkedIn: www.linkedin.com/in/deicool
>> >> >> >> >
>> >> >> >> > "Plant a Tree, Go Green"
>> >> >> >> >
>> >> >> >> > Make In India : http://www.makeinindia.com/home
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski <mkempski@xxxxxxxxxxxx.invalid> wrote:
>> >> >> >> >>
>> >> >> >> >> Hi all,
>> >> >> >> >> I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7
>> >> >> >> >> install, the other is fresh Rocky 8. I installed httpd (on Centos 7
>> >> >> >> >> it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to
>> >> >> >> >> point to the same static default html file and enabled mpm event on
>> >> >> >> >> Centos (mpm event is default on Rocky). Then I added following options
>> >> >> >> >> to default config on both servers:
>> >> >> >> >> ```
>> >> >> >> >> <IfModule mpm_event_module>
>> >> >> >> >> ThreadsPerChild 25
>> >> >> >> >> StartServers 3
>> >> >> >> >> ServerLimit 120
>> >> >> >> >> MinSpareThreads 75
>> >> >> >> >> MaxSpareThreads 3000
>> >> >> >> >> MaxRequestWorkers 3000
>> >> >> >> >> MaxConnectionsPerChild 0
>> >> >> >> >> </IfModule>
>> >> >> >> >> ```
>> >> >> >> >> After this is done I performed ab tests with keepalive using different
>> >> >> >> >> Centos 7 VM in the same local network. Can you help me understand
>> >> >> >> >> these results? On Centos I am able to complete 1 milion requests at
>> >> >> >> >> 1000 concurrent connections with little to no errors, however with
>> >> >> >> >> version 2.4.37 on Rocky I get a lot of failed requests due to length
>> >> >> >> >> and exceptions. Served content is static so I am assuming this is
>> >> >> >> >> because keepalive connections are closed by the server. I tried
>> >> >> >> >> various configurations of KeeAaliveTimeout, MaxKeepAliveRequests,
>> >> >> >> >> AsyncRequestWorkerFactor and other mpm_event options but I got no
>> >> >> >> >> better results. The only thing that seems to improve the stability of
>> >> >> >> >> keepalive connections is setting threads and servers to the moon for
>> >> >> >> >> example:
>> >> >> >> >> ```
>> >> >> >> >> <IfModule mpm_event_module>
>> >> >> >> >> ThreadsPerChild 50
>> >> >> >> >> StartServers 120
>> >> >> >> >> ServerLimit 120
>> >> >> >> >> MinSpareThreads 6000
>> >> >> >> >> MaxSpareThreads 6000
>> >> >> >> >> MaxRequestWorkers 6000
>> >> >> >> >> MaxConnectionsPerChild 0
>> >> >> >> >> </IfModule>
>> >> >> >> >> ```
>> >> >> >> >> However it stills throws more errors (~8k) than 2.4.6 on the first set
>> >> >> >> >> of settings. This problem occurs only when using keepalive. There are
>> >> >> >> >> no errors when using ab without -k option, although speed is lower. I
>> >> >> >> >> can replicate this issue on newest httpd build from source (2.4.57).
>> >> >> >> >> What is causing this difference of behavior? How can I achieve
>> >> >> >> >> performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing much more
>> >> >> >> >> resources at httpd? These errors seems to be a root cause of a problem
>> >> >> >> >> we have with 502 errors thrown from downstream reverse proxy server
>> >> >> >> >> when httpd kills keepalive connection prematurely and the proxy is
>> >> >> >> >> trying to use this connection.
>> >> >> >> >>
>> >> >> >> >> Below results of ab tests:
>> >> >> >> >>
>> >> >> >> >> Centos 7 VM:
>> >> >> >> >> ```
>> >> >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://centos/
>> >> >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
>> >> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> >> >> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/
>> >> >> >> >>
>> >> >> >> >> Benchmarking 10.1.3.3 (be patient)
>> >> >> >> >> Completed 100000 requests
>> >> >> >> >> Completed 200000 requests
>> >> >> >> >> Completed 300000 requests
>> >> >> >> >> Completed 400000 requests
>> >> >> >> >> Completed 500000 requests
>> >> >> >> >> Completed 600000 requests
>> >> >> >> >> Completed 700000 requests
>> >> >> >> >> Completed 800000 requests
>> >> >> >> >> Completed 900000 requests
>> >> >> >> >> Completed 1000000 requests
>> >> >> >> >> Finished 1000000 requests
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Server Software: Apache/2.4.6
>> >> >> >> >> Server Hostname: 10.1.3.3
>> >> >> >> >> Server Port: 80
>> >> >> >> >>
>> >> >> >> >> Document Path: /
>> >> >> >> >> Document Length: 7620 bytes
>> >> >> >> >>
>> >> >> >> >> Concurrency Level: 1000
>> >> >> >> >> Time taken for tests: 15.285 seconds
>> >> >> >> >> Complete requests: 1000000
>> >> >> >> >> Failed requests: 67
>> >> >> >> >> (Connect: 0, Receive: 0, Length: 67, Exceptions: 0)
>> >> >> >> >> Write errors: 0
>> >> >> >> >> Keep-Alive requests: 990567
>> >> >> >> >> Total transferred: 7919057974 bytes
>> >> >> >> >> HTML transferred: 7619489460 bytes
>> >> >> >> >> Requests per second: 65422.95 [#/sec] (mean)
>> >> >> >> >> Time per request: 15.285 [ms] (mean)
>> >> >> >> >> Time per request: 0.015 [ms] (mean, across all concurrent requests)
>> >> >> >> >> Transfer rate: 505945.41 [Kbytes/sec] received
>> >> >> >> >>
>> >> >> >> >> Connection Times (ms)
>> >> >> >> >> min mean[+/-sd] median max
>> >> >> >> >> Connect: 0 0 4.7 0 1042
>> >> >> >> >> Processing: 3 15 16.8 13 467
>> >> >> >> >> Waiting: 0 14 16.2 13 433
>> >> >> >> >> Total: 3 15 17.7 13 1081
>> >> >> >> >>
>> >> >> >> >> Percentage of the requests served within a certain time (ms)
>> >> >> >> >> 50% 13
>> >> >> >> >> 66% 15
>> >> >> >> >> 75% 16
>> >> >> >> >> 80% 17
>> >> >> >> >> 90% 21
>> >> >> >> >> 95% 23
>> >> >> >> >> 98% 32
>> >> >> >> >> 99% 44
>> >> >> >> >> 100% 1081 (longest request)
>> >> >> >> >> ```
>> >> >> >> >>
>> >> >> >> >> Rocky 8 VM:
>> >> >> >> >> ```
>> >> >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/
>> >> >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
>> >> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> >> >> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/
>> >> >> >> >>
>> >> >> >> >> Benchmarking 10.1.3.11 (be patient)
>> >> >> >> >> Completed 100000 requests
>> >> >> >> >> Completed 200000 requests
>> >> >> >> >> Completed 300000 requests
>> >> >> >> >> Completed 400000 requests
>> >> >> >> >> Completed 500000 requests
>> >> >> >> >> Completed 600000 requests
>> >> >> >> >> Completed 700000 requests
>> >> >> >> >> Completed 800000 requests
>> >> >> >> >> Completed 900000 requests
>> >> >> >> >> Completed 1000000 requests
>> >> >> >> >> Finished 1000000 requests
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Server Software: Apache/2.4.37
>> >> >> >> >> Server Hostname: 10.1.3.11
>> >> >> >> >> Server Port: 80
>> >> >> >> >>
>> >> >> >> >> Document Path: /
>> >> >> >> >> Document Length: 7620 bytes
>> >> >> >> >>
>> >> >> >> >> Concurrency Level: 1000
>> >> >> >> >> Time taken for tests: 19.101 seconds
>> >> >> >> >> Complete requests: 1000000
>> >> >> >> >> Failed requests: 93159
>> >> >> >> >> (Connect: 0, Receive: 0, Length: 85029, Exceptions: 8130)
>> >> >> >> >> Write errors: 0
>> >> >> >> >> Keep-Alive requests: 912753
>> >> >> >> >> Total transferred: 7248228337 bytes
>> >> >> >> >> HTML transferred: 6973694460 bytes
>> >> >> >> >> Requests per second: 52352.12 [#/sec] (mean)
>> >> >> >> >> Time per request: 19.101 [ms] (mean)
>> >> >> >> >> Time per request: 0.019 [ms] (mean, across all concurrent requests)
>> >> >> >> >> Transfer rate: 370566.51 [Kbytes/sec] received
>> >> >> >> >>
>> >> >> >> >> Connection Times (ms)
>> >> >> >> >> min mean[+/-sd] median max
>> >> >> >> >> Connect: 0 4 54.4 0 3022
>> >> >> >> >> Processing: 0 15 20.6 13 1400
>> >> >> >> >> Waiting: 0 13 17.7 12 1400
>> >> >> >> >> Total: 0 19 59.0 13 3048
>> >> >> >> >>
>> >> >> >> >> Percentage of the requests served within a certain time (ms)
>> >> >> >> >> 50% 13
>> >> >> >> >> 66% 15
>> >> >> >> >> 75% 17
>> >> >> >> >> 80% 19
>> >> >> >> >> 90% 24
>> >> >> >> >> 95% 41
>> >> >> >> >> 98% 53
>> >> >> >> >> 99% 106
>> >> >> >> >> 100% 3048 (longest request)
>> >> >> >> >> ```
>> >> >> >> >>
>> >> >> >> >> ---
>> >> >> >> >> Mateusz Kempski
>> >> >> >> >>
>> >> >> >> >> ---------------------------------------------------------------------
>> >> >> >> >> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
>> >> >> >> >> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
>> >> >> >> >>
>> >> >> >>
>> >> >> >> ---------------------------------------------------------------------
>> >> >> >> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
>> >> >> >> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
>> >> >> >>
>> >> >>
>> >> >> ---------------------------------------------------------------------
>> >> >> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
>> >> >> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
>> >> >>
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
>> >> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
>> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|