On 14.10.23 17:40, Alex Rousskov wrote:
On 2023-10-13 16:01, Julian Taylor wrote:
When using squid for caching using the rock cache_dir setting the
performance is pretty poor with multiple workers.
The reason for this is due to the very high number of systemcalls
involved in the IPC between the disker and workers.
Please allow me to rephrase your conclusion to better match (expected)
reality and avoid misunderstanding:
By design, a mostly idle SMP Squid should use a lot more system calls
per disk cache hit than a busy SMP Squid would:
* Mostly idle Squid: Every disk I/O may require a few IPC messages.
* Busy Squid: Bugs notwithstanding, disk I/Os require no IPC messages.
In your single-request test, you are observing the expected effects
described in the first bullet. That does not imply those effects are
"good" or "desirable" in your use case, of course. It only means that
SMP Squid was no optimized for that use case; SMP rock design was
explicitly targeting the opposite use case (i.e. a busy Squid).
The reproducer uses as single request, the same very thing can be
observed on a very busy squid and workaround improves both the single
request case and the actual heavy loaded production squid in the same way.
The hardware involved has a 10G card, not ssds but lots of ram so it has
a very high page cache hit rate and the squid is very busy, so much it
is overloaded by system cpu usage in default configuration with the rock
cache. The network or disk bandwidth is barely ever utilized more than
10% with all 8 cpus busy on system load.
The only way to get the squid to utilize the machine is to increase the
IO size via the request buffer change or not use the rock cache. UFS
cache works ok in comparison, but requires multiple independent squid
instances as it does not support SMP.
Increasing the IO size to 32KiB as I mentioned does allow the squid
workers to utilize a good 60% of the hardware network and disk capabilities.
Roughly speaking, here, "busy" means "there are always some messages in
the disk I/O queue [maintained by Squid in shared memory]".
You may wonder how it is possible that an increase in I/O work results
in decrease (and, hopefully, elimination) of related IPC messages.
Roughly speaking, a worker must send an IPC "you have a new I/O request"
message only when its worker->disker queue is empty. If the queue is not
empty, then there is no reason to send an IPC message to wake up disker
because disker will see the new message when dequeuing the previous one.
Same for the opposite direction: disker->worker...
This is probably true if you have slow disks and are actually IO bound,
but on fast disks or high page cache hit rate you essential see this ipc
ping pong and very little actual work being done.
> Is it necessary to have these read chunks so small
It is not. Disk I/O size should be at least the system I/O page size,
but it can be larger. The optimal I/O size is probably very dependent on
traffic patterns. IIRC, Squid I/O size is at most one Squid page
(SM_PAGE_SIZE or 4KB).
FWIW, I suspect there are significant inefficiencies in disk I/O related
request alignment: The code does not attempt to read from and write to
disk page boundaries, probably resulting in multiple low-level disk I/Os
per one Squid 4KB I/O in some (many?) cases. With modern non-rotational
storage these effects are probably less pronounced, but they probably
still exist.
The kernel drivers will mostly handle this for you if multiple requests
are available, but this is also almost irrelevant with current hardware,
typically it will be so fast software overhead will make it hard to
utilize modern large disk arrays properly you probably need to look at
other approaches like io_ring to get rid of the classical read/write
systemcall overhead dominating your performance.
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
https://lists.squid-cache.org/listinfo/squid-users
