Thank you for the fast reply!
read_ops and write_ops is AWS EBS metric and in general it correlates with OS-level reads/s writes/s stats which iostat shows.
So if I understand you correctly max-swap-rate doesn't limit disk IOPS but limits squid swap ops instead and every squid operation could in theory use more than 1 disk IO operation. This means we can't really say "limit swap ops to 1500 because our disk can handle 1500 iops" but should figure out the number after testing different values.
Ok, I suppose I'll just do what Rock documentation says – will test different values and figure out what works for us.
Thanks again.
On Thu, Jan 18, 2018 at 2:54 PM, Alex Rousskov <rousskov@xxxxxxxxxxxxxxxxxxxxxxx> wrote:
On 01/18/2018 03:16 PM, Ivan Larionov wrote:
> cache_dir max-swap-rate documentation says that swap in requests
> contribute to measured swap rate. However in our squid 4 load test we
> see that read_ops + write_ops significantly exceeds the max-swap-rate we
> set and squid doesn't limit it.
In this context, a single Squid "op" is a read or write request from
worker to the disker process. These requests are up to one I/O page in
size. A single I/O page is 32*1024 bytes. See Ipc::Mem::PageSize().
* A single read request usually ends up being a single pread(2) system
call that reads at most one I/O page worth of data from disk. See
diskerRead().
* A single write request usually ends up being a single pwrite(2) system
call that writes at most one I/O page worth of data to disk. However, if
that single pwrite() does not write everything a worker has requested to
write, then Squid will make more pwrite() calls, up to 10 calls total.
See diskerWrite().
Within a single cache miss transaction, the rock code should accumulate
small swapout requests from Store into page-size write requests to
disker, but I do not remember how complete those optimizations are: It
is possible that smaller-than-page writes get through to diskers,
increasing the number of write requests. Same for reading cache hits.
What is the "op" in read_ops and write_ops you have measured?
Since Squid does not (and does not want to) have access to low-level
disk stats and since Squid cannot assume exlusive disk ownership, the
rate-limiting feature for rock cache_dirs does not know how many
low-level disk operations the disk is doing and how those operations
correspond to what Squid is asking the disk to do.
HTH,
Alex.
> ______________________________
> I tried to set it to 200 to confirm that it actually works and saw that
> it does. Squid started warning about exceeding max-swap-rate. But looks
> like real limit is higher than the value we set in configuration.
>
> Hardware:
>
> AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops
> burstable.
>
> Config:
>
> cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500
> max-swap-rate=1200 slot-size=16384
>
> IOPS squid pushes under our load test:
>
> read ~800 ops/sec
> write ~1100 ops/sec
>
> In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500
> ops/sec and after spending too much "burst balance" we started getting
> throttled from AWS side.
>
> Could you please comment on this behavior? What the limit should we set
> to stay under 1500 ops/sec for swap in + swap out operations?
>
> Thanks.
>
> Squid version:
>
> Squid Cache: Version 4.0.22
> Service Name: squid
> configure options: '--program-prefix=' '--prefix=/usr'
> '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
> '--sysconfdir=/etc/squid' '--libdir=/usr/lib'
> '--libexecdir=/usr/lib/squid' '--includedir=/usr/include'
> '--datadir=/usr/share' '--sharedstatedir=/usr/com'
> '--localstatedir=/var' '--mandir=/usr/share/man'
> '--infodir=/usr/share/info' '--enable-epoll'
> '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
> '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
> '--with-large-files' '--with-maxfd=16384' '--enable-htcp'
>
> --
> With best regards, Ivan Larionov.
>
>
_________________
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
With best regards, Ivan Larionov.
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
