On 01/18/2018 04:04 PM, Ivan Larionov wrote: > So if I understand you correctly max-swap-rate doesn't limit disk IOPS Correct. Squid does not know what the disk- or OS-level stats are. > but limits squid swap ops instead If you define a "swap op" as reading or writing a single I/O page for the purpose of swapping in or swapping out a cached object, then yes. Processing an HTTP transaction may involve reading and writing many I/O pages. See my original response for a more precise definition. > and every squid operation could in > theory use more than 1 disk IO operation. Yes, depending on the file system, disk, etc. Ideally, rock cache_dirs should use raw disk partitions, but we do not have a sponsor for that work. > This means we can't really say > "limit swap ops to 1500 because our disk can handle 1500 iops" but > should figure out the number after testing different values. Correct. > Ok, I suppose I'll just do what Rock documentation says – will test > different values and figure out what works for us. Sounds like a good plan! You may even find a good/stable correlation between Squid swap I/O ops and low-level disk I/O ops. Please consider reporting your findings for others to reuse. Cheers, Alex. > On Thu, Jan 18, 2018 at 2:54 PM, Alex Rousskov wrote: > > On 01/18/2018 03:16 PM, Ivan Larionov wrote: > > > cache_dir max-swap-rate documentation says that swap in requests > > contribute to measured swap rate. However in our squid 4 load test we > > see that read_ops + write_ops significantly exceeds the max-swap-rate we > > set and squid doesn't limit it. > > In this context, a single Squid "op" is a read or write request from > worker to the disker process. These requests are up to one I/O page in > size. A single I/O page is 32*1024 bytes. See Ipc::Mem::PageSize(). > > * A single read request usually ends up being a single pread(2) system > call that reads at most one I/O page worth of data from disk. See > diskerRead(). > > * A single write request usually ends up being a single pwrite(2) system > call that writes at most one I/O page worth of data to disk. However, if > that single pwrite() does not write everything a worker has requested to > write, then Squid will make more pwrite() calls, up to 10 calls total. > See diskerWrite(). > > Within a single cache miss transaction, the rock code should accumulate > small swapout requests from Store into page-size write requests to > disker, but I do not remember how complete those optimizations are: It > is possible that smaller-than-page writes get through to diskers, > increasing the number of write requests. Same for reading cache hits. > > > What is the "op" in read_ops and write_ops you have measured? > > > Since Squid does not (and does not want to) have access to low-level > disk stats and since Squid cannot assume exlusive disk ownership, the > rate-limiting feature for rock cache_dirs does not know how many > low-level disk operations the disk is doing and how those operations > correspond to what Squid is asking the disk to do. > > > HTH, > > Alex. > > > > I tried to set it to 200 to confirm that it actually works and saw > that > > it does. Squid started warning about exceeding max-swap-rate. But > looks > > like real limit is higher than the value we set in configuration. > > > > Hardware: > > > > AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops > > burstable. > > > > Config: > > > > cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500 > > max-swap-rate=1200 slot-size=16384 > > > > IOPS squid pushes under our load test: > > > > read ~800 ops/sec > > write ~1100 ops/sec > > > > In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500 > > ops/sec and after spending too much "burst balance" we started getting > > throttled from AWS side. > > > > Could you please comment on this behavior? What the limit should > we set > > to stay under 1500 ops/sec for swap in + swap out operations? > > > > Thanks. > > > > Squid version: > > > > Squid Cache: Version 4.0.22 > > Service Name: squid > > configure options: '--program-prefix=' '--prefix=/usr' > > '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin' > > '--sysconfdir=/etc/squid' '--libdir=/usr/lib' > > '--libexecdir=/usr/lib/squid' '--includedir=/usr/include' > > '--datadir=/usr/share' '--sharedstatedir=/usr/com' > > '--localstatedir=/var' '--mandir=/usr/share/man' > > '--infodir=/usr/share/info' '--enable-epoll' > > '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock' > > '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests' > > '--with-large-files' '--with-maxfd=16384' '--enable-htcp' > > > > -- > > With best regards, Ivan Larionov. > > > > > > _______________________________________________ > > squid-users mailing list > > squid-users@xxxxxxxxxxxxxxxxxxxxx > <mailto:squid-users@xxxxxxxxxxxxxxxxxxxxx> > > http://lists.squid-cache.org/listinfo/squid-users > <http://lists.squid-cache.org/listinfo/squid-users> > > > > > > > -- > With best regards, Ivan Larionov. _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
