The message indicates that the numbers of membufs should be
because there are insufficent membufs to use for caching
objects. The reason for having 'insufficient membufs'
is explained below.
Given the fact that the average object size is 13 KB, the given
configuration effectively puts a very large percentage of objects,
most likely more than 90% in the COSS-based cache dir. This puts
a high pressure on (the disk with) COSS and I bet that the disk
with COSS (/cache3) is 100% busy while the other three are mostly idle.
COSS is very good for small objects and AUFS is fine with larger objects.
There is one larger disk. But this larger disk is not faster.
It will perform worse with more objects on it than the other disks.
To find out more about disk I/O and pressure on the disk with COSS, one
can evaluate the output of iostat or 'vmstat -d 5 5'
I recommend to change the configuration, to utilise all disks in a
more balanced way. Be sure to also look at the output of iostat.
My suggestion is to use COSS only for objects smaller than 64 KB.
Depending on the average object size of your cache, this limit
may be set lower.
So I suggest:
cache_dir coss /cache3 110000 max-size=65535 max-stripe-waste=32768 block-size=8192 membufs=15
cache_dir aufs /cache1 115000 16 256 min-size=65536
cache_dir aufs /cache2 115000 16 256 min-size=65536
cache_dir aufs /cache4 115000 16 256 min-size=65536
And to observe the log and output of iostat.
If the disk I/O is balanced and the message about membufs reappears and
you have sufficient free memory, you may increase membufs. If the I/O is
not balanced, the limit of 64KB may be decreased to 16KB.
Depending on the results and iostat, it may be better to
have 2 disks with COSS and 2 disks with AUFS:
cache_dir coss /cache3 110000 max-size=16383 max-stripe-waste=32768 block-size=8192 membufs=15
cache_dir aufs /cache1 110000 max-size=16383 max-stripe-waste=32768 block-size=8192 membufs=15
cache_dir aufs /cache2 115000 16 256 min-size=16384
cache_dir aufs /cache4 115000 16 256 min-size=16384
Marcus
Ragheb Rustom wrote:
Dear All,
I have a squid cache proxy which is delivering content to around 3000+
users. After some problems with performance on AUFS under peak hour loads I
have used one of my cache_dirs as COSS disk following Amos Settings on
squid-cache website while left all others as AUFS that hold files bigger
than 1MB. After running perfectly for some time with coss and very beautiful
results I started seeing the below messages in my cache.log file:
storeCossCreateMemOnlyBuf: no free membufs. You may need to increase the
value of membufs on the /cache3/coss1 cache_dir
here are my squid.conf settings:
cache_dir coss /cache3/coss1 110000 max-size=1048575 max-stripe-waste=32768
block-size=8192 membufs=15
cache_dir aufs /cache1 115000 16 256 min-size=1048576
cache_dir aufs /cache2 115000 16 256 min-size=1048576
cache_dir aufs /cache4/cache1 240000 16 256 min-size=1048576
Please note all my Hdds are SAS 15k drives sizes as follows:
/cache1 147GB
/cache2 147GB
/cache3 147GB
/cache4 450GB
The system is dual Xeon quad core intel server with 16GB of physical Ram
Do you think I should increase the membufs value and what do you think the
best or optimal value for a such system should be?
Sincerely,
Ragheb Rustom
Smart Telecom S.A.R.L
