----- Original Message -----
> From: "John A. Sullivan III" <jsullivan@xxxxxxxxxxxxxxxxxxx>
> To: lartc@xxxxxxxxxxxxxxx
> Sent: Monday, February 6, 2012 2:48:56 PM
> Subject: Traffic shaping for 95th percentile
>
> Hello, all. We are trying to figure out how to shape our colocated
> traffic to conform to 95th percentile billing, i.e., where the top 5%
> of
> traffic samples are discarded but the very next rate in the samples
> for
> the month becomes the billable rate for the month. So, if I have 100
> samples, 5 of which are 45 Mbps and the rest under 1.5 Mbps, I am
> only
> charged for a T1 but if I have 6 samples at 45 Mbps, I am charged for
> a
> T3.
>
> How can I shape my traffic to ensure I do not violate my guaranteed
> rate
> (we'll use the above numbers for this email)?
>
> I think this may be an interesting application for using a dilinear
> HFSC
> curve for ul but that is only a best guess and presents some serious
> issues. Let's assume a five minute sampling interval so I can have
> up
> to 36 hours in a month which are over my committed rate.
>
> Let's say I set my ul to m1 45mbits d 20m m2 1.5mbits, I'd get a nice
> 20
> minute burst for major downloads, backups, etc., but what if my queue
> continually clears and backlogs? Theoretically, I could run at 45
> mbits
> for 20 minutes, have an instant of dead time where the queue clears,
> and
> then start at 45mbits again crashing through my 95th percentile.
> Conversely, if the queue never fully drains, I will have capped my
> queue
> for the entire month when that was unnecessary. I'll also need to
> ensure
> I use SFQ on the back end or I'll have massive buffer bloat when the
> capacity changes suddenly.
>
> I suppose I could try to get our monitoring system to monitor 95th
> percentile and dynamically change ul as we get dangerously close but
> that is going to be somewhat complicated and does not give a
> stand-alone
> solution.
>
> Is there a viable way to do this with traffic shaping alone or is the
> only way by integrating it with a monitoring system? Thanks - John
>
><snip>
Hello, all. Here is our first stab at this. I'll post our script below in case it is helpful to anyone and also to gain feedback for improvements, oversights, errors, etc. It is intended to run continuously although I have not yet put in any init script headers. There is a second script to be run as a cron job to start the daemon in case the daemon fails. I would appreciate any and all feedback. Thanks - John
percentileshaper:
#!/bin/bash
# Copyright 2012 John A. Sullivan III
# Email: user:jsullivan domain:pacifera.com
# This script will shape traffic using HFSC to conform to percentile levels
# such as are often used in colocation billing. It will default to 95th
# percentile.
# One must specify the egress interface to be regulated.
# One can optionally specify, in order, target bandwidth in kbits (default
# 1544), burst bandwidth in kbits (default 45000), ingress interface (default
# ifb0), initial egress (TX) state (B for Burst or L for Limited - defaults to "B")
# initial ingress (RX) state (same values as TX), sample interval in seconds
# (default 300 - 5 minutes), percentile as
# a percent (default 95).
# If TXstate or RXstate are incorrect, the bandwidth will not be adjusted
if [ -z ${1} ];then
echo "Usage $(basename ${0}) egress-interface [targetbandwidth burst ingress-interface TXstate RXstate sampleinterval percentile]"
echo "For example, $(basename ${0}) eth0 1544 45000 ifb0 B B 300 95"
exit 5
fi
# ***************************
# PATHS
BASEP="/var/log/tcdata"
IP="/sbin/ip" # Path to ip utility
TC="/sbin/tc" # Path to tc utility
# Paths to egress and ingress traffic shaping scripts.
# These will be passed the interface name and the requested bandwidth
ESCRIPT=""
ISCRIPT=""
# ***************************
# ***************************
# Notification options
LOGS="Y" # Requires logger
CONSOLE="Y"
EMAIL="N" # Requires an email client with a mail command
RECIPIENTS="someone@xxxxxxxxxxxx"
# ***************************
# ***************************
# Parameters
# Mininum number of samples before we start changing traffic shaping
# Defaults to 8 hours at a 5 minutes sample rate
MINSAMPLES=96
# Value to add to wrapped counters
# Defaults to 32 bit counters
WRAP=4294967295
# Number by which TARGET and BURST are reduced to account for
# packet overhead such as CRC, Preamble, IFG, and Ethernet header
BWREDUCER=0.975
# The number by which the percentile will be raised
# This is to account for the inexactitude of the method, e.g.,
# it is possible our sample timing is slightly different from the
# billing entity or the last sample before the month closes shoots
# way over and we violate the percentile by a single entry.
# The default value is calculated assuming a five minute sample
# interval and a shortest month of 28 days yielding 8064 samples.
# We round this down to 8000. Offsetting this by 8 samples yields
# a nice round number of 0.1%. To illustrate, at 95th percentile,
# we discard 5% or sameples, i.e., 8000 * 0.05 = 400. Using the
# default offset, we have 8000 * 0.049 = 392 discarded samples.
PTILEOFFSET=0.1
# ***************************
message() {
if [ "${LOGS}" = "Y" ];then
logger -t "PERCENTILE SHAPER: " "${MSG}"
fi
if [ "${CONSOLE}" = "Y" ];then
echo -e "${MSG}"
fi
if [ "${EMAIL}" = "Y" ];then
echo -e "${MSG}" | mail -s "PERCENTILE SHAPER Message" ${RECIPIENTS}
fi
}
lastmonth()
{
((MONTH2 = MONTH1 - 1))
if [ ${MONTH2: -1:1} -eq 0 ];then
((MONTH2 = MONTH2 - 88))
fi
}
MSG=""
IF=${1}
TARGET=${2:-1544}
BURST=${3:-45000}
IIF=${4:-ifb0}
TXSTATE=${5:-B}
RXSTATE=${6:-B}
INTERVAL=${7:-300}
PTILE=${8:-95}
# We need to use bc since we are using non whole numbers
TARGET="$( echo "${TARGET} * ${BWREDUCER}" | bc)"
TARGET=${TARGET%%.*} # Truncate the decimals
BURST="$( echo "${BURST} * ${BWREDUCER}" | bc)"
BURST=${BURST%%.*}
PMULT="$( echo "scale=4;(100 - ${PTILE} - ${PTILEOFFSET}) / 100" | bc)"
case ${PTILE: -1:1} in
1 )
TH="st";;
2 )
TH="nd";;
3 )
TH="rd";;
* )
TH="th";;
esac
# Initialize data on first start
MONTH=$(date +%Y%m)
OLDMONTH=
declare -a RAW
declare -a SORTED
if [ -f ${BASEP}/${MONTH}/${IF}/bw/LastRXBytes ];then
OLDMONTH=${MONTH}
# Utilization will not exist of LastRXBytes does not exist
if [ -f ${BASEP}/${MONTH}/${IF}/bw/Utilization ];then
RAW=($(cat ${BASEP}/${MONTH}/${IF}/bw/Utilization | tr '\n' ' '))
fi
else
MONTH1=${MONTH}
lastmonth
LASTMONTH=${MONTH2}
if [ -f ${BASEP}/${LASTMONTH}/${IF}/bw/LastRXBytes ];then
OLDMONTH=${LASTMONTH}
fi
fi
if [ -n "${OLDMONTH}" ];then
if [ "$(wc -w ${BASEP}/${OLDMONTH}/${IF}/bw/LastRXBytes | cut -f 1 -d " ")" != "1" ];then
MSG="There is a critical problem with the ${BASEP}/${OLDMONTH}/${IF}/bw/LastRXBytes file"
message
exit 6
fi
if [ "$(wc -w ${BASEP}/${OLDMONTH}/${IF}/bw/LastTXBytes | cut -f 1 -d " ")" != "1" ];then
MSG="There is a critical problem with the ${BASEP}/${OLDMONTH}/${IF}/bw/LastTXBytes file"
message
exit 6
fi
OLDRXB="$(cat ${BASEP}/${OLDMONTH}/${IF}/bw/LastRXBytes)"
OLDTXB="$(cat ${BASEP}/${OLDMONTH}/${IF}/bw/LastTXBytes)"
TXSTATE="$(cat ${BASEP}/${OLDMONTH}/${IF}/bw/TXState)"
RXSTATE="$(cat ${BASEP}/${OLDMONTH}/${IF}/bw/RXState)"
fi
if [ "${TXSTATE}" != "B" -a "${TXSTATE}" != "L" -a "${TXSTATE}" != "b" -a "${TXSTATE}" != "l" ];then
MSG="Received invalid value for TXSTATE"
message
exit 6
fi
if [ "${RXSTATE}" != "B" -a "${RXSTATE}" != "L" -a "${RXSTATE}" != "b" -a "${RXSTATE}" != "l" ];then
MSG="Received invalid value for RXSTATE"
message
exit 6
fi
c=${#RAW[@]}
while [ true ]
do
NEWMONTH=0
LASTMONTH=""
ARCHIVEMONTH=""
if [ ! -d ${BASEP}/${MONTH}/${IF}/bw ];then # We could change months while running
mkdir -p ${BASEP}/${MONTH}/${IF}/bw
if [ $? -ne 0 ];then
MSG="Could not create ${BASEP}/${MONTH}/${IF}/bw directory"
message
exit 6
fi
NEWMONTH=1
MONTH1=${MONTH}
lastmonth
LASTMONTH=${MONTH2}
MONTH1=${LASTMONTH}
lastmonth
ARCHIVEMONTH=${MONTH2}
fi
cd ${BASEP}/${MONTH}/${IF}/bw
if [ $? -ne 0 ];then
MSG="Could not change to ${BASEP}/${MONTH}/${IF}/bw directory"
message
exit 6
fi
BYTES="$(${IP} -s link ls ${IF})"
RXB="$(echo $BYTES | sed 's/.*RX:[^0-9]*\([0-9][0-9]*\) .*/\1/')"
TXB="$(echo $BYTES | sed 's/.*TX:[^0-9]*\([0-9][0-9]*\) .*/\1/')"
if [ ${NEWMONTH} -eq 1 ];then
if [ -d ${BASEP}/${LASTMONTH}/${IF}/bw ];then
if [ -d ${BASEP}/${ARCHIVEMONTH}/${IF}/bw ];then
tar --remove-files -czPf ${BASEP}/${ARCHIVEMONTH}/${IF}/bw.tar.gz ${BASEP}/${ARCHIVEMONTH}/${IF}/bw
# Remember to untar with -P
fi
else
echo -e "${RXB}" > LastRXBytes
echo -e "${TXB}" > LastTXBytes
echo -e "${TXSTATE}" > TXState
echo -e "${RXSTATE}" > RXState
OLDRXB=${RXB}
OLDTXB=${TXB}
sleep ${INTERVAL}
continue
fi
fi
((RXBCOUNT = RXB - OLDRXB))
if [ ${RXBCOUNT} -lt 0 ];then
((RXBCOUNT = RXBCOUNT + WRAP))
fi
((TXBCOUNT = TXB - OLDTXB))
if [ ${TXBCOUNT} -lt 0 ];then
((TXBCOUNT = TXBCOUNT + WRAP))
fi
echo -e "${RXBCOUNT}\n${TXBCOUNT}" >> Utilization
echo -e "${RXB}" > LastRXBytes
echo -e "${TXB}" > LastTXBytes
OLDRXB=${RXB}
OLDTXB=${TXB}
RAW[${c}]=${RXBCOUNT}
((c++))
RAW[${c}]=${TXBCOUNT}
((c++)) # c now reflects the number of elements in the array since it is zero indexed
if [ ${c} -lt ${MINSAMPLES} ];then
sleep ${INTERVAL}
continue
fi
PTILENUM="$(echo "${c} * ${PMULT}" | bc)"
PTILENUM=${PTILENUM%%.*} # Strip decimals
((PTILENUM--)) # To account for zero indexing of the SORTED array
SORTED=($(printf '%d\n' ${RAW[@]} | sort -gr | tr '\n' ' '))
PTILEVAL="$(echo "${SORTED[${PTILENUM}]} * 8 / ${INTERVAL} / 1000" | bc)" # kbps
PTILEVAL=${PTILEVAL%%.*} # Strip the decimals and add 1 to round up to be safe
((PTILEVAL++))
CHANGE=0
if [ ${PTILEVAL} > ${TARGET} ];then
MSG="I am in danger of violating ${PTILE}${TH} percentile bandwidth on ${IF}."
TOVER="$(echo "(${TXBCOUNT} * 8 / ${INTERVAL} / 1000) > ${TARGET}" | bc)"
ROVER="$(echo "(${RXBCOUNT} * 8 / ${INTERVAL} / 1000) > ${TARGET}" | bc)"
if [ "${TXSTATE}" = "B" -a ${TOVER} -eq 1 ];then
MSG="${MSG}\nI will bandwitdth limit egress traffic."
CHANGE=1
eval ${ESCRIPT} ${IF} ${TARGET}
if [ $? -ne 0 ];then
MSG="${MSG}\nSERIOUS ERROR: I was not able to adjust egress bandwidth on ${IF}!!!"
else
TXSTATE="L" # Limited
echo "${TXSTATE}" > TXState
fi
fi
if [ "${RXSTATE}" = "B" -a ${ROVER} -eq 1 ];then
if [ ${CHANGE} -eq 1 ];then
ALSO="also "
else
ALSO=""
fi
MSG="${MSG}\nI will ${ALSO}bandwitdth limit ingress traffic."
CHANGE=1
eval ${ISCRIPT} ${IIF} ${TARGET}
if [ $? -ne 0 ];then
MSG="${MSG}\nSERIOUS ERROR: I was not able to adjust ingress bandwidth on ${IIF}!!!"
else
RXSTATE="L" # Limited
echo "${RXSTATE}" > RXState
fi
fi
else
MSG=""
if [ "${TXSTATE}" = "L" ];then
MSG="I am restoring egress traffic to burst mode."
CHANGE=1
eval ${ESCRIPT} ${IF} ${BURST}
if [ $? -ne 0 ];then
MSG="${MSG}\nSERIOUS ERROR: I was not able to adjust egress bandwidth on ${IF}!!!"
else
TXSTATE="B" # Limited
echo "${TXSTATE}" > TXState
fi
fi
if [ "${RXSTATE}" = "L" ];then
if [ ${CHANGE} -eq 1 ];then
NL="\n"
ALSO="also "
else
NL=""
ALSO=""
fi
MSG="${MSG}${NL}I am ${ALSO}restoring ingress traffic to burst mode."
CHANGE=1
eval ${ISCRIPT} ${IIF} ${TARGET}
if [ $? -ne 0 ];then
MSG="${MSG}\nSERIOUS ERROR: I was not able to adjust ingress bandwidth on ${IIF}!!!"
else
RXSTATE="B" # Limited
echo "${RXSTATE}" > RXState
fi
fi
fi
if [ ${CHANGE} -ne 0 ];then
message
fi
sleep ${INTERVAL}
done
pshapercheck:
#!/bin/bash
# Copyright 2012 John A. Sullivan III
# Email: user:jsullivan domain:pacifera.com
# This script checks if the percentileshaper script is running and
# starts it if it is not
if [ -z ${1} ];then
echo "Usage $(basename ${0}) egress-interface [targetbandwidth burst ingress-interface TXstate RXstate sampleinterval percentile]"
echo "For example, $(basename ${0}) eth0 1544 45000 ifb0 B B 300 95"
exit 5
fi
if [ -z "$(ps x | egrep "(^|[[:space:]])/root/percentileshaper $@([[:space:]]|$)")" ];then
/root/percentileshaper $@ &
fi
exit 0
--
To unsubscribe from this list: send the line "unsubscribe lartc" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
