Jeff Spaleta wrote:
On 6/7/05, david <dfarning@xxxxxxxxxxxxx> wrote:
I'm looking at pulling the timestamp from the master about every hour.
If a master version/arch timestamp changes, I will check evey hour
until it becomes up to date.
If a prior version/arch timestamp fails, I will check evey hour until
it becomes up to date.
Every {hour | couple of hours] I will check a random version/arch
timestamp per mirror to ensure that the entire mirror isn't dead.
So you are checking every mirror once an hour after you see the master
mirror timestamp change? So the time between checking the master and
the time to check an individual a mirror is at maximum 1 hour? You
need to have a state of "unknown" for each mirror between when you
know the master has changed and you have yet to check to see if the
mirror has been updated. You can't assume that a mirror is out of date
until you check that mirror. Nor can you assume the mirror is up2date
either. Mirrors are simply in an unverified state in the meantime.
Good heavens man, I just working on this a few hours ago. So as they
say, "The concrete ain't set up real firm yet."
I originally started looking at the mirror status problem to set up some
sort of geoip/ mirror_list/dns_server for a repository and wanted to
ensure that I am not referring to dead or out of date mirrors. But,
that a good why off.
In the mean time--
My intentions are to probe all master timestamps each hour. So, for the
development branch
1 mirror X 6 arch X ~1.1K (size of repomd.xml) = 250K traffic per hour
If a master timestamp has change - probe all mirrors
~250 mirror X 1 arch X ~1.1K = 1.5M traffic per arch per change
(development changes ~ once per day)
If a master timestamp has not changed - probe only those mirrors that
did not report op2date last probe
~150 mirror X 1 arch X ~1.1K = 150K traffic first hour after change
(sync delay)
~75 mirror X 1 arch X ~1.1K = 750K traffic second hour after sage
(sync delay)
...
Every time a master timestamp is changed all mirrors are immediately
reprobed to eliminate "the dreaded unknown state."
[dfarning@localhost m3d]$ time /home/dfarning/workspace/m3d/mirmon.pl -v
-get all -c development/core.conf -probes 25
real 13m12.098s
user 0m51.962s
sys 0m31.346s
An update should be pretty quick because the master timestamp seldom
changes.
that 25 means 25 different mirrors? How long does it take to do the
whole set of mirrors assuming just x86? Thinking again about the
unverified state for the web page summary... if it takes 15 minutes to
work through a list of mirrors.. you want to make sure that for those
15 minutes the webpage isnt misleading.
A probe is a little chunk of c code ( part of a repository tool kit that
I am working on) that return the primary.xml timestamp for a given repo
ie. rtk_probe
ftp://ftp.linux.ncsu.edu/pub/fedora/linux/core//development/ returns
1118144219
hmm.. lets rerun for only one arch (x86) and up the probes to 250 and
see the results on my dsl line
Took 23 seconds to get all but the last 14 mirrors. Those reported time
out failure are 360 seconds as set in the probe.
Not bad
How would you feel if I stated the time since the last probe so user are
aware of that fact.
Also, the mirror master could (with the help of m3d) clean up the mirror
list so not as much effort is spent contacting dead mirrors.
-jef
thanks
-dtf
--
fedora-devel-list mailing list
fedora-devel-list@xxxxxxxxxx
http://www.redhat.com/mailman/listinfo/fedora-devel-list
