Jad madi wrote:
I'm building an RSS aggregator so I'm trying to find out the best way to
parse users account feeds equally so Lets say we have 20.000 user with
average of 10 feeds in account so we have about
200.000 feed
How would you schedule the parsing process to keep all accounts always
updated without killing the server? NOTE: that some of the 200.000 feeds
might be shared between more than one user
this is just an idle thought, but could you base it on prime numbers?
for instance, let's say you have 6 feeds which in the last week have generated
(a=10, b=8, c=5, d=7, e=13, f=2) articles.
order them by article amount descending in last week: e, a, b, d, c, f
assign primes to them: e=1, a=2, b=3, d=5, c=7, f=11
let $i=1
every 6 hours, check feeds who's primes are divisible by $i, then add 1 to $i
at the end of the week, recalculate the primes.
in this example, the feed checks would be:
monday (1,2,3,4) = (e), (e,a), (e,b), (e,a)
tuesday (5,6,7,8) = (e,d), (e,a,b), (e,c), (e,a)
wednesday (9,10,11,12) = (e,b), (e,a,d), (e,f), (e,a,b)
thursday (13,14,15,16) = (e), (e,a,c), (e,b,d), (e,a)
friday (17,18,19,20) = (e), (e,a,b), (e), (e,a,d)
saturday (21,22,23,24) = (e,b,c), (e,a,f), (e), (e,a,b)
sunday (25,26,27,28) = (e,d), (e,a), (e,a,b), (e,a,c)
in this example, at most three out of six are being checked for at a time, but
everything gets at least a chance.
e had 13 articles, is checked 28 times
a had 10 articles, is checked 14 times
b had 8 articles, is checked 9 times
d had 7 articles, is checked 5 times
c had 5 articles, is checked 4 times
f had 2 articles, is checked 2 times
Kae
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
