> On Tue, 2004-03-23 at 04:39, fedora@xxxxxxxxxxxxxxxxxxxxxxxxx wrote: > > I'd disagree about updates via BT not having a use. > > I'll be glad to see you prove me wrong but I don't think that will > happen. > It's good to see you have an open mind, hopefully I can at least make you understand why I wish to "play around" with the concept. > > > > >From the various comments relating to slow/failed downloads, and the > > fedoralegacy.org people having trouble with bandwidth usage I'd say > > that there are a reasonable number of concurrent downloaders, which is > > an ideal environment for BT. > > You just restated the problem we all know - mirrors need to be more > easily synchronized and people need to be able to use them with ease. > Another solution is that there needs to be a method of reduceing the bandwidth requirements on each machine acting as a mirror (e.g. Using BT to distribute the load between mirrors and clients). > > > Given that most people will only download updates from the last few > > days on a regular basis (i.e. do an update once a day, week, or > > whatever), it's highly likley that most of the concurrent downloads > > are for a small set of files. > > This is where your assumptions are getting flawed. That mostly works for > rawhide but since people are in and out of sync you're going to have a > hard time counting on this assumption. > Everyone doesn't need to be in sync, if a client offers what it has others will take it if they need it. With the number of clients involved in downloading Fedora updates the critical mass for clients needed to distribute load should be reached quite soon after a patch is released. > Moreover if the mirrors can't get the data there won't be anyone to seed > the torrents. > The mirrors don't need to be permanently in sync, they will just be another node in the P2P network. When they are up to date they will share the load along with the other available nodes. In the worst case the main side acts as the seed and as others download they in turn act as temporary seeds. I've written my own BT tracker to which I can assign priorities to seeds which allows for "core" servers to be far down the list available to clients so that they act as last hope peers. > > > The paper I sent the link for previously shows how the list of > > available updates can be rolled up to make the most of BT features. > > Combined with the ability to tune the size of download chunks it > > should be reasonably easy to create a configuration which makes > > effective use of the limited time people are online (e.g. for a 500kb > > update use 50kb chunks would mean that each user would offer upto 9 > > completed chunks before their download is complete). > > I read the paper, I don't think you're counting on: > > 1. startup overhead The only overhead would be the download of the updates file which can be reduced via various strategies (delta transfers, using a single torrent so the clients can pick what they don't have, etc.). > 2. cpu overhead Minimal, The developers of BT clients have been working on this. > 3. proxies/routers/port blocks upstream of the user This is a problem with any system. Not everyone needs to upload, and if they are unable to download then it is an issue they need to take up with their local admins. > 4. complexity of doing this all in the backend of up2date/yum/etc > I've already been looking into it and it's not such a huge task. It'll take time, but I have some spare so I'm going to plough on. > I've gone through the 'use bittorrent to download updates' concept > before and it just doesn't pay off. Ask Adrian Likins what he thinks of > it. He wrote up2date and worked on a bittorrent back end. I've had very > long discussions with michael stenner who works on urlgrabber(the module > yum uses to download) and it seems like bittorrent will be more pain > than benefit, both programmatically and speed-wise. > Has anyone constructed a prototype?, I'd be interested to see their work and learn from it. > > > I hope this helps you understand my motivation. > > I understand your motivation, definitely, but I'm concerned that you're > focusing your attention in the wrong place. > > Again, I'd encourage you to do your work to make bittorrent a useful > tool for synchronizing the official mirrors. It's a smaller problem to > start with and a good place to cut your teeth in this area. > This has already been covered by a number of people. One of the most promising ideas is to use RSS as a method for syndicating new updates which are available, the RSS contains the URL of the new files .torrent, and the mirrors download the new files using the torrent, which in turn uses the other mirrors as peers. The reason this isn't so hot for client downloads is that the XML RSS doesn't contain enough information to make a call on the download of dependancies. Al.