Hi all! InstantMirror currently works fine but involves a lot of configuration and that might actually stop someone from use it. I came across this project recently and found it really interesting. After reading the discussions on fedora-devel, warren's journal, InstantMirror wiki, I thought that I would come up with a redesign that would be more yum friendly and will involve minimal configuration for setup. Below is my point of view of new InstantMirror design. Note: You can access the design proposal here (http://iyum.saini.co.in/index.php/InstantMirror) also. Redesign Proposal ***************** Basically the most obvious use case of InstantMirror is to be used by Yum for updates. I propose to develop InstantMirror keeping in mind that it should integrate with Yum. In the new design I am using not multiple programs to get the job done as proposed by warren here (https://fedorahosted.org/InstantMirror/wiki/InstantMirrorDaemon). There will be only one daemon that will continuously listen to requests and if necessary it will fork itself. Below is the method of operation for the daemon. Method of Operation ******************* 1. InstantMirror gets a client request for a URL. 2. Check: if URL is not in (RPM, metadata file) * Then its none of our business. * Let proxy handle it the normal way. * Done and exit. 3. Error Check: if remote host is not reachable * Check: if RPM/metadata is available in cache 1. Stream the RPM/metadata from cache. 2. Done and exit. * else 1. Throw a "No route to host" error. 2. Done and exit. 4. Check: if RPM/metadata is available in cache * Check: if RPM/metadata in cache is older than upstream 1. Delete RPM/metadata from cache. 2. Download and stream. 3. Done and exit. * Check: if RPM/metadata matches upstream or newer than upstream 1. Stream the RPM/metadata from cache. 2. Done and exit. * Check: if RPM/metadata does not exsit upstream 1. Delete RPM/metadata from cache. 2. Throw a "Not found" error. 3. Done and exit. 5. Check: if RPM/metadata is not available in cache * Download and stream. * Done and exit. Download Process **************** In the above operation everything is clear except the download process. If a file is already being downloaded from upstream and another client request come in for the same file, then we have two options to continue downloading 1. Download via only first instance (master) and let the other instances (slaves) copy the partial content to the client. The disadvantage is that the slaves will be throttled by the master's download speed. 2. The other instance also starts downloading from upstream and append data to the local file. Stream the data to clients when download is finished. This is quite complicated and as of now I don't even know how to do it or even if its feasible to do it. I am currently trying to get the hang of downloading process. The above design is more or less same as the previous design with may be a bit of improvements. Now, we have decided to have two types of InstantMirror. 1. InstantMirror to be used by a small group of people. In this case we have to get rid of all the dependencies like squid, apache because for a small setup nobody is going to configure squid and apache. So, we use a proxy server implemented in python for this kind of setup and integrate InstantMirror with it in caching mode. So that it becomes easy to setup and don't require squid, apache or whatever else. 2. InstantMirror to be used by an organization. As almost all the organization ( i am focusing more on institutes/universities here) use a common proxy server to access the Internet, we will have the InstantMirror which can be integrated with squid. There will be no difficulty in setup as the people already use squid (assuming squid is widely used in Unix/Linux world) and know how to configure it. We can't use proxy server implemented in python here because no organization would ever agree to use a stripped down version of proxy server instead of squid. If the above sounds interesting, then I also propose to build a Yum plugin (say Yum Client) which will interact with InstantMirror sitting near the proxy server. Yum Client will periodically update the InstantMirror with the information like which packages are very frequently updated by user (kernel, kde, yum), which packages are never updated or updated rarely (vim). Using this information InstantMirror will optimize the download queue and using some very simple techniques for queuing a download etc., we can optimize the bandwidth usage. Imagine a university with thousands of Linux users and everyone is updating their system weekly. GBs of bandwidth is being wasted every week due to subsequent downloads of the same package. If you have any suggestions for improvements, comments on the current design or you want to criticize the design, please reply back. They would really help me to improve. InstantMirror - https://fedorahosted.org/InstantMirror/wiki InstantMirrorDaemon - https://fedorahosted.org/InstantMirror/wiki/InstantMirrorDaemon InstantMirror needs a rethink - http://www.redhat.com/archives/rhl-devel-list/2008-January/msg02341.html Warren Togami's Journal - http://wtogami.livejournal.com/20536.html PS : This is my first RFC, if i wrote it badly please forgive me :) ------------------------------------------------------- Thank you, Kulbir Saini, Computer Science and Engineering, International Institute of Information Technology, Hyderbad, India - 500032. My Home-Page: http://saini.co.in/ My Institute: http://www.iiit.ac.in/ My Linux-Blog: http://linux.saini.co.in/ IRC nick : generalBordeaux Channels : #fedora, #fedora-devel, #yum on freenode ------------------------------------------------------- -- fedora-devel-list mailing list fedora-devel-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-devel-list
