On Mon, Apr 11, 2005 at 03:13:17AM -0400, Chuck R. Anderson wrote: > On Mon, Apr 11, 2005 at 12:45:17AM -0400, seth vidal wrote: > > I wanna see what sort of bandwidth saving is possible. Rather than just > > take your word and jigdo's website word for it. > > Well, for the mirrors, the bandwidth savings would be pretty obvious > since they would only need to downloaded the distro once, not thrice > like many of them do now (tree + CDs + DVD). > > > Let's get some numbers, shall we? > > More interesting would be rsyncing the tree from rawhide -> test -> > final release, then using master-server-provided jigdo files to create > the CD and DVD ISOs. I agree that getting numbers for this would be > worthwhile. I'll see what I can do here. FC4T2 size: test/3.91/i386/ test/3.91/SRPMS/ Total 15 GiB For anyone that maintains a rawhide mirror already (most mirrors), going to FC4T2 you only need to download this much: 183 MiB That includes 1.1 MiB of jigdo/template files. You can find the jigdo/template files I generated here. Feel free to edit the .jigdo files themselves and change/add more lines to the [Servers] sections. Right now I left them pointing at the master server. Or you can specify a local file path during reassembly (below). http://download.wpi.edu/pub/jigdo/ For jigdo itself, the packages which I intend to submit to Extras are here (built/tested on FC3): http://angus.ind.wpi.edu/~cra/fedora/extras/jigdo/ To recreate the original CD/DVD ISO images from your file tree: jigdo-lite http://download.wpi.edu/pub/jigdo/FC4-test2-i386-disc1.jigdo Jigsaw Download "lite" Copyright (C) 2001-2004 | jigdo@ Richard Atterer | atterer.net Downloading .jigdo file --10:56:26-- http://download.wpi.edu/pub/jigdo/FC4-test2-i386-disc1.jigdo => `FC4-test2-i386-disc1.jigdo' Resolving download.wpi.edu... 130.215.32.86 Connecting to download.wpi.edu[130.215.32.86]:80... connected. HTTP request sent, awaiting response... 200 OK Length: 60,698 [text/plain] 0K 100% 165.39 MB/s 10:56:26 (165.39 MB/s) - `FC4-test2-i386-disc1.jigdo' saved [60,698/60,698] ----------------------------------------------------------------- Images offered by `http://download.wpi.edu/pub/jigdo/FC4-test2-i386-disc1.jigdo': 1: FC4-test2-i386-disc1.iso ----------------------------------------------------------------- If you already have a previous version of the CD you are downloading, jigdo can re-use files on the old CD that are also present in the new image, and you do not need to download them again. Mount the old CD ROM and enter the path it is mounted under (e.g. `/mnt/cdrom'). Alternatively, just press enter if you want to start downloading the remaining files. Files to scan: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/os Downloading .template file --10:56:38-- http://download.wpi.edu/pub/jigdo/FC4-test2-i386-disc1.template => `FC4-test2-i386-disc1.template' Resolving download.wpi.edu... 130.215.32.86 Connecting to download.wpi.edu[130.215.32.86]:80... connected. HTTP request sent, awaiting response... 200 OK Length: 129,022 [text/plain] 0K . 100% 5.16 MB/s 10:56:38 (5.16 MB/s) - `FC4-test2-i386-disc1.template' saved [129,022/129,022] general: Image file: general: Jigdo: general: Template: general: [exit(0)] general: Image file: FC4-test2-i386-disc1.iso general: Jigdo: FC4-test2-i386-disc1.jigdo general: Template: FC4-test2-i386-disc1.template 100% 0k/0k scanning Successfully created `FC4-test2-i386-disc1.iso' general: [exit(0)] ----------------------------------------------------------------- Finished! The fact that you got this far is a strong indication that `FC4-test2-i386-disc1.iso' was generated correctly. I will perform an additional, final check, which you can interrupt safely with Ctrl-C if you do not want to wait. general: Image file: FC4-test2-i386-disc1.iso general: Jigdo: FC4-test2-i386-disc1.jigdo general: Template: FC4-test2-i386-disc1.template 0% 256k/633206k verifying image general: [exit(0)] 26.82user 9.41system 1:32.50elapsed 39%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (3major+10954minor)pagefaults 0swaps (repeat for the other images) Here is the procedure I used and the timings of the runs: cd test/3.91/SRPMS/ cp -avx ../../../development/SRPMS/ . cd ../i386/os/ cp -avx ../../../development/i386/ . cd ../debug/ mv ../os/debug . cd test/3.91/SRPMS/ /usr/bin/time rsync -av --delete-after \ rsync://[your-neighborhood-friendly-mirror]/test/3.91/SRPMS/ . receiving file list ... done ./ comps-3.91-0.20050407.src.rpm fedora-release-3.91-1.src.rpm deleting [a bunch of obsolete rawhide files] sent 148 bytes received 103420 bytes 3395.67 bytes/sec total size is 2605906175 speedup is 25161.31 0.03user 1.56system 0:29.52elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+490minor)pagefaults 0swaps cd test/3.91/i386/debug/ /usr/bin/time rsync -av --delete-after \ rsync://[your-neighborhood-friendly--mirror]/test/3.91/i386/debug/ . receiving file list ... done ./ headers/ repodata/ fedora-release-debuginfo-3.91-1.i386.rpm headers/[a bunch of hdr files] repodata/filelists.xml.gz repodata/other.xml.gz repodata/primary.xml.gz repodata/repomd.xml deleting [a bunch of obsolete rawhide files] sent 15153 bytes received 9247918 bytes 173141.51 bytes/sec total size is 1885583549 speedup is 203.56 0.11user 2.96system 0:52.33elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+574minor)pagefaults 0swaps cd test/3.91/i386/os/ /usr/bin/time rsync -av --delete-after \ rsync://[your-neighborhood-friendly-mirror]/test/3.91/i386/os/ . receiving file list ... done ./ Fedora/ Fedora/RPMS/ Fedora/base/ .discinfo Fedora/RPMS/comps-3.91-0.20050407.i386.rpm Fedora/RPMS/fedora-release-3.91-1.i386.rpm Fedora/base/comps.rpm Fedora/base/comps.xml Fedora/base/hdlist Fedora/base/hdlist2 Fedora/base/hdstg2.img Fedora/base/netstg2.img SRPMS -> ../../SRPMS headers/ images/ images/pxeboot/ Fedora/base/stage2.img isolinux/ repodata/ GPL README-Accessibility README-en RELEASE-NOTES-en RPM-GPG-KEY RPM-GPG-KEY-beta RPM-GPG-KEY-fedora RPM-GPG-KEY-fedora-rawhide RPM-GPG-KEY-fedora-test RPM-GPG-KEY-rawhide autorun eula.txt headers/comps-1-3.91-0.20050407.i386.hdr headers/comps-1-3.91-0.20050407.src.hdr headers/fedora-release-0-3.91-1.i386.hdr headers/fedora-release-0-3.91-1.src.hdr headers/header.info headers/header.src.info images/README images/boot.iso images/diskboot.img images/pxeboot/README images/pxeboot/initrd.img images/pxeboot/vmlinuz images/rescuecd.iso isolinux/boot.msg isolinux/general.msg isolinux/initrd.img isolinux/isolinux.bin isolinux/isolinux.cfg isolinux/memtest isolinux/options.msg isolinux/param.msg isolinux/rescue.msg isolinux/snake.msg isolinux/splash.lss isolinux/vmlinuz repodata/comps.xml repodata/filelists.xml.gz repodata/other.xml.gz repodata/primary.xml.gz repodata/repomd.xml deleting yumgroups.xml deleting i386/images/boot.iso deleting i386/images/.listing deleting directory i386/images deleting directory i386 deleting [a bunch of obsolete rawhide files] sent 264720 bytes received 180311140 bytes 1059095.95 bytes/sec total size is 2699311849 speedup is 14.95 3.54user 12.18system 2:50.71elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+2123minor)pagefaults 0swaps /usr/bin/time jigdo-file make-template \ --cache=/var/cache/jigdo-cache.db \ --image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-DVD.iso \ --jigdo=FC4-test2-i386-DVD.jigdo \ --template=FC4-test2-i386-DVD.template \ --label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \ --uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \ /srv/ftp/pub/fedora/linux/core//test/3.91/ general: Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-DVD.iso general: Jigdo: FC4-test2-i386-DVD.jigdo general: Template: FC4-test2-i386-DVD.template general: URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/' Finished - image size is 2657673216 bytes. general: [exit(0)] 1487.39user 18.08system 26:38.37elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (10major+2949minor)pagefaults 0swaps /usr/bin/time jigdo-file make-template \ --cache=/var/cache/jigdo-cache.db \ --image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc1.iso \ --jigdo=FC4-test2-i386-disc1.jigdo \ --template=FC4-test2-i386-disc1.template \ --label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \ --uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \ /srv/ftp/pub/fedora/linux/core//test/3.91/ general: Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc1.iso general: Jigdo: FC4-test2-i386-disc1.jigdo general: Template: FC4-test2-i386-disc1.template general: URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/' Finished - image size is 648402944 bytes. general: [exit(0)] 563.70user 1.80system 9:28.03elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (17major+3153minor)pagefaults 0swaps /usr/bin/time jigdo-file make-template \ --cache=/var/cache/jigdo-cache.db \ --image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc2.iso \ --jigdo=FC4-test2-i386-disc2.jigdo \ --template=FC4-test2-i386-disc2.template \ --label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \ --uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \ /srv/ftp/pub/fedora/linux/core//test/3.91/ general: Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc2.iso general: Jigdo: FC4-test2-i386-disc2.jigdo general: Template: FC4-test2-i386-disc2.template general: URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/' Finished - image size is 656263168 bytes. general: [exit(0)] 311.40user 1.58system 5:13.65elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+3034minor)pagefaults 0swaps /usr/bin/time jigdo-file make-template \ --cache=/var/cache/jigdo-cache.db \ --image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc3.iso \ --jigdo=FC4-test2-i386-disc3.jigdo \ --template=FC4-test2-i386-disc3.template \ --label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \ --uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \ /srv/ftp/pub/fedora/linux/core//test/3.91/ general: Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc3.iso general: Jigdo: FC4-test2-i386-disc3.jigdo general: Template: FC4-test2-i386-disc3.template general: URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/' Finished - image size is 669345792 bytes. general: [exit(0)] 260.72user 1.65system 4:23.26elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+3033minor)pagefaults 0swaps /usr/bin/time jigdo-file make-template \ --cache=/var/cache/jigdo-cache.db \ --image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc4.iso \ --jigdo=FC4-test2-i386-disc4.jigdo \ --template=FC4-test2-i386-disc4.template \ --label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \ --uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \ /srv/ftp/pub/fedora/linux/core//test/3.91/ general: Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc4.iso general: Jigdo: FC4-test2-i386-disc4.jigdo general: Template: FC4-test2-i386-disc4.template general: URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/' Finished - image size is 603430912 bytes. general: [exit(0)] 358.96user 1.64system 6:01.24elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+3037minor)pagefaults 0swaps /usr/bin/time jigdo-file make-template \ --cache=/var/cache/jigdo-cache.db \ --image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc1.iso \ --jigdo=FC4-test2-i386-SRPMS-disc1.jigdo \ --template=FC4-test2-i386-SRPMS-disc1.template \ --label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \ --uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \ /srv/ftp/pub/fedora/linux/core//test/3.91/ general: Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc1.iso general: Jigdo: FC4-test2-i386-SRPMS-disc1.jigdo general: Template: FC4-test2-i386-SRPMS-disc1.template general: URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/' Finished - image size is 650930176 bytes. general: [exit(0)] 210.17user 2.75system 3:44.35elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (2major+3096minor)pagefaults 0swaps /usr/bin/time jigdo-file make-template \ --cache=/var/cache/jigdo-cache.db \ --image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc2.iso \ --jigdo=FC4-test2-i386-SRPMS-disc2.jigdo \ --template=FC4-test2-i386-SRPMS-disc2.template \ --label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \ --uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \ /srv/ftp/pub/fedora/linux/core//test/3.91/ general: Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc2.iso general: Jigdo: FC4-test2-i386-SRPMS-disc2.jigdo general: Template: FC4-test2-i386-SRPMS-disc2.template general: URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/' Finished - image size is 650944512 bytes. general: [exit(0)] 219.23user 2.76system 3:53.31elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+3217minor)pagefaults 0swaps /usr/bin/time jigdo-file make-template \ --cache=/var/cache/jigdo-cache.db \ --image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc3.iso \ --jigdo=FC4-test2-i386-SRPMS-disc3.jigdo \ --template=FC4-test2-i386-SRPMS-disc3.template \ --label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \ --uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \ /srv/ftp/pub/fedora/linux/core//test/3.91/ general: Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc3.iso general: Jigdo: FC4-test2-i386-SRPMS-disc3.jigdo general: Template: FC4-test2-i386-SRPMS-disc3.template general: URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/' Finished - image size is 650930176 bytes. general: [exit(0)] 190.68user 2.77system 3:25.46elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+3180minor)pagefaults 0swaps /usr/bin/time jigdo-file make-template \ --cache=/var/cache/jigdo-cache.db \ --image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc4.iso \ --jigdo=FC4-test2-i386-SRPMS-disc4.jigdo \ --template=FC4-test2-i386-SRPMS-disc4.template \ --label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \ --uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \ /srv/ftp/pub/fedora/linux/core//test/3.91/ general: Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc4.iso general: Jigdo: FC4-test2-i386-SRPMS-disc4.jigdo general: Template: FC4-test2-i386-SRPMS-disc4.template general: URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/' Finished - image size is 650913792 bytes. general: [exit(0)] 217.25user 2.80system 3:52.51elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+3298minor)pagefaults 0swaps