Attached is an up to date revision of my turboLiveInst patch which
incorporates the suggestions made during MarkMC's review.
Mark's executive summary of the feature:
"Reduce installation time by not copying unused data to disk"
"We avoid copying unused data by copying a filesystem image that has
been reduced to the minimal possible size. This minimal image is not
sufficient for a running LiveCD as applications need room to write
more data, but the minimal image is efficiently created just before
copying by applying a pre-calculated set of deltas to the original
large filesystem image."
2nd revision post with performance numbers and decent descritpion:
(first two copy times are typo swapped)
http://www.redhat.com/archives/anaconda-devel-list/2007-July/msg00065.html
Mark's comments thread:
http://www.redhat.com/archives/fedora-livecd-list/2007-September/msg00007.html
The anaconda patch applies on top of the selinux bugfix I sent to
anaconda-devel this morning, but there is no overlap.
The main things I'll mention that aren't directly related to the review
thread above-
- I included a couple typo fixes to the documentation, as well as adding
what seemed to be some appropriate additions.
- I included a slight cleanup of livecd-creator's resize2fsToMinimal.
As discussed in the above thread, since the dumpe2fs code is going into
anaconda for this patch, it seems to make sense to go ahead and include
it in livecd-creator as well to remove the blocksize argument to
resize2fsToMinimal, and have it calculate implicitly instead.
- I had to move the anaconda resize2fs invocation, as now the filesystem
starts out full, and before resize2fs cannot even have mountpoint
directories made in it. Even outside turboLiveInst this is a valid
thing to do.
- renamed the main function from turboLiveInst to genMinInstDelta
- removed option for ignore-deleted, and didn't bother with option for
turboliveinst. The liveinst.sh code gracefully handles the legacy
configuration of no delta file.
- still not yet being a rawhide user, I tested this on an F7 livecd
spin, generated with this git livecd-tools. I used the patch to patch
the installed anaconda. It worked in a test under qemu. (sparse disk
file with du -cms showed that only ~2.2G of data was written).
- I expect, that even just looking at it myself over the next couple
days I'll probably find one or two things to change. Please give me any
feedback about any possible improvements you can think of. If you have
a rawhide livecd spinning enironment handy and want to test it, that
would be greatly appreciated.
Enjoy...
-dmc/jdog
diff -Naur livecd.200709140049/creator/isotostick.sh livecd.turboliveinst/creator/isotostick.sh
--- livecd.200709140049/creator/isotostick.sh 2007-09-14 00:49:42.000000000 -0500
+++ livecd.turboliveinst/creator/isotostick.sh 2007-09-14 11:00:12.000000000 -0500
@@ -195,6 +195,10 @@
elif [ -f $CDMNT/ext3fs.img ]; then
cp $CDMNT/ext3fs.img $USBMNT/LiveOS/ext3fs.img || exitclean
fi
+if [ -f $CDMNT/osmin.gz ]; then
+ cp $CDMNT/osmin.gz $USBMNT/LiveOS/osmin.gz || exitclean
+fi
+
cp $CDMNT/isolinux/* $USBMNT/$SYSLINUXPATH
echo "Updating boot config file"
diff -Naur livecd.200709140049/creator/livecd-creator livecd.turboliveinst/creator/livecd-creator
--- livecd.200709140049/creator/livecd-creator 2007-09-14 00:49:42.000000000 -0500
+++ livecd.turboliveinst/creator/livecd-creator 2007-09-14 12:40:21.000000000 -0500
@@ -343,6 +343,7 @@
self.image_size = 4096 # in megabytes
self.blocksize = 4096 # in kilobytes
+ self.minimized_image_size = 0 # in kilobytes
def _getRequiredPackages(self):
return []
@@ -930,6 +931,21 @@
shutil.move("%s/data/os.img" %(self.build_dir,),
"%s/out/ext3fs.img" %(self.build_dir,))
+ def parseField(self, output, field):
+ for line in output.split("\n"):
+ if line.startswith(field + ":"):
+ return line[len(field) + 1:].strip()
+
+ raise KeyError("Failed to find field '%s' in output" % field)
+
+ def getBlockCountOfExt2FS(self, filesystem):
+ output = subprocess.Popen(['/sbin/dumpe2fs', '-h', filesystem],
+ stdout=subprocess.PIPE,
+ stderr=open('/dev/null', 'w')
+ ).communicate()[0]
+
+ return int(self.parseField(output, "Block count"))
+
def resize2fs(self, image, n_blocks):
dev_null = os.open("/dev/null", os.O_WRONLY)
try:
@@ -943,9 +959,9 @@
# resize2fs doesn't have any kind of minimal setting, so use
# a binary search to get it to minimal size.
#
- def resize2fsToMinimal(self, image, n_blocks):
+ def resize2fsToMinimal(self, image):
bot = 0
- top = n_blocks
+ top = self.getBlockCountOfExt2FS(image)
while top != (bot + 1):
t = bot + ((top - bot) / 2)
@@ -968,17 +984,95 @@
n_blocks = os.stat(image)[stat.ST_SIZE] / self.blocksize
- min_blocks = self.resize2fsToMinimal(image, n_blocks)
+ min_blocks = self.resize2fsToMinimal(image)
# truncate the unused excess portion of the sparse file
fd = os.open(image, os.O_WRONLY )
os.ftruncate(fd, min_blocks * self.blocksize)
os.close(fd)
- print >> sys.stderr, "Installation target minimized to %dK" % (min_blocks * self.blocksize / 1024L)
+ self.minimized_image_size = min_blocks * self.blocksize / 1024L
+ print >> sys.stderr, "Installation target minimized to %dK" % (self.minimized_image_size)
self.resize2fs(image, n_blocks)
+
+ #
+ # genMinInstDelta: generates an osmin overlay file to sit alongside
+ # os.img. liveinst may then detect the existence of
+ # osmin, and use it to create a minimized os.img
+ # which can be installed more quickly, and to smaller
+ # destination volumes.
+ #
+ def genMinInstDelta(self):
+ # create the sparse file for the minimized overlay
+ fd = os.open("%s/out/osmin" %(self.build_dir,),
+ os.O_WRONLY | os.O_CREAT)
+ off = long(64L * 1024L * 1024L)
+ os.lseek(fd, off, 0)
+ os.write(fd, '\x00')
+ os.close(fd)
+
+ # associate os image with loop device
+ osloop = LoopbackMount("%s/data/os.img" %(self.build_dir,),
+ "not_going_to_actually_get_mounted")
+ osloop.loopsetup()
+
+ # associate overlay with loop device
+ minloop = LoopbackMount("%s/out/osmin" %(self.build_dir,),
+ "not_going_to_actually_get_mounted")
+ minloop.loopsetup()
+
+ # create a snapshot device
+ rc = subprocess.call(["/sbin/dmsetup",
+ "--table",
+ "0 %d snapshot %s %s p 8"
+ %(self.image_size * 1024L * 2L,
+ osloop.loopdev, minloop.loopdev),
+ "create",
+ "livecd-creator-%d" %(os.getpid(),) ])
+ if rc != 0:
+ raise InstallationError("Could not create genMinInstDelta snapshot device")
+ # resize snapshot device back to minimal (self.minimized_image_size)
+ rc = subprocess.call(["/sbin/resize2fs",
+ "/dev/mapper/livecd-creator-%d" %(os.getpid(),),
+ "%dK" %(self.minimized_image_size,)])
+ if rc != 0:
+ raise InstallationError("Could not shrink ext3fs image")
+
+ # calculate how much delta data to keep
+ dmsetupOutput = subprocess.Popen(['/sbin/dmsetup', 'status',
+ "livecd-creator-%d" %(os.getpid(),)],
+ stdout=subprocess.PIPE,
+ stderr=open('/dev/null', 'w')
+ ).communicate()[0]
+
+ try:
+ minInstDeltaDataLength = int((dmsetupOutput.split()[3]).split('/')[0])
+ print >> sys.stderr, "genMinInstDelta data length is %d 512 byte sectors" % (minInstDeltaDataLength)
+ except ValueError:
+ raise InstallationError("Could not calculate amount of data used by genMinInstDelta")
+
+ # tear down snapshot and loop devices
+ rc = subprocess.call(["/sbin/dmsetup", "remove",
+ "livecd-creator-%d" %(os.getpid(),) ])
+ if rc != 0:
+ raise InstallationError("Could not remove genMinInstDelta snapshot device")
+ osloop.lounsetup()
+ minloop.lounsetup()
+
+ # truncate the unused excess portion of the sparse file
+ fd = os.open("%s/out/osmin" %(self.build_dir,), os.O_WRONLY )
+ os.ftruncate(fd, minInstDeltaDataLength * 512)
+ os.close(fd)
+
+ # the delta data is *extremely* compressible (e.g. 1.2M->7kb)
+ rc = subprocess.call(["/usr/bin/gzip", "osmin"],
+ cwd="%s/out" %(self.build_dir,),
+ env={"PWD": "%s/out" %(self.build_dir,)})
+ if rc != 0:
+ raise InstallationError("Could not compress genMinInstDelta data")
+
def package(self):
self.createSquashFS()
self.createIso()
@@ -1300,10 +1394,6 @@
# Don't compress the image.
parser.add_option("-s", "--skip-compression", action="store_true", dest="skip_compression",
help=optparse.SUPPRESS_HELP)
- # Don't run resize2fs to clean up wasted blocks
- parser.add_option("", "--ignore-deleted", action="store_true", dest="ignore_deleted",
- help=optparse.SUPPRESS_HELP)
-
(options, args) = parser.parse_args()
if not options.kscfg or not os.path.isfile(options.kscfg):
@@ -1374,10 +1464,12 @@
target.unmount()
- if not options.ignore_deleted:
- target.cleanupDeleted()
+ target.cleanupDeleted()
+
+ target.genMinInstDelta()
target.package()
+
except InstallationError, e:
print >> sys.stderr, "Error creating Live CD : %s" % e
target.teardown()
diff -Naur livecd.200709140049/creator/mayflower livecd.turboliveinst/creator/mayflower
--- livecd.200709140049/creator/mayflower 2007-09-14 00:49:42.000000000 -0500
+++ livecd.turboliveinst/creator/mayflower 2007-09-14 13:28:20.000000000 -0500
@@ -625,6 +625,24 @@
mount -n -o ro,remount /sysroot
}
+modprobe loop max_loop=128
+
+# we might have a genMinInstDelta delta file for anaconda to take advantage of
+if [ -e /sysroot/LiveOS/osmin.gz ]; then
+ mknod /dev/loop118 b 7 118
+ # note: osmin.gz should typically only be about 7kb.
+ dd if=/sysroot/LiveOS/osmin.gz of=/osmin.gz bs=512 2> /dev/null
+ # pad to at least next sector boundry
+ dd if=/dev/zero of=/osmin.gz bs=512 count=1 oflag=append conv=notrunc 2> /dev/null
+ losetup /dev/loop118 /osmin.gz
+elif [ -e /sysroot/osmin.gz ] ; then
+ mknod /dev/loop118 b 7 118
+ dd if=/sysroot/osmin.gz of=/osmin.gz bs=512 2> /dev/null
+ # pad to at least next sector boundry
+ dd if=/dev/zero of=/osmin.gz bs=512 count=1 oflag=append conv=notrunc 2> /dev/null
+ losetup /dev/loop118 /osmin.gz
+fi
+
# we might have an uncompressed embedded ext3 to use as rootfs (uncompressed live)
#
if [ -e /sysroot/LiveOS/ext3fs.img ]; then
@@ -638,13 +656,11 @@
echo "setting up embedded ext3 fs "
fi
- mknod /dev/loop118 b 7 118
mknod /dev/loop119 b 7 119
mknod /dev/loop120 b 7 120
mknod /dev/loop121 b 7 121
mkdir -p /dev/mapper
mknod /dev/mapper/control c 10 63
- modprobe loop max_loop=128
modprobe dm_snapshot
losetup /dev/loop121 \$EXT3FS
@@ -667,13 +683,11 @@
echo "setting up embedded squash -> ext3 fs "
fi
- mknod /dev/loop118 b 7 118
mknod /dev/loop119 b 7 119
mknod /dev/loop120 b 7 120
mknod /dev/loop121 b 7 121
mkdir -p /dev/mapper
mknod /dev/mapper/control c 10 63
- modprobe loop max_loop=128
modprobe dm_snapshot
if [ "\$live_ram" == "1" ] ; then
diff -Naur livecd.200709140049/README livecd.turboliveinst/README
--- livecd.200709140049/README 2007-09-14 00:49:42.000000000 -0500
+++ livecd.turboliveinst/README 2007-09-14 12:13:34.000000000 -0500
@@ -74,6 +74,13 @@
o Unmounts the installation root
+ o Runs resize2fs to minimize and unminimize the ext3 file to remove data
+ from deleted files
+
+ o Runs resize2fs to minimize on a devicemapper snapshot, to generate a
+ small minimized delta image file which can be used by anaconda to
+ reduce installation time by not copying unused data to disk
+
o Creates a squashfs file system containing only the ext3 file (compression)
o Configures the boot loader
@@ -118,7 +125,7 @@
--fslabel=Fedora-7-LiveCD-1-foo
will create a live CD called "Fedora-7-LiveCD-1-foo". The name
-given by --fs-label is used.
+given by --fslabel is used.
o as a file system label on the ext3 and iso960 file systems
(as such it's visible on the desktop as the CD name)
@@ -155,7 +162,7 @@
use live images. You can take a live CD iso image and transform it so
that it can be used on a USB stick. To do so, use the
livecd-iso-to-disk script.
- /usr/bin/livecd-iso-to-stick /path/to/live.iso /dev/sdb1
+ /usr/bin/livecd-iso-to-disk /path/to/live.iso /dev/sdb1
Replace /dev/sdb1 with the (unmounted) partition where you wish to put
the live image. This is not a destructive process; any data you
diff -Naur anaconda.200709140049.bugfix_selinux/livecd.py anaconda.turboliveinst/livecd.py
--- anaconda.200709140049.bugfix_selinux/livecd.py 2007-09-14 12:41:41.000000000 -0500
+++ anaconda.turboliveinst/livecd.py 2007-09-14 12:44:17.000000000 -0500
@@ -133,25 +133,24 @@
def getLiveBlockDevice(self):
return self.osimg
- def getLiveSizeMB(self):
- lnk = os.readlink(self.osimg)
- if lnk[0] != "/":
- lnk = os.path.join(os.path.dirname(self.osimg), lnk)
- blk = os.path.basename(lnk)
-
- if not os.path.exists("/sys/block/%s/size" %(blk,)):
- log.debug("Unable to determine the actual size of the live image")
- return 0
-
- size = open("/sys/block/%s/size" %(blk,), "r").read()
- try:
- size = int(size)
- except ValueError:
- log.debug("Unable to handle live size conversion: %s" %(size,))
- return 0
+ def parseField(self, output, field):
+ for line in output.split("\n"):
+ if line.startswith(field + ":"):
+ return line[len(field) + 1:].strip()
+
+ raise KeyError("Failed to find field '%s' in output" % field)
- return (size * 512) / 1024 / 1024
+ def getSectorCountOfExt2FS(self, filesystem):
+ output = subprocess.Popen(['/sbin/dumpe2fs', '-h', filesystem],
+ stdout=subprocess.PIPE,
+ stderr=open('/dev/null', 'w')
+ ).communicate()[0]
+ return (int(self.parseField(output, "Block count")) *
+ int(self.parseField(output, "Block size")) / 512)
+
+ def getLiveSizeMB(self):
+ return (self.getSectorCountOfExt2FS(self.osimg) * 512) / 1024 / 1024
class LiveCDCopyBackend(backend.AnacondaBackend):
def __init__(self, method, instPath):
@@ -215,6 +214,9 @@
wait = anaconda.intf.waitWindow(_("Doing post-installation"),
_("Performing post-installation filesystem changes. This may take several minutes..."))
+ # resize rootfs first, since it is 100% full due to genMinInstDelta
+ self._resizeRootfs(anaconda, wait)
+
# remount filesystems
anaconda.id.fsset.mountFilesystems(anaconda)
@@ -291,7 +293,6 @@
log.error("error mounting selinuxfs: %s" %(e,))
isys.mount("/dev", "%s/dev" %(anaconda.rootPath,), bindMount = 1)
- self._resizeRootfs(anaconda, wait)
wait.pop()
def _resizeRootfs(self, anaconda, win = None):
diff -Naur anaconda.200709140049.bugfix_selinux/liveinst/liveinst.sh anaconda.turboliveinst/liveinst/liveinst.sh
--- anaconda.200709140049.bugfix_selinux/liveinst/liveinst.sh 2007-04-04 13:05:42.000000000 -0500
+++ anaconda.turboliveinst/liveinst/liveinst.sh 2007-09-14 12:20:13.000000000 -0500
@@ -4,7 +4,29 @@
#
if [ -z "$LIVE_BLOCK" ]; then
- LIVE_BLOCK="/dev/live-osimg"
+ #
+ # We avoid copying unused data by copying a filesystem image that has
+ # been reduced to the minimal possible size. This minimal image is not
+ # sufficient for a running LiveCD as applications need room to write
+ # more data, but the minimal image is efficiently created just before
+ # copying by applying a pre-calculated set of deltas to the original
+ # large filesystem image.
+ #
+
+ # did mayflower find and expose the delta data via loop118?
+ if ( losetup /dev/loop118 > /dev/null 2>&1 ); then
+ # the delta data is exposed as gzipped data in a loop device
+ zcat /dev/loop118 > /dev/shm/osmin.img
+ # devicemapper needs a loop device with the uncompressed data
+ losetup /dev/loop117 /dev/shm/osmin.img
+ # set up the devicemapper snapshot device, which will merge
+ # the normal live fs image, and the delta, into a minimzied fs image
+ echo "0 $( blockdev --getsize /dev/loop121 ) snapshot /dev/loop121 /dev/loop117 p 8" | dmsetup create --readonly live-osimg-min
+ LIVE_BLOCK="/dev/mapper/live-osimg-min"
+ else
+ # fall back to legacy pre-genMinInstDelta behavior
+ LIVE_BLOCK="/dev/live-osimg"
+ fi
fi
if [ ! -b $LIVE_BLOCK ]; then
@@ -42,3 +64,10 @@
if [ -n $current ]; then
/usr/sbin/setenforce $current
fi
+
+# cleanup genMinInstDelta stuff if needed
+if ( losetup /dev/loop118 > /dev/null 2>&1 ); then
+ dmsetup remove live-osimg-min
+ losetup -d /dev/loop117
+ rm -f /dev/shm/osmin.img
+fi
