Hi,
I'd like to introduce the bmap-tools project. I am sending this e-mail
to this forum since here have people who may find it useful, or give
good feed-back/criticism. This does not really have much to do with the
kernel.
Anyway, I am working with Tizen IVI and we use raw FS images there, and
there has always be a problem that flashing the images to USB sticks
takes very long time - we used dd for this.
I addressed this issue with the bmaptool, which increased flashing speed
by factor of 8 for Tizen IVI back at that time.
In short, the idea is that raw images in Tizen IVI were very sparse, and
there was a lot of 'gaps' in them (unmapped areas). Of course the
sparseness information is lost as soon as the image is compressed and
published.
What I implemented is so-called bmap file, which I generate on the build
server just after the raw image is generated. I use FIEMAP to record
where the gaps are. And later this information is used to speed-up
copying and flashing of the image - instead of copying all the zeroes,
we just skip the area where there were gaps.
This increases flashing speed dramatically for reasonably spars images.
On top of that I implement various goodies in bmaptool which make it a
lot more usable than plain 'dd', not only much faster:
* it can read images from http/https/ssh/ftp urls, also via proxy
* it can read compressed images and stream-uncompress them on-the-fly,
supported formats are .gz, .xz, .bz2, .tgz, tar.{gz|xz|bz2}
* it won't let you write to a mounted block device
* it uses tricks like switching to noop scheduler for faster sequential
writing to a block device, so it is faster than dd even without the
bmap file
* it reacts on ctrl-c almost immediately, unlike dd, because we fsync
the block device after every 4MiB or something, so that ctrl-c won't
need to flush huge amount of data before exiting (kernel forces full
flush for bdevs on the last close)
* check SHA1 of the data and bmap file (the plan is to switch to sha256
and support gpg signature verification)
* some other things I guess :-)
So, for example, in Tizen IVI we flash raw images like this:
bmaptool copy http://download.tizen.org/.../image.raw.bz2 /dev/usb_stick
This will read data from the URL, uncompress the data on-the-fly, and
write to USB stick. It will also auto-discover the bmap file, and use it
for ultra-fast writing (providing there is sparseness, which is often
the case).
It can also be use for copying large images around, not necessarily
writing to block devices.
FYI, this technique can also be used for Factory-flashing of images.
Indeed, when have a product pipe-line, you want to populate them with
data as fast as possible, and the fastest way to do this is to write
directly to the block device and only those blocks which have data.
Well, enough for first introduction. If someone is interested, see the
docs here:
https://source.tizen.org/documentation/articles/bmaptool
It is a bit out-of-date, but not much, will be updated soon.
Please, see the information about the project here as well:
http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/devel:/docs/README
An example of a Tizen IVI image and the bmap file can be found here:
http://linux-ftp.fi.intel.com/pub/mirrors/tizen/snapshots/tizen/ivi/latest/images/ivi-efi-i586/
Feed-back is welcome. I hope the project can be useful for others (it is
GPL-2.0).
And here is an example of a bmap file which I created today for a Fedora
image (I sent a similar announcement to the Fedora community):
<?xml version="1.0" ?>
<!-- This file contains the block map for an image file, which is basically
a list of useful (mapped) block numbers in the image file. In other words,
it lists only those blocks which contain data (boot sector, partition
table, file-system metadata, files, directories, extents, etc). These
blocks have to be copied to the target device. The other blocks do not
contain any useful data and do not have to be copied to the target
device.
The block map an optimization which allows to copy or flash the image to
the image quicker than copying of flashing the entire image. This is
because with bmap less data is copied: <MappedBlocksCount> blocks instead
of <BlocksCount> blocks.
Besides the machine-readable data, this file contains useful commentaries
which contain human-readable information like image size, percentage of
mapped data, etc.
The 'version' attribute is the block map file format version in the
'major.minor' format. The version major number is increased whenever an
incompatible block map format change is made. The minor number changes
in case of minor backward-compatible changes. -->
<bmap version="1.3">
<!-- Image size in bytes: 2.0 GiB -->
<ImageSize> 2147483648 </ImageSize>
<!-- Size of a block in bytes -->
<BlockSize> 4096 </BlockSize>
<!-- Count of blocks in the image file -->
<BlocksCount> 524288 </BlocksCount>
<!-- Count of mapped blocks: 688.6 MiB or 33.6% -->
<MappedBlocksCount> 176288 </MappedBlocksCount>
<!-- The checksum of this bmap file. When it is calculated, the value of
the SHA1 checksum has be zero (40 ASCII "0" symbols). -->
<BmapFileSHA1> 75b48dc596a5e92d7cc4935d8fcc3a91c2e48b0f </BmapFileSHA1>
<!-- The block map which consists of elements which may either be a
range of blocks or a single block. The 'sha1' attribute (if present)
is the SHA1 checksum of this blocks range. -->
<BlockMap>
<Range sha1="ce5cc4f31d623a34c01f791e6e5c1ee65456044b"> 0-15 </Range>
<Range sha1="4ca6ee9c01354785502ba201a7c03cdd4ffbaedb"> 240-1967 </Range>
<Range sha1="6f0869f38618931042495803f5b4c02f2d7d03e9"> 8224-9583 </Range>
<Range sha1="b7650fa6a6bb703e30438ae22ca7a53c107f72ad"> 9632-11599 </Range>
<Range sha1="4e04cad0efb9194e7e00d407caaf89c9bc19042d"> 11632-11775 </Range>
<Range sha1="feebd3924ce0db12f1d7186577ca54985984ab69"> 33008-33023 </Range>
<Range sha1="ce74e311d9c1316f03b3b5a64225b607196419b9"> 33136-33567 </Range>
<Range sha1="1e6a1f19428243b09da296e5e5494d9443b90c64"> 34032-34095 </Range>
<Range sha1="9b2b5218b79d1b51e00d67814a8eea41d5cc430f"> 34544-34591 </Range>
<Range sha1="f36da167d811feba274928d5c293b569b9f8c0aa"> 34656-34671 </Range>
<Range sha1="2ffd3f90346ba2d0a9841865ccca0adfa19b830e"> 35056-35087 </Range>
<Range sha1="48a82266b1351a543a169df69274c7f82c743988"> 35568-35615 </Range>
<Range sha1="71ffc02cadefd321af333fa4b8e54ca6a2a48fbe"> 35888-35903 </Range>
<Range sha1="e3b51cd7304df8df536e9ebd6d84d6d0d516d0a0"> 36080-36255 </Range>
<Range sha1="6939de8eed4c6c35691f6b6ccf805e7d7a141b70"> 36304-36511 </Range>
<Range sha1="9d2b2e2c0cc771f2a2684317ffdaa8557d5962e1"> 36592-37071 </Range>
<Range sha1="5ccf6bca3a0140f3f549b67cf114e8847d8fd82e"> 37104-55551 </Range>
<Range sha1="a3259735e513eb38f27058e6c87bd555d060615a"> 62656-62687 </Range>
<Range sha1="27c3ac0865d0b12180395e2d083f7bbdc0440bb3"> 62992-63023 </Range>
<Range sha1="bfe6dbe0c325a0a95a6cf6d62226287f4dd0b99d"> 64144-64175 </Range>
<Range sha1="ac97574508a8bd42f39bed6ccbe0ab1e57cbfd54"> 65120-98559 </Range>
<Range sha1="2351cae4967fe643d5003b6f526c811cd311df45"> 98672-100975 </Range>
<Range sha1="6a47e91dc5be78d69761ab762fc2e3ae1ea9bc8e"> 101040-101615 </Range>
<Range sha1="4e9bbbb0f3b487efd37cc54808a371d27eac4c78"> 101808-123759 </Range>
<Range sha1="05a6b82bcd9672a04252bc090b368e6deb3084b1"> 123952-124015 </Range>
<Range sha1="531f3dcb1cefdc515233da007069761849058b13"> 124064-141151 </Range>
<Range sha1="626d4a7ed0e01815230b1fc95e22e7095b3c2c5a"> 141392-143327 </Range>
<Range sha1="04c93f6e63aec7320088037fab3b5d0edb994d23"> 143360-146319 </Range>
<Range sha1="949d1861d7dde9c2ad916a0807ac1b2db6ee2002"> 146624-155391 </Range>
<Range sha1="4e5395a152a382e5e487e24af9d2ed8930eb4531"> 155888-164095 </Range>
<Range sha1="484548b0536d1276575f620139ff3596a455b017"> 164208-165983 </Range>
<Range sha1="e44b975f3d627e4f8df3442d27ca0aeaa7329af5"> 166016-166047 </Range>
<Range sha1="ce95c6eb68528b4d310d0dc087ba595b43473fef"> 166080-167103 </Range>
<Range sha1="d42280d545afdc3c5fb6ae5a0e6e7e3430cb4d23"> 167568-196143 </Range>
<Range sha1="2856f3fac20726e5c647d32fe4757f1b091309b0"> 196848-213247 </Range>
<Range sha1="10614aa162d95797e64dff049707ae788a19d40f"> 221424-223311 </Range>
<Range sha1="a4c0fef5c6c9a013fc641016face92e92329864b"> 223344-223391 </Range>
<Range sha1="b7375c8568a14fb1217a1e31f52cfdffbdd9ce30"> 223472-227375 </Range>
<Range sha1="855e049fb6ace103d4569465b0b06df47628b76f"> 229616-229631 </Range>
<Range sha1="0c900792e5cc9613496ab53d1f30bb94b1c9ad9d"> 295152-295167 </Range>
<Range sha1="2efc62831e5d49db8550c7ef401bc1c1c2a27a78"> 295280-295295 </Range>
</BlockMap>
</bmap>
--
Best Regards,
Artem Bityutskiy
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
