Hi Andreas,
Thanks for the reply. And yes, I did come across the usage of debugfs for updating certain metadata and it works fine.
On the same note, I also came across another observation. I noticed that number of free blocks in some of the block groups are different for the 2 ext4 images in comparison although the filesystem contents are identical. Is this an expected behaviour? Any hints on what would lead to this could be helpful.
I am including a diff of dumpe2fs from the 2 ext4 images I am comparing below:
host@debian12:~/work/temp$ diff image1.dumpe2fs image2.dumpe2fs
< Lifetime writes: 316 MB
---
> Lifetime writes: 314 MB
47c47
< Checksum: 0x6a0e9f0e
---
> Checksum: 0xbba8f574
53c53
< Journal sequence: 0x00000008
---
> Journal sequence: 0x00000007
56c56
< Journal checksum: 0x5c50e847
---
> Journal checksum: 0xba912c29
68c68
< Group 1: (Blocks 32768-65535) csum 0x9bf2 [ITABLE_ZEROED]
---
> Group 1: (Blocks 32768-65535) csum 0x98a6 [ITABLE_ZEROED]
71c71
< Block bitmap at 65 (bg #0 + 65), csum 0x9ecad502
---
> Block bitmap at 65 (bg #0 + 65), csum 0xe702afc8
74,75c74,75
< 93 free blocks, 6070 free inodes, 1049 directories, 6070 unused inodes
< Free blocks: 33264-33279, 47035, 51832-51839, 52923-52927, 53244-53247, 54773-54783, 56723-56735, 58622-58623, 62006-62015, 62463, 64752-64767, 65530-65535
---
> 538 free blocks, 6070 free inodes, 1049 directories, 6070 unused inodes
> Free blocks: 33264-33279, 47035, 52732-52735, 53951, 54782-54783, 54823-55295, 56275-56287, 58622-58623, 59389-59391, 62521-62527, 62975, 65275-65279, 65526-65535
77,78c77,78
< Group 2: (Blocks 65536-98303) csum 0x0943 [ITABLE_ZEROED]
< Block bitmap at 66 (bg #0 + 66), csum 0xc4b03f5a
---
> Group 2: (Blocks 65536-98303) csum 0x9332 [ITABLE_ZEROED]
> Block bitmap at 66 (bg #0 + 66), csum 0x8b8b9691
81,82c81,82
< 328 free blocks, 7996 free inodes, 4 directories, 7996 unused inodes
< Free blocks: 73714-73727, 77307-77311, 79347-79359, 80371-80383, 81404-81407, 82430-82431, 83954-83967, 88575, 90621-90623, 93497-93503, 94719, 95731-95743, 96020-96255, 96670-96671
---
> 811 free blocks, 7996 free inodes, 4 directories, 7996 unused inodes
> Free blocks: 70652-70655, 72177-72191, 73215, 74239, 75775, 77311, 78559, 79358-79359, 83959-83967, 85499-85503, 86522-86527, 87257-87263, 88028-88031, 89586-89599, 90489-90495, 92444-92447, 93694-93695, 94966-94975, 95226-95231, 95890-95903, 96253-96255, 97531-97791, 97871-98303
84c84
< Group 3: (Blocks 98304-127999) csum 0xceff [ITABLE_ZEROED]
---
> Group 3: (Blocks 98304-127999) csum 0xb1ee [ITABLE_ZEROED]
87c87
< Block bitmap at 67 (bg #0 + 67), csum 0x2c1ce6ca
---
> Block bitmap at 67 (bg #0 + 67), csum 0x4e67b818
90,91c90,91
< 14411 free blocks, 7994 free inodes, 6 directories, 7994 unused inodes
< Free blocks: 98803-98815, 100852-100863, 102393-102399, 102910-102911, 104958-104959, 106487-114687, 116223, 116669-116735, 116914-117247, 117354-122879, 123391, 123688-123903, 126830-126847, 126967-126975, 127487, 127999
---
> 13483 free blocks, 7994 free inodes, 6 directories, 7994 unused inodes
> Free blocks: 103224-103423, 104447, 109582-110079, 110128-122879, 126960-126975, 127984-127999
Thanks and Regards,
Adithya Balakumar
________________________________________
From: Andreas Dilger
Sent: Thursday, February 20, 2025 1:28 AM
To: balakumar adithya(ＴＳＩＰ DITC_DIT-OST)
Cc: linux-ext4@xxxxxxxxxxxxxxx; kunijadar shivanand(ＴＳＩＰ DITC_DIT-OST); dinesh kumar(ＴＳＩＰ DITC_DIT-OST); hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４); iwamatsu nobuhiro(岩松 信洋 ○ＤＩＴＣ□ＤＩＴ○ＯＳＴ)
Subject: Re: Is it possible to make ext4 images reproducible even after filesystem operations ?
On Jan 21, 2025, at 5:29 AM, Adithya.Balakumar@xxxxxxxxxxxxxxxx wrote:
> I am working towards reproducible builds for a project that I am involved in. We use a few ext4 partitions in our disk images and I am trying to make the ext4 filesystems reproducible.
>
> I understand that from e2fsprogs v1.47.1 onwards we can create a reproducible ext4 filesystem image. We can indeed create a reproducible ext4 filesystem image when we use the "-d" option in "mke2fs" command to pass the contents of the filesystem at the time of creation of the filesystem itself. I understand that there are a few other parameters that needs to passed to the "mke2fs" command like a deterministic UUID and hash_seed values to make the filesystem image reproducible.
>
> In the project that I am working on, there are some mount operations done on the filesystem to copy certain files into the file system. This updates the "Last mount" and "Last write" timestamps in the filesystem metadata (confirmed this with dumpe2fs) thereby making the images generated not reproducible.
>
> I would like to understand if its possible to make the ext4 images reproducible even after filesystem operations like mounting and unmounting the filesystem ?
It should be possible to use debugfs commands to change the timestamps (and other
fields) in the superblock to an arbitrary value, something like:
{
echo "ssv wtime 123456789"
echo "ssv mtime 123456789"
} | debugfs -w -F /dev/stdin $IMAGE_FILE
Depending on what changes are being made while the filesystem is mounted, you
may also need to modify the inode timestamps directly as well:
{
echo "sif $PATHNAME ctime 123456789"
echo "sif $PATHNAME2 ctime 123456789"
:
} | debugfs -w -F /dev/stdin $IMAGE_FILE
The debugfs commands could all be combined into a single debugfs invocation,
and are just shown here as separate commands for clarity. If the commands
are always the same, they could also be written into a command file instead
of read from stdin each time:
debugfs -w -f $COMMANDS $IMAGE_FILE
but for scripting purposes it can be convenient to generate debugfs commands
on the fly (e.g. with looping, etc.) and pipe it to debugfs via stdin, and
this is not obvious, so I thought it would be good to show an example.
Cheers, Andreas
