Re: A tool that allows changing inode table sizes

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi everyone!

(After > 2.5 years have passed :)!)

At last, I've integrated my inode table resize tool into resize2fs. The code is not very clean yet so I'm not sending it here, but it seems it works on testcases which I've used to test my previous tool. It also includes patch_io manager as `-T` option for e2fsck and resize2fs (in separate commits, of course).

It can even change inode count and resize the filesystem simultaneously (shrink fs + extend inode tables gives some errors now, but I think it should also be easy to fix and I'll do it shortly).

I'll clean it up and send for review, but at the current point you can take a look at it here: https://github.com/vitalif/e2fsprogs (better check a squashed diff of last commits if you want to look at the code... some lines are rewritten, I'll send it squashed)

My only questions are:
1) now I'm using my own move_inode_tables() instead of move_itables(). I see there's some strange logic in there - which cases does it handle? Is it something to handle overlaps during move? 2) interesting point of inode table allocation is that in case of a bigalloc FS it's done based on per-block (not per-cluster) bitmap during mke2fs. so, to reproduce that behavior I should either allocate similar per-block bitmap or force their position by a kind of hack (do not mark allocated inode tables in bitmap and set inode_table_loc to inode_table_loc of previous BG + inode_blocks_per_group). the second works, but the first seems more correct, so should I use that approach?

On Thu, Jan 16, 2014 at 05:05:45PM -0700, Andreas Dilger wrote:

On Jan 15, 2014, at 6:28 AM, vitalif@xxxxxxxxxx wrote:
> As I understand it was a well-known fact that ext2/3/4 does not allow changing inode table size without recreating the filesystem. And I didn't have any experience in linux filesystem internals until recently, when I've discovered that inode tables take 45 GB on one of my hard drives (3 TB in size) :-):-) that hard drive is, of course, full of movies, not 16Kb files, so the inode tables are almost 100% unused.
>
> So, I've thought it would be good if it it would possible to change inode table sizes. So I've written a tool that in fact allows to do it, and I want to present it to the community! :)

Interesting.  I did something years ago for ext2/3 filesystem resizing
(ext2resize), but that has since become obsolete as the functionality
was included into e2fsprogs.  I'd recommend that you also work to get
your functionality included into e2fsprogs sooner rather than later.

Ideally this would be part of resize2fs, but I'm not sure it would be
easily implemented there.

I don't think it would be too difficult, since there's already code to move blocks and inodes around. I guess the big question is how well does it respond
to having inodes_per_group change?

<shrug>

--D

> Anyone is welcome to test it of course if it's of any interest for you - the source is here http://svn.yourcmc.ru/viewvc.py/vitalif/trunk/ext4-realloc-inodes/ ('download tarball') (maybe it would be better to move it into a separate git repo, of course)
>
> I didn't test it on a real hard drive yet :-D, only on small fs images with different settings (block, block group, flex_bg size, ext2/3/4, bigalloc and etc). There are even some auto-tests (ran by 'make test').

Note that it is critical to refuse to do anything on filesystems that
have any feature that your tool doesn't understand.  Otherwise, it has
a good possibility to corrupt the filesystem.

> The tools works without problem on all small test images that I've created, though I didn't try to run it on bigger filesystems (of course I'll do it in the nearest future).
>
> As this is a highly destructive process that involves overwriting ALL inode numbers in ALL directory entries across the whole filesystem, I've also implemented a simple method of safely applying/rolling back changes. First I've tried to use undo_io_manager, but it appears to be very slow because of frequent commits, which are of course needed for it to be safe.

Would it be possible to speed up undo_io_manager if it had larger IO
groups or similar?  How does the speed of running with undo_io_manager
compare to running your patch_io_manager doing both a backup and apply?

> My method is called patch_io_manager and does a different thing - it does not overwrite the initial FS image, but writes all modified blocks into a separate sparse file + writes a bitmap of modified blocks in the end when it finishes. I.e. the initial filesystem stays unmodified.

This is essentially implementing a journal in userspace for e2fsprogs.
You could even use the journal file in the filesystem.  The journal
MUST be clean before the inode renumbering, or journal replay will
corrupt the filesystem after your resize.  Does your tool check this?

That said, there may not be enough space in the journal for full data
journaling, but it might be enough for logical journaling of the inodes
to be moved and the directories that need to be updated?

> Then, using e2patch utility (it's in the same repository), you can a) backup the blocks that will be modified into another patch file (e2patch backup <fs> <patch> <backup>) and b) apply the patch to real filesystem. If the applying process gets interrupted (for example by the power outage) it can be restarted from the beginning because it does nothing except just overwriting some blocks.

This is exactly like journal replay.

> And if the FS changes appear to be bad at all, you can restore the backup in a same way. So the process should be safe at least to some extent.

Looks interesting.  Of course, I always recommend doing a full backup
before any operation like this.  At that point, it would also be
possible to just format a new filesystem and copy the data over.  That
has the advantage of also allowing other filesystem features to be
enabled and defragmenting the data, but could be slower if the files
are large (as in your case) and relatively few inodes are moved.

Cheers, Andreas

--
With best regards,
  Vitaliy Filippov
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [Reiser Filesystem Development]     [Ceph FS]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Linux FS]     [Yosemite National Park]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Device Mapper]     [Linux Media]

  Powered by Linux