Andreas, thanks for such a prompt reply.
On Fri, 12 Nov 2021, Andreas Dilger wrote:
> On Nov 12, 2021, at 11:37, Mark Hills <mark@xxxxxxxx> wrote:
> >
> > Surprised to hit a limit when handling a modest Maildir case; does
> > this reflect a bug?
> >
> > rsync'ing to a new mail server, after fewer than 100,000 files there
> > are intermittent failures:
>
> This is probably because you are using 1KB blocksize instead of 4KB,
> which reduces the size of each tree level by the cube of the ratio, so
> 64x. I guess that was selected because of very small files in the
> maildir?
Interesting! The 1Kb block size was not explicitly chosen. There was no
plan other than using the defaults.
However I did forget that this is a VM installed from a base image. The
root cause is likely to be that the /home partition has been enlarged from
a small size to 32Gb.
Is block size the only factor? If so, a patch like below (untested) could
make it clear it's relevant, and saved the question in this case.
[...]
> If you have a relatively recent kernel, you can enable the "large_dir"
> feature to allow 3-level htree, which would be enough for another factor
> of 1024/8 = 128 more entries than now (~12M).
The system is not yet in use, so I think it's better we reformat here, and
get a block size chosen by the experts :)
These days I think VMs make it more common to enlarge a filesystem from a
small size. We could have picked this up earlier with a warning from
resize2fs; eg. if the block size will no longer match the one that would
be chosen by default. That would pick it up before anyone puts 1Kb block
size into production.
Thanks for identifying the issue.
--
Mark
From 8604c50be77a4bc56a91099598c409d5a3c1fdbe Mon Sep 17 00:00:00 2001
From: Mark Hills <mark@xxxxxxxx>
Date: Sat, 13 Nov 2021 11:46:50 +0000
Subject: [PATCH] Block size has an effect on the index size
---
fs/ext4/namei.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c
index f3bbcd4efb56..8965bed4d7ff 100644
--- a/fs/ext4/namei.c
+++ b/fs/ext4/namei.c
@@ -2454,8 +2454,9 @@ static int ext4_dx_add_entry(handle_t *handle, struct ext4_filename *fname,
}
if (add_level && levels == ext4_dir_htree_level(sb)) {
ext4_warning(sb, "Directory (ino: %lu) index full, "
- "reach max htree level :%d",
- dir->i_ino, levels);
+ "reach max htree level :%d"
+ "with block size %lu",
+ dir->i_ino, levels, sb->s_blocksize);
if (ext4_dir_htree_level(sb) < EXT4_HTREE_LEVEL) {
ext4_warning(sb, "Large directory feature is "
"not enabled on this "
--
2.33.1
