On Nov 13, 2021, at 04:05, Mark Hills <mark@xxxxxxxx> wrote:
>
> Andreas, thanks for such a prompt reply.
>
>> On Fri, 12 Nov 2021, Andreas Dilger wrote:
>>
>>> On Nov 12, 2021, at 11:37, Mark Hills <mark@xxxxxxxx> wrote:
>>>
>>> Surprised to hit a limit when handling a modest Maildir case; does
>>> this reflect a bug?
>>>
>>> rsync'ing to a new mail server, after fewer than 100,000 files there
>>> are intermittent failures:
>>
>> This is probably because you are using 1KB blocksize instead of 4KB,
>> which reduces the size of each tree level by the cube of the ratio, so
>> 64x. I guess that was selected because of very small files in the
>> maildir?
>
> Interesting! The 1Kb block size was not explicitly chosen. There was no
> plan other than using the defaults.
>
> However I did forget that this is a VM installed from a base image. The
> root cause is likely to be that the /home partition has been enlarged from
> a small size to 32Gb.
>
> Is block size the only factor? If so, a patch like below (untested) could
> make it clear it's relevant, and saved the question in this case.
The patch looks reasonable, but should be submitted separately with
[patch] in the subject so that it will not be lost.
You can also add on your patch:
Reviewed-by: Andreas Dilger <adilger@xxxxxxxxx>
Cheers, Andreas
>
> [...]
>> If you have a relatively recent kernel, you can enable the "large_dir"
>> feature to allow 3-level htree, which would be enough for another factor
>> of 1024/8 = 128 more entries than now (~12M).
>
> The system is not yet in use, so I think it's better we reformat here, and
> get a block size chosen by the experts :)
>
> These days I think VMs make it more common to enlarge a filesystem from a
> small size. We could have picked this up earlier with a warning from
> resize2fs; eg. if the block size will no longer match the one that would
> be chosen by default. That would pick it up before anyone puts 1Kb block
> size into production.
>
> Thanks for identifying the issue.
>
> --
> Mark
>
>
> From 8604c50be77a4bc56a91099598c409d5a3c1fdbe Mon Sep 17 00:00:00 2001
> From: Mark Hills <mark@xxxxxxxx>
> Date: Sat, 13 Nov 2021 11:46:50 +0000
> Subject: [PATCH] Block size has an effect on the index size
>
> ---
> fs/ext4/namei.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c
> index f3bbcd4efb56..8965bed4d7ff 100644
> --- a/fs/ext4/namei.c
> +++ b/fs/ext4/namei.c
> @@ -2454,8 +2454,9 @@ static int ext4_dx_add_entry(handle_t *handle, struct ext4_filename *fname,
> }
> if (add_level && levels == ext4_dir_htree_level(sb)) {
> ext4_warning(sb, "Directory (ino: %lu) index full, "
> - "reach max htree level :%d",
> - dir->i_ino, levels);
> + "reach max htree level :%d"
> + "with block size %lu",
> + dir->i_ino, levels, sb->s_blocksize);
> if (ext4_dir_htree_level(sb) < EXT4_HTREE_LEVEL) {
> ext4_warning(sb, "Large directory feature is "
> "not enabled on this "
> --
> 2.33.1
