在 2024/10/1 11:07, Shakeel Butt 写道:
On Tue, Oct 01, 2024 at 07:30:38AM GMT, Qu Wenruo wrote:
[...]
Although btrfs has error handling already for all the possible ENOMEMs,
hitting ENOMEMs for metadata may still be a big problem, thus all my
previous attempt to remove NOFAIL flag all got rejected.
__GFP_NOFAIL for memcg charging is reasonable in many scenarios. Memcg
oom-killer is enabled for __GFP_NOFAIL and going over limit and getting
oom-killed is totally reasonable. Orthogonal to the discussion though.
2. What the normal overhead of these metadata in real world production
environment? I see 4 to 32 bytes per 4k but what's the most used one and
does it depend on the data of 4k or something else?
What did you mean by the "overhead" part? Did you mean the checksum?
To me this metadata is overhead, so yes checksum is something not the
actual data stored on the storage.
Oh, by "metadata" it means everything not data.
It includes all the info like directory layout, file layout, data
checksum and all the other needed info to represent a btrfs.
If so, there is none, because btrfs store metadata checksum inside the tree
block (thus the page cache).
The first 32 bytes of a tree block are always reserved for metadata
checksum.
The tree block size depends on the mkfs time option nodesize, is 16K by
default, and that's the most common value.
Sorry I am not very familiar with btrfs. What is tree block?
A tree block of btrfs is a fixed block, containing metadata (aka,
everything other than the data), organized in a B-tree structure.
A tree block can be a node, containing the pointers to the next level
nodes/leaves.
Or a leave, contains the key and the extra info bound to that key.
And btrfs uses the same tree block structure for all different kind of
info.
E.g. an inode is stored with (<ino> INODE_ITEM 0) as the key, with a
btrfs_inode_item structure as the extra data bound to that key.
And a file extent is stored with (<ino> EXTENT_DATA <file pos>) as the
key, with a btrfs_file_extent_item structure bound to that key.
3. Most probably multiple metadata values are colocated on a single 4k
page of the btrfs page cache even though the corresponding page cache
might be charged to different cgroups. Is that correct?
Not always a single 4K page, it depends on the nodesize, which is 16K by
default.
Otherwise yes, the metadata page cache can be charged to different cgroup,
depending on the caller's context.
And we do not want to charge the metadata page cache to the caller's cgroup,
since it's really a shared resource and the caller has no way to directly
accessing the page cache.
Not charging the metadata page cache will align btrfs more to the ext4/xfs,
which all uses regular page allocation without attaching to a filemap.
Can you point me to ext4/xfs code where they are allocating uncharged
memory for their metadata?
For xfs, it's inside fs/xfs/xfs_buf.c.
E.g. xfs_buf_alloc_pages(), which goes with kzalloc() to allocate needed
pages.
For ext4 it's using buffer header, which is I'm not familiar at all.
But it looks like the bh folios are from the block device mapping, which
may still be charged by cgroup.
Thanks,
Qu
4. What is stopping us to use reclaimable slab cache for this metadata?
Josef has tried this before, the attempt failed on the shrinker part, and
partly due to the size.
Btrfs has very large metadata compared to all other fses, not only due to
the COW nature and a larger tree block size (16K by default), but also the
extra data checksum (4 bytes per 4K by default, 32 bytes per 4K maximum).
On a real world system, the metadata itself can easily go hundreds of GiBs,
thus a shrinker is definitely needed.
This amount of uncharged memory is concerning which becomes part of
system overhead and may impact the schedulable memory for the datacenter
environment.
Overall the code seems fine and no pushback from me if btrfs maintainers
are ok with this. I think btrfs should move to slab+shrinker based
solution for this metadata unless there is deep technical reason not to.
thanks,
Shakeel
