From: Darrick J. Wong <djwong@xxxxxxxxxx> Recently, the upstream maintainers have been taking a lot of heat on account of writer threads encountering high latency when asking for log grant space when the log is small. The reported use case is a heavily threaded indexing product logging trace information to a filesystem ranging in size between 20 and 250GB. The meetings that result from the complaints about latency and stall warnings in dmesg both from this use case and also a large well known cloud product are now consuming 25% of the maintainer's weekly time and have been for months. For small filesystems, the log is small by default because we have defaulted to a ratio of 1:2048 (or even less). For grown filesystems, this is even worse, because big filesystems generate big metadata. However, the log size is still insufficient even if it is formatted at the larger size. On a 220GB filesystem, the 99.95% latencies observed with a 200-writer file synchronous append workload running on a 44-AG filesystem (with 44 CPUs) spread across 4 hard disks showed: 99.5% Log(MB) Latency(ms) BW (MB/s) xlog_grant_head_wait 10 520 243 1875 20 220 308 540 40 140 360 6 80 92 363 0 160 86 364 0 For 4 NVME, the results were: 10 201 409 898 20 177 488 144 40 122 550 0 80 120 549 0 160 121 545 0 This shows pretty clearly that we could reduce the amount of time that threads spend waiting on the XFS log by increasing the log size to at least 40MB regardless of size. We then repeated the benchmark with a cloud system and an old machine to see if there were any ill effects on less stable hardware. For cloudy iscsi block storage, the results were: 10 390 176 2584 20 173 186 357 40 37 187 0 80 40 183 0 160 37 183 0 A decade-old machine w/ 24 CPUs and a giant spinning disk RAID6 array produced this: 10 55 5.4 0 20 40 5.9 0 40 62 5.7 0 80 66 5.7 0 160 25 5.4 0
