On Thu, Aug 15, 2024 at 2:25 PM Mateusz Guzik <mjguzik@xxxxxxxxx> wrote:
>
> I have an ext4-based system where xfs got mounted on tmpfs for testing
erm, i mean on /tmp :)
I also used noatime.
> purposes. The directory is being used a lot by gcc when compiling.
>
> I'm testing with 24 compilers running in parallel, each operating on
> their own hello world source file, listed at the end for reference.
>
> Both ext4 and btrfs backing the directory result in 100% cpu
> utilization and about 1500 compiles/second. With xfs I see about 20%
> idle(!) and about 1100 compiles/second.
>
> According to offcputime-bpfcc -K the time is spent waiting on the buf
> thing, sample traces:
>
> finish_task_switch.isra.0
> __schedule
> schedule
> schedule_timeout
> __down_common
> down
> xfs_buf_lock
> xfs_buf_find_lock
> xfs_buf_get_map
> xfs_buf_read_map
> xfs_trans_read_buf_map
> xfs_read_agi
> xfs_ialloc_read_agi
> xfs_dialloc
> xfs_create
> xfs_generic_create
> path_openat
> do_filp_open
> do_sys_openat2
> __x64_sys_openat
> do_syscall_64
> entry_SYSCALL_64_after_hwframe
> - cc (602142)
> 10639
>
> finish_task_switch.isra.0
> __schedule
> schedule
> schedule_timeout
> __down_common
> down
> xfs_buf_lock
> xfs_buf_find_lock
> xfs_buf_get_map
> xfs_buf_read_map
> xfs_trans_read_buf_map
> xfs_read_agi
> xfs_iunlink
> xfs_dir_remove_child
> xfs_remove
> xfs_vn_unlink
> vfs_unlink
> do_unlinkat
> __x64_sys_unlink
> do_syscall_64
> entry_SYSCALL_64_after_hwframe
> - as (598688)
> 12050
>
> The fact that this is contended aside, I'll note the stock semaphore
> code does not do adaptive spinning, which avoidably significantly
> worsens the impact. You can probably convert this to a rw semaphore
> and only ever writelock, which should sort out this aspect. I did not
> check what can be done to contend less to begin with.
>
> reproducing:
> create a hello world .c file (say /tmp/src.c) and plop into /src:
> for i in $(seq 0 23); do cp /tmp/src.c /src/src${i}.c; done
>
> plop the following into will-it-scale/tests/cc.c && ./cc_processes -t 24
>
> #include <sys/types.h>
> #include <unistd.h>
>
> char *testcase_description = "compile";
>
> void testcase(unsigned long long *iterations, unsigned long nr)
> {
> char cmd[1024];
>
> sprintf(&cmd, "cc -c -o /tmp/out.%d /src/src%d.c", nr, nr);
>
> while (1) {
> system(cmd);
>
> (*iterations)++;
> }
> }
>
> --
> Mateusz Guzik <mjguzik gmail.com>
--
Mateusz Guzik <mjguzik gmail.com>
