On Thu, Nov 26, 2020 at 11:44 AM Dmitry Antipov <dmantipov@xxxxxxxxx> wrote:
BTW, did someone try to profile the brick process? I do, and got this
for the default replica 3 volume ('perf record -F 2500 -g -p [PID]'):
I run a slightly different command, which hides the kernel stuff and focuses on the user mode functions:
sudo perf record --call-graph dwarf -j any --buildid-all --all-user -p `pgrep -d\, gluster` -F 2000 -ag
Y.
+ 3.29% 0.02% glfs_epoll001 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 3.17% 0.01% glfs_epoll001 [kernel.kallsyms] [k] do_syscall_64
+ 3.17% 0.02% glfs_epoll000 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 3.06% 0.02% glfs_epoll000 [kernel.kallsyms] [k] do_syscall_64
+ 2.75% 0.01% glfs_iotwr00f [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.74% 0.01% glfs_iotwr00b [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.74% 0.01% glfs_iotwr001 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.73% 0.00% glfs_iotwr003 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.72% 0.00% glfs_iotwr000 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.72% 0.01% glfs_iotwr00c [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.70% 0.01% glfs_iotwr003 [kernel.kallsyms] [k] do_syscall_64
+ 2.69% 0.00% glfs_iotwr001 [kernel.kallsyms] [k] do_syscall_64
+ 2.69% 0.01% glfs_iotwr008 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.68% 0.00% glfs_iotwr00b [kernel.kallsyms] [k] do_syscall_64
+ 2.68% 0.00% glfs_iotwr00c [kernel.kallsyms] [k] do_syscall_64
+ 2.68% 0.00% glfs_iotwr00f [kernel.kallsyms] [k] do_syscall_64
+ 2.68% 0.01% glfs_iotwr000 [kernel.kallsyms] [k] do_syscall_64
+ 2.67% 0.00% glfs_iotwr00a [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.65% 0.00% glfs_iotwr008 [kernel.kallsyms] [k] do_syscall_64
+ 2.64% 0.00% glfs_iotwr00e [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.64% 0.01% glfs_iotwr00d [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.63% 0.01% glfs_iotwr00a [kernel.kallsyms] [k] do_syscall_64
+ 2.63% 0.01% glfs_iotwr007 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.63% 0.00% glfs_iotwr005 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.63% 0.01% glfs_iotwr006 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.63% 0.00% glfs_iotwr009 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.61% 0.01% glfs_iotwr004 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.61% 0.01% glfs_iotwr00e [kernel.kallsyms] [k] do_syscall_64
+ 2.60% 0.00% glfs_iotwr006 [kernel.kallsyms] [k] do_syscall_64
+ 2.59% 0.00% glfs_iotwr005 [kernel.kallsyms] [k] do_syscall_64
+ 2.59% 0.00% glfs_iotwr00d [kernel.kallsyms] [k] do_syscall_64
+ 2.58% 0.00% glfs_iotwr002 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.58% 0.01% glfs_iotwr007 [kernel.kallsyms] [k] do_syscall_64
+ 2.58% 0.00% glfs_iotwr004 [kernel.kallsyms] [k] do_syscall_64
+ 2.57% 0.00% glfs_iotwr009 [kernel.kallsyms] [k] do_syscall_64
+ 2.54% 0.00% glfs_iotwr002 [kernel.kallsyms] [k] do_syscall_64
+ 1.65% 0.00% glfs_epoll000 [unknown] [k] 0x0000000000000001
+ 1.65% 0.00% glfs_epoll001 [unknown] [k] 0x0000000000000001
+ 1.48% 0.01% glfs_rpcrqhnd [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 1.44% 0.08% glfs_rpcrqhnd libpthread-2.32.so [.] pthread_cond_wait@@GLIBC_2.3.2
+ 1.40% 0.01% glfs_rpcrqhnd [kernel.kallsyms] [k] do_syscall_64
+ 1.36% 0.01% glfs_rpcrqhnd [kernel.kallsyms] [k] __x64_sys_futex
+ 1.35% 0.03% glfs_rpcrqhnd [kernel.kallsyms] [k] do_futex
+ 1.34% 0.01% glfs_iotwr00a libpthread-2.32.so [.] __libc_pwrite64
+ 1.32% 0.00% glfs_iotwr00a [kernel.kallsyms] [k] __x64_sys_pwrite64
+ 1.32% 0.00% glfs_iotwr001 libpthread-2.32.so [.] __libc_pwrite64
+ 1.31% 0.01% glfs_iotwr002 libpthread-2.32.so [.] __libc_pwrite64
+ 1.31% 0.00% glfs_iotwr00b libpthread-2.32.so [.] __libc_pwrite64
+ 1.31% 0.01% glfs_iotwr00a [kernel.kallsyms] [k] vfs_write
+ 1.30% 0.00% glfs_iotwr001 [kernel.kallsyms] [k] __x64_sys_pwrite64
+ 1.30% 0.00% glfs_iotwr008 libpthread-2.32.so [.] __libc_pwrite64
+ 1.30% 0.00% glfs_iotwr00a [kernel.kallsyms] [k] new_sync_write
+ 1.30% 0.00% glfs_iotwr00c libpthread-2.32.so [.] __libc_pwrite64
+ 1.29% 0.00% glfs_iotwr00a [kernel.kallsyms] [k] xfs_file_write_iter
+ 1.29% 0.01% glfs_iotwr00a [kernel.kallsyms] [k] xfs_file_dio_aio_write
And on replica 3 with storage.linux-aio enabled:
+ 11.76% 0.05% glfs_posixaio [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 11.42% 0.01% glfs_posixaio [kernel.kallsyms] [k] do_syscall_64
+ 8.81% 0.00% glfs_posixaio [unknown] [k] 0x00000000baadf00d
+ 8.81% 0.00% glfs_posixaio [unknown] [k] 0x0000000000000004
+ 8.74% 0.06% glfs_posixaio libc-2.32.so [.] __GI___writev
+ 8.33% 0.02% glfs_posixaio [kernel.kallsyms] [k] do_writev
+ 8.23% 0.03% glfs_posixaio [kernel.kallsyms] [k] vfs_writev
+ 8.12% 0.05% glfs_posixaio [kernel.kallsyms] [k] do_iter_write
+ 8.02% 0.05% glfs_posixaio [kernel.kallsyms] [k] do_iter_readv_writev
+ 7.96% 0.04% glfs_posixaio [kernel.kallsyms] [k] sock_write_iter
+ 7.92% 0.01% glfs_posixaio [kernel.kallsyms] [k] sock_sendmsg
+ 7.86% 0.01% glfs_posixaio [kernel.kallsyms] [k] tcp_sendmsg
+ 7.28% 0.15% glfs_posixaio [kernel.kallsyms] [k] tcp_sendmsg_locked
+ 6.49% 0.01% glfs_posixaio [kernel.kallsyms] [k] __tcp_push_pending_frames
+ 6.48% 0.10% glfs_posixaio [kernel.kallsyms] [k] tcp_write_xmit
+ 6.31% 0.02% glfs_posixaio [unknown] [k] 0000000000000000
+ 6.05% 0.13% glfs_posixaio [kernel.kallsyms] [k] __tcp_transmit_skb
+ 5.71% 0.06% glfs_posixaio [kernel.kallsyms] [k] __ip_queue_xmit
+ 4.15% 0.03% glfs_rpcrqhnd [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 4.07% 0.08% glfs_posixaio [kernel.kallsyms] [k] ip_finish_output2
+ 3.75% 0.02% glfs_posixaio [kernel.kallsyms] [k] asm_call_sysvec_on_stack
+ 3.75% 0.01% glfs_rpcrqhnd [kernel.kallsyms] [k] do_syscall_64
+ 3.70% 0.03% glfs_rpcrqhnd [kernel.kallsyms] [k] __x64_sys_futex
+ 3.68% 0.06% glfs_posixaio [kernel.kallsyms] [k] __local_bh_enable_ip
+ 3.67% 0.07% glfs_rpcrqhnd [kernel.kallsyms] [k] do_futex
+ 3.62% 0.05% glfs_posixaio [kernel.kallsyms] [k] do_softirq
+ 3.61% 0.01% glfs_posixaio [kernel.kallsyms] [k] do_softirq_own_stack
+ 3.59% 0.06% glfs_posixaio [kernel.kallsyms] [k] __softirqentry_text_start
+ 3.44% 0.06% glfs_posixaio [kernel.kallsyms] [k] net_rx_action
+ 3.34% 0.04% glfs_posixaio [kernel.kallsyms] [k] process_backlog
+ 3.28% 0.02% glfs_posixaio [kernel.kallsyms] [k] __netif_receive_skb_one_core
+ 3.08% 0.02% glfs_epoll000 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 3.02% 0.03% glfs_epoll001 [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.97% 0.01% glfs_epoll000 [kernel.kallsyms] [k] do_syscall_64
+ 2.89% 0.01% glfs_epoll001 [kernel.kallsyms] [k] do_syscall_64
+ 2.73% 0.08% glfs_posixaio [kernel.kallsyms] [k] nf_hook_slow
+ 2.25% 0.04% glfs_posixaio libc-2.32.so [.] fgetxattr
+ 2.16% 0.14% glfs_rpcrqhnd [kernel.kallsyms] [k] futex_wake
According to these tables, the brick process is just a thin wrapper for the system calls
and kernel network subsystem behind them.
To whom it may be interesting, the following replica 3 volume options:
performance.io-cache-pass-through: on
performance.iot-pass-through: on
performance.md-cache-pass-through: on
performance.nl-cache-pass-through: on
performance.open-behind-pass-through: on
performance.read-ahead-pass-through: on
performance.readdir-ahead-pass-through: on
performance.strict-o-direct: on
features.ctime: off
features.selinux: off
performance.write-behind: off
performance.open-behind: off
performance.quick-read: off
storage.linux-aio: on
storage.fips-mode-rchecksum: off
are likely to improve the I/O performance of GFAPI clients (fio with gfapi and gfapi_async
engines, qemu -drive file=gluster://XXX, etc.) by ~20%. But beware of killing I/O performance
of FUSE clients.
Dmitry
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
