Hi xfsers:
I got some troubles on the performance of xfs.
The environment is ,
xfs version is 3.2.1,
centos 7.1,
kernel version:3.10.0-229.el7.x86_64.
pcie-ssd card,
mkfs: mkfs.xfs /dev/hioa2 -f -n size=64k -i size=512 -d agcount=40 -l size=1024m.
mount: mount /dev/hioa2 /mnt/ -t xfs -o rw,noexec,nodev,noatime,nodiratime,nobarrier,discard,inode64,logbsize=256k,delaylog
I use the following command to test iops: fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=50G -filename=/mnt/test -name="EBS 4KB randwrite test" -iodepth=64 -runtime=60
The results is normal at the beginning which is about 210k±，but some seconds later, the results down to 19k±.
I did a senond test ,
umount the /dev/hioa2,
fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -filename=/dev/hioa2 -name="EBS 8KB randwrite test" -iodepth=64 -runtime=60
The results was normal, the iops is about 210k± all the time.
Then I use perf-tools to trace xfs:
# ./funccount -i 1 'xfs*'
In the first test case , when the iops seems normal , the results is as follows:
FUNC COUNT
xfs_free_eofblocks 1
xfs_attr_get 5
xfs_attr_get_int 5
xfs_attr_name_to_xname 5
xfs_attr_shortform_getvalue 5
xfs_da_hashname 5
xfs_ilock_attr_map_shared 5
xfs_xattr_get 5
xfs_fs_statfs 7
xfs_icsb_count 7
xfs_icsb_sync_counters 7
xfs_icsb_sync_counters_locked 7
xfs_can_free_eofblocks 9
xfs_file_release 9
xfs_release 9
xfs_icsb_counter_disabled 21
xfs_file_open 30
xfs_file_aio_read 33
xfs_readlink 37
xfs_vn_follow_link 37
xfs_vn_put_link 37
xfs_buf_delwri_submit_nowait 41
xfs_trans_ail_cursor_first 41
xfs_trans_ail_cursor_init 41
xfs_vn_getattr 44
xfs_file_mmap 48
xfs_cil_prepare_item.isra.1 1004
xfs_extent_busy_clear 1004
xfs_inode_item_committing 1004
xfs_inode_item_data_fork_size. 1004
xfs_inode_item_format 1004
xfs_inode_item_format_data_for 1004
xfs_inode_item_size 1004
xfs_inode_item_unlock 1004
xfs_log_calc_unit_res 1004
xfs_log_commit_cil 1004
xfs_log_done 1004
xfs_log_reserve 1004
xfs_log_space_wake 1004
xfs_log_ticket_put 1004
xfs_trans_add_item 1004
xfs_trans_alloc 1004
xfs_trans_apply_dquot_deltas 1004
xfs_trans_commit 1004
xfs_trans_free 1004
xfs_trans_free_dqinfo 1004
xfs_trans_free_item_desc 1004
xfs_trans_free_items 1004
xfs_trans_ijoin 1004
xfs_trans_log_inode 1004
xfs_trans_reserve 1004
xfs_trans_unreserve_and_mod_sb 1004
xfs_vn_update_time 1004
xfs_destroy_ioend 84845
xfs_end_io_direct_write 84845
xfs_finish_ioend 84845
xfs_file_aio_write 84880
xfs_file_dio_aio_write 84882
xfs_vm_direct_IO 84882
xfs_get_blocks_direct 84883
xfs_bmapi_read 84885
xfs_bmap_search_extents 84886
xfs_bmap_search_multi_extents 84886
xfs_iext_bno_to_ext 84887
xfs_iext_bno_to_irec 84887
xfs_iext_get_ext 84890
xfs_iext_idx_to_irec 84890
xfs_ilock_data_map_shared 84891
xfs_map_buffer.isra.9 84892
xfs_file_aio_write_checks 84893
xfs_alloc_ioend 84895
xfs_bmapi_trim_map.isra.11 84900
xfs_fsb_to_db 84902
xfs_bmapi_update_map 84903
xfs_bmbt_get_all 169793
xfs_find_bdev_for_inode 169794
xfs_ilock 170868
xfs_iunlock 170872
xfs_bmbt_get_blockcount 555016
xfs_bmbt_get_startoff 2277490
when the results down to 19k±， the trace results are:
FUNC COUNT
xfs_bitmap_empty 1
xfs_free_eofblocks 1
xfs_attr_get 5
xfs_attr_get_int 5
xfs_attr_name_to_xname 5
xfs_attr_shortform_getvalue 5
xfs_da_hashname 5
xfs_ilock_attr_map_shared 5
xfs_xattr_get 5
xfs_fs_statfs 7
xfs_icsb_count 7
xfs_icsb_sync_counters 7
xfs_icsb_sync_counters_locked 7
xfs_can_free_eofblocks 9
xfs_file_release 9
xfs_release 9
xfs_file_open 30
xfs_file_aio_read 33
xfs_readlink 37
xfs_vn_follow_link 37
xfs_vn_put_link 37
xfs_buf_delwri_submit_nowait 41
xfs_trans_ail_cursor_first 41
xfs_trans_ail_cursor_init 41
xfs_alloc_ag_vextent 43
xfs_alloc_ag_vextent_near 43
xfs_alloc_find_best_extent 43
xfs_alloc_fix_freelist 43
xfs_alloc_fixup_trees 43
xfs_alloc_read_agf 43
xfs_alloc_read_agfl 43
xfs_alloc_update 43
xfs_alloc_update_counters.isra 43
xfs_alloc_vextent 43
xfs_bmbt_alloc_block 43
xfs_btree_get_buf_block.constp 43
xfs_btree_reada_bufs 43
xfs_btree_readahead_sblock.isr 43
xfs_btree_split 43
xfs_btree_split_worker 43
xfs_read_agf 43
xfs_trans_get_buf_map 43
xfs_allocbt_dup_cursor 44
xfs_allocbt_get_minrecs 44
xfs_allocbt_init_rec_from_cur 44
xfs_allocbt_update_lastrec 44
xfs_alloc_fix_minleft 44
xfs_alloc_log_agf 44
xfs_alloc_longest_free_extent 44
xfs_bmbt_init_rec_from_key 44
xfs_bmbt_update_cursor 44
xfs_btree_buf_to_ptr.isra.17 44
xfs_btree_check_sblock 44
xfs_btree_init_block_cur.isra. 44
xfs_btree_init_block_int 44
xfs_btree_islastblock 44
xfs_trans_mod_dquot_byino 44
xfs_trans_mod_sb 44
xfs_vn_getattr 44
xfs_file_mmap 48
xfs_iext_irec_new 57
xfs_iext_realloc_indirect 57
xfs_btree_copy_ptrs 66
xfs_btree_shift_keys.isra.23 66
xfs_btree_shift_ptrs 66
xfs_btree_log_ptrs 72
xfs_alloc_get_rec 86
xfs_alloc_lookup_eq 86
xfs_alloc_compute_aligned 88
xfs_btree_get_rec 88
xfs_extent_busy_trim 88
xfs_iext_add_indirect_multi 108
xfs_alloc_compute_diff 110
xfs_alloc_fix_len 110
xfs_allocbt_init_cursor 132
xfs_iext_irec_compact 163
xfs_iext_remove 163
xfs_iext_remove_indirect 163
xfs_buf_item_relse 173
xfs_allocbt_init_ptr_from_cur 176
xfs_btree_set_sibling.isra.12 176
xfs_buf_item_free 177
xfs_buf_item_free_format 177
xfs_buf_hold 248
xfs_buf_item_get_format 248
xfs_log_item_init 248
xfs_allocbt_get_maxrecs 264
xfs_bmbt_get_minrecs 326
xfs_btree_dec_cursor 369
xfs_btree_delete 369
xfs_btree_delrec 369
xfs_bmbt_set_allf 633
xfs_bmbt_set_startblock 1007
xfs_bmbt_set_startoff 1007
xfs_trans_alloc 1029
xfs_vn_update_time 1029
xfs_btree_lshift 1048
xfs_btree_decrement 1332
xfs_allocbt_init_key_from_rec 1481
xfs_allocbt_key_diff 2427
xfs_btree_check_lblock 3653
xfs_btree_lastrec 3653
xfs_buf_free 3660
xfs_bmbt_dup_cursor 3696
xfs_btree_check_block 3696
xfs_buf_allocate_memory 3703
xfs_btree_dup_cursor 3738
xfs_btree_increment 4324
xfs_btree_make_block_unfull 4699
xfs_btree_rshift 4699
xfs_btree_updkey 4738
xfs_btree_copy_keys.isra.22 4824
xfs_btree_log_keys 4832
xfs_iext_add 4876
xfs_iext_insert 4876
xfs_btree_reada_bufl 5465
xfs_btree_readahead_lblock.isr 5465
xfs_buf_readahead_map 5508
xfs_btree_readahead 5670
xfs_inode_item_format 6071
xfs_inode_item_format_data_for 6072
xfs_inode_item_size 6072
xfs_inode_item_data_fork_size. 6073
xfs_trans_log_inode 6115
xfs_buf_item_pin 6315
xfs_bmapi_write 6674
xfs_end_io 6674
xfs_iomap_write_unwritten 6674
xfs_bmap_add_extent_unwritten_ 6675
xfs_bmapi_convert_unwritten 6675
xfs_destroy_ioend 6675
xfs_end_io_direct_write 6676
xfs_finish_ioend 6677
xfs_bmap_finish 6683
xfs_file_aio_write 6784
xfs_bmapi_read 6785
xfs_file_aio_write_checks 6785
xfs_file_dio_aio_write 6785
xfs_get_blocks_direct 6785
xfs_ilock_data_map_shared 6785
xfs_map_buffer.isra.9 6785
xfs_vm_direct_IO 6785
xfs_alloc_ioend 6786
xfs_fsb_to_db 6786
xfs_bmbt_set_blockcount 7691
xfs_log_commit_cil 7704
xfs_trans_commit 7704
xfs_trans_free_items 7704
xfs_trans_reserve 7705
xfs_trans_unreserve_and_mod_sb 7705
xfs_log_done 7706
xfs_trans_free 7706
xfs_log_reserve 7708
xfs_trans_ijoin 7708
xfs_extent_busy_clear 7712
xfs_inode_item_unlock 7712
xfs_log_space_wake 7712
xfs_log_ticket_put 7712
xfs_trans_apply_dquot_deltas 7712
xfs_inode_item_committing 7713
xfs_log_calc_unit_res 7713
xfs_trans_free_dqinfo 7713
xfs_bmbt_update 8311
xfs_btree_update 8354
xfs_bmbt_init_rec_from_cur 9758
xfs_bmbt_disk_set_all 9760
xfs_bmbt_set_all 9760
xfs_btree_insert 9795
xfs_btree_insrec 9838
xfs_bmbt_init_cursor 10379
xfs_btree_del_cursor 10498
xfs_trans_del_item 11101
xfs_buf_item_dirty 11109
xfs_bmbt_lookup_eq 11551
xfs_bmbt_init_ptr_from_cur 11563
xfs_btree_get_iroot.isra.8 11563
xfs_buf_item_format 11690
xfs_buf_item_size 11690
xfs_buf_item_size_segment.isra 11690
xfs_btree_lookup 11723
xfs_btree_get_sibling.isra.11 11974
xfs_buf_item_unlock 12229
xfs_buf_item_committing 12237
xfs_icsb_modify_counters 13351
xfs_icsb_lock_cntr 13365
xfs_icsb_unlock_cntr 13365
xfs_iext_bno_to_irec 13458
xfs_bmap_search_extents 13459
xfs_bmap_search_multi_extents 13459
xfs_iext_bno_to_ext 13459
xfs_bmapi_trim_map.isra.11 13469
xfs_bmapi_update_map 13469
xfs_find_bdev_for_inode 13573
xfs_btree_shift_recs.isra.24 14798
xfs_cil_prepare_item.isra.1 17769
xfs_bmbt_disk_set_allf 18126
xfs_btree_is_lastrec 18581
xfs_btree_log_block 19648
xfs_btree_set_ptr_null.isra.10 19649
xfs_btree_offsets 19703
xfs_iunlock 21348
xfs_ilock 21349
xfs_btree_copy_recs.isra.21 22849
xfs_buf_item_init 23331
xfs_buf_unlock 23345
xfs_btree_setbuf 23948
xfs_btree_log_recs 24190
xfs_icsb_counter_disabled 26751
xfs_buf_read_map 28786
xfs_buf_get_map 28829
xfs_btree_ptr_is_null.isra.9 29274
xfs_btree_read_buf_block.isra. 29684
xfs_btree_set_refs.isra.13 29712
xfs_trans_add_item 31037
xfs_trans_free_item_desc 31052
xfs_buf_trylock 32463
xfs_perag_put 32606
xfs_perag_get 32633
xfs_buf_rele 32693
xfs_trans_brelse 35148
xfs_btree_ptr_addr 38591
xfs_btree_ptr_offset 38738
xfs_trans_read_buf_map 40938
xfs_trans_buf_item_match 41000
xfs_btree_ptr_to_daddr 44562
xfs_btree_lookup_get_block 46548
xfs_trans_log_buf 48783
xfs_buf_item_log 48806
xfs_trans_buf_set_type 48815
xfs_iext_get_ext 51712
xfs_bmbt_get_all 53669
xfs_btree_get_block 54424
xfs_iext_idx_to_irec 56768
xfs_bmbt_get_maxrecs 57774
xfs_bmbt_get_blockcount 89650
xfs_bmbt_disk_get_startoff 98285
xfs_bmbt_init_key_from_rec 98285
xfs_btree_rec_addr 115199
xfs_btree_rec_offset 163635
xfs_btree_key_addr 176837
xfs_btree_key_offset 186525
xfs_bmbt_key_diff 254763
xfs_lookup_get_search_key 257190
xfs_bmbt_get_startoff 363158
xfs_next_bit 631334
xfs_buf_offset 1166477
Compare with the two results above, I found when the iops down to 19k±, lots of functions were traced, such as xfs_buf_trylock, xfs_iext_insert, xfs_btree_insert, etc.
I cannot find what cause the iops down to 19k±, any suggestion ?
_______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
