From: Dave Chinner <dchinner@xxxxxxxxxx> The biggest problem with large directory block sizes is the CPU overhead in maintaining the buffer log item direty region bitmap. The bit manipulations and buffer region mapping calls are right at the top of the profiles when running tests on 64k directory buffers: + 16.65% [kernel] [k] memcpy + 11.99% [kernel] [k] xfs_next_bit + 5.87% [kernel] [k] xfs_buf_item_format + 5.85% [kernel] [k] xfs_buf_item_size_segment.isra.4 + 5.72% [kernel] [k] xfs_buf_offset The memcpy is the copying of the dirty regions into the log vec array, but almost twice as much CPU time is spent working out what needs to be copied and where it needs to be copied from. As a result, on a production kernel creating 100,000 entries in a 64k directory runs at about 9,000 files/s while on a 4k directory block size it runs at 19,000 files/s - about half the speed. Switching this to just track the first and last modified bytes in the block and only converting that to a dirty bitmap in the buffer log item at format time, however, gets rid of most of this dirty bitmap overhead without increasing memcpy time at all. the result is that peformance on a 64k directory block size increases to roughly 16,000 files/s with memcpy() overhead only slightly increasing. The profile now looks like: 10.41% [kernel] [k] __memcpy 0.42% [kernel] [k] xfs_buf_item_init 0.27% [kernel] [k] xfs_buf_item_size 0.21% [kernel] [k] xfs_buf_item_format_segment 0.15% [kernel] [k] xfs_buf_item_log And the majority of the memcpy load is coming from: - 9.43% 9.43% [kernel] [k] __memcpy - __memcpy + 95.63% xfs_buf_item_format_segment + 1.93% xfs_buf_item_format The buffer item formatting code. Hence we can see that there is major reduction in CPU overhead as well as an improvement in performance for 64k buffers with this change. The current implementation tracks, at most, four dirty regions per buffer. The nature of directory operations result in almost operation modifying a header in the buffer, a tail section in the buffer and then some number of bytes/regions in the middle of the buffer. Hence if we just track a single region, it will almost always cover the entire directory buffer - if we only modify a single entry in the buffer, then that's a fairly large cost in terms of log space and CPU overhead for random individual operations. We also have to consider non-directory buffer modification patterns. freespace, inode and extent btrees are the other major types of buffers that get logged, but they also have modification patterns that lend themselves well to a small number of ranges for dirty tracking. That is, each btree block is kept compact, so when we insert or remove a record or pointer we shift then higher records/ptrs up or down as a block, and then log the lot of them. And they also often have a header that is dirtied with each insert/delete, so typically there are usually only one or two dirty ranges in a btree block. The only metadata type that really seems to benefit from fine grained dirty range logging is the inode buffers. Specifically, for v4 superblocks the create transaction only dirties the regions of the inode core, so for 256 byte inodes only dirties every alternate bitmap segment. Dirty range tracking will double the required log bandwidth of inode buffers during create (roughly 25% increase on a 4k directory block size filesystem). There isn't any performance differential noticable on typical systems because the log is far from being bandwidth bound. For v5 filesystems, this isn't an issue because the initialised inode buffers are XFS_BLI_ORDERED buffers and so their contents aren't logged. The same problem happens with unlinks due to the unlinked list being logged via the inode buffer. Again this results in an increase in log bandwidth on both v4 and v5 filesystems, but there isn't any performance differential that occurs because, again, the log isn't bandwidth bound. As it is, there is an existing plan of improvement to the unlinked list logging (moving the unlinked list logging into the inode core transaction) and hence that will avoid any extra overhead here as well. Hence the overall CPU reduction benefits of minimal dirty range tracking versus fine grained dirty bit tracking are overall going to be beneficial to performance and throughput on current (v5) format filesystems. Signed-off-by: Dave Chinner <dchinner@xxxxxxxxxx> --- fs/xfs/xfs_buf.c | 2 + fs/xfs/xfs_buf_item.c | 422 ++++++++++++++++++++++++++------------------------ fs/xfs/xfs_buf_item.h | 19 +++ 3 files changed, 244 insertions(+), 199 deletions(-) diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 9a2191b..467a636 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1462,6 +1462,8 @@ xfs_buf_iomove( page = bp->b_pages[page_index]; csize = min_t(size_t, PAGE_SIZE - page_offset, BBTOB(bp->b_io_length) - boff); + if (boff + csize > bend) + csize = bend - boff; ASSERT((csize + page_offset) <= PAGE_SIZE); diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c index 55b5ad0..53fd2dc 100644 --- a/fs/xfs/xfs_buf_item.c +++ b/fs/xfs/xfs_buf_item.c @@ -65,50 +65,12 @@ xfs_buf_item_size_segment( int *nvecs, int *nbytes) { - struct xfs_buf *bp = bip->bli_buf; - int next_bit; - int last_bit; - - last_bit = xfs_next_bit(blfp->blf_data_map, blfp->blf_map_size, 0); - if (last_bit == -1) - return; - /* * initial count for a dirty buffer is 2 vectors - the format structure - * and the first dirty region. + * and the dirty region. Dirty region is accounted for separately. */ *nvecs += 2; - *nbytes += xfs_buf_log_format_size(blfp) + XFS_BLF_CHUNK; - - while (last_bit != -1) { - /* - * This takes the bit number to start looking from and - * returns the next set bit from there. It returns -1 - * if there are no more bits set or the start bit is - * beyond the end of the bitmap. - */ - next_bit = xfs_next_bit(blfp->blf_data_map, blfp->blf_map_size, - last_bit + 1); - /* - * If we run out of bits, leave the loop, - * else if we find a new set of bits bump the number of vecs, - * else keep scanning the current set of bits. - */ - if (next_bit == -1) { - break; - } else if (next_bit != last_bit + 1) { - last_bit = next_bit; - (*nvecs)++; - } else if (xfs_buf_offset(bp, next_bit * XFS_BLF_CHUNK) != - (xfs_buf_offset(bp, last_bit * XFS_BLF_CHUNK) + - XFS_BLF_CHUNK)) { - last_bit = next_bit; - (*nvecs)++; - } else { - last_bit++; - } - *nbytes += XFS_BLF_CHUNK; - } + *nbytes += xfs_buf_log_format_size(blfp); } /* @@ -135,7 +97,9 @@ xfs_buf_item_size( int *nbytes) { struct xfs_buf_log_item *bip = BUF_ITEM(lip); - int i; + struct xfs_buf *bp = bip->bli_buf; + uint offset; + int i, j; ASSERT(atomic_read(&bip->bli_refcount) > 0); if (bip->bli_flags & XFS_BLI_STALE) { @@ -154,6 +118,7 @@ xfs_buf_item_size( } ASSERT(bip->bli_flags & XFS_BLI_LOGGED); + ASSERT(bip->bli_flags & XFS_BLI_DIRTY); if (bip->bli_flags & XFS_BLI_ORDERED) { /* @@ -168,17 +133,45 @@ xfs_buf_item_size( /* * the vector count is based on the number of buffer vectors we have - * dirty bits in. This will only be greater than one when we have a + * dirty ranges in. This will only be greater than one when we have a * compound buffer with more than one segment dirty. Hence for compound - * buffers we need to track which segment the dirty bits correspond to, - * and when we move from one segment to the next increment the vector - * count for the extra buf log format structure that will need to be - * written. + * buffers we need to track which segment the dirty ranges correspond + * to, and when we move from one segment to the next increment the + * vector count for the extra buf log format structure that will need to + * be written. */ - for (i = 0; i < bip->bli_format_count; i++) { - xfs_buf_item_size_segment(bip, &bip->bli_formats[i], - nvecs, nbytes); + ASSERT(bip->bli_range[0].last != 0); + if (bip->bli_range[0].last == 0) { + /* clean! */ + ASSERT(bip->bli_range[0].first == 0); + return; } + + for (i = 0, offset = 0; + i < bip->bli_format_count; + i++, offset += BBTOB(bp->b_maps[i].bm_len)) { + /* Only format dirty regions */ + for (j = 0; j < bip->bli_ranges; j++) { + struct xfs_bli_range *rp = &bip->bli_range[j]; + + /* range ends before segment start, check next range */ + if (rp->last < offset) + continue; + + /* range beyond segment end, check next segment */ + if (rp->first > offset + BBTOB(bp->b_maps[i].bm_len)) + break; + + /* dirty range overlaps segment, need headers */ + xfs_buf_item_size_segment(bip, &bip->bli_formats[i], + nvecs, nbytes); + } + } + + for (j = 0; j < bip->bli_ranges; j++) + *nbytes += bip->bli_range[j].last - bip->bli_range[j].first; + + trace_xfs_buf_item_size(bip); } @@ -191,7 +184,6 @@ xfs_buf_item_copy_iovec( int first_bit, uint nbits) { - offset += first_bit * XFS_BLF_CHUNK; xlog_copy_iovec(lv, vecp, XLOG_REG_TYPE_BCHUNK, xfs_buf_offset(bp, offset), nbits * XFS_BLF_CHUNK); @@ -214,14 +206,18 @@ xfs_buf_item_format_segment( struct xfs_buf_log_item *bip, struct xfs_log_vec *lv, struct xfs_log_iovec **vecp, + struct xfs_bli_range *rp, uint offset, + uint length, struct xfs_buf_log_format *blfp) { struct xfs_buf *bp = bip->bli_buf; + char *buf; uint base_size; + uint start; + uint end; int first_bit; int last_bit; - int next_bit; uint nbits; /* copy the flags across from the base format item */ @@ -233,16 +229,6 @@ xfs_buf_item_format_segment( * memory structure. */ base_size = xfs_buf_log_format_size(blfp); - - first_bit = xfs_next_bit(blfp->blf_data_map, blfp->blf_map_size, 0); - if (!(bip->bli_flags & XFS_BLI_STALE) && first_bit == -1) { - /* - * If the map is not be dirty in the transaction, mark - * the size as zero and do not advance the vector pointer. - */ - return; - } - blfp = xlog_copy_iovec(lv, vecp, XLOG_REG_TYPE_BFORMAT, blfp, base_size); blfp->blf_size = 1; @@ -257,46 +243,40 @@ xfs_buf_item_format_segment( return; } + blfp->blf_size++; /* - * Fill in an iovec for each set of contiguous chunks. + * Now we need to set the bits in the bitmap and set up the iovecs + * appropriately. We know there is a contiguous range in this buffer + * than needs to be set, so find the first bit, the last bit, and + * go from there. */ - last_bit = first_bit; - nbits = 1; - for (;;) { - /* - * This takes the bit number to start looking from and - * returns the next set bit from there. It returns -1 - * if there are no more bits set or the start bit is - * beyond the end of the bitmap. - */ - next_bit = xfs_next_bit(blfp->blf_data_map, blfp->blf_map_size, - (uint)last_bit + 1); - /* - * If we run out of bits fill in the last iovec and get out of - * the loop. Else if we start a new set of bits then fill in - * the iovec for the series we were looking at and start - * counting the bits in the new one. Else we're still in the - * same set of bits so just keep counting and scanning. - */ - if (next_bit == -1) { - xfs_buf_item_copy_iovec(lv, vecp, bp, offset, - first_bit, nbits); - blfp->blf_size++; - break; - } else if (next_bit != last_bit + 1 || - xfs_buf_item_straddle(bp, offset, next_bit, last_bit)) { - xfs_buf_item_copy_iovec(lv, vecp, bp, offset, - first_bit, nbits); - blfp->blf_size++; - first_bit = next_bit; - last_bit = next_bit; - nbits = 1; - } else { - last_bit++; - nbits++; - } - } + start = 0; + if (offset < rp->first) + start = rp->first - offset; + end = length - 1; + if (offset + length > rp->last) + end = rp->last - offset - 1; + + start &= ~((1 << XFS_BLF_SHIFT) - 1); + first_bit = start >> XFS_BLF_SHIFT; + last_bit = end >> XFS_BLF_SHIFT; + nbits = last_bit - first_bit + 1; + bitmap_set((unsigned long *)blfp->blf_data_map, first_bit, nbits); + + ASSERT(end <= length); + ASSERT(start <= length); + ASSERT(length >= nbits * XFS_BLF_CHUNK); + /* + * Copy needs to be done a buffer page at a time as we can be logging + * unmapped buffers. hence we have to use xfs_buf_iomove() rather than a + * straight memcpy here. + */ + offset += first_bit * XFS_BLF_CHUNK; + length = nbits * XFS_BLF_CHUNK; + buf = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_BCHUNK); + xfs_buf_iomove(bp, offset, length, buf, XBRW_READ); + xlog_finish_iovec(lv, *vecp, length); } /* @@ -313,8 +293,8 @@ xfs_buf_item_format( struct xfs_buf_log_item *bip = BUF_ITEM(lip); struct xfs_buf *bp = bip->bli_buf; struct xfs_log_iovec *vecp = NULL; - uint offset = 0; - int i; + uint offset; + int i, j; ASSERT(atomic_read(&bip->bli_refcount) > 0); ASSERT((bip->bli_flags & XFS_BLI_LOGGED) || @@ -323,7 +303,6 @@ xfs_buf_item_format( (xfs_blft_from_flags(&bip->__bli_format) > XFS_BLFT_UNKNOWN_BUF && xfs_blft_from_flags(&bip->__bli_format) < XFS_BLFT_MAX_BUF)); - /* * If it is an inode buffer, transfer the in-memory state to the * format flags and clear the in-memory state. @@ -356,10 +335,37 @@ xfs_buf_item_format( return; } - for (i = 0; i < bip->bli_format_count; i++) { - xfs_buf_item_format_segment(bip, lv, &vecp, offset, - &bip->bli_formats[i]); - offset += bp->b_maps[i].bm_len; + for (i = 0, offset = 0; + i < bip->bli_format_count; + i++, offset += BBTOB(bp->b_maps[i].bm_len)) { + + /* stale regions cover the entire segment */ + if (bip->bli_flags & XFS_BLI_STALE) { + xfs_buf_item_format_segment(bip, lv, &vecp, NULL, offset, + BBTOB(bp->b_maps[i].bm_len), + &bip->bli_formats[i]); + continue; + } + + /* only format dirty ranges over the current segment */ + for (j = 0; j < bip->bli_ranges; j++) { + struct xfs_bli_range *rp = &bip->bli_range[j]; + + /* range ends before segment start, check next range */ + if (rp->last < offset) + continue; + + /* range beyond segment end, check next segment */ + if (rp->first > offset + BBTOB(bp->b_maps[i].bm_len)) + break; + + /* dirty range overlaps segment, need headers */ + xfs_buf_item_format_segment(bip, lv, &vecp, rp, offset, + BBTOB(bp->b_maps[i].bm_len), + &bip->bli_formats[i]); + + } + } /* @@ -800,6 +806,9 @@ xfs_buf_item_init( bip->bli_formats[i].blf_map_size = map_size; } + for (i = 0; i < XFS_BLI_RANGES; i++) + bip->bli_range[i].first = UINT_MAX; + /* * Put the buf item into the list of items attached to the * buffer at the front. @@ -811,124 +820,139 @@ xfs_buf_item_init( return 0; } - /* * Mark bytes first through last inclusive as dirty in the buf - * item's bitmap. + * record dirty regions on the buffer. */ -static void -xfs_buf_item_log_segment( +void +xfs_buf_item_log( + xfs_buf_log_item_t *bip, uint first, - uint last, - uint *map) + uint last) { - uint first_bit; - uint last_bit; - uint bits_to_set; - uint bits_set; - uint word_num; - uint *wordp; - uint bit; - uint end_bit; - uint mask; + struct xfs_bli_range *rp = NULL; + int i; - /* - * Convert byte offsets to bit numbers. - */ - first_bit = first >> XFS_BLF_SHIFT; - last_bit = last >> XFS_BLF_SHIFT; + ASSERT(last != 0); + ASSERT(first <= last); + ASSERT(last < BBTOB(bip->bli_buf->b_length)); - /* - * Calculate the total number of bits to be set. - */ - bits_to_set = last_bit - first_bit + 1; - /* - * Get a pointer to the first word in the bitmap - * to set a bit in. - */ - word_num = first_bit >> BIT_TO_WORD_SHIFT; - wordp = &map[word_num]; + /* simple case - first range being stored */ + if (!bip->bli_ranges) { + bip->bli_ranges = 1; + bip->bli_range[0].first = rounddown(first, XFS_BLF_CHUNK); + bip->bli_range[0].last = roundup(last, XFS_BLF_CHUNK); + ASSERT(bip->bli_range[0].last != 0); + ASSERT(bip->bli_range[0].first <= bip->bli_range[0].last); + return; + } - /* - * Calculate the starting bit in the first word. - */ - bit = first_bit & (uint)(NBWORD - 1); + /* 2nd case: search for overlaps and extend */ + for (i = 0; i < bip->bli_ranges; i++) { + rp = &bip->bli_range[i]; - /* - * First set any bits in the first word of our range. - * If it starts at bit 0 of the word, it will be - * set below rather than here. That is what the variable - * bit tells us. The variable bits_set tracks the number - * of bits that have been set so far. End_bit is the number - * of the last bit to be set in this word plus one. - */ - if (bit) { - end_bit = MIN(bit + bits_to_set, (uint)NBWORD); - mask = ((1 << (end_bit - bit)) - 1) << bit; - *wordp |= mask; - wordp++; - bits_set = end_bit - bit; - } else { - bits_set = 0; + /* wholly within an existing dirty range, we're done */ + if (first >= rp->first && last <= rp->last) + return; + + /* no overlap, continue */ + if (first > rp->last || last < rp->first) + continue; + + /* left edge overlap, extend */ + if (first < rp->first) + rp->first = rounddown(first, XFS_BLF_CHUNK); + + /* right edge overlap, extend */ + if (last > rp->last) + rp->last = roundup(last, XFS_BLF_CHUNK) - 1; + + goto merge; } + /* 3rd case: not found, insert or extend */ + ASSERT(i == bip->bli_ranges); + /* - * Now set bits a whole word at a time that are between - * first_bit and last_bit. + * Case 3a: Extend last slot. + * + * If the range is beyond the last slot, extend the last slot to + * cover it. This treated the same as if an overlap existed with + * the last range. */ - while ((bits_to_set - bits_set) >= NBWORD) { - *wordp |= 0xffffffff; - bits_set += NBWORD; - wordp++; + + if (i == XFS_BLI_RANGES) { + ASSERT(bip->bli_ranges == XFS_BLI_RANGES); + rp = &bip->bli_range[XFS_BLI_RANGES - 1]; + + if (first < rp->first) + rp->first = rounddown(first, XFS_BLF_CHUNK); + if (last > rp->last) + rp->last = roundup(last, XFS_BLF_CHUNK) - 1; + goto merge; } - /* - * Finally, set any bits left to be set in one last partial word. + /* Case 3b: insert new range. + * + * Find the insertion point for the new range, then make a hole + * and insert the new range. */ - end_bit = bits_to_set - bits_set; - if (end_bit) { - mask = (1 << end_bit) - 1; - *wordp |= mask; + for (i = 0; i < bip->bli_ranges; i++) { + rp = &bip->bli_range[i]; + + /* order ranges by ascending offset */ + if (last < rp->first) + break; } -} -/* - * Mark bytes first through last inclusive as dirty in the buf - * item's bitmap. - */ -void -xfs_buf_item_log( - xfs_buf_log_item_t *bip, - uint first, - uint last) -{ - int i; - uint start; - uint end; - struct xfs_buf *bp = bip->bli_buf; + /* shift down and insert */ + ASSERT(i < XFS_BLI_RANGES); + rp = &bip->bli_range[i]; + if (i < XFS_BLI_RANGES - 1) + memmove(rp + 1, rp, sizeof(*rp) * (bip->bli_ranges - i)); + bip->bli_ranges++; + rp->first = rounddown(first, XFS_BLF_CHUNK); + rp->last = roundup(last, XFS_BLF_CHUNK) - 1; +merge: /* - * walk each buffer segment and mark them dirty appropriately. + * Check for overlaping ranges and merge them. If there is only one + * range, there is nothing to merge so bail early. */ - start = 0; - for (i = 0; i < bip->bli_format_count; i++) { - if (start > last) - break; - end = start + BBTOB(bp->b_maps[i].bm_len); - if (first > end) { - start += BBTOB(bp->b_maps[i].bm_len); - continue; - } - if (first < start) - first = start; - if (end > last) - end = last; + if (bip->bli_ranges == 1) + return; - xfs_buf_item_log_segment(first, end, - &bip->bli_formats[i].blf_data_map[0]); + for (i = 0; i < bip->bli_ranges - 1; i++) { + struct xfs_bli_range *rp_next; - start += bp->b_maps[i].bm_len; + rp = &bip->bli_range[i]; + rp_next = &bip->bli_range[i + 1]; + + +check_merge: + ASSERT(rp->last != 0); + ASSERT(rp->first <= rp->last); + + /* no overlap or adjacent, move on */ + if (rp->last < rp_next->first - 1) + continue; + + /* + * overlap: select lowest first, highest last, remove the merged + * range (rp_next) and then go back and check the next range for + * whether it can be merged (e.g. we have 4 separate ranges, + * then something logs the buffer entirely. This merges all + * ranges into one). + */ + rp->first = min(rp->first, rp_next->first); + rp->last = max(rp->last, rp_next->last); + if (i + 2 < bip->bli_ranges) + memmove(rp_next, rp_next + 1, sizeof(*rp) * + (bip->bli_ranges - i - 2)); + bip->bli_ranges--; + if (i < bip->bli_ranges - 1) + goto check_merge; } } diff --git a/fs/xfs/xfs_buf_item.h b/fs/xfs/xfs_buf_item.h index f7eba99..16e8d31 100644 --- a/fs/xfs/xfs_buf_item.h +++ b/fs/xfs/xfs_buf_item.h @@ -57,6 +57,25 @@ typedef struct xfs_buf_log_item { unsigned int bli_recur; /* lock recursion count */ atomic_t bli_refcount; /* cnt of tp refs */ int bli_format_count; /* count of headers */ + + /* + * logging ranges. Keep a small number of distinct ranges rather than a + * bitmap which is expensive to maintain. + * 4 separate ranges s probably optimal so that we + * can log separate header, tail and content changes (e.g. for dir + * structures) without capturing the entire buffer unnecessarily for + * isolated changes. + * + * Note: ranges are 32 bit values because we have to support an end + * range value of 0x10000.... + */ +#define XFS_BLI_RANGES 4 + struct xfs_bli_range { + uint32_t first; + uint32_t last; + } bli_range[XFS_BLI_RANGES]; + int bli_ranges; + struct xfs_buf_log_format *bli_formats; /* array of in-log header ptrs */ struct xfs_buf_log_format __bli_format; /* embedded in-log header */ } xfs_buf_log_item_t; -- 2.7.0 _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs