On 2018/4/17 6:45 AM, Eric Wheeler wrote:
> On Sat, 14 Apr 2018, Coly Li wrote:
>
>> On 2018/4/14 6:46 AM, Eric Wheeler wrote:
>>> Hello all,
>>>
>>> We are running bcache in 4.1.49 with both the cache and backing device
>>> having 4k blocks. The disk stack is DRBD->dm-thin->bcache->[sdc->sdb]
>>> Where sdc is the cache.
>>>
>>> Sometimes we get errors like the following:
>>>
>>> [432015.934869] block drbd8065: Began resync as SyncTarget (will sync 880 KB [220 bits set]).
>>> [432015.949469] sd 0:0:0:1: [sdb] Unaligned block number requested: sector_size=4096, block=15724561783, blk_rq=9
>>> [432015.950347] sd 0:0:0:2: [sdc] Unaligned block number requested: sector_size=4096, block=353041040, blk_rq=7
>>> [432015.951146] bcache: bch_count_io_errors() dm-6: IO error on reading from cache, recovering
>>> [432015.952015] block drbd8065: read: error=-5 s=19281488s
>>> [432015.952866] block drbd8065: Local IO failed in drbd_endio_read_sec_final.
>>> [432015.953777] sd 0:0:0:2: [sdc] Unaligned block number requested: sector_size=4096, block=387084784, blk_rq=7
>>> [432015.954710] bcache: bch_count_io_errors() dm-6: IO error on reading from cache, recovering
>>> [432015.959037] sd 0:0:0:1: [sdb] Unaligned block number requested: sector_size=4096, block=15725385535, blk_rq=1
>>> [432015.959938] block drbd8065: read: error=-5 s=19391384s
>>> [432015.960862] block drbd8065: Local IO failed in drbd_endio_read_sec_final.
>>>
>>>
>>> Note that 15724561783 is not divisible by 8, thus it is unaligned to 4k
>>> blocks.
>>>
>>> Does anyone know if the bcache code is enforcing correct alignment?
>>>
>>> Is there any way that bcache could introduce misalignment?
>>>
>>> We ran blockdev --getbsz and --getpbsz all the way down the stack and
>>> everything reports 4k.
>>
>> Hi Eric,
>>
>> Do you use 4.1 stable tree, or with your extra patches ? It would be
>> helpful if I may access your kernel tree.
>>
>> So far I cannot tell where the problem is, I just feel there might be
>> some hidden issue triggered by 4KB sector size hard drive. Maybe adding
>> a garden code to detect unaligned I/O request from bcache will be
>> helpful to diagnose the root cause.
>
> Hi Coly,
>
> I just pushed the branch that we built for this system to bitbucket:
> https://bitbucket.org/ewheelerinc/linux/branch/ewi-4.1.49-rpmbuild
>
> There are a few changes, but probably nothing that you haven't seen:
>
> 1. We use the BFQ scheduler (but the problem presents in CFQ also)
> 2. dmthin fixes were backported from 4.2/4.3 to fix pool space issues on
> rolling snapshots
> 3. bcache and dmcrypt have my ioprio patches, but no dmcrypt in this bug
> 4. ibrs patches from Oracle are included for Spectre mitigation
> 5. We attempted to use Mauricio's patch to fix 4k alignment issues, which
> shows in our tree as 4a595ccc, but it did not fix the issue.
> 6. We updated the error strings in sd.c with our commits aa372d91 and
> f603bf7e while troubleshooting this issue.
>
> Thank you for your help, let me know if there is anything else that you
> need to help troubleshoot. I can produce this pretty reliably at the
> moment.
Eric,
Is it possible to apply the attached patch ? This patch will dump call
trace if the request to backing device is not 4KB aligned. Then we can
see what happens indeed.
Thanks.
Coly Li
From 3d4daa9c24402978cd43479cd8aa429f37fe2416 Mon Sep 17 00:00:00 2001
From: Coly Li <colyli@xxxxxxx>
Date: Tue, 17 Apr 2018 15:58:11 +0800
Subject: [PATCH] bcache: check 4KB alignment to backing device I/O requests
This patch checks whetehr the backing device address of a key is 4KB
address aligned, also it checks whetehr the incoming I/O request is 4KB address
aligned. If not, call trace will be printed for further debugging.
Signed-by: Coly Li <colyli@xxxxxxx>
---
drivers/md/bcache/alloc.c | 1 +
drivers/md/bcache/bcache.h | 8 ++++++++
drivers/md/bcache/bset.c | 5 ++++-
drivers/md/bcache/btree.c | 1 +
drivers/md/bcache/io.c | 1 +
drivers/md/bcache/journal.c | 1 +
drivers/md/bcache/request.c | 1 +
7 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/drivers/md/bcache/alloc.c b/drivers/md/bcache/alloc.c
index 8eeab72b93e2..12a2f32ba82a 100644
--- a/drivers/md/bcache/alloc.c
+++ b/drivers/md/bcache/alloc.c
@@ -632,6 +632,7 @@ bool bch_alloc_sectors(struct cache_set *c, struct bkey *k, unsigned sectors,
for (i = 0; i < KEY_PTRS(&b->key); i++) {
SET_PTR_OFFSET(&b->key, i, PTR_OFFSET(&b->key, i) + sectors);
+ check_4k_alignment(PTR_OFFSET(&b->key, i));
atomic_long_add(sectors,
&PTR_CACHE(c, &b->key, i)->sectors_written);
diff --git a/drivers/md/bcache/bcache.h b/drivers/md/bcache/bcache.h
index 6b420a55c745..8834248452a0 100644
--- a/drivers/md/bcache/bcache.h
+++ b/drivers/md/bcache/bcache.h
@@ -847,6 +847,14 @@ static inline void wake_up_allocators(struct cache_set *c)
wake_up_process(ca->alloc_thread);
}
+static inline void check_4k_alignment(size_t sector)
+{
+ if (unlikely(sector & 7)) {
+ pr_err("PTR_OFFSET is not 4KB aligned: %lu", sector);
+ dump_stack();
+ }
+}
+
/* Forward declarations */
void bch_count_io_errors(struct cache *, int, const char *);
diff --git a/drivers/md/bcache/bset.c b/drivers/md/bcache/bset.c
index 646fe85261c1..b0f7e2c57d48 100644
--- a/drivers/md/bcache/bset.c
+++ b/drivers/md/bcache/bset.c
@@ -9,6 +9,7 @@
#include "util.h"
#include "bset.h"
+#include "bcache.h"
#include <linux/console.h>
#include <linux/random.h>
@@ -202,8 +203,10 @@ bool __bch_cut_front(const struct bkey *where, struct bkey *k)
else
bkey_copy_key(k, where);
- for (i = 0; i < KEY_PTRS(k); i++)
+ for (i = 0; i < KEY_PTRS(k); i++) {
SET_PTR_OFFSET(k, i, PTR_OFFSET(k, i) + KEY_SIZE(k) - len);
+ check_4k_alignment(PTR_OFFSET(k, i));
+ }
BUG_ON(len > KEY_SIZE(k));
SET_KEY_SIZE(k, len);
diff --git a/drivers/md/bcache/btree.c b/drivers/md/bcache/btree.c
index 22b9e34ceb75..66b97153d773 100644
--- a/drivers/md/bcache/btree.c
+++ b/drivers/md/bcache/btree.c
@@ -419,6 +419,7 @@ static void do_btree_node_write(struct btree *b)
bkey_copy(&k.key, &b->key);
SET_PTR_OFFSET(&k.key, 0, PTR_OFFSET(&k.key, 0) +
bset_sector_offset(&b->keys, i));
+ check_4k_alignment(PTR_OFFSET(&k.key, 0));
if (!bio_alloc_pages(b->bio, __GFP_NOWARN|GFP_NOWAIT)) {
int j;
diff --git a/drivers/md/bcache/io.c b/drivers/md/bcache/io.c
index 86a0bb87124e..0bc087c76527 100644
--- a/drivers/md/bcache/io.c
+++ b/drivers/md/bcache/io.c
@@ -38,6 +38,7 @@ void __bch_submit_bbio(struct bio *bio, struct cache_set *c)
bio->bi_iter.bi_sector = PTR_OFFSET(&b->key, 0);
bio->bi_bdev = PTR_CACHE(c, &b->key, 0)->bdev;
+ check_4k_alignment(bio->bi_iter.bi_sector);
b->submit_time_us = local_clock_us();
closure_bio_submit(bio, bio->bi_private);
diff --git a/drivers/md/bcache/journal.c b/drivers/md/bcache/journal.c
index 29eba7219b01..863017ab3a65 100644
--- a/drivers/md/bcache/journal.c
+++ b/drivers/md/bcache/journal.c
@@ -637,6 +637,7 @@ static void journal_write_unlocked(struct closure *cl)
bio_list_add(&list, bio);
SET_PTR_OFFSET(k, i, PTR_OFFSET(k, i) + sectors);
+ check_4k_alignment(PTR_OFFSET(k, i));
ca->journal.seq[ca->journal.cur_idx] = w->data->seq;
}
diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
index 25fa8445bb24..1ed24bcc7b89 100644
--- a/drivers/md/bcache/request.c
+++ b/drivers/md/bcache/request.c
@@ -970,6 +970,7 @@ static blk_qc_t cached_dev_make_request(struct request_queue *q,
bio->bi_bdev = dc->bdev;
bio->bi_iter.bi_sector += dc->sb.data_offset;
+ check_4k_alignment(bio->bi_iter.bi_sector);
if (cached_dev_get(dc)) {
s = search_alloc(bio, d);
--
2.16.2
