On 04/23/2013 08:40 PM, Kent Overstreet wrote:
> Well _that_ interesting, the implication is that garbage collection is
> getting stuck somehow and unable to finish.
>
> I'll probably have another debug patch for you later today.
Hi Kent,
Any possible patch, please don't hesitate to send to me , I can help to
test and hope we can fix that finally.
Regards,
Jack
>
> On Tue, Apr 23, 2013 at 4:56 AM, Jack Wang <jinpu.wang@xxxxxxxxxxxxxxxx> wrote:
>> Hi Kent,
>>
>> After some test, we saw log in dmesg like below:
>>
>>
>>
>> [ 1505.282400] bcache: bch_allocator_thread() bcache: invalidating
>> buckets: free_inc 0/127
>> [ 1505.285843] bcache: bch_allocator_thread() bcache: not writing prios:
>> free_inc 0/127
>> [ 1508.005957] bcache: bch_allocator_thread() bcache: invalidating
>> buckets: free_inc 0/127
>> [ 1508.009357] bcache: bch_allocator_thread() bcache: not writing prios:
>> free_inc 0/127
>> [ 1512.493609] bcache: bch_allocator_thread() bcache: invalidating
>> buckets: free_inc 0/127
>> [ 1512.497070] bcache: bch_allocator_thread() bcache: not writing prios:
>> free_inc 0/127
>>
>> Does this show some clue to you?
>>
>> Regards,
>> Jack
>>
>> On 04/22/2013 11:51 PM, Kent Overstreet wrote:
>>> On Mon, Apr 22, 2013 at 10:27:03PM +0200, Jack Wang wrote:
>>>> Thanks for reply, Kent, two of my colleagues saw this behaviour, so I
>>>> think we can reproduce this.
>>>> If you could give me more detailed guide to narrow it down, I can try it
>>>> on my side.
>>>
>>> So, my current hypothesis is that the problem is the allocator spinning,
>>> and the IO is from it continually rewriting prios/gens.
>>>
>>> But I'm still not sure what's causing the allocator to spin, that's what
>>> the last patch was supposed to fix.
>>>
>>> Can you see if you can reproduce it with this patch, and then tell me
>>> what shows up in the dmesg log? I expect you'll get a _lot_ of output -
>>> flip timestamps on in your kernel config, if they're not already on.
>>> Thanks!
>>>
>>> commit 60a09d37d301f88dd0f0f413408821a067966d1a
>>> Author: Kent Overstreet <koverstreet@xxxxxxxxxx>
>>> Date: Mon Apr 22 14:49:33 2013 -0700
>>>
>>> bcache: Allocator debug patch
>>>
>>> diff --git a/drivers/md/bcache/alloc.c b/drivers/md/bcache/alloc.c
>>> index 2879487..37c22c6 100644
>>> --- a/drivers/md/bcache/alloc.c
>>> +++ b/drivers/md/bcache/alloc.c
>>> @@ -393,12 +393,15 @@ void bch_allocator_thread(struct closure *cl)
>>> allocator_wait(ca, !list_empty(&ca->discards));
>>> do_discard(ca, bucket);
>>> } else {
>>> - fifo_push(&ca->free, bucket);
>>> + BUG_ON(!fifo_push(&ca->free, bucket));
>>> closure_wake_up(&ca->set->bucket_wait);
>>> }
>>> }
>>>
>>> allocator_wait(ca, ca->set->gc_mark_valid);
>>> +
>>> + printk(KERN_DEBUG "bcache: invalidating buckets: free_inc %zu/%zu\n",
>>> + fifo_used(&ca->free_inc), ca->free_inc.size);
>>> invalidate_buckets(ca);
>>>
>>> allocator_wait(ca, !atomic_read(&ca->set->prio_blocked) ||
>>> @@ -407,8 +410,12 @@ void bch_allocator_thread(struct closure *cl)
>>> if (CACHE_SYNC(&ca->set->sb) &&
>>> (!fifo_empty(&ca->free_inc) ||
>>> ca->need_save_prio > 64)) {
>>> + printk(KERN_DEBUG "bcache: writing prios: free_inc %zu/%zu\n",
>>> + fifo_used(&ca->free_inc), ca->free_inc.size);
>>> bch_prio_write(ca);
>>> - }
>>> + } else
>>> + printk(KERN_DEBUG "bcache: not writing prios: free_inc %zu/%zu\n",
>>> + fifo_used(&ca->free_inc), ca->free_inc.size);
>>> }
>>> }
>>>
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
