I've been busy lately. Looks like Sam wrote the BackoffThrottle; not
sure if he did the tests as well or not. Maybe check the git logs on
them. :)
-Greg
On Wed, May 11, 2016 at 2:46 AM, Willem Jan Withagen <wjw@xxxxxxxxxxx> wrote:
> On 9-5-2016 10:44, Willem Jan Withagen wrote:
>>
>> On 6-5-2016 23:04, Willem Jan Withagen wrote:
>>>
>>> On 6-5-2016 20:41, Gregory Farnum wrote:
>
>
> Hi Greg,
>
> If you are not the one to ask about the Throttle stuff, would you know
> anybody else I could ask this question.
> Because the more runs I make, the more it seems like it is a livelock.
>
> I'm now going to do the annotated runs on Linux as well, and try and
> find out why it doesn't happen there.
>
> --WjW
>
>> Oke,
>>
>> I'm going to need some help understanding some things...
>>
>> I've taken the oversaturated test_backoff from test/common/Throttle.cc,
>> since that one wasn't working as well. And I suspect that it might be
>> for about the same/similar reason.
>>
>> I start 1 putter and one getter, and with certain parameters the test
>> does not complete. When
>> expected_throughput * put_delay_per_count ~> 10 (max_multiple??)
>> then the process get into a loop, I'm printing over and over:
>> _get_delay return r > high_threshhold (queue is full): time = 0.0172
>>
>> Note that the parameters to test_backoff are:
>> ====
>> // double expected_throughput,
>> 500,
>> // double max_multiple,
>> 10,
>> // double put_delay_per_count,
>> 0.025,
>> ====
>>
>> And that loop is in BackoffThrottle::get, around line 390 (I have a lot
>> of extra lines for annotation and tracing.)
>> ====
>> while (((start + delay) > std::chrono::system_clock::now()) ||
>> !((max == 0) || (current == 0) || ((current + c) <= max))) {
>> assert(ticket == waiters.begin());
>> (*ticket)->wait_until(l, start + delay);
>> delay = _get_delay(c);
>> }
>> ====
>>
>> The reason that this loop never completes is that it executed under the
>> lock in the getter thread in test/common/Throttle.cc:test_backoff where
>> it is executed under g.lock(). And the putter is held on the same lock,
>> so nothing gets in or out.... Not a deadlock, but a live-lock.
>>
>> Thusfar the analysis.
>>
>> Now the question(s)....
>> 1) How do I break this livelock.
>> 1a) Should the extra while() { wait } loop actually be in
>> BackoffThrottle:get without releasing the lock?
>>
>> 2) The thread are called getter and putter.
>> So I'd expect the getter to get messages from the list and the putter so
>> write things to the list.
>>
>> Why does the getter:
>> while(!stop) {
>> throttle.get(to_get);
>> in_queue.push_back(to_get)
>> assert(total <= max);
>> }
>> And the putter:
>> while(!stop) {
>> while(empty)
>> wait;
>> c = in_queue.front();
>> in_queue.pop();
>> "wait time to queue some more"
>> throttle.put(c)
>> }
>>
>> Am I getting it wrong around, and would I not expect the
>> throttle.{get,put} to match the similar operations on the in_queue?
>>
>> And what is this c value that we retrieve from the in_queue that is then
>> used to be put in the throttle???
>>
>> --WjW
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
