Great. Makes sense. Thanks for the help here. Regards, Jeff -----Original Message----- From: Jens Axboe [mailto:axboe@xxxxxxxxx] Sent: Wednesday, March 21, 2018 2:51 PM To: Jeff Furlong <jeff.furlong@xxxxxxx>; fio@xxxxxxxxxxxxxxx Subject: Re: fio server/client disconnect bug Done: http://git.kernel.dk/cgit/fio/commit/?id=b647a293d70949b8c401772936a8000d10eb1bb9 On 3/21/18 3:47 PM, Jens Axboe wrote: > OK good, I will commit the fix. > > The summary is not to do those commands inline. Usually we don't have > issues with inline vs out-of-line ordering, since we don't queue > deeply or depend on ordering. But for this specific case, it's pretty > unfortunate if commands we already queued up are not processed, before > the client sends a stop/quit and finishes. > > > On 3/21/18 3:45 PM, Jeff Furlong wrote: >> Ran over 600 iterations so far without issue. Will keep running to get a higher confidence. That way this issue will be fresh in our minds, given the fun sequence we just traced through. >> >> So the summary is to not add the QUIT command to the inline queue, and just process it like most other typical commands? The only thing that gets processed separately are errors or triggers? >> >> Regards, >> Jeff >> >> >> -----Original Message----- >> From: Jens Axboe [mailto:axboe@xxxxxxxxx] >> Sent: Wednesday, March 21, 2018 12:50 PM >> To: Jeff Furlong <jeff.furlong@xxxxxxx>; fio@xxxxxxxxxxxxxxx >> Subject: Re: fio server/client disconnect bug >> >> On 3/21/18 1:45 PM, Jeff Furlong wrote: >>> There is noticeable slowdown with this patch. Seems some iterations (even successful) are taking several seconds of usleep(). Eventually hit a new issue: >>> >>> fio: client host1, timeout on cmd SEND_ETA >>> fio: client host1 timed out >>> >>> But host1 still logged the iops/bw/lat logs, but did not send summary to output. >> >> Can you try this one? >> >> >> diff --git a/server.c b/server.c >> index 65d4484edae5..6064d8172a3b 100644 >> --- a/server.c >> +++ b/server.c >> @@ -616,7 +616,7 @@ static int fio_net_queue_quit(void) { >> dprint(FD_NET, "server: sending quit\n"); >> >> - return fio_net_queue_cmd(FIO_NET_CMD_QUIT, NULL, 0, NULL, SK_F_SIMPLE | SK_F_INLINE); >> + return fio_net_queue_cmd(FIO_NET_CMD_QUIT, NULL, 0, NULL, >> +SK_F_SIMPLE); >> } >> >> int fio_net_send_quit(int sk) >> @@ -636,7 +636,7 @@ static int fio_net_send_ack(struct fio_net_cmd >> *cmd, int error, int signal) >> >> epdu.error = __cpu_to_le32(error); >> epdu.signal = __cpu_to_le32(signal); >> - return fio_net_queue_cmd(FIO_NET_CMD_STOP, &epdu, sizeof(epdu), &tag, SK_F_COPY | SK_F_INLINE); >> + return fio_net_queue_cmd(FIO_NET_CMD_STOP, &epdu, sizeof(epdu), >> +&tag, SK_F_COPY); >> } >> >> static int fio_net_queue_stop(int error, int signal) >> >> -- >> Jens Axboe >> > > -- Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
