On Thu, Oct 26, 2023 at 07:41:45PM +0200, mwilck@xxxxxxxx wrote:
> From: Martin Wilck <mwilck@xxxxxxxx>
>
> Currently the number of iocbs per path to test is the same as the
> total number of iocbs in the ioctx. This can easily cause iocb
> starvation, in particular if some IOs are hanging. In that case
> io_submit() will fail, and some paths under test will use much
> less IOs as intended, or in the worst case, none at all.
>
> The total number of iocbs reserved in the kernel should be higher.
> With this patch, we will be able to run the marginal path test for
> at least NR_IOSTAT_PATHS=32 paths at the same time. This is not an upper
> limit, because kernel IOCBs can be reused between paths.
>
> Increase the log levels of io_setup and io_submit to make it sure
> we catch problems with this approach.
>
> Signed-off-by: Martin Wilck <mwilck@xxxxxxxx>
Reviewed-by: Benjamin Marzinski <bmarzins@xxxxxxxxxx>
> Cc: Li Xiao Keng <lixiaokeng@xxxxxxxxxx>
> Cc: Miao Guanqin <miaoguanqin@xxxxxxxxxx>
> Cc: Guan Junxiong <guanjunxiong@xxxxxxxxxx>
> ---
> libmultipath/io_err_stat.c | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/libmultipath/io_err_stat.c b/libmultipath/io_err_stat.c
> index 3f32e32..5749003 100644
> --- a/libmultipath/io_err_stat.c
> +++ b/libmultipath/io_err_stat.c
> @@ -38,6 +38,7 @@
> #define TIMEOUT_NO_IO_NSEC 10000000 /*10ms = 10000000ns*/
> #define FLAKY_PATHFAIL_THRESHOLD 2
> #define CONCUR_NR_EVENT 32
> +#define NR_IOSTAT_PATHS 32
>
> #define PATH_IO_ERR_IN_CHECKING -1
> #define PATH_IO_ERR_WAITING_TO_CHECK -2
> @@ -477,7 +478,7 @@ static int send_each_async_io(struct dio_ctx *ct, int fd, char *dev)
> get_monotonic_time(&ct->io_starttime);
> io_prep_pread(&ct->io, fd, ct->buf, ct->blksize, 0);
> if (io_submit(ioctx, 1, ios) != 1) {
> - io_err_stat_log(5, "%s: io_submit error %i",
> + io_err_stat_log(2, "%s: io_submit error %i",
> dev, errno);
> return rc;
> }
> @@ -703,8 +704,8 @@ int start_io_err_stat_thread(void *data)
> if (uatomic_read(&io_err_thread_running) == 1)
> return 0;
>
> - if (io_setup(CONCUR_NR_EVENT, &ioctx) != 0) {
> - io_err_stat_log(4, "io_setup failed");
> + if (io_setup(CONCUR_NR_EVENT * NR_IOSTAT_PATHS, &ioctx) != 0) {
> + io_err_stat_log(1, "io_setup failed - increase /proc/sys/fs/aio-nr ?");
> return 1;
> }
>
> --
> 2.42.0
