Currently, using smaller nr_events in io_setup(), the max number of
allocation requests is inversely proportional to num_possible_cpus()
(which is a problem on larger/non-virtualized systems with many CPUs).
This happens because, internally, nr_events must be increased for the
percpu allocation scheme (according to the number of CPUs), but it is
being reflected on the max_reqs value (aio-nr), which is used to count
against the global limit (aio-max-nr).
This patchset decouples ctx->max_reqs and ctx->nr_events, so that the
extra nr_events which are internally required do not reflect into the
'aio-nr' counter which is externally visible & checked to 'aio-max-nr'.
This ensures that userspace can allocate most of the aio-max-nr value
that is made available/visible to it -- regardless of the number of
possible CPUs in the system.
P.S.: this patchset has been 'relatively acked' by Benjamin LaHaise [2],
but for timing difficulties, it hasn't been queued nor re-submitted
in a while; sorry.
v2:
- adjust/move comment in PATCH 2/2 to commit message (Jeff Moyer [3])
Test-case:
// 64k request (ie, aio-max-nr) to io_setup with nr_events = 1
for (i = 0; i < 65536; i++)
if (rc = io_setup(1, &ioctx[i]))
break;
printf("rc = %d, i = %d\n", rc, i);
- on v4.12-rc2, it fails w/ -EAGAIN on request #102 (160 CPUs),
and aio-nr value is ~2x the aio-max-nr value (wrong):
# ./test-case &
rc = -11, i = 102
# grep . /proc/sys/fs/aio-*
/proc/sys/fs/aio-max-nr:65536
/proc/sys/fs/aio-nr:130560
- on v4.12-rc2 with patches, it fails w/ -ENOMEM on request #65515,
thus going way further toward the actual max limit:
# ./test-case &
rc = -12, i = 65515
# grep . /proc/sys/fs/aio-*
/proc/sys/fs/aio-max-nr:65536
/proc/sys/fs/aio-nr:65515
Links:
[1] http://marc.info/?l=linux-aio&m=147562172703469&w=2
[2] http://marc.info/?l=linux-aio&m=147757894603034&w=2
[3] http://marc.info/?l=linux-aio&m=147769074702475&w=2
Mauricio Faria de Oliveira (2):
aio: make nr_events a parameter for aio_setup_ring()
aio: use ctx->max_reqs only for counting against the global limit
fs/aio.c | 19 ++++++++++++-------
1 file changed, 12 insertions(+), 7 deletions(-)
--
1.8.3.1
