Hi all,
We're reading from a Ceph Luminous pool using the librados asychronous I/O API. We're seeing some concerning memory usage patterns when we read many objects in sequence.
The expected behaviour is that our memory usage stabilises at a small amount, since we're just fetching objects and ignoring their data. What we instead find is that the memory usage of our program grows linearly with the amount of data read for an interval of time, and then continues to grow at a much slower but still consistent pace. This memory is not freed until program termination. My guess is that this is an issue with Ceph's memory allocator.
To demonstrate, we create 20000 objects of size 10KB, and of size 100KB, and of size 1MB:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <rados/librados.h>
int main() {
rados_t cluster;
rados_create(&cluster, "test");
rados_conf_read_file(cluster, "/etc/ceph/ceph.conf");
rados_connect(cluster);
rados_ioctx_t io;
rados_ioctx_create(cluster, "test", &io);
char data[1000000];
memset(data, 'a', 1000000);
char smallobj_name[16], mediumobj_name[16], largeobj_name[16];
int i;
for (i = 0; i < 20000; i++) {
sprintf(smallobj_name, "10kobj_%d", i);
rados_write(io, smallobj_name, data, 10000, 0);
sprintf(mediumobj_name, "100kobj_%d", i);
rados_write(io, mediumobj_name, data, 100000, 0);
sprintf(largeobj_name, "1mobj_%d", i);
rados_write(io, largeobj_name, data, 1000000, 0);
printf("wrote %s of size 10000, %s of size 100000, %s of size 1000000\n",
smallobj_name, mediumobj_name, largeobj_name);
}
return 0;
}
$ gcc create.c -lrados -o create
$ ./create
wrote 10kobj_0 of size 10000, 100kobj_0 of size 100000, 1mobj_0 of size 1000000
wrote 10kobj_1 of size 10000, 100kobj_1 of size 100000, 1mobj_1 of size 1000000
[...]
wrote 10kobj_19998 of size 10000, 100kobj_19998 of size 100000, 1mobj_19998 of size 1000000
wrote 10kobj_19999 of size 10000, 100kobj_19999 of size 100000, 1mobj_19999 of size 1000000
Now we read each of these objects with the async API, into the same buffer. First we read just the the 10KB objects first:
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <rados/librados.h>
void readobj(rados_ioctx_t* io, char objname[]);
int main() {
rados_t cluster;
rados_create(&cluster, "test");
rados_conf_read_file(cluster, "/etc/ceph/ceph.conf");
rados_connect(cluster);
rados_ioctx_t io;
rados_ioctx_create(cluster, "test", &io);
char smallobj_name[16];
int i, total_bytes_read = 0;
for (i = 0; i < 20000; i++) {
sprintf(smallobj_name, "10kobj_%d", i);
readobj(&io, smallobj_name);
total_bytes_read += 10000;
printf("Read %s for total %d\n", smallobj_name, total_bytes_read);
}
getchar();
return 0;
}
void readobj(rados_ioctx_t* io, char objname[]) {
char data[1000000];
unsigned long bytes_read;
rados_completion_t completion;
int retval;
rados_read_op_t read_op = rados_create_read_op();
rados_read_op_read(read_op, 0, 10000, data, &bytes_read, &retval);
retval = rados_aio_create_completion(NULL, NULL, NULL, &completion);
assert(retval == 0);
retval = rados_aio_read_op_operate(read_op, *io, completion, objname, 0);
assert(retval == 0);
rados_aio_wait_for_complete(completion);
rados_aio_get_return_value(completion);
}
$ gcc read.c -lrados -o read_small -Wall -g && ./read_small
Read 10kobj_0 for total 10000
Read 10kobj_1 for total 20000
[...]
Read 10kobj_19998 for total 199990000
Read 10kobj_19999 for total 200000000
We read 200MB. A graph of the resident set size of the program is attached as mem-graph-10k.png, with seconds on x axis and KB on the y axis. You can see that the memory usage increases throughout, which itself is unexpected since that memory should be freed over time and we should only hold 10KB of object data in memory at a time. The rate of growth decreases and eventually stabilises, and by the end we've used 60MB of RAM.
We repeat this experiment for the 100KB and 1MB objects and find that after all reads they use 140MB and 500MB of RAM, and memory usage presumably would continue to grow if there were more objects. This is orders of magnitude more memory than what I would expect these programs to use.
- We do not get this behaviour with the synchronous API, and the memory usage remains stable at just a few MB.
- We've found that for some reason, this doesn't happen (or doesn't happen as severely) if we intersperse large reads with much smaller reads. In this case, the memory usage seems to stabilise at a reasonable number.
- Valgrind only reports a trivial amount of unreachable memory.
- Memory usage doesn't increase in this manner if we repeatedly read the same object over and over again. It hovers around 20MB.
- In other experiments we've done, with different object data and distributions of object sizes, we've seen memory usage grow even larger in proportion to the amount of data read.
We maintain a long-running (order of weeks) services that read objects from Ceph and send them elsewhere. Over time, the memory usage of some of these services have grown to more than 6GB, which is unreasonable.
Regards,
Dan G
Attachment:
mem-graph-10k.png
Description: PNG image
Attachment:
mem-graph-1m.png
Description: PNG image
Attachment:
mem-graph-100k.png
Description: PNG image
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
