Did you check if your OS allows you to limit memory usage per user / process basis?
I'll definitely dig more into this. I found some stuff saying it isn't as easy as one would expect, i.e. not just a ulimit and done. Or maybe it is, in this case. I will play around.
Did you either profiled or debugged it to see what is going on?
I would love to learn more about how to do this, to get to the bottom of the memory usage. I can google around, or can you suggest any reads?
When memory consumption grows , Doesn't ever swap out?
Yes, it consumes all swap.