Hi Phil, On Tue, Nov 12, 2024 at 09:52:35PM +0100, Phil Sutter wrote: > Hi Pablo, > > On Thu, Oct 31, 2024 at 11:04:11PM +0100, Pablo Neira Ayuso wrote: > > Side note: While profiling, I can still see lots json objects, this > > results in memory consumption that is 5 times than native > > representation. Error reporting is also lagging behind, it should be > > possible to add a json_t pointer to struct location to relate > > expressions and json objects. > > I can't quite reproduce this. When restoring a ruleset with ~12.7k > elements in individual standard syntax commands, valgrind prints: > > | HEAP SUMMARY: > | in use at exit: 59,802 bytes in 582 blocks > | total heap usage: 954,970 allocs, > | 954,388 frees, > | 18,300,874 bytes allocated > > Repeating the same in JSON syntax, I get: > > | HEAP SUMMARY: > | in use at exit: 61,592 bytes in 647 blocks > | total heap usage: 1,200,164 allocs, > | 1,199,517 frees, > | 38,612,257 bytes allocated > > So this is 38MB vs 18MB? At least far from the mentioned 5 times. Would > you mind sharing how you got to that number? > > Please kindly find my reproducers attached for reference. I am using valgrind --tool=massif to measure memory consumption in userspace. I used these two files: - set-init.json-nft, to create the table and set. - set-65535.nft-json, to create a small set with 64K elements. then I run: valgrind --tool=massif nft -f set-65535.nft-json there is a tool: ms_print massif.out.XYZ At "peak time" in heap memory consumption, I can see 60% is consumed in json objects. I am looking at the commands and expressions to reduce memory consumption there. The result of that work will also help json support.
