Roger, Any idea what the hash size does to memory usage? I wonder if we can scale this based on the directory count, or if the memory usage is minimal (only needed in case of tdb) then just make it the default. It definitely appears to have been a major performance boost. Another possible optuzatiom is to use the in-memory icount list (preferably with the patch to reduce realloc size) until the allocations fail and only then dump the list into tdb? That would allow people to run with a swapfile configured by default, but only pay the cost of on-disk operations if really needed. Cheers, Andreas On 2011-02-22, at 6:54, Rogier Wolff <R.E.Wolff@xxxxxxxxxxxx> wrote: > On Tue, Feb 22, 2011 at 02:36:52PM +0100, Rogier Wolff wrote: >> On Tue, Feb 22, 2011 at 11:20:56AM +0100, Rogier Wolff wrote: >>> I wouldn't be surprised if I'd need more than 3G of RAM. When I >>> extrapolated "more than a few days" it was at under 20% of the >>> filesystem and had already allocated on the order of 800Gb of >>> memory. Now I'm not entirely sure that this is fair: memory use seems >>> to go up quickly in the beginning, and then stabilize: as if it has >>> decided that 800M of memory use is "acceptable" and somehow uses a >>> different strategy once it hits that limit. >> >> OK. Good news. It's finished pass1. It is currently using about 2100Mb >> of RAM (ehh. mostly swap, I have only 1G in there). Here is the patch. > > Forgot the patch. > > Roger. > > -- > ** R.E.Wolff@xxxxxxxxxxxx ** http://www.BitWizard.nl/ ** +31-15-2600998 ** > ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** > *-- BitWizard writes Linux device drivers for any device you may have! --* > Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. > Does it sit on the couch all day? Is it unemployed? Please be specific! > Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ > <cputimefix.patch> -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
