> On 2 Jul 2024, at 05:08, Pierre Rogier <progier@xxxxxxxxxx> wrote: > > Hi, > Thank you for your interesting feedback. > > Indeed we should improve the https://www.port389.org/docs/389ds/FAQ/Berkeley-DB-deprecation.html FAQ > Clearly we should add more detail about nsslapd-mdb-max-size as it is the main configuration parameters. > Yes it is the maximum database file size on disk (and also in memory as the file is memory mapped) > > So yes if you greatly oversize the size, the memory map creation needs more time to write the db file > and the system is likely to swap. lmdb should write a sparse file by default, it shouldn't need to fully allocate on the disk. Something seems off that it's trying to write the whole thing out ... > If the size limit is reached, any further writes operation will fail until the size has been increased and the instance restarted. > > The main change between 2.5 and 3.0.1 is the database type that is used by default when creating a new instance (now mdb instead of bdb) > So yes, instances created with bdb are still using bdb after migrating to 3.0.1 > And in 3.0.1 dscreate can still create a new instance with bdb > > But beware that the underlying libdb library that we are relying on, has been tagged obsolete in 2020 > and will disappear soon. Especially for container users, we should automate the bdb to mdb migration I think in these cases. > > On Mon, Jul 1, 2024 at 7:34 PM <tdarby@xxxxxxxxxxx> wrote: > The mdb change came as a worrying surprise when the container pulled down 3.0.1 without me realizing it. I've read as much as I can find (which is not much), but I don't understand all the changes. Questions: > > 1. Is there a way to continue to pull the 2.5 version of the container? > 2. If I update an existing 2.5 container instance to 3.0.1, will it continue to use bdb? > 3. The nsslapd-mdb-max-size attribute: > - Does this only set the size of the db on disk? What if it exceeds that limit? > - I was going to set it to "use all available space" until I saw these comments, but not sure what to make of them and I don't feel any closer to understanding what I should do with this attribute: > > - the file will tend to grow up to its maximum > - if real memory is smaller than the file then pages will be swappped The way that lmdb works is that it's a memory mapped file. It just means that the content of the file is exposed to the process as though it's a block of memory. Nothing more, nothing less. So the concept of it "swapping" isn't really necessarily the case. Swap implies the process where the kernel VFS cache is under pressure and so process memory pages are swapped to disk in order to allow other memory pages to be available for other processes. This is *NOT* what is occuring in lmdb. In lmdb reads and writes via the memory mapped file use pages in the kernel VFS. So when a page is read it's transfered from disk to the VFS. When you write it's written to the VFS, and on an fsync those are flushed to the disk storage. In addition the kernel begins page writeback in the background based on a number of tunables. (This is pretty much identical to any other read/write on a filesystem). This means that it's not "swapping" as much as "the database file's pages are cached in ram, and limited by the size of the VFS cache available to your system". Where this dynamic is important is that under memory pressure, the kernel will evict VFS pages *before* application pages. So if your container is running on a host with 8GB of ram, 389-ds is taking 2GB of ram (and you have 4GB of memory assigned to the container), and you have another process on the system that begins to consume 5GB of ram. The docker memory limits are not a reservation, they are a limit on *application pages*. So this means that due to the memory pressure from the external application (5GB + 2GB), the VFS pages begin to be pressured limiting the lmdb cache size that is available (1GB instead of the expected 2GB). Because of this, it's critical that you ensure when you run with lmdb you run on a host which is not under memory pressure to prevent page evictions (I forsee this being extremely problematic for FreeIPA but that's unrelated to this discussion). > - My first experience about configuring oversized database was interresting: > at first I did not capped the value to 1Gb and the basic import scenario was spending > hours before failing because disk was full (with a real db size around 10Gb on the test VM) > while once capped to 1 GB the import was successful in less than 10 minutes > ==> *Correct sizing matters*" -- Sincerely, William Brown Senior Software Engineer, Identity and Access Management SUSE Labs, Australia -- _______________________________________________ 389-users mailing list -- 389-users@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to 389-users-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
