lmdb fails to map large DBs on i686/s390x

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

I've been having some issues with LMDB on i686/s390x when building
python-zarr and its dependants. I filed a bug report [1], but the
maintainer suggested asking here to get more reach. I have replicated
my original message from the report below:

The documentation for LMDB [2] states that the map size should be
"chosen as large as possible". Consequently, the zarr developers have
chosen a map size of 2**40 on 64-bit systems and 2**28 on 32-bit
systems [3]. When the package is built on a 64-bit system [4],
everything works fine. However, if the build is run on a 32-bit system
[5], it fails with a MemoryError.

I have narrowed this down to the mmap call at [6] which attempts to
map the backing file into memory. AFAICT, the mapped size is far below
the RAM on the build machine as well as far below the normal 32-bit VM
limit. So I don't know why the call is failing.

I can reproduce this in mock with the fedora-rawhide-i386 config, but
what's weird is it also fails in the fedora-rawhide-s390x config. This
machine has 16G of RAM, so it's enough for 2**28, but not 2**40. So it
fails on 32-bit x86 with more than enough RAM, works on 64-bit x86
with nowhere near enough RAM, yet fails with 64-bit s390x.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1695525
[2] http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5
[3] https://github.com/zarr-developers/zarr/blob/f6ced1e31b919065f8834d813ec081d2a85195b3/zarr/storage.py#L1585-L1587
[4] https://koji.fedoraproject.org/koji/taskinfo?taskID=33910305
[5] https://koji.fedoraproject.org/koji/taskinfo?taskID=33881973
[6] http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=libraries/liblmdb/mdb.c;h=e12af4482a0172da8d759c1da1530339d9095510;hb=2a5eaad6919ce6941dec4f0d5cce370707a00ba7#l4019

-- 
Elliott
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux