I sent the previous mail too early!
I got it to work for root user by setting "/proc/sys/vm/nr_hugepages" to 20 as suggested in Documentation/vm/hugetlbpage.txt. Now I'm able to mmap it and it reflects the same in /proc/meminfo as well! :)
HTH,
-mandeep
On Mon, Aug 18, 2014 at 1:23 PM, Mandeep Sandhu <mandeepsandhu.chd@xxxxxxxxx> wrote:
I tried your program and got the same result, i.e "Resource temporarily unavailable" for an ordinary user and "Cannot allocate memory" if run as root.However, reading the Huge TLB pages doc (Documentation/vm/hugetlbpage.txt), I saw the following para which might explain the error:...Once a number of huge pages have been pre-allocated to the kernel huge page
pool, a user with appropriate privilege can use either the mmap system callor shared memory system calls to use the huge pages. See the discussion of
Using Huge Pages, below.
The administrator can allocate persistent huge pages on the kernel bootcommand line by specifying the "hugepages=N" parameter, where 'N' = the
number of huge pages requested. This is the most reliable method ofallocating huge pages as memory has not yet become fragmented.
......I don't have my kernel booted with it, so that might explain why mmap fails for MAP_HUGETLB.I can't restart my machine with the new flag right now, but maybe you can try.HTH,-mandeep
Now I'm befuddled. I modified the source so that it was easy to try different combinations. If I sudo it, I can map a 1GiB (1<<30) private anonymous region if I sudo the program. I cannot find any combination with MAP_HUGETLB that works at all.Here it is in working form, with an anonymous map and no backing file. The mmap request is for a single byte, so it works for ordinary users Try uncommenting the MAP_HUGETLB line, and I'd love to hear about a version that works, even if it's only for root.
Since I'm aiming for a huge memory region, I figured HUGETLB would make sense to limit the overhead.============================ CUT HERE ================
/**
* @file
* <pre>"Find out the limits on locked memory"
* Last Modified: Mon Aug 18 11:21:23 PDT 2014
* @author Kevin O'Gorman
*/
#define _GNU_SOURCE /* enable some of the mmap flags */
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
/* for getrlimit(2) */
#include <sys/time.h>
#include <sys/resource.h>
/* for open(2) */
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/fcntl.h>
int
main(int argc, char *argv[])
{
void *where;
size_t length = 1;
struct rlimit rlim;
int flags = 0;
int fd = -1;
// fd = open("solvedtaq", O_RDWR); if (fd == -1) perror("open");
printf("fd is %d\n", fd);
// flags |= MAP_HUGETLB;
flags |= MAP_PRIVATE;
flags |= MAP_LOCKED;
flags |= MAP_ANONYMOUS;
if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) {
printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long long)rlim.rlim_max, (long long)rlim.rlim_cur);
} else {
perror("getrlimit failed");
}
where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0);
if (where != MAP_FAILED) {
printf("Mapped at %p\n", where);
} else {
perror("Mapping failed");
}
return EXIT_SUCCESS;
}
============================ CUT HERE ================On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <kogorman@xxxxxxxxx> wrote:
On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks@xxxxxx> wrote:
On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:What problem are you trying to solve by doing this? It usually doesn't make
> maybe someone here can help me figure out how to map a big (really big)
> work area and lock it in memory.
sense to lock more than a small chunk in memory - that's mostly for crypto
programs like PGP that are paranoid about keys being written to swap spaces,
and a *very* few highly performance oriented programs. 99.8% of the time,
the kernel will do a reasonable job of making sure the right pages are in
memory.
Plus, programmers almost always over-estimate what they *really* need to lock
down. The end result is that performance ends up being *worse*, because they
lock out more than needed, leaving all the rest of the processes on the system
fighting over a too-small pool of pages. Really sucks when you actually have
to read from disk every I/O because you no longer have an in-core file cache :)
I'm solving a huge alpha-beta search problem, and want a huge cache table tocheck permutations. That's why I increased RAM from 8 to 32 GB, and would
be happy to use the extra 24GB all for the cache. (Cache table is like a hash table but with collisions causing replacement rather than chaining).
Even in my small tests, the cache is giving me speedups in the neighborhood ofa million, so I'm going big. Because this is about speed, I do not *ever* want it toswap.
> I wrote a little test program to try this out, and it fails. As a regular
> user, perror() tells me some "resource is temporarily unavailable". As
> root, it says it "cannot allocate memory". I'm only asking for 1 byte.
> where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE |Remember that no matter what "length" you pass, even if it's 1 byte, the
> MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
kernel has to allocate at least enough integer pages to cover the request.
You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't
quite big enough.
And if you ask for MAP_HUGETLB, you're going to get at least 1 page. Which
is probably 2M in size. And that can be painful if the default max lockable
memory is less than 2M (at least on my Fedora Rawhide box, it's all of
64K). There's also some interaction danger with MAP_POPULATE and memory
fragmentation. But both POPULATE and HUGETLB have a high chance of
not playing well with LOCKED.
Good catch. Indeed, removing MAP_HUGETLB allows the program to succeed. I'll try increasing the resource limit, but may have to have to sudo it, do the mmap() and then chusr to my real ID.
The fd of -1 that some folks thought might be a problem is suggested in the man page for use with MAP_ANONYMOUS. That flag means I'm not using a backing file, after all.
--Kevin O'Gorman
programmer, n. an organism that transmutes caffeine into software.
Please consider the environment before printing this email.
--Kevin O'Gorman
programmer, n. an organism that transmutes caffeine into software.
Please consider the environment before printing this email. _______________________________________________
Kernelnewbies mailing list
Kernelnewbies@xxxxxxxxxxxxxxxxx
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@xxxxxxxxxxxxxxxxx http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
