On Tue, Apr 15, 2008 at 04:23:15PM +0200, Andi Kleen wrote: > Cliff Wickman wrote: > > On Tue, Apr 15, 2008 at 01:35:42AM +0200, Andi Kleen wrote: > >> Cliff Wickman wrote: > >>> Per Andi's post, the current version of numactl/libnuma changes the > >>> application programming interface, so it should be bumped to version 2. > >>> (the symbol versioning preserves the version 1 ABI) > >>> > >>> And he also suggested we supply some compatibility wrappers with a compat > >>> define for old programs. > >>> > >>> This patch provides numacompat1.h and additions to numa.h, which provide > >>> source code compatibility to libnuma version 1. > >>> > >>> Makefile is changed to install as version 2, and install numacompat1.h > >> Can you change numa.h to include numacompat1.h when some define > >> is set? This way Makefiles can be easier changed. > > > > I run into a problem here. If numa.h includes numacompat1.h then you > > get conflicts with other include files that appear after numa.h in some > > source code (like numactl.c for example). > > I don't see an easy solution to that -- suggestions welcome. > > > > There should be only conflicts when the compat define is set. > So don't set it for numactl.c? > > Basically just > > #ifdef NUMACTL_COMPAT1 > #include <numacompat1.h> > #endif I was using numactl as an example, but that's a bad example I agree. In user codes they would only be including numa.h. I tried some of the test suite codes ( ftok.c mynode.c pagesize.c prefered.c randmap.c tshared.c) and they worked fine with numa.h including numacompat1.h. So I'll do it that way. So, the current version of the patch: --------------------------------------------------------------------- Andi suggested we supply some compatibility wrappers with a compat define for old programs. This patch provides numacompat1.h and additions to numa.h, which provide source code compatibility to libnuma version 1. (The application progamming interface changes, but the ABI is preserved through the use of symbol versioning. So the library stays libnuma.so.1) numa.h includes numacompat1.h if VERSION1_COMPATIBILITY is defined. Makefile is changed to install numacompat1.h The libnuma.c change fixes a bug in numa_node_to_cpus_v2() and adds a bitmask-to-bitmask copy for unequal-size masks. The numa.3 man page documents the bitmask copies and the source code compatibility. I plan to include this patch (when I get your feedback) and post the current source as numactl-2.0.0-rc1.tar.gz on http://oss.sgi.com/projects/libnuma/ Signed-off-by: Cliff Wickman <cpw@xxxxxxx> --- CHANGES | 33 +++++ Makefile | 5 libnuma.c | 25 ++++ numa.3 | 43 +++++++ numa.h | 298 +++++++++++++++++++++++++++++++++++++++++++++++++++--- numacompat1.h | 25 ++++ versions.ldscript | 3 7 files changed, 412 insertions(+), 20 deletions(-) Index: numactl-dev/numa.h =================================================================== --- numactl-dev.orig/numa.h +++ numactl-dev/numa.h @@ -22,6 +22,7 @@ #include <stddef.h> #include <string.h> #include <sys/types.h> +#include <stdlib.h> #if defined(__x86_64__) || defined(__i386__) #define NUMA_NUM_NODES 128 @@ -57,6 +58,8 @@ unsigned int numa_bitmask_nbytes(struct struct bitmask *numa_bitmask_alloc(unsigned int); void numa_bitmask_free(struct bitmask *); int numa_bitmask_equal(const struct bitmask *, const struct bitmask *); +void copy_nodemask_to_bitmask(nodemask_t *, struct bitmask *); +void copy_bitmask_to_nodemask(struct bitmask *, nodemask_t *); /* compatibility for codes that used them: */ static inline void nodemask_zero(struct bitmask *mask) @@ -64,20 +67,6 @@ static inline void nodemask_zero(struct numa_bitmask_clearall(mask); } -static inline void nodemask_set(struct bitmask *mask, int node) -{ - numa_bitmask_setbit(mask, node); -} - -static inline void nodemask_clr(struct bitmask *mask, int node) -{ - numa_bitmask_clearbit(mask, node); -} - -static inline int nodemask_isset(struct bitmask *mask, int node) -{ - return numa_bitmask_isbitset(mask, node); -} static inline int nodemask_equal(struct bitmask *a, struct bitmask *b) { return numa_bitmask_equal(a, b); @@ -104,12 +93,18 @@ int numa_pagesize(void); /* Set with all nodes. Only valid after numa_available. */ extern struct bitmask *numa_all_nodes_ptr; +/* For source compatibility */ +extern nodemask_t numa_all_nodes; + /* Set with all cpus. */ extern struct bitmask *numa_all_cpus_ptr; /* Set with no nodes */ extern struct bitmask *numa_no_nodes_ptr; +/* Source compatibility */ +extern nodemask_t numa_no_nodes; + /* Only run and allocate memory from a specific set of nodes. */ void numa_bind(struct bitmask *nodes); @@ -243,9 +238,282 @@ struct bitmask *numa_parse_nodestring(ch /* Convert an ascii list of cpu to a bitmask */ struct bitmask *numa_parse_cpustring(char *); -#ifdef __cplusplus +/* + * The following functions are for source code compatibility + * with releases prior to version 2. + * Such codes must include numacompat1.h (the #include should follow that + * of numa.h). + */ + +static inline void nodemask_zero_compat(nodemask_t *mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = sizeof(nodemask_t) * 8; + numa_bitmask_clearall(&tmp); +} + +static inline void nodemask_set(struct bitmask *mask, int node) +{ + numa_bitmask_setbit(mask, node); +} + +static inline void nodemask_set_compat(nodemask_t *mask, int node) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = sizeof(nodemask_t) * 8; + numa_bitmask_setbit(&tmp, node); +} + +static inline void nodemask_clr(struct bitmask *mask, int node) +{ + numa_bitmask_clearbit(mask, node); +} + +static inline void nodemask_clr_compat(nodemask_t *mask, int node) +{ + mask->n[node / (8*sizeof(unsigned long))] &= + ~(1UL<<(node%(8*sizeof(unsigned long)))); +} + +static inline int nodemask_isset(struct bitmask *mask, int node) +{ + return numa_bitmask_isbitset(mask, node); +} + +static inline int nodemask_isset_compat(nodemask_t *mask, int node) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = sizeof(nodemask_t) * 8; + return numa_bitmask_isbitset(&tmp, node); +} + +static inline int nodemask_equal_compat(nodemask_t *a, nodemask_t *b) +{ + struct bitmask tmpa, tmpb; + + tmpa.maskp = (unsigned long *)a; + tmpa.size = sizeof(nodemask_t) * sizeof(unsigned long); + tmpb.maskp = (unsigned long *)b; + tmpb.size = sizeof(nodemask_t) * sizeof(unsigned long); + return numa_bitmask_equal(&tmpa, &tmpb); +} + +static inline void numa_set_interleave_mask_compat(nodemask_t *nodemask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)nodemask; + tmp.size = sizeof(nodemask_t) * 8; + numa_set_interleave_mask(&tmp); +} + +static inline nodemask_t numa_get_interleave_mask_compat() +{ + struct bitmask *tp; + nodemask_t mask; + + tp = numa_get_interleave_mask(); + copy_bitmask_to_nodemask(tp, &mask); + numa_bitmask_free(tp); + return mask; +} + +static inline void numa_bind_compat(nodemask_t *mask) +{ + struct bitmask *tp; + + tp = numa_allocate_nodemask(); + copy_nodemask_to_bitmask(mask, tp); + numa_bind(tp); + numa_bitmask_free(tp); +} + +static inline void numa_set_membind_compat(nodemask_t *mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = sizeof(nodemask_t) * 8; + numa_set_membind(&tmp); } + +static inline nodemask_t numa_get_membind_compat() +{ + struct bitmask *tp; + nodemask_t mask; + + tp = numa_get_membind(); + copy_bitmask_to_nodemask(tp, &mask); + numa_bitmask_free(tp); + return mask; +} + +static inline void *numa_alloc_interleaved_subset_compat(size_t size, + const nodemask_t *mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = sizeof(nodemask_t) * 8; + return numa_alloc_interleaved_subset(size, &tmp); +} + +static inline int numa_run_on_node_mask_compat(const nodemask_t *mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = sizeof(nodemask_t) * 8; + return numa_run_on_node_mask(&tmp); +} + +static inline nodemask_t numa_get_run_node_mask_compat() +{ + struct bitmask *tp; + nodemask_t mask; + + tp = numa_get_run_node_mask(); + copy_bitmask_to_nodemask(tp, &mask); + numa_bitmask_free(tp); + return mask; +} + +static inline void numa_interleave_memory_compat(void *mem, size_t size, + const nodemask_t *mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = sizeof(nodemask_t) * 8; + numa_interleave_memory(mem, size, &tmp); +} + +static inline void numa_tonodemask_memory_compat(void *mem, size_t size, + const nodemask_t *mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = sizeof(nodemask_t) * 8; + numa_tonodemask_memory(mem, size, &tmp); +} + +static inline int numa_sched_getaffinity_compat(pid_t pid, unsigned len, + unsigned long *mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = len * 8; + return numa_sched_getaffinity(pid, &tmp); +} + +static inline int numa_sched_setaffinity_compat(pid_t pid, unsigned len, + unsigned long *mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = len * 8; + return numa_sched_setaffinity(pid, &tmp); +} + +void printcpumask(char *name, struct bitmask *mask); +static inline void printcpumask_compat(char *name, unsigned long *mask, + int size) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = size * 8; + printcpumask(name, &tmp); +} + +void printmask(char *name, struct bitmask *mask); +static inline void printmask_compat(char *name, nodemask_t *mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)mask; + tmp.size = sizeof(nodemask_t) * 8; + printmask(name, &tmp); +} + +static inline int numa_node_to_cpus_compat(int node, unsigned long *buffer, + int buffer_len) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)buffer; + tmp.size = buffer_len * 8; + return numa_node_to_cpus(node, &tmp); +} + +void verify_shm(int policy, struct bitmask *nodes); +static inline void verify_shm_compat(int policy, nodemask_t mask) +{ + struct bitmask tmp; + + tmp.maskp = (unsigned long *)&mask; + tmp.size = sizeof(nodemask_t) * 8; + verify_shm(policy, &tmp); +} + +static inline nodemask_t nodemask_compat(char *c) +{ + struct bitmask *tp; + nodemask_t mask; + + tp = numa_parse_nodestring(c); + copy_bitmask_to_nodemask(tp, &mask); + numa_bitmask_free(tp); + return mask; +} + +void complain(char *fmt, ...); +static inline unsigned long *cpumask_compat(char *c, int *ncpus) +{ + struct bitmask *tp; + unsigned long *cpubuf; + + tp = numa_parse_cpustring(c); + *ncpus = tp->size; + cpubuf = calloc(tp->size,1); + if (!cpubuf) + complain("Out of memory"); + memcpy(cpubuf, tp->maskp, tp->size/8); + numa_bitmask_free(tp); + return cpubuf; +} + +#include <stdio.h> +static inline int test_bit_compat(int bit, unsigned long *mask) +{ + struct bitmask tmp; + + tmp.maskp = mask; + tmp.size = sizeof(nodemask_t) * 8; + return numa_bitmask_isbitset(&tmp, bit); +} + +/* end of version 1 compatibility functions */ + +/* + * To compile an application that uses libnuma version 1: + * add -DVERSION1_COMPATIBILITY to your Makefile's CFLAGS + */ +#ifdef VERSION1_COMPATIBILITY +#include <numacompat1.h> #endif +#ifdef __cplusplus +} #endif +#endif Index: numactl-dev/versions.ldscript =================================================================== --- numactl-dev.orig/versions.ldscript +++ numactl-dev/versions.ldscript @@ -71,6 +71,9 @@ libnuma_1.1 { libnuma_1.2 { global: + copy_bitmask_to_nodemask; + copy_nodemask_to_bitmask; + copy_bitmask_to_bitmask; set_mempolicy; get_mempolicy; mbind; Index: numactl-dev/libnuma.c =================================================================== --- numactl-dev.orig/libnuma.c +++ numactl-dev/libnuma.c @@ -72,6 +72,7 @@ struct bitmask *numa_allocate_nodemask(v void set_sizes(void); void copy_nodemask_to_bitmask(nodemask_t *, struct bitmask *); void copy_bitmask_to_nodemask(struct bitmask *, nodemask_t *); +void copy_bitmask_to_bitmask(struct bitmask *, struct bitmask *); static inline void nodemask_set_v1(nodemask_t *mask, int node) @@ -984,6 +985,28 @@ copy_bitmask_to_nodemask(struct bitmask } /* + * copy a bitmask map body to another bitmask body + * fill a larger destination with zeroes + */ +void +copy_bitmask_to_bitmask(struct bitmask *bmpfrom, struct bitmask *bmpto) +{ + int bytes; + + if (bmpfrom->size >= bmpto->size) { + memcpy(bmpto->maskp, bmpfrom->maskp, CPU_BYTES(bmpto->size)); + } else if (bmpfrom->size < bmpto->size) { + bytes = CPU_BYTES(bmpfrom->size); + memcpy(bmpto->maskp, bmpfrom->maskp, bytes); + memset(((char *)bmpto->maskp)+bytes, 0, + CPU_BYTES(bmpto->size)-bytes); + } else { + bytes = CPU_BYTES(bmpfrom->size); + memcpy(bmpto->maskp, bmpfrom->maskp, bytes); + } +} + +/* * copy a numa.h nodemask_t structure to a bitmask map body */ void @@ -1262,7 +1285,7 @@ numa_node_to_cpus_v2(int node, struct bi } free(line); - memcpy(buffer->maskp, mask->maskp, bufferlen); + copy_bitmask_to_bitmask(mask, buffer); /* slightly racy, see above */ /* save the mask we created */ Index: numactl-dev/numa.3 =================================================================== --- numactl-dev.orig/numa.3 +++ numactl-dev/numa.3 @@ -137,6 +137,12 @@ numa \- NUMA policy library .BI "struct bitmask *numa_bitmask_setall(struct bitmask *" bmp ); .br .BI "struct bitmask *numa_bitmask_setbit(struct bitmask *" bmp ", unsigned int " n ); +.br +.BI "void copy_bitmask_to_nodemask(struct bitmask *" bmp ", nodemask_t *" nodemask ) +.br +.BI "void copy_nodemask_to_bitmask(nodemask_t *" nodemask ", struct bitmask *" bmp ) +.br +.BI "void copy_bitmask_to_bitmask(struct bitmask *" bmpfrom ", struct bitmask *" bmpto ) .sp .BI "int numa_move_pages(int " pid ", unsigned long " count ", void **" pages ", const int *" nodes ", int *" status ", int " flags ); .br @@ -802,6 +808,33 @@ returned). The value of .I bmp is always returned. +.BR copy_bitmask_to_nodemask() +copies the body (the bit map itself) of the bitmask structure pointed +to by +.I bmp +to the nodemask_t structure pointed to by the +.I nodemask +pointer. If the two areas differ in size, the copy is truncated to the size +of the receiving field or zero-filled. + +.BR copy_nodemask_to_bitmask() +copies the nodemask_t structure pointed to by the +.I nodemask +pointer to the body (the bit map itself) of the bitmask structure pointed +to by the +.I bmp +pointer. If the two areas differ in size, the copy is truncated to the size +of the receiving field or zero-filled. + +.BR copy_bitmask_to_bitmask() +copies the body (the bit map itself) of the bitmask structure pointed +to by the +.I bmpfrom +pointer to the body of the bitmask structure pointed to by the +.I bmpto +pointer. If the two areas differ in size, the copy is truncated to the size +of the receiving field or zero-filled. + .br .BR numa_move_pages() moves a list of pages in the address space of the currently @@ -893,6 +926,14 @@ number identifying each warning. After t .BR printf (3)-style format string and a variable number of arguments. +.SH Compatibility with libnuma version 1 +Binaries that were compiled for libnuma version 1 need not be re-compiled +to run with libnuma version 2. +.br +Source codes written for libnuma version 1 may be re-compiled without +change with version 2 installed. To do so, in the code's Makefile add +this option to CFLAGS: -DVERSION1_COMPATIBILITY + .SH THREAD SAFETY .I numa_set_bind_policy and @@ -900,7 +941,7 @@ and are process global. The other calls are thread safe. .SH COPYRIGHT -Copyright 2002, 2004, 2007, Andi Kleen, SuSE Labs. +Copyright 2002, 2004, 2007, 2008 Andi Kleen, SuSE Labs. .I libnuma is under the GNU Lesser General Public License, v2.1. Index: numactl-dev/Makefile =================================================================== --- numactl-dev.orig/Makefile +++ numactl-dev/Makefile @@ -1,5 +1,4 @@ # these can (and should) be overridden on the make command line for production -# use CFLAGS := -g -Wall -O2 # these are used for the benchmarks in addition to the normal CFLAGS. # Normally no need to overwrite unless you find a new magic flag to make @@ -117,7 +116,7 @@ tonodemask_memory distance MANPAGES := numa.3 numactl.8 numastat.8 migratepages.8 migspeed.8 -install: numactl migratepages migspeed numademo.c numamon memhog libnuma.so.1 numa.h numaif.h numastat ${MANPAGES} +install: numactl migratepages migspeed numademo.c numamon memhog libnuma.so.1 numa.h numaif.h numacompat1.h numastat ${MANPAGES} mkdir -p ${prefix}/bin cp numactl ${prefix}/bin cp migratepages ${prefix}/bin @@ -133,7 +132,7 @@ install: numactl migratepages migspeed n cp libnuma.so.1 ${libdir} cd ${libdir} ; ln -sf libnuma.so.1 libnuma.so mkdir -p ${prefix}/include - cp numa.h numaif.h ${prefix}/include + cp numa.h numaif.h numacompat1.h ${prefix}/include cp numastat ${prefix}/bin if [ -d ${docdir} ] ; then \ mkdir -p ${docdir}/numactl/examples ; \ Index: numactl-dev/numacompat1.h =================================================================== --- /dev/null +++ numactl-dev/numacompat1.h @@ -0,0 +1,25 @@ +#define nodemask_clr(m,n) nodemask_clr_compat(m,n) +#define nodemask_isset(m,n) nodemask_isset_compat(m,n) +#define nodemask_equal(m,n) nodemask_equal_compat(m,n) +#define numa_set_interleave_mask(m) numa_set_interleave_mask_compat(m) +#define nodemask_set(m,n) nodemask_set_compat(m,n) +#define nodemask_zero(m) nodemask_zero_compat(m) +#define numa_get_interleave_mask() numa_get_interleave_mask_compat() +#define numa_bind(m) numa_bind_compat(m) +#define numa_get_membind(m) numa_get_membind_compat(m) +#define numa_set_membind(m) numa_set_membind_compat(m) +#define numa_alloc_interleaved_subset(s,m) numa_alloc_interleaved_subset_compat(s,m) +#define numa_run_on_node_mask(m) numa_run_on_node_mask_compat(m) +#define numa_get_run_node_mask() numa_get_run_node_mask_compat() +#define numa_interleave_memory(st,si,m) numa_interleave_memory_compat(st,si,m) +#define numa_tonodemask_memory(st,si,m) numa_tonodemask_memory_compat(st,si,m) +#define numa_sched_getaffinity(p,l,m) numa_sched_getaffinity_compat(p,l,m) +#define numa_sched_setaffinity(p,l,m) numa_sched_setaffinity_compat(p,l,m) +#define printcpumask(n,m,s) printcpumask_compat(n,m,s) +#define printmask(n,m) printmask_compat(n,m) +#define numa_node_to_cpus(n,b,bl) numa_node_to_cpus_compat(n,b,bl) +#define verify_shm(p,m) verify_shm_compat(p,m) +#define nodemask(c) nodemask_compat(c) +#define cpumask(c,n) cpumask_compat(c,n) +#undef test_bit +#define test_bit(i,p) test_bit_compat(i,p) Index: numactl-dev/CHANGES =================================================================== --- numactl-dev.orig/CHANGES +++ numactl-dev/CHANGES @@ -179,3 +179,36 @@ updating in newer kernels (Mel Gorman) - Fix parallel Makefile build (Andreas Herrmann) - Fix target command argument parsing for numactl (no -- needed again anymore) - Clarify numa_node_to_cpus() manpage + +2.0.0[-rc1] + +- Version 2 (symbol versioning) +- Modify libnuma to use variable-length bit masks (the bitmask structure) +- Modify numactl, migspeed, memhog, migratepages, numademo and stream_main + to use variable-length bit masks +- Modify the test/ programs to use the libnuma that uses variable size bit masks +- Man page changes with the change to variable-length bit masks, move_pages + migrate_pages and others +- Add the migspeed test program to test the speed of migrating pages from + one node to another +- Support for move_pages(2) to numactl, and numa_move_pages() to libnuma +- Add the move_pages test command to exercise the move_pages(2) system call +- Add the mbind_mig_pages test command to verify the moving of a task's + pages with move_pages(2) +- Add the migrate_pages test command to test that a task's pages can be + moved with move_pages(2) +- Support numactl +nn syntax for cpuset_relative cpu and node numbers +- L.S. : +- libnuma man pages -- General Cleanup +- add numa_get_mems_allowed() +- fix checktopology regression test +- numa_maps man page clarifications and cleanup +- minor cleanups of numademo.c +- numactl - fix numastat sysfs scanning +- numactl - reorganize regress test script. +- fix mempolicy regression test for assymetric platforms and memoryless nodes. +- numactl - fix checkaffinity regression test +- numactl - fix/cleanup libnuma local allocation behavior +- P.M. : +- fix __NR_migrate_pages reference. +- Provide numacompat1.h for source code compatibility with version 1 -- Cliff Wickman Silicon Graphics, Inc. cpw@xxxxxxx (651) 683-3824 -- To unsubscribe from this list: send the line "unsubscribe linux-numa" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
