Shared Policy Infrastructure - fix show_numa_maps()
This patch updates the procfs numa_maps display to handle multiple
shared policy ranges on a single vma. numa_maps() still uses the
procfs task maps infrastructure, but provides wrappers around the
maps seq_file ops to handle shared policy "submaps", if any.
This fixes a problem with numa_maps for shared mappings:
Before this [shared policy] patch series, numa_maps could show you
different results for shared mappings depending on which task you
examined. A task which has installed shared policies on sub-ranges
of the shared region will show the policies on the sub-ranges, as the
vmas for that task were split when the policies were installed.
Another task that shares the region, but didn't install any policies,
or installed policies on a different region or set of regions will
show a different policy/range or set thereof, based on the VMAs
of that task. By displaying the policies directly from the shared
policy structure, we now see the same info--the correct effective
mempolicy for each address range--from each task that maps the segment.
The patch expands the proc_maps_private struct [#ifdef CONFIG_NUMA]
to track the existence of and progress through a submap for the
"current" vma. For vmas with shared policy submaps, a new
function--get_numa_submap()--in mm/mempolicy.c allocates and
populates an array of the policy ranges in the shared policy.
To facilitate this, the shared policy struct tracks the number
of ranges [sp_nodes] in the tree.
The nm_* numa_map seq_file wrappers pass the range to be displayed
to show_numa_map() via the saddr and eaddr members added to the
proc_maps_private struct. The patch modifies show_numa_map() to
use these members, where appropriate, instead of vm_start, vm_end.
As before, once the internal page size buffer is full, seq_read()
suspends the display, drops the mmap_sem and exits the read.
During this time the vma list can change. However, even within a
single seq_read(), the shared_policy "submap" can be changed by
other mappers. We could prevent this by holding the shared policy
spin_lock or otherwise holding off other mappers. That would also
hold off other tasks faulting in pages, attempting to look up the
policy for that offset, unless we convert the lock to reader/writer.
It doesn't seem worth the effort, as the numa_map is only a snap_shot
in any case. So, this patch makes a best effort [at least as good as
unpatched task map code, I think] to perform a single scan over the
address space, displaying the policies and page state/location
for policy ranges "snapped" under spin lock into the "submap"
array mentioned above.
NOTE: this patch adds a fair bit of code to the numa maps display.
If necessary, I can make numa_maps a separately configurable option.
E.g., in support of, say, embedded systems that use numa [sh?] and
want/need /proc, but don't need numa_maps nor want the overhead.
Signed-off-by: Lee Schermerhorn <lee.schermerhorn@xxxxxx>
Documentation/vm/numa_memory_policy.txt | 20 +--
fs/proc/task_mmu.c | 186 +++++++++++++++++++++++++++++++-
include/linux/mempolicy.h | 5
include/linux/mm.h | 6 +
include/linux/proc_fs.h | 12 ++
include/linux/shared_policy.h | 3
mm/mempolicy.c | 58 +++++++++
7 files changed, 271 insertions(+), 19 deletions(-)
Index: linux-2.6.36-mmotm-101103-1217/include/linux/proc_fs.h
===================================================================
--- linux-2.6.36-mmotm-101103-1217.orig/include/linux/proc_fs.h
+++ linux-2.6.36-mmotm-101103-1217/include/linux/proc_fs.h
@@ -286,12 +286,24 @@ static inline struct net *PDE_NET(struct
return pde->parent->data;
}
+struct mpol_range {
+ unsigned long saddr;
+ unsigned long eaddr;
+};
+
struct proc_maps_private {
struct pid *pid;
struct task_struct *task;
#ifdef CONFIG_MMU
struct vm_area_struct *tail_vma;
#endif
+
+#ifdef CONFIG_NUMA
+ struct vm_area_struct *vma; /* preserved over seq_reads */
+ unsigned long saddr;
+ unsigned long eaddr; /* preserved over seq_reads */
+ struct mpol_range *range, *ranges; /* preserved ... */
+#endif
};
#endif /* _LINUX_PROC_FS_H */
Index: linux-2.6.36-mmotm-101103-1217/include/linux/mm.h
===================================================================
--- linux-2.6.36-mmotm-101103-1217.orig/include/linux/mm.h
+++ linux-2.6.36-mmotm-101103-1217/include/linux/mm.h
@@ -1243,6 +1243,12 @@ static inline pgoff_t vma_mpol_pgoff(str
return ((addr - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;
}
+static inline pgoff_t vma_mpol_addr(struct vm_area_struct *vma,
+ pgoff_t pgoff)
+{
+ return ((pgoff - vma->vm_pgoff) << PAGE_SHIFT) + vma->vm_start;
+}
+
extern void zone_pcp_update(struct zone *zone);
/* nommu.c */
Index: linux-2.6.36-mmotm-101103-1217/include/linux/mempolicy.h
===================================================================
--- linux-2.6.36-mmotm-101103-1217.orig/include/linux/mempolicy.h
+++ linux-2.6.36-mmotm-101103-1217/include/linux/mempolicy.h
@@ -235,6 +235,11 @@ static inline int vma_migratable(struct
return 1;
}
+struct seq_file;
+extern int show_numa_map(struct seq_file *, void *);
+struct mpol_range;
+extern struct mpol_range *get_numa_submap(struct vm_area_struct *);
+
#else
struct mempolicy {};
Index: linux-2.6.36-mmotm-101103-1217/include/linux/shared_policy.h
===================================================================
--- linux-2.6.36-mmotm-101103-1217.orig/include/linux/shared_policy.h
+++ linux-2.6.36-mmotm-101103-1217/include/linux/shared_policy.h
@@ -26,7 +26,8 @@ struct sp_node {
struct shared_policy {
struct rb_root root;
- spinlock_t lock; /* protects rb tree */
+ spinlock_t lock; /* protects rb tree */
+ int nr_sp_nodes; /* for numa_maps */
};
extern struct shared_policy *mpol_shared_policy_new(
Index: linux-2.6.36-mmotm-101103-1217/mm/mempolicy.c
===================================================================
--- linux-2.6.36-mmotm-101103-1217.orig/mm/mempolicy.c
+++ linux-2.6.36-mmotm-101103-1217/mm/mempolicy.c
@@ -2108,6 +2108,8 @@ static void sp_insert(struct shared_poli
}
rb_link_node(&new->nd, parent, p);
rb_insert_color(&new->nd, &sp->root);
+ ++sp->nr_sp_nodes;
+
pr_debug("inserting %lx-%lx: %d\n", new->start, new->end,
new->policy ? new->policy->mode : 0);
}
@@ -2137,6 +2139,7 @@ static void sp_delete(struct shared_poli
rb_erase(&n->nd, &sp->root);
mpol_put(n->policy);
kmem_cache_free(sn_cache, n);
+ --sp->nr_sp_nodes;
}
static struct sp_node *sp_alloc(unsigned long start, unsigned long end,
@@ -2256,6 +2259,7 @@ put_free:
}
sp->root = RB_ROOT; /* empty tree == default mempolicy */
spin_lock_init(&sp->lock);
+ sp->nr_sp_nodes = 0;
if (new) {
err = mpol_set_shared_policy(sp, 0UL, TASK_SIZE >> PAGE_SHIFT,
@@ -2740,11 +2744,11 @@ int show_numa_map(struct seq_file *m, vo
if (!md)
return 0;
- pol = get_vma_policy(priv->task, vma, vma->vm_start);
+ pol = get_vma_policy(priv->task, vma, priv->saddr);
mpol_to_str(buffer, sizeof(buffer), pol, 0);
mpol_cond_put(pol);
- seq_printf(m, "%08lx %s", vma->vm_start, buffer);
+ seq_printf(m, "%08lx %s", priv->saddr, buffer);
if (file) {
seq_printf(m, " file=");
@@ -2757,10 +2761,10 @@ int show_numa_map(struct seq_file *m, vo
}
if (is_vm_hugetlb_page(vma)) {
- check_huge_range(vma, vma->vm_start, vma->vm_end, md);
+ check_huge_range(vma, priv->saddr, priv->eaddr, md);
seq_printf(m, " huge");
} else {
- check_pgd_range(vma, vma->vm_start, vma->vm_end,
+ check_pgd_range(vma, priv->saddr, priv->eaddr,
&node_states[N_HIGH_MEMORY], MPOL_MF_STATS, md);
}
@@ -2799,3 +2803,49 @@ out:
m->version = (vma != priv->tail_vma) ? vma->vm_start : 0;
return 0;
}
+
+/*
+ * alloc/populate array of shared policy ranges for show_numa_map()
+ */
+struct mpol_range *get_numa_submap(struct vm_area_struct *vma)
+{
+ struct shared_policy *sp;
+ struct mpol_range *ranges, *range;
+ struct rb_node *rbn;
+ int nranges;
+
+ BUG_ON(!vma->vm_file);
+ sp = mapping_shared_policy(vma->vm_file->f_mapping);
+ if (!sp)
+ return NULL;
+
+ nranges = sp->nr_sp_nodes;
+ if (!nranges)
+ return NULL;
+
+ ranges = kzalloc((nranges + 1) * sizeof(*ranges), GFP_KERNEL);
+ if (!ranges)
+ return NULL; /* pretend there are none */
+
+ range = ranges;
+ spin_lock(&sp->lock);
+ /*
+ * # of ranges could have changes since we checked, but that is
+ * unlikely, so this is close enough [as long as it's safe].
+ */
+ rbn = rb_first(&sp->root);
+ /*
+ * count nodes to ensure we leave one empty range struct
+ * in case node added between check and alloc
+ */
+ while (rbn && nranges--) {
+ struct sp_node *spn = rb_entry(rbn, struct sp_node, nd);
+ range->saddr = vma_mpol_addr(vma, spn->start);
+ range->eaddr = vma_mpol_addr(vma, spn->end);
+ range++;
+ rbn = rb_next(rbn);
+ }
+
+ spin_unlock(&sp->lock);
+ return ranges;
+}
Index: linux-2.6.36-mmotm-101103-1217/fs/proc/task_mmu.c
===================================================================
--- linux-2.6.36-mmotm-101103-1217.orig/fs/proc/task_mmu.c
+++ linux-2.6.36-mmotm-101103-1217/fs/proc/task_mmu.c
@@ -818,12 +818,190 @@ const struct file_operations proc_pagema
#endif /* CONFIG_PROC_PAGE_MONITOR */
#ifdef CONFIG_NUMA
-extern int show_numa_map(struct seq_file *m, void *v);
+/*
+ * numa_maps uses procfs task maps file operations, with wrappers
+ * to handle mpol submaps--policy ranges within a vma
+ */
+
+/*
+ * start processing a new vma for show_numa_maps
+ */
+static void nm_vma_start(struct proc_maps_private *priv,
+ struct vm_area_struct *vma)
+{
+ if (!vma)
+ return;
+ priv->vma = vma; /* saved across read()s */
+
+ priv->saddr = vma->vm_start;
+ if (!(vma->vm_flags & VM_SHARED) || !vma->vm_file ||
+ !vma->vm_file->f_mapping->spolicy) {
+ /*
+ * usual case: no submap
+ */
+ priv->eaddr = vma->vm_end;
+ return;
+ }
+
+ priv->range = priv->ranges = get_numa_submap(vma);
+ if (!priv->range) {
+ priv->eaddr = vma->vm_end; /* empty shared policy */
+ return;
+ }
+
+ /*
+ * restart suspended submap where we left off
+ */
+ while (priv->range->eaddr && priv->range->eaddr < priv->eaddr)
+ ++priv->range;
+
+ if (!priv->range->eaddr)
+ priv->eaddr = vma->vm_end;
+ else if (priv->saddr < priv->range->saddr)
+ priv->eaddr = priv->range->saddr; /* show gap [default pol] */
+ else
+ priv->eaddr = priv->range->eaddr; /* show range */
+}
+
+/*
+ * done with numa_maps vma: reset so we start a new
+ * vma on next seq_read.
+ */
+static void nm_vma_stop(struct proc_maps_private *priv)
+{
+ kfree(priv->ranges);
+ priv->ranges = priv->range = NULL;
+ priv->vma = NULL;
+}
+
+/*
+ * Advance to next vma in mm or next subrange in vma.
+ * mmap_sem held during a single seq_read(), but shared
+ * policy ranges can be modified at any time by other
+ * mappers. We just continue to display the ranges we
+ * found when we started the vma.
+ */
+static void *nm_next(struct seq_file *m, void *v, loff_t *pos)
+{
+ struct proc_maps_private *priv = m->private;
+ struct vm_area_struct *vma = v;
+
+ if (!priv->range || priv->eaddr >= vma->vm_end) {
+ /*
+ * usual case: no submap or end of vma
+ * re: '>=' -- in case we got here from nm_start()
+ * and vma @ pos truncated to < priv->eaddr
+ */
+ nm_vma_stop(priv);
+ vma = m_next(m, v, pos);
+ nm_vma_start(priv, vma);
+ return vma;
+ }
+
+ /*
+ * Advance to next range in submap
+ */
+ priv->saddr = priv->eaddr;
+ if (priv->eaddr == priv->range->saddr) {
+ /*
+ * just processed a gap in the submap
+ */
+ priv->eaddr = min(priv->range->eaddr, vma->vm_end);
+ return vma; /* show the range */
+ }
+
+ ++priv->range;
+ if (!priv->range->eaddr)
+ priv->eaddr = vma->vm_end; /* past end of ranges */
+ else if (priv->saddr < priv->range->saddr)
+ priv->eaddr = priv->range->saddr; /* gap in submap */
+ else
+ priv->eaddr = min(priv->range->eaddr, vma->vm_end);
+
+ return vma;
+}
+
+/*
+ * [Re]start scan for new seq_read().
+ * N.B., much could have changes in mm, as we dropped the mmap_sem
+ * between reads(). Need to call m_start() to find vma at pos.
+ */
+static void *nm_start(struct seq_file *m, loff_t *pos)
+{
+ struct proc_maps_private *priv = m->private;
+ struct vm_area_struct *vma;
+
+ if (!priv->range) {
+ /*
+ * usual case: 1st after open, or finished prev vma
+ */
+ vma = m_start(m, pos);
+ nm_vma_start(priv, vma);
+ return vma;
+ }
+
+ /*
+ * Continue with submap of "current" vma. However, vma could have
+ * been unmapped, split, truncated, ... between read()s.
+ * Reset "last_addr" to simulate seek; find vma by 'pos'.
+ */
+ m->version = 0;
+ --(*pos); /* seq_read() incremented it */
+ vma = m_start(m, pos);
+ if (vma != priv->vma)
+ goto new_vma;
+ /*
+ * Same vma address as where we left off, but could have different
+ * ranges or could be entirely different vma.
+ */
+ if (vma->vm_start > priv->eaddr)
+ goto new_vma; /* starts past last range displayed */
+ if (priv->eaddr < vma->vm_end) {
+ /*
+ * vma at pos still covers eaddr--where we left off. Submap
+ * could have changed, but we'll keep reporting ranges we found
+ * earlier up to vm_end.
+ * We hope it is very unlikely that submap changed.
+ */
+ return nm_next(m, vma, pos);
+ }
+
+ /*
+ * Already reported past end of vma; find next vma past eaddr
+ */
+ while (vma && vma->vm_end < priv->eaddr)
+ vma = m_next(m, vma, pos);
+
+new_vma:
+ /*
+ * new vma at pos; continue from ~ last eaddr
+ */
+ nm_vma_stop(priv);
+ nm_vma_start(priv, vma);
+ return vma;
+}
+
+/*
+ * Suspend display of numa_map--e.g., buffer full?
+ */
+static void nm_stop(struct seq_file *m, void *v)
+{
+ struct proc_maps_private *priv = m->private;
+ struct vm_area_struct *vma = v;
+
+ if (!vma || priv->eaddr >= vma->vm_end)
+ nm_vma_stop(priv);
+ /*
+ * leave state in priv for nm_start(); but drop the
+ * mmap_sem and unref the mm
+ */
+ m_stop(m, v);
+}
static const struct seq_operations proc_pid_numa_maps_op = {
- .start = m_start,
- .next = m_next,
- .stop = m_stop,
+ .start = nm_start,
+ .next = nm_next,
+ .stop = nm_stop,
.show = show_numa_map,
};
Index: linux-2.6.36-mmotm-101103-1217/Documentation/vm/numa_memory_policy.txt
===================================================================
--- linux-2.6.36-mmotm-101103-1217.orig/Documentation/vm/numa_memory_policy.txt
+++ linux-2.6.36-mmotm-101103-1217/Documentation/vm/numa_memory_policy.txt
@@ -130,16 +130,16 @@ most general to most specific:
task policy, if any, else System Default Policy.
The shared policy infrastructure supports different policies on subset
- ranges of the shared object. However, Linux still splits the VMA of
- the task that installs the policy for each range of distinct policy.
- Thus, different tasks that attach to a shared memory object can have
- different VMA configurations mapping that one shared object. This
- can be seen by examining the /proc/<pid>/numa_maps of tasks sharing
- a shared memory region. When one task has installed shared policy on
- one or more ranges of the region, the numa_maps of that task will
- show different policies than the numa_maps of other tasks mapping the
- shared object. However, the installed shared policy with be used for
- all pages allocated for the shared object by any of the attached tasks.
+ ranges of the shared object. However, before Linux 2.6.XX, the kernel
+ still split the VMA of the task that installed the policy for each
+ range of distinct policy. Thus, different tasks that attach to a
+ shared memory segment could have different VMA configurations mapping
+ that one shared object. This was visible by examining the
+ /proc/<pid>/numa_maps of tasks sharing the shared memory region.
+ As of 2.6.XX, Linux no longer splits the VMA that maps a shared object
+ to install a memory policy on a sub-range of the object. The
+ /proc/<pid>/numa_maps of all tasks sharing a shared memory region now
+ show the same set of memory policy ranges.
When installing shared policy on a shared object, the virtual address
range specified can be viewed as a "direct mapped", linear window onto
--
To unsubscribe from this list: send the line "unsubscribe linux-numa" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
