Fwd: [PATCH 2/2] mm: zcache: core functions added

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



-------- Original Message --------
Subject: [PATCH 2/2] mm: zcache: core functions added
Date: Sat, 20 Jul 2013 22:36:58 +0800
From: Bob Liu <lliubbo@xxxxxxxxx>
CC: linux-kernel@xxxxxxxxxxxxxxx, sjenning@xxxxxxxxxxxxxxxxxx,
gregkh@xxxxxxxxxxxxxxxxxxx, ngupta@xxxxxxxxxx, minchan@xxxxxxxxxx,
  konrad.wilk@xxxxxxxxxx, rcj@xxxxxxxxxxxxxxxxxx, mgorman@xxxxxxx,
  riel@xxxxxxxxxx, penberg@xxxxxxxxxx, akpm@xxxxxxxxxxxxxxxxxxxx,
 Bob Liu <bob.liu@xxxxxxxxxx>

This patch adds the cleancache backend for clean file pages compression.

Nitin Gupta have already done many works on this topic in 2010.
You can see his work from below links:
http://lwn.net/Articles/396467/
http://thread.gmane.org/gmane.linux.kernel.mm/50523
But at that time neither the allocation layer nor cleancache have been
merged
into upstream.

(Most of below comments are copyed from his patch.)
Frequently accessed filesystem data is stored in memory to reduce access to
(much) slower backing disks. Under memory pressure, these pages are
freed and
when needed again, they have to be read from disks again. When combined
working set of all running application exceeds amount of physical RAM,
we get
extereme slowdown as reading a page from disk can take time in order of
milliseconds.

Memory compression increases effective memory size and allows more pages to
stay in RAM. Since de/compressing memory pages is several orders of
magnitude
faster than disk I/O, this can provide signifant performance gains for many
workloads. Also, with multi-cores becoming common, benefits of reduced disk
I/O should easily outweigh the problem of increased CPU usage.

It is implemented as a "backend" for cleancache which provides
callbacks for events such as when a page is to be removed from the page
cache
and when it is required again. We use them to implement a 'second chance'
cache for these evicted page cache pages by compressing and storing them in
memory itself.

We use zbud memory allocator which is already merged and used by zswap
for the
same purpose.

A separate "pool" is created for each mount instance for a cleancache-aware
filesystem. Each incoming page is identified with <pool_id, inode_no, index>
where inode_no identifies file within the filesystem corresponding to
pool_id
and index is offset of the page within this inode. Within a pool, inodes are
maintained in an rb-tree and each of its nodes points to a separate
radix-tree
which maintains list of pages within that inode.

Tests were done on physical machine (not in a VM)
Only played with iozone for some simple tests.
Kernel booted with command line "single mem=1G zcache.enabled=1"
Kernel 3.10.0+ (with Seth's zswap patches)
CPU: Core-i5 4-Core
RAM: 1G
Boot into single mode
Swapoff -a to avoid I/O interference
Command for testing:
iozone -a -M -B -s 1G -y 4k -+u -R -b iozone.xls

Results:
Without zcache
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read
         1048576       4   73070   36713    75187    74916    1292
674   26724    45514     8491
         1048576       8   68777   34407    71162    73407    2703
1233   27800    42417     5548
         1048576      16   70030   35156    71496    71186    4462
2181   31480    42040     8737
         1048576      32   68796   35442    72557    72734    6955
3615   40783    44627    10959
         1048576      64   76576   35247    68685    72279   11383
6615   49913    44620    14777
         1048576     128   77516   33878    72538    71231   17897
10896   64393    48613    35265
         1048576     256   73465   37450    22857    22854   22845
16655   61152    49170    40571
         1048576     512   68096   35303    22265    22398   36154
23952   58286    48594    39972
         1048576    1024   69442   36192    22206    22541   38927
29643   60695    50483    44120
         1048576    2048   69404   34891    22951    23286   40036
35984   62095    50704    43764
         1048576    4096   68871   35716    23147    23272   41587
37619   66809    47613    45229
         1048576    8192   66775   36409    22997    23070   39023
39674   69262    55548    49324
         1048576   16384   68728   34792    23829    24022   39922
42496   65714    44732    39104
Average:		   70734   35507    45529    45938   23760   19326   52700
47283    29682

With zcache
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read
         1048576       4   80330   35970   120942   115559    1491
686   29923    47864    11177
         1048576       8   69553   36658   134402   105351    2978
1332   33291    45885     6331
         1048576      16   79538   36719   149905   155649    5970
2305   38136    42365    14070
         1048576      32   68326   37064   150126   151728   10462
4128   44693    44975    15607
         1048576      64   74450   35379   147918   130771   19199
7468   54065    43630    22104
         1048576     128   71655   36125   137384   139080   27495
11507   65742    47471    49833
         1048576     256   78867   36030    51814    42656   41578
17829   77058    44247    54368
         1048576     512   68671   35378    52359    44516   52296
25944   80121    47270    57271
         1048576    1024   69495   37064    53579    56057   60121
31833   83420    46752    65267
         1048576    2048   69014   35589    52216    49919   69441
37169   92552    45746    68838
         1048576    4096   67338   36194    56068    56855   71569
39895   98462    49440    72496
         1048576    8192   65153   35345    54983    58602   78402
43841   98809    47334    74803
         1048576   16384   63579   34842    50888    54429   68070
42518   87534    45665    64967
Average:                   71228   36027    93345    89321   39159
20497   67985    46050    44395
Chnage :                      1%      1%     105%      94%     65%
6%     29%      -3%      50%

You can see that reading side performance get improved a lot after using
zcache.
Because pages are cached by zcache, we only need to decompress them
instead of
reading from disk.

Signed-off-by: Bob Liu <bob.liu@xxxxxxxxxx>
---
 mm/Kconfig  |   18 ++
 mm/Makefile |    1 +
 mm/zcache.c |  840
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 859 insertions(+)
 create mode 100644 mm/zcache.c

diff --git a/mm/Kconfig b/mm/Kconfig
index eec97f2..2b68103 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -507,3 +507,21 @@ config ZSWAP
 	  interactions don't cause any known issues on simple memory setups,
 	  they have not be fully explored on the large set of potential
 	  configurations and workloads that exist.
+
+config ZCACHE
+	bool "Compressed cache for swap and clean file pages (EXPERIMENTAL)"
+	depends on FRONTSWAP && CRYPTO && CLEANCACHE
+	select CRYPTO_LZO
+	select ZBUD
+	default n
+	help
+	  A compressed cache for swap and clean file pages.
+
+	  It can takes pages that are in the process of being swapped out and
+	  attempts to compress them into a dynamically allocated RAM-based
memory pool.
+	  This can result in a significant I/O reduction on swap device and,
+	  in the case where decompressing from RAM is faster that swap device
+	  reads, can also improve workload performance.
+
+	  Besides swap pages, zcache can also compress the clean file pages
into a memory pool.
+	  This can reduce the refaults of reading those file pages back from
disks.
diff --git a/mm/Makefile b/mm/Makefile
index f008033..a29232b 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -33,6 +33,7 @@ obj-$(CONFIG_BOUNCE)	+= bounce.o
 obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o
 obj-$(CONFIG_FRONTSWAP)	+= frontswap.o
 obj-$(CONFIG_ZSWAP)	+= zswap.o
+obj-$(CONFIG_ZCACHE)	+= zcache.o
 obj-$(CONFIG_HAS_DMA)	+= dmapool.o
 obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o
 obj-$(CONFIG_NUMA) 	+= mempolicy.o
diff --git a/mm/zcache.c b/mm/zcache.c
new file mode 100644
index 0000000..a2408e8
--- /dev/null
+++ b/mm/zcache.c
@@ -0,0 +1,840 @@
+/*
+ * zcache.c - zcache driver file
+ *
+ * The goal of zcache is implement a generic memory compression layer.
+ * It's a backend of both frontswap and cleancache.
+ *
+ * This file only implemented cleancache part currently.
+ * Concepts based on original zcache by Dan Magenheimer.
+ *
+ * Copyright (C) 2013  Bob Liu <bob.liu@xxxxxxxxxx>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+*/
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/module.h>
+#include <linux/cpu.h>
+#include <linux/highmem.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+#include <linux/atomic.h>
+#include <linux/cleancache.h>
+#include <linux/rbtree.h>
+#include <linux/radix-tree.h>
+#include <linux/swap.h>
+#include <linux/crypto.h>
+#include <linux/mempool.h>
+#include <linux/zbud.h>
+
+#include <linux/mm_types.h>
+#include <linux/page-flags.h>
+#include <linux/swapops.h>
+#include <linux/writeback.h>
+#include <linux/pagemap.h>
+
+/* Enable/disable zcache (disabled by default) */
+static bool zcache_enabled __read_mostly;
+module_param_named(enabled, zcache_enabled, bool, 0);
+
+/* Enable/disable cleancache part of zcache */
+static bool zcache_nocleancache __read_mostly;
+module_param_named(nocleancache, zcache_nocleancache, bool, 0);
+
+/* Compressor to be used by zcache */
+#define ZCACHE_COMPRESSOR_DEFAULT "lzo"
+static char *zcache_compressor = ZCACHE_COMPRESSOR_DEFAULT;
+module_param_named(compressor, zcache_compressor, charp, 0);
+
+/* The maximum percentage of memory that the compressed pool can occupy */
+static unsigned int zcache_max_pool_percent = 10;
+module_param_named(max_pool_percent,
+			zcache_max_pool_percent, uint, 0644);
+
+/* zcache cleancache part statistics */
+static u64 zcache_cleancache_pool_pages;
+static u64 zcache_cleancache_pool_limit_hit;
+static u64 zcache_cleancache_written_back_pages;
+static u64 zcache_cleancache_dup_entry;
+static u64 zcache_cleancache_reclaim_fail;
+static u64 zcache_cleancache_zbud_alloc_fail;
+static atomic_t zcache_cleancache_stored_pages = ATOMIC_INIT(0);
+
+struct zcache_cleancache_meta {
+	int ra_index;
+	int length;	/* compressed page size */
+};
+
+#define MAX_ZCACHE_POOLS 32 /* arbitrary */
+
+/* Red-Black tree node. Maps inode to its page-tree */
+struct zcache_rb_entry {
+	int rb_index;
+	struct kref refcount;
+
+	struct radix_tree_root ra_root; /* maps inode index to page */
+	spinlock_t ra_lock;		/* protects radix tree */
+	struct rb_node rb_node;
+};
+
+/* One zcache pool per (cleancache aware) filesystem mount instance */
+struct zcache_pool {
+	struct rb_root rb_root;		/* maps inode number to page tree */
+	rwlock_t rb_lock;		/* protects inode_tree */
+	struct zbud_pool *pool;         /* zbud pool used */
+};
+
+/* Manage all zcache pools */
+struct _zcache {
+	struct zcache_pool *pools[MAX_ZCACHE_POOLS];
+	u32 num_pools;		/* current no. of zcache pools */
+	spinlock_t pool_lock;	/* protects pools[] and num_pools */
+};
+struct _zcache zcache;
+
+static struct kmem_cache *zcache_cleancache_entry_cache;
+
+/*********************************
+* compression functions
+**********************************/
+/* per-cpu compression transforms */
+static struct crypto_comp * __percpu *zcache_comp_pcpu_tfms;
+
+enum comp_op {
+	ZCACHE_COMPOP_COMPRESS,
+	ZCACHE_COMPOP_DECOMPRESS
+};
+
+static int zcache_comp_op(enum comp_op op, const u8 *src, unsigned int
slen,
+				u8 *dst, unsigned int *dlen)
+{
+	struct crypto_comp *tfm;
+	int ret;
+
+	tfm = *per_cpu_ptr(zcache_comp_pcpu_tfms, get_cpu());
+	switch (op) {
+	case ZCACHE_COMPOP_COMPRESS:
+		ret = crypto_comp_compress(tfm, src, slen, dst, dlen);
+		break;
+	case ZCACHE_COMPOP_DECOMPRESS:
+		ret = crypto_comp_decompress(tfm, src, slen, dst, dlen);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	put_cpu();
+	return ret;
+}
+
+static int __init zcache_comp_init(void)
+{
+	if (!crypto_has_comp(zcache_compressor, 0, 0)) {
+		pr_info("%s compressor not available\n", zcache_compressor);
+		/* fall back to default compressor */
+		zcache_compressor = ZCACHE_COMPRESSOR_DEFAULT;
+		if (!crypto_has_comp(zcache_compressor, 0, 0))
+			/* can't even load the default compressor */
+			return -ENODEV;
+	}
+	pr_info("using %s compressor\n", zcache_compressor);
+
+	/* alloc percpu transforms */
+	zcache_comp_pcpu_tfms = alloc_percpu(struct crypto_comp *);
+	if (!zcache_comp_pcpu_tfms)
+		return -ENOMEM;
+	return 0;
+}
+
+static void zcache_comp_exit(void)
+{
+	/* free percpu transforms */
+	if (zcache_comp_pcpu_tfms)
+		free_percpu(zcache_comp_pcpu_tfms);
+}
+
+/*********************************
+* per-cpu code
+**********************************/
+static DEFINE_PER_CPU(u8 *, zcache_dstmem);
+
+static int __zcache_cpu_notifier(unsigned long action, unsigned long cpu)
+{
+	struct crypto_comp *tfm;
+	u8 *dst;
+
+	switch (action) {
+	case CPU_UP_PREPARE:
+		tfm = crypto_alloc_comp(zcache_compressor, 0, 0);
+		if (IS_ERR(tfm)) {
+			pr_err("can't allocate compressor transform\n");
+			return NOTIFY_BAD;
+		}
+		*per_cpu_ptr(zcache_comp_pcpu_tfms, cpu) = tfm;
+		dst = kmalloc(PAGE_SIZE * 2, GFP_KERNEL);
+		if (!dst) {
+			pr_err("can't allocate compressor buffer\n");
+			crypto_free_comp(tfm);
+			*per_cpu_ptr(zcache_comp_pcpu_tfms, cpu) = NULL;
+			return NOTIFY_BAD;
+		}
+		per_cpu(zcache_dstmem, cpu) = dst;
+		break;
+	case CPU_DEAD:
+	case CPU_UP_CANCELED:
+		tfm = *per_cpu_ptr(zcache_comp_pcpu_tfms, cpu);
+		if (tfm) {
+			crypto_free_comp(tfm);
+			*per_cpu_ptr(zcache_comp_pcpu_tfms, cpu) = NULL;
+		}
+		dst = per_cpu(zcache_dstmem, cpu);
+		kfree(dst);
+		per_cpu(zcache_dstmem, cpu) = NULL;
+		break;
+	default:
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static int zcache_cpu_notifier(struct notifier_block *nb,
+				unsigned long action, void *pcpu)
+{
+	unsigned long cpu = (unsigned long)pcpu;
+	return __zcache_cpu_notifier(action, cpu);
+}
+
+static struct notifier_block zcache_cpu_notifier_block = {
+	.notifier_call = zcache_cpu_notifier
+};
+
+static int zcache_cpu_init(void)
+{
+	unsigned long cpu;
+
+	get_online_cpus();
+	for_each_online_cpu(cpu)
+		if (__zcache_cpu_notifier(CPU_UP_PREPARE, cpu) != NOTIFY_OK)
+			goto cleanup;
+	register_cpu_notifier(&zcache_cpu_notifier_block);
+	put_online_cpus();
+	return 0;
+
+cleanup:
+	for_each_online_cpu(cpu)
+		__zcache_cpu_notifier(CPU_UP_CANCELED, cpu);
+	put_online_cpus();
+	return -ENOMEM;
+}
+
+/*********************************
+* helpers
+**********************************/
+static bool zcache_is_full(void)
+{
+	return (totalram_pages * zcache_max_pool_percent / 100 <
+		zcache_cleancache_pool_pages);
+}
+
+static int zcache_cleancache_entry_cache_create(void)
+{
+	zcache_cleancache_entry_cache = KMEM_CACHE(zcache_rb_entry, 0);
+	return (zcache_cleancache_entry_cache == NULL);
+}
+static void zcache_cleancache_entry_cache_destory(void)
+{
+	kmem_cache_destroy(zcache_cleancache_entry_cache);
+}
+
+static struct zcache_rb_entry *zcache_find_rb_entry(struct rb_root *root,
+		int index, struct rb_node **rb_parent, struct rb_node ***rb_link)
+{
+	struct zcache_rb_entry *entry;
+	struct rb_node **__rb_link, *__rb_parent, *rb_prev;
+
+	__rb_link = &root->rb_node;
+	rb_prev = __rb_parent = NULL;
+
+	while (*__rb_link) {
+		__rb_parent = *__rb_link;
+		entry = rb_entry(__rb_parent, struct zcache_rb_entry, rb_node);
+		if (entry->rb_index > index)
+			__rb_link = &__rb_parent->rb_left;
+		else if (entry->rb_index < index) {
+			rb_prev = __rb_parent;
+			__rb_link = &__rb_parent->rb_right;
+		} else
+			return entry;
+	}
+
+	if (rb_parent)
+		*rb_parent = __rb_parent;
+	if (rb_link)
+		*rb_link = __rb_link;
+	return NULL;
+}
+
+static struct zcache_rb_entry *zcache_find_get_rb_entry(struct
zcache_pool *zpool,
+					int rb_index)
+{
+	unsigned long flags;
+	struct zcache_rb_entry *rb_entry;
+
+	read_lock_irqsave(&zpool->rb_lock, flags);
+	rb_entry = zcache_find_rb_entry(&zpool->rb_root, rb_index, 0, 0);
+	if (rb_entry)
+		kref_get(&rb_entry->refcount);
+	read_unlock_irqrestore(&zpool->rb_lock, flags);
+	return rb_entry;
+}
+
+/*
+ * kref_put callback for zcache rb_entry.
+ *
+ * The entry must have been isolated from rbtree already.
+ */
+static void zcache_rb_entry_release(struct kref *kref)
+{
+	struct zcache_rb_entry *rb_entry;
+
+	rb_entry = container_of(kref, struct zcache_rb_entry, refcount);
+	BUG_ON(rb_entry->ra_root.rnode);
+	kmem_cache_free(zcache_cleancache_entry_cache, rb_entry);
+}
+
+/*
+ * Called under zcache_rb_entry->ra_lock
+ */
+static int zcache_rb_entry_is_empty(struct zcache_rb_entry *rb_entry)
+{
+	return rb_entry->ra_root.rnode == NULL;
+}
+
+/* Remove rb_entry from rbtree */
+static void zcache_rb_entry_isolate(struct zcache_pool *zpool,
+		struct zcache_rb_entry *rb_entry, bool hold_rblock)
+{
+	unsigned long flags;
+
+	if (!hold_rblock)
+		write_lock_irqsave(&zpool->rb_lock, flags);
+	/*
+	 * Someone can get reference on this node before we could
+	 * acquire write lock above. We want to remove it from its
+	 * inode_tree when only the caller and corresponding inode_tree
+	 * holds a reference to it. This ensures that a racing zcache
+	 * put will not end up adding a page to an isolated node and
+	 * thereby losing that memory.
+	 *
+	 */
+	if (atomic_read(&rb_entry->refcount.refcount) == 2) {
+		rb_erase(&rb_entry->rb_node, &zpool->rb_root);
+		RB_CLEAR_NODE(&rb_entry->rb_node);
+		kref_put(&rb_entry->refcount, zcache_rb_entry_release);
+	}
+	if (!hold_rblock)
+		write_unlock_irqrestore(&zpool->rb_lock, flags);
+}
+
+
+static int zcache_store_handle(struct zcache_pool *zpool,
+		unsigned long handle, int rb_index, int ra_index)
+{
+	unsigned long flags;
+	struct zcache_rb_entry *rb_entry, *tmp;
+	struct rb_node **link = NULL, *parent = NULL;
+	int ret;
+	void *dup_handlep;
+
+	rb_entry = zcache_find_get_rb_entry(zpool, rb_index);
+	if (!rb_entry) {
+		/* alloc new rb_entry */
+		rb_entry = kmem_cache_alloc(zcache_cleancache_entry_cache, GFP_KERNEL);
+		if (!rb_entry)
+			return -ENOMEM;
+
+		INIT_RADIX_TREE(&rb_entry->ra_root, GFP_ATOMIC|__GFP_NOWARN);
+		spin_lock_init(&rb_entry->ra_lock);
+		rb_entry->rb_index = rb_index;
+		kref_init(&rb_entry->refcount);
+		RB_CLEAR_NODE(&rb_entry->rb_node);
+
+		/* add new entry to rb tree */
+		write_lock_irqsave(&zpool->rb_lock, flags);
+
+		tmp = zcache_find_rb_entry(&zpool->rb_root, rb_index, &parent, &link);
+		if (tmp) {
+			/* somebody else allocated new entry */
+			kmem_cache_free(zcache_cleancache_entry_cache, rb_entry);
+			rb_entry = tmp;
+		} else {
+			rb_link_node(&rb_entry->rb_node, parent, link);
+			rb_insert_color(&rb_entry->rb_node, &zpool->rb_root);
+		}
+
+		kref_get(&rb_entry->refcount);
+		write_unlock_irqrestore(&zpool->rb_lock, flags);
+	}
+
+	/* Succ get rb_entry and refcount after arrived here */
+	spin_lock_irqsave(&rb_entry->ra_lock, flags);
+	dup_handlep = radix_tree_delete(&rb_entry->ra_root, ra_index);
+	if (unlikely(dup_handlep)) {
+		WARN_ON("duplicated entry, will be replaced!\n");
+		zbud_free(zpool->pool, (unsigned long)dup_handlep);
+		atomic_dec(&zcache_cleancache_stored_pages);
+		zcache_cleancache_pool_pages = zbud_get_pool_size(zpool->pool);
+		zcache_cleancache_dup_entry++;
+	}
+	ret = radix_tree_insert(&rb_entry->ra_root, ra_index, (void *)handle);
+
+	if (unlikely(ret))
+		if (zcache_rb_entry_is_empty(rb_entry))
+			zcache_rb_entry_isolate(zpool, rb_entry, 0);
+	spin_unlock_irqrestore(&rb_entry->ra_lock, flags);
+
+	kref_put(&rb_entry->refcount, zcache_rb_entry_release);
+	return ret;
+}
+
+/* Load the handle, and delete it */
+static unsigned long *zcache_load_delete_handle(struct zcache_pool
*zpool, int rb_index,
+				int ra_index)
+{
+	struct zcache_rb_entry *rb_entry;
+	void *handlep = NULL;
+	unsigned long flags;
+
+	rb_entry = zcache_find_get_rb_entry(zpool, rb_index);
+	if (!rb_entry)
+		goto out;
+
+	BUG_ON(rb_entry->rb_index != rb_index);
+
+	spin_lock_irqsave(&rb_entry->ra_lock, flags);
+	handlep = radix_tree_delete(&rb_entry->ra_root, ra_index);
+	if (zcache_rb_entry_is_empty(rb_entry))
+		/* If no more nodes in the rb_entry->radix_tree,
+		 * rm rb_entry from the rbtree and drop the refcount
+		 */
+		zcache_rb_entry_isolate(zpool, rb_entry, 0);
+	spin_unlock_irqrestore(&rb_entry->ra_lock, flags);
+
+	/* After arrive here, rb_entry have dropped from rbtree */
+	kref_put(&rb_entry->refcount, zcache_rb_entry_release);
+out:
+	return handlep;
+}
+
+static void zcache_cleancache_store_page(int pool_id, struct
cleancache_filekey key,
+			pgoff_t index, struct page *page)
+{
+	unsigned int dlen = PAGE_SIZE, len;
+	unsigned long handle;
+	char *buf;
+	u8 *src, *dst;
+	struct zcache_cleancache_meta *zmeta;
+	int ret;
+
+	struct zcache_pool *zpool = zcache.pools[pool_id];
+
+	/* reclaim space if needed */
+	if (zcache_is_full()) {
+		/* Reclaim will be implemented in following version */
+		zcache_cleancache_pool_limit_hit++;
+		return;
+	}
+
+	/* compress */
+	dst = get_cpu_var(zcache_dstmem);
+	src = kmap_atomic(page);
+	ret = zcache_comp_op(ZCACHE_COMPOP_COMPRESS, src, PAGE_SIZE, dst, &dlen);
+	kunmap_atomic(src);
+	if (ret) {
+		pr_err("zcache_cleancache compress error ret %d\n", ret);
+		put_cpu_var(zcache_dstmem);
+		return;
+	}
+
+	/* store handle with meta data */
+	len = dlen + sizeof(struct zcache_cleancache_meta);
+	ret = zbud_alloc(zpool->pool, len, __GFP_NORETRY | __GFP_NOWARN, &handle);
+	if (ret) {
+		zcache_cleancache_zbud_alloc_fail++;
+		put_cpu_var(zcache_dstmem);
+		return;
+	}
+
+	zmeta = zbud_map(zpool->pool, handle);
+	zmeta->ra_index = index;
+	zmeta->length = dlen;
+	buf = (u8 *)(zmeta + 1);
+	memcpy(buf, dst, dlen);
+	zbud_unmap(zpool->pool, handle);
+	put_cpu_var(zcache_dstmem);
+
+	/* populate entry */
+	ret = zcache_store_handle(zpool, handle, key.u.ino, index);
+	if (ret) {
+		pr_err("%s: store handle error %d\n", __func__, ret);
+		zbud_free(zpool->pool, handle);
+	}
+
+	/* update stats */
+	atomic_inc(&zcache_cleancache_stored_pages);
+	zcache_cleancache_pool_pages = zbud_get_pool_size(zpool->pool);
+	return;
+}
+
+static int zcache_cleancache_load_page(int pool_id, struct
cleancache_filekey key,
+			pgoff_t index, struct page *page)
+{
+	struct zcache_pool *zpool = zcache.pools[pool_id];
+	u8 *src, *dst;
+	unsigned int dlen;
+	int ret;
+	unsigned long *handlep;
+	struct zcache_cleancache_meta *zmeta;
+
+	handlep = zcache_load_delete_handle(zpool, key.u.ino, index);
+	if (!handlep)
+		return -1;
+
+	zmeta = (struct zcache_cleancache_meta *)zbud_map(zpool->pool,
(unsigned long)handlep);
+	src = (u8 *)(zmeta + 1);
+
+	/* decompress */
+	dlen = PAGE_SIZE;
+	dst = kmap_atomic(page);
+	ret = zcache_comp_op(ZCACHE_COMPOP_DECOMPRESS, src, zmeta->length,
dst, &dlen);
+	kunmap_atomic(dst);
+	zbud_unmap(zpool->pool, (unsigned long)handlep);
+	zbud_free(zpool->pool, (unsigned long)handlep);
+
+	WARN_ON(ret);	/* decompress err, will fetch from real disk */
+	/* update stats */
+	atomic_dec(&zcache_cleancache_stored_pages);
+	zcache_cleancache_pool_pages = zbud_get_pool_size(zpool->pool);
+	return ret;
+}
+
+static void zcache_cleancache_flush_page(int pool_id, struct
cleancache_filekey key,
+			pgoff_t index)
+{
+	struct zcache_pool *zpool = zcache.pools[pool_id];
+	unsigned long *handlep = NULL;
+
+	handlep = zcache_load_delete_handle(zpool, key.u.ino, index);
+	if (handlep) {
+		zbud_free(zpool->pool, (unsigned long)handlep);
+		atomic_dec(&zcache_cleancache_stored_pages);
+		zcache_cleancache_pool_pages = zbud_get_pool_size(zpool->pool);
+	}
+}
+
+#define FREE_BATCH 16
+static void zcache_cleancache_flush_ratree(struct zcache_pool *zpool,
+				struct zcache_rb_entry *entry)
+{
+	int count, i;
+	unsigned long index = 0;
+
+	do {
+		struct zcache_cleancache_meta *handles[FREE_BATCH];
+
+		count = radix_tree_gang_lookup(&entry->ra_root,
+				(void **)handles, index, FREE_BATCH);
+
+		for (i = 0; i < count; i++) {
+			index = handles[i]->ra_index;
+			radix_tree_delete(&entry->ra_root, index);
+			zbud_free(zpool->pool, (unsigned long)handles[i]);
+			atomic_dec(&zcache_cleancache_stored_pages);
+			zcache_cleancache_pool_pages = zbud_get_pool_size(zpool->pool);
+		}
+
+		index++;
+	} while (count == FREE_BATCH);
+}
+
+static void zcache_cleancache_flush_inode(int pool_id,
+					struct cleancache_filekey key)
+{
+	struct zcache_rb_entry *rb_entry;
+	unsigned long flags1, flags2;
+	struct zcache_pool *zpool = zcache.pools[pool_id];
+
+	/* refuse new pages added in to the same inode */
+	write_lock_irqsave(&zpool->rb_lock, flags1);
+	rb_entry = zcache_find_rb_entry(&zpool->rb_root, key.u.ino, 0, 0);
+	if (!rb_entry) {
+		write_unlock_irqrestore(&zpool->rb_lock, flags1);
+		return;
+	}
+
+	kref_get(&rb_entry->refcount);
+
+	spin_lock_irqsave(&rb_entry->ra_lock, flags2);
+	zcache_cleancache_flush_ratree(zpool, rb_entry);
+	if (zcache_rb_entry_is_empty(rb_entry))
+		zcache_rb_entry_isolate(zpool, rb_entry, 1);
+	spin_unlock_irqrestore(&rb_entry->ra_lock, flags2);
+
+	write_unlock_irqrestore(&zpool->rb_lock, flags1);
+	kref_put(&rb_entry->refcount, zcache_rb_entry_release);
+}
+
+static void zcache_destroy_pool(struct zcache_pool *zpool);
+static void zcache_cleancache_flush_fs(int pool_id)
+{
+	struct zcache_rb_entry *entry = NULL;
+	struct rb_node *node;
+	unsigned long flags1, flags2;
+	struct zcache_pool *zpool = zcache.pools[pool_id];
+
+	if (!zpool)
+		return;
+
+	/* refuse new pages added in to the same inode */
+	write_lock_irqsave(&zpool->rb_lock, flags1);
+
+	node = rb_first(&zpool->rb_root);
+	while (node) {
+		entry = rb_entry(node, struct zcache_rb_entry, rb_node);
+		node = rb_next(node);
+		if (entry) {
+			kref_get(&entry->refcount);
+			spin_lock_irqsave(&entry->ra_lock, flags2);
+			zcache_cleancache_flush_ratree(zpool, entry);
+			if (zcache_rb_entry_is_empty(entry))
+				zcache_rb_entry_isolate(zpool, entry, 1);
+			spin_unlock_irqrestore(&entry->ra_lock, flags2);
+			kref_put(&entry->refcount, zcache_rb_entry_release);
+		}
+	}
+
+	write_unlock_irqrestore(&zpool->rb_lock, flags1);
+
+	zcache_destroy_pool(zpool);
+}
+
+static int zcache_cleancache_evict_entry(struct zbud_pool *pool,
+		unsigned long handle)
+{
+	return -1;
+}
+
+static struct zbud_ops zcache_cleancache_zbud_ops = {
+	.evict = zcache_cleancache_evict_entry
+};
+
+static void zcache_destroy_pool(struct zcache_pool *zpool)
+{
+	int i;
+
+	if (!zpool)
+		return;
+
+	spin_lock(&zcache.pool_lock);
+	zcache.num_pools--;
+	for (i = 0; i < MAX_ZCACHE_POOLS; i++)
+		if (zcache.pools[i] == zpool)
+			break;
+	zcache.pools[i] = NULL;
+	spin_unlock(&zcache.pool_lock);
+
+	if (!RB_EMPTY_ROOT(&zpool->rb_root)) {
+		WARN_ON("Memory leak detected. Freeing non-empty pool!\n");
+	}
+
+	zbud_destroy_pool(zpool->pool);
+	kfree(zpool);
+}
+
+/* return pool id */
+static int zcache_create_pool(void)
+{
+	int ret;
+	struct zcache_pool *zpool;
+
+	zpool = kzalloc(sizeof(*zpool), GFP_KERNEL);
+	if (!zpool) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	zpool->pool = zbud_create_pool(GFP_KERNEL, &zcache_cleancache_zbud_ops);
+	if (!zpool->pool) {
+		kfree(zpool);
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	spin_lock(&zcache.pool_lock);
+	if (zcache.num_pools == MAX_ZCACHE_POOLS) {
+		pr_info("Cannot create new pool (limit: %u)\n",
+					MAX_ZCACHE_POOLS);
+		zbud_destroy_pool(zpool->pool);
+		kfree(zpool);
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	rwlock_init(&zpool->rb_lock);
+	zpool->rb_root = RB_ROOT;
+
+	/* Add to pool list */
+	for (ret = 0; ret < MAX_ZCACHE_POOLS; ret++)
+		if (!zcache.pools[ret])
+			break;
+	zcache.pools[ret] = zpool;
+	zcache.num_pools++;
+	pr_info("New pool created id:%d\n", ret);
+
+out_unlock:
+	spin_unlock(&zcache.pool_lock);
+out:
+	return ret;
+}
+
+static int zcache_cleancache_init_fs(size_t pagesize)
+{
+	int ret;
+
+	if (pagesize != PAGE_SIZE) {
+		pr_info("Unsupported page size: %zu", pagesize);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = zcache_create_pool();
+	if (ret < 0) {
+		pr_info("Failed to create new pool\n");
+		ret = -ENOMEM;
+		goto out;
+	}
+
+out:
+	return ret;
+}
+
+static int zcache_cleancache_init_shared_fs(char *uuid, size_t pagesize)
+{
+	/* shared pools are unsupported and map to private */
+	return zcache_cleancache_init_fs(pagesize);
+}
+
+static struct cleancache_ops zcache_cleancache_ops = {
+	.put_page = zcache_cleancache_store_page,
+	.get_page = zcache_cleancache_load_page,
+	.invalidate_page = zcache_cleancache_flush_page,
+	.invalidate_inode = zcache_cleancache_flush_inode,
+	.invalidate_fs = zcache_cleancache_flush_fs,
+	.init_shared_fs = zcache_cleancache_init_shared_fs,
+	.init_fs = zcache_cleancache_init_fs
+};
+
+/*********************************
+* debugfs functions
+**********************************/
+#ifdef CONFIG_DEBUG_FS
+#include <linux/debugfs.h>
+
+static struct dentry *zcache_cleancache_debugfs_root;
+
+static int __init zcache_debugfs_init(void)
+{
+	if (!debugfs_initialized())
+		return -ENODEV;
+
+	if (!zcache_nocleancache) {
+		zcache_cleancache_debugfs_root =
debugfs_create_dir("zcache_cleancache", NULL);
+		if (!zcache_cleancache_debugfs_root)
+			return -ENOMEM;
+
+		debugfs_create_u64("pool_limit_hit", S_IRUGO,
+				zcache_cleancache_debugfs_root, &zcache_cleancache_pool_limit_hit);
+		debugfs_create_u64("reclaim_fail", S_IRUGO,
+				zcache_cleancache_debugfs_root, &zcache_cleancache_reclaim_fail);
+		debugfs_create_u64("reject_alloc_fail", S_IRUGO,
+				zcache_cleancache_debugfs_root, &zcache_cleancache_zbud_alloc_fail);
+		debugfs_create_u64("written_back_pages", S_IRUGO,
+				zcache_cleancache_debugfs_root, &zcache_cleancache_written_back_pages);
+		debugfs_create_u64("duplicate_entry", S_IRUGO,
+				zcache_cleancache_debugfs_root, &zcache_cleancache_dup_entry);
+		debugfs_create_u64("pool_pages", S_IRUGO,
+				zcache_cleancache_debugfs_root, &zcache_cleancache_pool_pages);
+		debugfs_create_atomic_t("stored_pages", S_IRUGO,
+				zcache_cleancache_debugfs_root, &zcache_cleancache_stored_pages);
+	}
+	return 0;
+}
+
+static void __exit zcache_debugfs_exit(void)
+{
+	debugfs_remove_recursive(zcache_cleancache_debugfs_root);
+}
+#else
+static int __init zcache_debugfs_init(void)
+{
+	return 0;
+}
+static void __exit zcache_debugfs_exit(void)
+{
+}
+#endif
+
+/*********************************
+* module init and exit
+**********************************/
+static int __init init_zcache(void)
+{
+	if (!zcache_enabled)
+		return 0;
+
+	pr_info("loading zcache..\n");
+	if (!zcache_nocleancache)
+		if (zcache_cleancache_entry_cache_create()) {
+			pr_err("entry cache creation failed\n");
+			goto error;
+		}
+
+	if (zcache_comp_init()) {
+		pr_err("compressor initialization failed\n");
+		goto compfail;
+	}
+	if (zcache_cpu_init()) {
+		pr_err("per-cpu initialization failed\n");
+		goto pcpufail;
+	}
+
+	spin_lock_init(&zcache.pool_lock);
+	if (!zcache_nocleancache)
+		cleancache_register_ops(&zcache_cleancache_ops);
+
+	if (zcache_debugfs_init())
+		pr_warn("debugfs initialization failed\n");
+	return 0;
+pcpufail:
+	zcache_comp_exit();
+compfail:
+	zcache_cleancache_entry_cache_destory();
+error:
+	return -ENOMEM;
+}
+/* must be late so crypto has time to come up */
+late_initcall(init_zcache);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Bob Liu <bob.liu@xxxxxxxxxx>");
+MODULE_DESCRIPTION("Compressed cache for clean file pages");
-- 
1.7.10.4


-- 
Regards,
-Bob


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>




[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]