[PATCH 07/12] ZDM: New DM target 'zoned'

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



ZDM presents a traditional block device for ZBC/ZAC zoned devices.

User space utilities in zdm-tools for creating, repairing and restore
DM instances at: https://github.com/Seagate

Signed-off-by: Shaun Tancheff <shaun.tancheff@xxxxxxxxxxx>
---
 Documentation/device-mapper/zoned.txt |   75 +
 MAINTAINERS                           |    7 +
 drivers/md/Kconfig                    |   11 +
 drivers/md/Makefile                   |    1 +
 drivers/md/dm-zoned.c                 | 2535 ++++++++++++
 drivers/md/dm-zoned.h                 |  714 ++++
 drivers/md/libzoned.c                 | 7149 +++++++++++++++++++++++++++++++++
 7 files changed, 10492 insertions(+)
 create mode 100644 Documentation/device-mapper/zoned.txt
 create mode 100644 drivers/md/dm-zoned.c
 create mode 100644 drivers/md/dm-zoned.h
 create mode 100644 drivers/md/libzoned.c

diff --git a/Documentation/device-mapper/zoned.txt b/Documentation/device-mapper/zoned.txt
new file mode 100644
index 0000000..2d1db28
--- /dev/null
+++ b/Documentation/device-mapper/zoned.txt
@@ -0,0 +1,75 @@
+Overview of Host Aware ZBC/ZAC Device Mapper
+  - Zone size (256MiB)
+  - Reset WP, Open Zone, Close Zone, Get Zone Info ...
+
+The Zoned DM treats a zoned device as a collection of 1024 zones 256GiB,
+referred to internally as 'megazones' as with zoned devices the last
+megazone may be less than 1024 zones in size. If the last zone is of
+different size it is ignored.
+
+What that means is that drives which report SAME=0 are not supported
+and likely won't be supported within this architecture. However drives
+which report a SAME code of: all same, last differs or same length
+different types, would all be supported by this architecture.
+
+The initial implementation focuses on drives with same sized zones of
+256MB which is 65536 4k blocks. In future the zone size of 256MB will
+be relaxed to allow any size of zone as long as they are all the same.
+
+Internally all addressing is on 4k boundaries. Currently a 4k PAGE_SIZE is
+assumed. Architectures with 8k (or other) PAGE_SIZE have not been tested
+and are likly broken at the moment.
+
+Host Managed drives should work if the zone type at the start of the partition
+is Conventional, or Preferred.
+
+
+Megazones:
+	Each megazone is managed independently and partitioned into
+	meta data and data. The Device Mapper Meta Data is logically
+	located in the 2nd and 3rd zones of the megazone. The 1st and
+	2nd zones are reserved for the megazone's minimally relocatable
+	super block which must be the first block of the first or
+	second zone. The most recent is determined by the generation
+	number embedded in the super block. The meta data is sized
+	for two zones and logically located in sectors non-addressable
+	to the upper device. The actual storage of the meta data is pooled
+	with the data using the same mapping scheme.
+	The device mapper internally is a COW device with a 4k per block
+	addressing scheme. There are some fix-ups to handle non-4k aligned
+	requests to support applications which read and write in 512 byte
+	blocks, however it is still desirable to submit patches for these
+	subsystems assuming the respective maintainers are willing to
+	accept 4k alignment patches.
+
+Address space:
+	The zoned device mapper presents a smaller block device than
+	the amount of data available on the physical media. The extra
+	space is used to hold the meta data needed for managing the
+	data being stored on the drive performing COW block [re]mapping.
+	The 'shrink' is done by appropriately sizing the device via
+	dmsetup.
+	See the zdmadm utility will detect and size the device appropriaty.
+
+Map Strategy:
+	Map incoming sector onto device mapper sector space.
+
+Read Strategy:
+	Check each block requested in the bio to determine if the data
+	blocks are consecutively stored on disk. Pass as much per-bio
+	as possible through to the backing block device.
+
+Write Strategy:
+	Allocate space for entire bio on the backing block device
+	redirecting all incoming write requests to the most recently
+	written zone until the zone is filled or the bio is too large
+	to fit and a new zone is opened. Note that if the zone is not
+	filled this zone will likely become used by meta data writes
+	which are typically single blocks.
+
+Sync Strategy:
+	On SYNC bios all the meta data need to restore the zoned device
+	mapper for disk is written to one of the well known zones at
+	the beginning of the mega zone. Data consistency is only
+	'guaranteed' to be on-disk and consistent following sync
+	events [same as ext4].
diff --git a/MAINTAINERS b/MAINTAINERS
index c6ad972..5a79f49 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12182,6 +12182,13 @@ L:	zd1211-devs@xxxxxxxxxxxxxxxxxxxxx (subscribers-only)
 S:	Maintained
 F:	drivers/net/wireless/zydas/zd1211rw/
 
+ZDM ZONED DEVICE MAPPER TARGET
+M:	Shaun Tancheff <shaun.tancheff@xxxxxxxxxxx>
+L:	dm-devel@xxxxxxxxxx
+S:	Maintained
+F:	drivers/md/dm-zoned.*
+F:	drivers/md/libzoned.c
+
 ZPOOL COMPRESSED PAGE STORAGE API
 M:	Dan Streetman <ddstreet@xxxxxxxx>
 L:	linux-mm@xxxxxxxxx
diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
index 0a2e727..d0894e4 100644
--- a/drivers/md/Kconfig
+++ b/drivers/md/Kconfig
@@ -345,6 +345,17 @@ config DM_ERA
          over time.  Useful for maintaining cache coherency when using
          vendor snapshots.
 
+config DM_ZONED
+       tristate "ZDM: Zoned based device target (EXPERIMENTAL)"
+       depends on BLK_DEV_DM
+       default n
+       select LIBCRC32C
+       ---help---
+         dm-zoned provides a rand access block device on top of a
+         ZBC/ZAC block device.
+         Forward writing within zones, garbage collection within zones.
+         Use zdmadm to create, repair and/or restore ZDM instances.
+
 config DM_MIRROR
        tristate "Mirror target"
        depends on BLK_DEV_DM
diff --git a/drivers/md/Makefile b/drivers/md/Makefile
index 62a6576..953bdda 100644
--- a/drivers/md/Makefile
+++ b/drivers/md/Makefile
@@ -60,6 +60,7 @@ obj-$(CONFIG_DM_CACHE_SMQ)	+= dm-cache-smq.o
 obj-$(CONFIG_DM_CACHE_CLEANER)	+= dm-cache-cleaner.o
 obj-$(CONFIG_DM_ERA)		+= dm-era.o
 obj-$(CONFIG_DM_LOG_WRITES)	+= dm-log-writes.o
+obj-$(CONFIG_DM_ZONED)		+= dm-zoned.o
 
 ifeq ($(CONFIG_DM_UEVENT),y)
 dm-mod-objs			+= dm-uevent.o
diff --git a/drivers/md/dm-zoned.c b/drivers/md/dm-zoned.c
new file mode 100644
index 0000000..4e04464
--- /dev/null
+++ b/drivers/md/dm-zoned.c
@@ -0,0 +1,2535 @@
+/*
+ * Kernel Device Mapper for abstracting ZAC/ZBC devices as normal
+ * block devices for linux file systems.
+ *
+ * Copyright (C) 2015 Seagate Technology PLC
+ *
+ * Written by:
+ * Shaun Tancheff <shaun.tancheff@xxxxxxxxxxx>
+ *
+ * This file is licensed under  the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include "dm.h"
+#include <linux/dm-io.h>
+#include <linux/init.h>
+#include <linux/mempool.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/random.h>
+#include <linux/crc32c.h>
+#include <linux/crc16.h>
+#include <linux/sort.h>
+#include <linux/ctype.h>
+#include <linux/types.h>
+#include <linux/blkzoned_api.h>
+#include <linux/timer.h>
+#include <linux/delay.h>
+#include <linux/kthread.h>
+#include <linux/freezer.h>
+#include <linux/proc_fs.h>
+#include <linux/seq_file.h>
+#include <linux/kfifo.h>
+#include <linux/bsearch.h>
+#include "dm-zoned.h"
+
+#define PRIu64 "llu"
+#define PRIx64 "llx"
+#define PRId32 "d"
+#define PRIx32 "x"
+#define PRIu32 "u"
+
+#define BIOSET_RESV 4
+
+/**
+ * _zdisk() - Return a pretty ZDM name.
+ * @znd: ZDM Instance
+ *
+ * Return: ZDM/backing device pretty name.
+ */
+static inline char *_zdisk(struct zoned *znd)
+{
+	return znd->bdev_name;
+}
+
+#define Z_ERR(znd, fmt, arg...) \
+	pr_err("dm-zoned(%s): " fmt "\n", _zdisk(znd), ## arg)
+
+#define Z_INFO(znd, fmt, arg...) \
+	pr_info("dm-zoned(%s): " fmt "\n", _zdisk(znd), ## arg)
+
+#define Z_DBG(znd, fmt, arg...) \
+	pr_debug("dm-zoned(%s): " fmt "\n", _zdisk(znd), ## arg)
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+static void do_io_work(struct work_struct *work);
+static int block_io(struct zoned *, enum dm_io_mem_type, void *, sector_t,
+		    unsigned int, int, int);
+static int znd_async_io(struct zoned *znd,
+			enum dm_io_mem_type dtype,
+			void *data,
+			sector_t block, unsigned int nDMsect, int rw, int queue,
+			io_notify_fn callback, void *context);
+static int zoned_bio(struct zoned *znd, struct bio *bio);
+static int zoned_map_write(struct zoned *znd, struct bio*, u64 s_zdm);
+static sector_t get_dev_size(struct dm_target *ti);
+static int dmz_reset_wp(struct zoned *znd, u64 z_id);
+static int dmz_open_zone(struct zoned *znd, u64 z_id);
+static int dmz_close_zone(struct zoned *znd, u64 z_id);
+static u32 dmz_report_count(struct zoned *znd, void *report, size_t bufsz);
+static int dmz_report_zones(struct zoned *znd, u64 z_id,
+			    struct page *pgs, size_t bufsz);
+static void activity_timeout(unsigned long data);
+static void zoned_destroy(struct zoned *);
+static int gc_can_cherrypick(struct zoned *znd, u32 sid, int delay, gfp_t gfp);
+static void bg_work_task(struct work_struct *work);
+static void on_timeout_activity(struct zoned *znd, int delay);
+static int zdm_create_proc_entries(struct zoned *znd);
+static void zdm_remove_proc_entries(struct zoned *znd);
+
+/**
+ * get_bdev_bd_inode() - Get primary backing device inode
+ * @znd: ZDM Instance
+ *
+ * Return: backing device inode
+ */
+static inline struct inode *get_bdev_bd_inode(struct zoned *znd)
+{
+	return znd->dev->bdev->bd_inode;
+}
+
+#include "libzoned.c"
+
+#define BIO_CACHE_SECTORS (IO_VCACHE_PAGES * Z_BLOCKS_PER_DM_SECTOR)
+
+/**
+ * bio_stream() - Decode stream id from BIO.
+ * @znd: ZDM Instance
+ *
+ * Return: stream_id
+ */
+static inline u32 bio_stream(struct bio *bio)
+{
+	u32 stream_id = 0x40;
+
+	/*
+	 * Since adding stream id to a BIO is not yet in mainline we just
+	 * use this heuristic to try to skip unnecessary co-mingling of data.
+	 */
+
+	if (bio->bi_rw & REQ_META)
+		stream_id = 0xff;
+
+	return stream_id;
+}
+
+/**
+ * zoned_map_discard() - Return a pretty ZDM name.
+ * @znd: ZDM Instance
+ * @bio: struct bio hold discard information
+ * @s_zdm: tlba being discarded.
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+static int zoned_map_discard(struct zoned *znd, struct bio *bio, u64 s_zdm)
+{
+	int rcode   = DM_MAPIO_SUBMITTED;
+	u64 blks    = bio->bi_iter.bi_size / Z_C4K;
+	int err;
+
+	if (znd->is_empty)
+		goto cleanup_out;
+
+	err = z_mapped_discard(znd, s_zdm, blks, CRIT);
+	if (err < 0) {
+		rcode = err;
+		goto out;
+	}
+
+cleanup_out:
+	bio->bi_iter.bi_sector = 8;
+	bio_endio(bio);
+out:
+	return rcode;
+}
+
+/**
+ * is_non_wp_zone() - Test zone # to see if it flagged as conventional.
+ * @znd: ZDM Instance
+ * @z_id: Zone #
+ *
+ * Return: 1 if conventional zone. 0 if sequentional write zone.
+ */
+static int is_non_wp_zone(struct zoned *znd, u64 z_id)
+{
+	u32 gzoff = z_id % 1024;
+	struct meta_pg *wpg = &znd->wp[z_id >> 10];
+	u32 wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+
+	return (wp & Z_WP_NON_SEQ) ? 1 : 0;
+}
+
+/**
+ */
+struct zone_action {
+	struct work_struct work;
+	struct zoned *znd;
+	u64 s_addr;
+	unsigned long bi_rw;
+	int wp_err;
+};
+
+/**
+ * do_zone_action_work() - Issue a 'zone action' to the backing device.
+ * @work: Work to do.
+ */
+static void do_zone_action_work(struct work_struct *work)
+{
+	struct zone_action *za = container_of(work, struct zone_action, work);
+	struct zoned *znd = za->znd;
+	struct block_device *bdev = znd->dev->bdev;
+	const gfp_t gfp = GFP_KERNEL;
+
+	/* Explicitly work on device lbas not partition offsets. */
+	if (bdev != bdev->bd_contains)
+		bdev = bdev->bd_contains;
+
+	za->wp_err = blkdev_issue_zone_action(bdev, za->bi_rw, za->s_addr, gfp);
+}
+
+/**
+ * dmz_zone_action() - Issue a 'zone action' to the backing device (via worker).
+ * @znd: ZDM Instance
+ * @z_id: Zone # to open.
+ * @rw: One of REQ_OPEN_ZONE, REQ_CLOSE_ZONE, or REQ_RESET_ZONE.
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static int dmz_zone_action(struct zoned *znd, u64 z_id, unsigned long rw)
+{
+	int wp_err = 0;
+
+	if (is_non_wp_zone(znd, z_id))
+		return wp_err;
+
+	if (znd->bdev_is_zoned) {
+		u64 z_offset = zone_to_sector(z_id + znd->zdstart);
+		struct zone_action za = {
+			.znd = znd,
+			.bi_rw = rw,
+			.s_addr = z_offset,
+			.wp_err = 0,
+		};
+		if (znd->ata_passthrough)
+			za.bi_rw |= REQ_META;
+		/*
+		 * Issue the synchronous I/O from a different thread
+		 * to avoid generic_make_request recursion.
+		 */
+		INIT_WORK_ONSTACK(&za.work, do_zone_action_work);
+		queue_work(znd->zone_action_wq, &za.work);
+		flush_workqueue(znd->zone_action_wq);
+		destroy_work_on_stack(&za.work);
+		wp_err = za.wp_err;
+
+		if (wp_err) {
+			Z_ERR(znd, "Zone Cmd: LBA: %" PRIx64
+			      " [Z:%" PRIu64 "] -> %d failed.",
+			      za.s_addr, z_id, wp_err);
+			Z_ERR(znd, "ZAC/ZBC support disabled.");
+			znd->bdev_is_zoned = 0;
+			wp_err = -ENOTSUPP;
+		}
+	}
+	return wp_err;
+}
+
+/**
+ * dmz_reset_wp() - Reset write pointer for zone z_id.
+ * @znd: ZDM Instance
+ * @z_id: Zone # to reset.
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static int dmz_reset_wp(struct zoned *znd, u64 z_id)
+{
+	return dmz_zone_action(znd, z_id, REQ_RESET_ZONE);
+}
+
+/**
+ * dmz_open_zone() - Open zone for writing.
+ * @znd: ZDM Instance
+ * @z_id: Zone # to open.
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static int dmz_open_zone(struct zoned *znd, u64 z_id)
+{
+	if (!znd->issue_open_zone)
+		return 0;
+	return dmz_zone_action(znd, z_id, REQ_OPEN_ZONE);
+
+}
+
+/**
+ * dmz_close_zone() - Close zone to writing.
+ * @znd: ZDM Instance
+ * @z_id: Zone # to close.
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static int dmz_close_zone(struct zoned *znd, u64 z_id)
+{
+	if (!znd->issue_close_zone)
+		return 0;
+	return dmz_zone_action(znd, z_id, REQ_CLOSE_ZONE);
+}
+
+/**
+ * dmz_report_count() - Read number of zones returned.
+ * @znd: ZDM Instance
+ * @rpt_in: Report data.
+ * @bufsz: size of space allocated for report
+ *
+ * Return: Number of zones in report.
+ */
+static u32 dmz_report_count(struct zoned *znd, void *rpt_in, size_t bufsz)
+{
+	u32 count;
+	u32 max_count = (bufsz - sizeof(struct bdev_zone_report))
+		      /	 sizeof(struct bdev_zone_descriptor);
+
+	if (znd->ata_passthrough) {
+		struct bdev_zone_report_le *report = rpt_in;
+
+		/* ZAC: ata results are little endian */
+		if (max_count > le32_to_cpu(report->descriptor_count))
+			report->descriptor_count = cpu_to_le32(max_count);
+		count = le32_to_cpu(report->descriptor_count);
+	} else {
+		struct bdev_zone_report *report = rpt_in;
+
+		/* ZBC: scsi results are big endian */
+		if (max_count > be32_to_cpu(report->descriptor_count))
+			report->descriptor_count = cpu_to_be32(max_count);
+		count = be32_to_cpu(report->descriptor_count);
+	}
+	return count;
+}
+
+/**
+ * dmz_report_zones() - issue report zones from z_id zones after zdstart
+ * @znd: ZDM Instance
+ * @z_id: Zone past zdstart
+ * @report: structure filled
+ * @bufsz: kmalloc()'d space reserved for report
+ *
+ * Return: -ENOTSUPP or 0 on success
+ */
+static int dmz_report_zones(struct zoned *znd, u64 z_id,
+			    struct page *pgs, size_t bufsz)
+{
+	int wp_err = -ENOTSUPP;
+
+	if (znd->bdev_is_zoned) {
+		u8  opt = ZOPT_NON_SEQ_AND_RESET;
+		unsigned long bi_rw = 0;
+		struct block_device *bdev = znd->dev->bdev;
+		u64 s_addr = zone_to_sector(z_id + znd->zdstart);
+
+		if (bdev != bdev->bd_contains)
+			s_addr -= znd->start_sect << Z_SHFT4K;
+
+		if (znd->ata_passthrough)
+			bi_rw = REQ_META;
+
+		wp_err = blkdev_issue_zone_report(bdev, bi_rw, s_addr, opt,
+						  pgs, bufsz, GFP_KERNEL);
+		if (wp_err) {
+			Z_ERR(znd, "Report Zones: LBA: %" PRIx64
+			      " [Z:%" PRIu64 " -> %d failed.",
+			      s_addr, z_id + znd->zdstart, wp_err);
+			Z_ERR(znd, "ZAC/ZBC support disabled.");
+			znd->bdev_is_zoned = 0;
+			wp_err = -ENOTSUPP;
+		}
+	}
+	return wp_err;
+}
+
+/**
+ * is_conventional() - Determine if zone is conventional.
+ * @dentry: Zone descriptor entry.
+ *
+ * Return: 1 if zone type is conventional.
+ */
+static inline int is_conventional(struct bdev_zone_descriptor *dentry)
+{
+	return (ZTYP_CONVENTIONAL == (dentry->type & 0x0F)) ? 1 : 0;
+}
+
+/**
+ * is_zone_reset() - Determine if zone is reset / ready for writing.
+ * @dentry: Zone descriptor entry.
+ *
+ * Return: 1 if zone condition is empty or zone type is conventional.
+ */
+static inline int is_zone_reset(struct bdev_zone_descriptor *dentry)
+{
+	u8 type = dentry->type & 0x0F;
+	u8 cond = (dentry->flags & 0xF0) >> 4;
+
+	return (ZCOND_ZC1_EMPTY == cond || ZTYP_CONVENTIONAL == type) ? 1 : 0;
+}
+
+/**
+ * get_wp_from_descriptor() - Decode write pointer as # of blocks from start
+ * @znd: ZDM Instance
+ * @dentry_in: Zone descriptor entry.
+ *
+ * Return: Write Pointer as number of blocks from start of zone.
+ */
+static inline u32 get_wp_from_descriptor(struct zoned *znd, void *dentry_in)
+{
+	u32 wp = 0;
+
+	/*
+	 * If ATA passthrough was used then ZAC results are little endian.
+	 * otherwise ZBC results are big endian.
+	 */
+
+	if (znd->ata_passthrough) {
+		struct bdev_zone_descriptor_le *lil = dentry_in;
+
+		wp = le64_to_cpu(lil->lba_wptr) - le64_to_cpu(lil->lba_start);
+	} else {
+		struct bdev_zone_descriptor *big = dentry_in;
+
+		wp = be64_to_cpu(big->lba_wptr) - be64_to_cpu(big->lba_start);
+	}
+	return wp;
+}
+
+/**
+ * _dec_wp_avail_by_lost() - Update free count due to lost/unusable blocks.
+ * @wpg: Write pointer metadata page.
+ * @gzoff: Zone entry in page.
+ * @lost: Number of blocks 'lost'.
+ */
+static inline
+void _dec_wp_avail_by_lost(struct meta_pg *wpg, u32 gzoff, u32 lost)
+{
+	wpg->zf_est[gzoff] = cpu_to_le32(le32_to_cpu(wpg->zf_est[gzoff])+lost);
+}
+
+/**
+ * zoned_wp_sync() - Re-Sync expected WP location with drive
+ * @znd: ZDM Instance
+ * @reset_non_empty: Reset the non-empty zones.
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static int zoned_wp_sync(struct zoned *znd, int reset_non_empty)
+{
+	int rcode = 0;
+	u32 rcount = 0;
+	u32 iter;
+	struct bdev_zone_report *report = NULL;
+	int order = REPORT_ORDER;
+	size_t bufsz = REPORT_FILL_PGS * Z_C4K;
+	struct page *pgs = alloc_pages(GFP_KERNEL, order);
+
+	if (pgs)
+		report = page_address(pgs);
+
+	if (!report) {
+		rcode = -ENOMEM;
+		goto out;
+	}
+
+	for (iter = 0; iter < znd->data_zones; iter++) {
+		u32 entry = iter % 4096;
+		u32 gzno  = iter >> 10;
+		u32 gzoff = iter & ((1 << 10) - 1);
+		struct meta_pg *wpg = &znd->wp[gzno];
+		struct bdev_zone_descriptor *dentry;
+		u32 wp_flgs;
+		u32 wp_at;
+		u32 wp;
+
+		if (entry == 0) {
+			int err = dmz_report_zones(znd, iter, pgs, bufsz);
+
+			if (err) {
+				Z_ERR(znd, "report zones-> %d", err);
+				if (err != -ENOTSUPP)
+					rcode = err;
+				goto out;
+			}
+			rcount = dmz_report_count(znd, report, bufsz);
+		}
+
+		dentry = &report->descriptors[entry];
+		if (reset_non_empty && !is_conventional(dentry)) {
+			int err = 0;
+
+			if (!is_zone_reset(dentry))
+				err = dmz_reset_wp(znd, iter);
+
+			if (err) {
+				Z_ERR(znd, "reset wp-> %d", err);
+				if (err != -ENOTSUPP)
+					rcode = err;
+				goto out;
+			}
+			wp = wp_at = 0;
+			wpg->wp_alloc[gzoff] = cpu_to_le32(0);
+			wpg->zf_est[gzoff] = cpu_to_le32(Z_BLKSZ);
+			continue;
+		}
+
+		wp = get_wp_from_descriptor(znd, dentry);
+		wp >>= Z_SHFT4K; /* 512 sectors to 4k sectors */
+		wp_at = le32_to_cpu(wpg->wp_alloc[gzoff]) & Z_WP_VALUE_MASK;
+		wp_flgs = le32_to_cpu(wpg->wp_alloc[gzoff]) & Z_WP_FLAGS_MASK;
+
+		if (is_conventional(dentry)) {
+			wp = wp_at;
+			wp_flgs |= Z_WP_NON_SEQ;
+		} else {
+			wp_flgs &= ~Z_WP_NON_SEQ;
+		}
+
+		if (wp > wp_at) {
+			u32 lost = wp - wp_at;
+
+			wp_at = wp;
+			_dec_wp_avail_by_lost(wpg, gzoff, lost);
+
+			Z_ERR(znd, "Z#%u z:%x [wp:%x rz:%x] lost %u blocks.",
+			      iter, gzoff, wp_at, wp, lost);
+		}
+		wpg->wp_alloc[gzoff] = cpu_to_le32(wp_at|wp_flgs);
+	}
+
+out:
+	if (pgs)
+		__free_pages(pgs, order);
+
+	return rcode;
+}
+
+#if USE_KTHREAD
+
+static inline int stop_or_data(struct zoned *znd)
+{
+	if (kthread_should_stop())
+		return 1;
+	return kfifo_is_empty(&znd->bio_fifo) ? 0 : 1;
+}
+
+static int znd_bio_kthread(void *arg)
+{
+	int err = 0;
+	struct zoned *znd = (struct zoned *)arg;
+
+	Z_ERR(znd, "znd_bio_kthread [started]");
+
+	while (!kthread_should_stop()) {
+		struct bio *bio = NULL;
+		int rcode;
+
+		if (kfifo_is_empty(&znd->bio_fifo)) {
+			wake_up(&znd->wait_fifo);
+			wait_event_freezable(znd->wait_bio, stop_or_data(znd));
+			continue;
+		}
+		if (!kfifo_get(&znd->bio_fifo, &bio)) {
+			wake_up(&znd->wait_fifo);
+			continue;
+		}
+		if (kfifo_avail(&znd->bio_fifo) > 5)
+			wake_up(&znd->wait_fifo);
+
+		rcode = zoned_bio(znd, bio);
+		if (rcode == DM_MAPIO_REMAPPED)
+			rcode = DM_MAPIO_SUBMITTED;
+
+		if (rcode < 0) {
+			znd->meta_result = err = rcode;
+			goto out;
+		}
+	}
+	err = 0;
+	Z_ERR(znd, "znd_bio_kthread [stopped]");
+
+out:
+	return err;
+}
+
+/**
+ * zoned_map() - Handle an incoming BIO
+ * @ti: Device Mapper Target Instance
+ * @bio: The BIO to disposition.
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static int zoned_map(struct dm_target *ti, struct bio *bio)
+{
+	struct zoned *znd = ti->private;
+	int rcode = DM_MAPIO_REQUEUE;
+
+	while (kfifo_put(&znd->bio_fifo, bio) == 0) {
+		wake_up_process(znd->bio_kthread);
+		wake_up(&znd->wait_bio);
+		wait_event_freezable(znd->wait_fifo,
+			kfifo_avail(&znd->bio_fifo) > 0);
+	}
+	rcode = DM_MAPIO_SUBMITTED;
+	wake_up_process(znd->bio_kthread);
+	wake_up(&znd->wait_bio);
+
+	return rcode;
+}
+
+#else
+
+/**
+ * zoned_map() - Handle an incoming BIO
+ * @ti: Device Mapper Target Instance
+ * @bio: The BIO to disposition.
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static int zoned_map(struct dm_target *ti, struct bio *bio)
+{
+	struct zoned *znd = ti->private;
+	int err = zoned_bio(znd, bio);
+
+	if (err < 0) {
+		znd->meta_result = err;
+		err = 0;
+	}
+
+	return err;
+}
+
+#endif
+
+/**
+ * zoned_actual_size() - Set number of 4k blocks available on block device.
+ * @ti: Device Mapper Target Instance
+ * @znd: ZDM Instance
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static void zoned_actual_size(struct dm_target *ti, struct zoned *znd)
+{
+	znd->nr_blocks = i_size_read(get_bdev_bd_inode(znd)) / Z_C4K;
+}
+
+/**
+ * zoned_ctr() - Create a ZDM Instance from DM Target Instance and args.
+ * @ti: Device Mapper Target Instance
+ * @argc: Number of args to handle.
+ * @argv: args to handle.
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static int zoned_ctr(struct dm_target *ti, unsigned argc, char **argv)
+{
+	const int reset_non_empty = 0;
+	int create = 0;
+	int force = 0;
+	int zbc_probe = 1;
+	int zac_probe = 1;
+	int r;
+	struct zoned *znd;
+	long long first_data_zone = 0;
+	long long mz_md_provision = MZ_METADATA_ZONES;
+
+	BUILD_BUG_ON(Z_C4K != (sizeof(struct map_sect_to_lba) * Z_UNSORTED));
+	BUILD_BUG_ON(Z_C4K != (sizeof(struct io_4k_block)));
+	BUILD_BUG_ON(Z_C4K != (sizeof(struct mz_superkey)));
+
+	znd = ZDM_ALLOC(NULL, sizeof(*znd), KM_00, NORMAL);
+	if (!znd) {
+		ti->error = "Error allocating zoned structure";
+		return -ENOMEM;
+	}
+
+	znd->trim = 1;
+
+	if (argc < 1) {
+		ti->error = "Invalid argument count";
+		return -EINVAL;
+	}
+
+	for (r = 1; r < argc; r++) {
+		if (isdigit(*argv[r])) {
+			int krc = kstrtoll(argv[r], 0, &first_data_zone);
+
+			if (krc != 0) {
+				DMERR("Failed to parse %s: %d", argv[r], krc);
+				first_data_zone = 0;
+			}
+		}
+		if (!strcasecmp("create", argv[r]))
+			create = 1;
+		if (!strcasecmp("load", argv[r]))
+			create = 0;
+		if (!strcasecmp("force", argv[r]))
+			force = 1;
+		if (!strcasecmp("nozbc", argv[r]))
+			zbc_probe = 0;
+		if (!strcasecmp("nozac", argv[r]))
+			zac_probe = 0;
+		if (!strcasecmp("discard", argv[r]))
+			znd->trim = 1;
+		if (!strcasecmp("nodiscard", argv[r]))
+			znd->trim = 0;
+
+		if (!strncasecmp("reserve=", argv[r], 8)) {
+			long long mz_resv;
+			int krc = kstrtoll(argv[r] + 8, 0, &mz_resv);
+
+			if (krc == 0) {
+				if (mz_resv > mz_md_provision)
+					mz_md_provision = mz_resv;
+			} else {
+				DMERR("Reserved 'FAILED TO PARSE.' %s: %d",
+					argv[r]+8, krc);
+				mz_resv = 0;
+			}
+		}
+	}
+
+	znd->ti = ti;
+	ti->private = znd;
+	znd->zdstart = first_data_zone;
+	znd->mz_provision = mz_md_provision;
+
+	r = dm_get_device(ti, argv[0], FMODE_READ | FMODE_WRITE, &znd->dev);
+	if (r) {
+		ti->error = "Error opening backing device";
+		zoned_destroy(znd);
+		return -EINVAL;
+	}
+
+	if (znd->dev->bdev) {
+		bdevname(znd->dev->bdev, znd->bdev_name);
+		znd->start_sect = get_start_sect(znd->dev->bdev) >> 3;
+	}
+
+	/*
+	 * Set if this target needs to receive flushes regardless of
+	 * whether or not its underlying devices have support.
+	 */
+	ti->flush_supported = true;
+
+	/*
+	 * Set if this target needs to receive discards regardless of
+	 * whether or not its underlying devices have support.
+	 */
+	ti->discards_supported = true;
+
+	/*
+	 * Set if the target required discard bios to be split
+	 * on max_io_len boundary.
+	 */
+	ti->split_discard_bios = false;
+
+	/*
+	 * Set if this target does not return zeroes on discarded blocks.
+	 */
+	ti->discard_zeroes_data_unsupported = false;
+
+	/*
+	 * Set if this target wants discard bios to be sent.
+	 */
+	ti->num_discard_bios = 1;
+
+	if (!znd->trim) {
+		ti->discards_supported = false;
+		ti->num_discard_bios = 0;
+	}
+
+	zoned_actual_size(ti, znd);
+
+	r = do_init_zoned(ti, znd);
+	if (r) {
+		ti->error = "Error in zoned init";
+		zoned_destroy(znd);
+		return -EINVAL;
+	}
+
+	znd->issue_open_zone = 1;
+	znd->issue_close_zone = 1;
+	znd->filled_zone = NOZONE;
+
+	if (zac_probe)
+		znd->bdev_is_zoned = znd->ata_passthrough = 1;
+	if (zbc_probe)
+		znd->bdev_is_zoned = 1;
+
+	r = zoned_init_disk(ti, znd, create, force);
+	if (r) {
+		ti->error = "Error in zoned init from disk";
+		zoned_destroy(znd);
+		return -EINVAL;
+	}
+	r = zoned_wp_sync(znd, reset_non_empty);
+	if (r) {
+		ti->error = "Error in zoned re-sync WP";
+		zoned_destroy(znd);
+		return -EINVAL;
+	}
+
+	znd->bio_set = bioset_create(BIOSET_RESV, 0);
+	if (!znd->bio_set)
+		return -ENOMEM;
+
+#if USE_KTHREAD
+	r = kfifo_alloc(&znd->bio_fifo, KFIFO_SIZE, GFP_KERNEL);
+	if (r)
+		return r;
+
+	znd->bio_kthread = kthread_run(znd_bio_kthread, znd, "zdm-io-%s",
+		znd->bdev_name);
+	if (IS_ERR(znd->bio_kthread)) {
+		r = PTR_ERR(znd->bio_kthread);
+		ti->error = "Couldn't alloc kthread";
+		zoned_destroy(znd);
+		return r;
+	}
+#endif
+
+	r = zdm_create_proc_entries(znd);
+	if (r) {
+		ti->error = "Failed to create /proc entries";
+		zoned_destroy(znd);
+		return -EINVAL;
+	}
+
+	mod_timer(&znd->timer, jiffies + msecs_to_jiffies(5000));
+
+	return 0;
+}
+
+/**
+ * zoned_dtr() - Deconstruct a ZDM Instance from DM Target Instance.
+ * @ti: Device Mapper Target Instance
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static void zoned_dtr(struct dm_target *ti)
+{
+	struct zoned *znd = ti->private;
+
+	if (znd->z_sballoc) {
+		struct mz_superkey *key_blk = znd->z_sballoc;
+		struct zdm_superblock *sblock = &key_blk->sblock;
+
+		sblock->flags = cpu_to_le32(0);
+		sblock->csum = sb_crc32(sblock);
+	}
+
+#if USE_KTHREAD
+	wake_up_process(znd->bio_kthread);
+	wait_event(znd->wait_bio, kfifo_is_empty(&znd->bio_fifo));
+	kthread_stop(znd->bio_kthread);
+	kfifo_free(&znd->bio_fifo);
+#endif
+	if (znd->bio_set)
+		bioset_free(znd->bio_set);
+	zdm_remove_proc_entries(znd);
+	zoned_destroy(znd);
+}
+
+
+/**
+ * do_io_work() - Read or write a data from a block device.
+ * @work: Work to be done.
+ */
+static void do_io_work(struct work_struct *work)
+{
+	struct z_io_req_t *req = container_of(work, struct z_io_req_t, work);
+	struct dm_io_request *io_req = req->io_req;
+	unsigned long error_bits = 0;
+
+	req->result = dm_io(io_req, 1, req->where, &error_bits);
+	if (error_bits)
+		DMERR("ERROR: dm_io error: %lx", error_bits);
+}
+
+/**
+ * znd_async_io() - Issue I/O via dm_io async or sync (using worker thread).
+ * @znd: ZDM Instance
+ * @dtype: Type of memory in data
+ * @data: Data for I/O
+ * @block: bLBA for I/O
+ * @nDMsect: Number of 512 byte blocks to read/write.
+ * @rw: READ or WRITE
+ * @queue: if true then use worker thread for I/O and wait.
+ * @callback: callback to use on I/O complete.
+ * context: context to be passed to callback.
+ *
+ * Return 0 on success, otherwise error.
+ */
+static int znd_async_io(struct zoned *znd,
+			enum dm_io_mem_type dtype,
+			void *data,
+			sector_t block, unsigned int nDMsect, int rw, int queue,
+			io_notify_fn callback, void *context)
+{
+	unsigned long error_bits = 0;
+	int rcode;
+	struct dm_io_region where = {
+		.bdev = znd->dev->bdev,
+		.sector = block,
+		.count = nDMsect,
+	};
+	struct dm_io_request io_req = {
+		.bi_rw = rw,
+		.mem.type = dtype,
+		.mem.offset = 0,
+		.mem.ptr.vma = data,
+		.client = znd->io_client,
+		.notify.fn = callback,
+		.notify.context = context,
+	};
+
+	switch (dtype) {
+	case DM_IO_KMEM:
+		io_req.mem.ptr.addr = data;
+		break;
+	case DM_IO_BIO:
+		io_req.mem.ptr.bio = data;
+		where.count = nDMsect;
+		break;
+	case DM_IO_VMA:
+		io_req.mem.ptr.vma = data;
+		break;
+	default:
+		Z_ERR(znd, "page list not handled here ..  see dm-io.");
+		break;
+	}
+
+	if (queue) {
+		struct z_io_req_t req;
+
+		/*
+		 * Issue the synchronous I/O from a different thread
+		 * to avoid generic_make_request recursion.
+		 */
+		INIT_WORK_ONSTACK(&req.work, do_io_work);
+		req.where = &where;
+		req.io_req = &io_req;
+		queue_work(znd->io_wq, &req.work);
+		flush_workqueue(znd->io_wq);
+		destroy_work_on_stack(&req.work);
+
+		rcode = req.result;
+		if (rcode < 0)
+			Z_ERR(znd, "ERROR: dm_io error: %d", rcode);
+		goto done;
+	}
+	rcode = dm_io(&io_req, 1, &where, &error_bits);
+	if (error_bits || rcode < 0)
+		Z_ERR(znd, "ERROR: dm_io error: %d -- %lx", rcode, error_bits);
+
+done:
+	return rcode;
+}
+
+/**
+ * block_io() - Issue sync I/O maybe using using a worker thread.
+ * @znd: ZDM Instance
+ * @dtype: Type of memory in data
+ * @data: Data for I/O
+ * @s: bLBA for I/O [512 byte resolution]
+ * @n: Number of 512 byte blocks to read/write.
+ * @rw: READ or WRITE
+ * @queue: if true then use worker thread for I/O and wait.
+ *
+ * Return 0 on success, otherwise error.
+ */
+static int block_io(struct zoned *znd,
+		    enum dm_io_mem_type dtype,
+		    void *data, sector_t s, unsigned int n, int rw, int queue)
+{
+	return znd_async_io(znd, dtype, data, s, n, rw, queue, NULL, NULL);
+}
+
+/**
+ * read_block() - Issue sync read maybe using using a worker thread.
+ * @ti: Device Mapper Target Instance
+ * @dtype: Type of memory in data
+ * @data: Data for I/O
+ * @lba: bLBA for I/O [4k resolution]
+ * @count: Number of 4k blocks to read/write.
+ * @queue: if true then use worker thread for I/O and wait.
+ *
+ * Return 0 on success, otherwise error.
+ */
+static int read_block(struct dm_target *ti, enum dm_io_mem_type dtype,
+		      void *data, u64 lba, unsigned int count, int queue)
+{
+	struct zoned *znd = ti->private;
+	sector_t block = lba << Z_SHFT4K;
+	unsigned int nDMsect = count << Z_SHFT4K;
+	int rc;
+
+	if (lba >= znd->nr_blocks) {
+		Z_ERR(znd, "Error reading past end of media: %llx.", lba);
+		rc = -EIO;
+		return rc;
+	}
+
+	rc = block_io(znd, dtype, data, block, nDMsect, READ, queue);
+	if (rc) {
+		Z_ERR(znd, "read error: %d -- R: %llx [%u dm sect] (Q:%d)",
+			rc, lba, nDMsect, queue);
+		dump_stack();
+	}
+
+	return rc;
+}
+
+/**
+ * writef_block() - Issue sync write maybe using using a worker thread.
+ * @ti: Device Mapper Target Instance
+ * @dtype: Type of memory in data
+ * @data: Data for I/O
+ * @lba: bLBA for I/O [4k resolution]
+ * @count: Number of 4k blocks to read/write.
+ * @queue: if true then use worker thread for I/O and wait.
+ *
+ * Return 0 on success, otherwise error.
+ */
+static int writef_block(struct dm_target *ti, int rw, enum dm_io_mem_type dtype,
+			void *data, u64 lba, unsigned int count, int queue)
+{
+	struct zoned *znd = ti->private;
+	sector_t block = lba << Z_SHFT4K;
+	unsigned int nDMsect = count << Z_SHFT4K;
+	int rc;
+
+	rc = block_io(znd, dtype, data, block, nDMsect, WRITE, queue);
+	if (rc) {
+		Z_ERR(znd, "write error: %d W: %llx [%u dm sect] (Q:%d)",
+			rc, lba, nDMsect, queue);
+		dump_stack();
+	}
+
+	return rc;
+}
+
+/**
+ * write_block() - Issue sync write maybe using using a worker thread.
+ * @ti: Device Mapper Target Instance
+ * @dtype: Type of memory in data
+ * @data: Data for I/O
+ * @lba: bLBA for I/O [4k resolution]
+ * @count: Number of 4k blocks to read/write.
+ * @queue: if true then use worker thread for I/O and wait.
+ *
+ * Return 0 on success, otherwise error.
+ */
+static int write_block(struct dm_target *ti, enum dm_io_mem_type dtype,
+		       void *data, u64 lba, unsigned int count, int queue)
+{
+	return writef_block(ti, WRITE, dtype, data, lba, count, queue);
+}
+
+/**
+ * struct zsplit_hook - Extra data attached to a hooked bio
+ * @znd: ZDM Instance to update on BIO completion.
+ * @endio: BIO's original bi_end_io handler
+ * @private: BIO's original bi_private data.
+ */
+struct zsplit_hook {
+	struct zoned *znd;
+	bio_end_io_t *endio;
+	void *private;
+};
+
+/**
+ * hook_bio() - Wrapper for hooking bio's endio function.
+ * @znd: ZDM Instance
+ * @bio: Bio to clone and hook
+ * @endiofn: End IO Function to hook with.
+ */
+static int hook_bio(struct zoned *znd, struct bio *split, bio_end_io_t *endiofn)
+{
+	struct zsplit_hook *hook = kmalloc(sizeof(*hook), GFP_NOIO);
+
+	if (!hook)
+		return -ENOMEM;
+
+	/*
+	 * On endio report back to ZDM Instance and restore
+	 * original the bi_private and bi_end_io.
+	 * Since all of our splits are also chain'd we also
+	 * 'know' that bi_private will be the bio we sharded
+	 * and that bi_end_io is the bio_chain_endio helper.
+	 */
+	hook->znd = znd;
+	hook->private = split->bi_private; /* = bio */
+	hook->endio = split->bi_end_io; /* = bio_chain_endio */
+
+	/*
+	 * Now on complete the bio will call endiofn which is 'zsplit_endio'
+	 * and we can record the update WP location and restore the
+	 * original bi_private and bi_end_io
+	 */
+	split->bi_private = hook;
+	split->bi_end_io = endiofn;
+
+	return 0;
+}
+
+
+/**
+ * TODO: On write error such as this we can incr wp-used but we need
+ * to re-queue/re-map the write to a new location on disk?
+ *
+ * sd 0:0:1:0: [sdb] tag#1 FAILED Result: hostbyte=DID_SOFT_ERROR
+ *	driverbyte=DRIVER_OK
+ * sd 0:0:1:0: [sdb] tag#1
+ *	CDB: Write(16) 8a 00 00 00 00 00 06 d4 92 a0 00 00 00 08 00 00
+ *	blk_update_request: I/O error, dev sdb, sector 114594464
+ * exec scsi cmd failed,opcode:133
+ * sdb: command 1 failed
+ * sd 0:0:1:0: [sdb] tag#1
+ *	CDB: Write(16) 8a 00 00 00 00 00 00 01 0a 70 00 00 00 18 00 00
+ * mpt3sas_cm0:    sas_address(0x4433221105000000), phy(5)
+ * mpt3sas_cm0:    enclosure_logical_id(0x500605b0074854d0),slot(6)
+ * mpt3sas_cm0:    enclosure level(0x0000), connector name(     )
+ * mpt3sas_cm0:    handle(0x000a), ioc_status(success)(0x0000), smid(17)
+ * mpt3sas_cm0:    request_len(12288), underflow(12288), resid(-1036288)
+ * mpt3sas_cm0:    tag(65535), transfer_count(1048576), sc->result(0x00000000)
+ * mpt3sas_cm0:    scsi_status(check condition)(0x02),
+ *	scsi_state(autosense valid )(0x01)
+ * mpt3sas_cm0:    [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
+ * Aborting journal on device dm-0-8.
+ * EXT4-fs error (device dm-0):
+ *	ext4_journal_check_start:56: Detected aborted journal
+ * EXT4-fs (dm-0): Remounting filesystem read-only
+ */
+
+/**
+ * _common_endio() - Bio endio tracking for update internal WP.
+ * @bio: Bio being completed.
+ *
+ * Bios that are split for writing are usually split to land on a zone
+ * boundary. Forward the bio along the endio path and update the WP.
+ */
+static void _common_endio(struct zoned *znd, struct bio *bio)
+{
+	u64 lba = bio->bi_iter.bi_sector >> Z_SHFT4K;
+	u32 blks = bio->bi_iter.bi_size / Z_C4K;
+
+	if (bio_data_dir(bio) == WRITE && lba > znd->start_sect) {
+		lba -= znd->start_sect;
+		if (lba > 0)
+			increment_used_blks(znd, lba - 1, blks + 1);
+	}
+}
+
+/**
+ * zoned_endio() - DM bio completion notification.
+ * @ti: DM Target instance.
+ * @bio: Bio being completed.
+ * @err: Error associated with bio.
+ *
+ * Non-split and non-dm_io bios end notification is here.
+ * Update the WP location for WRITE bios.
+ */
+static int zoned_endio(struct dm_target *ti, struct bio *bio, int err)
+{
+	struct zoned *znd = ti->private;
+
+	_common_endio(znd, bio);
+	return 0;
+}
+
+/**
+ * zsplit_endio() - Bio endio tracking for update internal WP.
+ * @bio: Bio being completed.
+ *
+ * Bios that are split for writing are usually split to land on a zone
+ * boundary. Forward the bio along the endio path and update the WP.
+ */
+static void zsplit_endio(struct bio *bio)
+{
+	struct zsplit_hook *hook = bio->bi_private;
+	struct bio *parent = hook->private;
+	struct zoned *znd = hook->znd;
+
+	_common_endio(znd, bio);
+
+	bio->bi_private = hook->private;
+	bio->bi_end_io = hook->endio;
+
+	/* On split bio's we are responsible for de-ref'ing and freeing */
+	bio_put(bio);
+	if (parent)
+		bio_endio(parent);
+
+	/* release our temporary private data */
+	kfree(hook);
+}
+
+/**
+ * zsplit_bio() - Split and chain a bio.
+ * @znd: ZDM Instance
+ * @bio: Bio to split
+ * @sectors: Number of sectors.
+ *
+ * Return: split bio.
+ */
+static struct bio *zsplit_bio(struct zoned *znd, struct bio *bio, int sectors)
+{
+	struct bio *split = bio;
+
+	if (bio_sectors(bio) > sectors) {
+		split = bio_split(bio, sectors, GFP_NOIO, znd->bio_set);
+		if (!split)
+			goto out;
+		bio_chain(split, bio);
+		if (bio_data_dir(bio) == WRITE)
+			hook_bio(znd, split, zsplit_endio);
+	}
+out:
+	return split;
+}
+
+/**
+ * zm_cow() - Read Modify Write to write less than 4k size blocks.
+ * @znd: ZDM Instance
+ * @bio: Bio to write
+ * @s_zdm: tLBA
+ * @blks: number of blocks to RMW (should be 1).
+ * @origin: Current bLBA
+ *
+ * Return: 0 on success, otherwise error.
+ */
+static int zm_cow(struct zoned *znd, struct bio *bio, u64 s_zdm, u32 blks,
+		  u64 origin)
+{
+	struct dm_target *ti = znd->ti;
+	int count = 1;
+	int use_wq = 1;
+	unsigned int bytes = bio_cur_bytes(bio);
+	u8 *data = bio_data(bio);
+	u8 *io = NULL;
+	u16 ua_off = bio->bi_iter.bi_sector & 0x0007;
+	u16 ua_size = bio->bi_iter.bi_size & 0x0FFF;	/* in bytes */
+	u32 mapped = 0;
+	u64 disk_lba = 0;
+
+	if (!znd->cow_block)
+		znd->cow_block = ZDM_ALLOC(znd, Z_C4K, PG_02, CRIT);
+
+	io = znd->cow_block;
+	if (!io)
+		return -EIO;
+
+	disk_lba = z_acquire(znd, Z_AQ_STREAM_ID, blks, &mapped);
+	if (!disk_lba || !mapped)
+		return -ENOSPC;
+
+	while (bytes) {
+		int ioer;
+		unsigned int iobytes = Z_C4K;
+
+		/* ---------------------------------------------------------- */
+		if (origin) {
+			if (s_zdm != znd->cow_addr) {
+				Z_ERR(znd, "Copy block from %llx <= %llx",
+				      origin, s_zdm);
+				ioer = read_block(ti, DM_IO_KMEM, io, origin,
+						count, use_wq);
+				if (ioer)
+					return -EIO;
+
+				znd->cow_addr = s_zdm;
+			} else {
+				Z_ERR(znd, "Cached block from %llx <= %llx",
+				      origin, s_zdm);
+			}
+		} else {
+			memset(io, 0, Z_C4K);
+		}
+
+		if (ua_off)
+			iobytes -= ua_off * 512;
+
+		if (bytes < iobytes)
+			iobytes = bytes;
+
+		Z_ERR(znd, "Moving %u bytes from origin [offset:%u]",
+		      iobytes, ua_off * 512);
+
+		memcpy(io + (ua_off * 512), data, iobytes);
+
+		/* ---------------------------------------------------------- */
+
+		ioer = write_block(ti, DM_IO_KMEM, io, disk_lba, count, use_wq);
+		if (ioer)
+			return -EIO;
+
+		MutexLock(&znd->mz_io_mutex);
+		ioer = z_mapped_addmany(znd, s_zdm, disk_lba, mapped, CRIT);
+		mutex_unlock(&znd->mz_io_mutex);
+		if (ioer) {
+			Z_ERR(znd, "%s: Journal MANY failed.", __func__);
+			return -EIO;
+		}
+		increment_used_blks(znd, disk_lba, mapped);
+
+		data += iobytes;
+		bytes -= iobytes;
+		ua_size -= (ua_size > iobytes) ? iobytes : ua_size;
+		ua_off = 0;
+		disk_lba++;
+
+		if (bytes && (ua_size || ua_off)) {
+			s_zdm++;
+			origin = current_mapping(znd, s_zdm, CRIT);
+		}
+	}
+	bio_endio(bio);
+
+	return DM_MAPIO_SUBMITTED;
+}
+
+/**
+ * Write 4k blocks from cache to lba.
+ * Move any remaining 512 byte blocks to the start of cache and update
+ * the @_blen count is updated
+ */
+static int zm_write_cache(struct zoned *znd, struct io_dm_block *dm_vbuf,
+			  u64 lba, u32 *_blen)
+{
+	int use_wq    = 1;
+	int cached    = *_blen;
+	int blks      = cached >> 3;
+	int sectors   = blks << 3;
+	int remainder = cached - sectors;
+	int err;
+
+	err = write_block(znd->ti, DM_IO_VMA, dm_vbuf, lba, blks, use_wq);
+	if (!err) {
+		if (remainder)
+			memcpy(dm_vbuf[0].data,
+			       dm_vbuf[sectors].data, remainder * 512);
+		*_blen = remainder;
+	}
+	return err;
+}
+
+/**
+ * zm_write_bios() - Map and write bios.
+ * @znd: ZDM Instance
+ * @bio: Bio to be written.
+ * @s_zdm: tLBA for mapping.
+ *
+ * Return: DM_MAPIO_SUBMITTED or negative on error.
+ */
+static int zm_write_bios(struct zoned *znd, struct bio *bio, u64 s_zdm)
+{
+	struct bio *split = NULL;
+	u32 acqflgs = Z_AQ_STREAM_ID | bio_stream(bio);
+	u64 lba     = 0;
+	u32 mapped  = 0;
+	int err     = -EIO;
+	int done    = 0;
+	int sectors;
+	u32 blks;
+
+	do {
+		blks = dm_div_up(bio->bi_iter.bi_size, Z_C4K);
+		lba = z_acquire(znd, acqflgs, blks, &mapped);
+		if (!lba && mapped)
+			lba = z_acquire(znd, acqflgs, mapped, &mapped);
+
+		if (!lba) {
+			if (atomic_read(&znd->gc_throttle) == 0) {
+				err = -ENOSPC;
+				goto out;
+			}
+
+			Z_ERR(znd, "Throttle input ... Mandatory GC.");
+			if (delayed_work_pending(&znd->gc_work)) {
+				mod_delayed_work(znd->gc_wq, &znd->gc_work, 0);
+				flush_delayed_work(&znd->gc_work);
+			}
+			continue;
+		}
+
+		sectors = mapped << Z_SHFT4K;
+		split = zsplit_bio(znd, bio, sectors);
+		if (split == bio)
+			done = 1;
+
+		if (!split) {
+			err = -ENOMEM;
+			goto out;
+		}
+		split->bi_iter.bi_sector = lba << Z_SHFT4K;
+		generic_make_request(split);
+		MutexLock(&znd->mz_io_mutex);
+		err = z_mapped_addmany(znd, s_zdm, lba, mapped, CRIT);
+		mutex_unlock(&znd->mz_io_mutex);
+		if (err) {
+			Z_ERR(znd, "%s: Journal MANY failed.", __func__);
+			err = DM_MAPIO_REQUEUE;
+			goto out;
+		}
+		s_zdm += mapped;
+	} while (!done);
+	err = DM_MAPIO_SUBMITTED;
+
+out:
+	return err;
+}
+
+/**
+ * zm_write_pages() - Copy bio pages to 4k aligned buffer. Write and map buffer.
+ * @znd: ZDM Instance
+ * @bio: Bio to be written.
+ * @s_zdm: tLBA for mapping.
+ *
+ * Return: DM_MAPIO_SUBMITTED or negative on error.
+ */
+static int zm_write_pages(struct zoned *znd, struct bio *bio, u64 s_zdm)
+{
+	u32 blks     = dm_div_up(bio->bi_iter.bi_size, Z_C4K);
+	u64 lba      = 0;
+	u32 blen     = 0; /* total: IO_VCACHE_PAGES * 8 */
+	u32 written  = 0;
+	int avail    = 0;
+	u32 acqflgs  = Z_AQ_STREAM_ID | bio_stream(bio);
+	int err;
+	struct bvec_iter start;
+	struct bvec_iter iter;
+	struct bio_vec bv;
+	struct io_4k_block *io_vcache;
+	struct io_dm_block *dm_vbuf = NULL;
+
+	MutexLock(&znd->vcio_lock);
+	io_vcache = get_io_vcache(znd, CRIT);
+	if (!io_vcache) {
+		Z_ERR(znd, "%s: FAILED to get SYNC CACHE.", __func__);
+		err = -ENOMEM;
+		goto out;
+	}
+
+	dm_vbuf = (struct io_dm_block *)io_vcache;
+
+	/* USE: dm_vbuf for dumping bio pages to disk ... */
+	start = bio->bi_iter; /* struct implicit copy */
+	do {
+		u64 alloc_ori = 0;
+		u32 mcount = 0;
+		u32 mapped = 0;
+
+reacquire:
+		/*
+		 * When lba is zero no blocks were not allocated.
+		 * Retry with the smaller request
+		 */
+		lba = z_acquire(znd, acqflgs, blks - written, &mapped);
+		if (!lba && mapped)
+			lba = z_acquire(znd, acqflgs, mapped, &mapped);
+
+		if (!lba) {
+			if (atomic_read(&znd->gc_throttle) == 0) {
+				err = -ENOSPC;
+				goto out;
+			}
+
+			Z_ERR(znd, "Throttle input ... Mandatory GC.");
+			if (delayed_work_pending(&znd->gc_work)) {
+				mod_delayed_work(znd->gc_wq, &znd->gc_work, 0);
+				flush_delayed_work(&znd->gc_work);
+			}
+			goto reacquire;
+		}
+
+		/* this may be redundant .. if we have lba we have mapped > 0 */
+		if (lba && mapped)
+			avail += mapped * 8; /* claimed pages in dm blocks */
+
+		alloc_ori = lba;
+
+		/* copy [upto mapped] pages to buffer */
+		__bio_for_each_segment(bv, bio, iter, start) {
+			int issue_write = 0;
+			unsigned int boff;
+			void *src;
+
+			if (avail <= 0) {
+				Z_ERR(znd, "%s: TBD: Close Z# %llu",
+					__func__, alloc_ori >> 16);
+				start = iter;
+				break;
+			}
+
+			src = kmap_atomic(bv.bv_page);
+			boff = bv.bv_offset;
+			memcpy(dm_vbuf[blen].data, src + boff, bv.bv_len);
+			kunmap_atomic(src);
+			blen   += bv.bv_len / 512;
+			avail  -= bv.bv_len / 512;
+
+			if ((blen >= (mapped * 8)) ||
+			    (blen >= (BIO_CACHE_SECTORS - 8)))
+				issue_write = 1;
+
+			/*
+			 * If there is less than 1 4k block in out cache,
+			 * send the available blocks to disk
+			 */
+			if (issue_write) {
+				int blks = blen / 8;
+
+				err = zm_write_cache(znd, dm_vbuf, lba, &blen);
+				if (err) {
+					Z_ERR(znd, "%s: bio-> %" PRIx64
+					      " [%d of %d blks] -> %d",
+					      __func__, lba, blen, blks, err);
+					bio->bi_error = err;
+					bio_endio(bio);
+					goto out;
+				}
+				lba     += blks;
+				written += blks;
+				mcount  += blks;
+				mapped  -= blks;
+
+				if (mapped == 0) {
+					bio_advance_iter(bio, &iter, bv.bv_len);
+					start = iter;
+					break;
+				}
+				if (mapped < 0) {
+					Z_ERR(znd, "ERROR: Bad write %"
+					      PRId32 " beyond alloc'd space",
+					      mapped);
+				}
+			}
+		} /* end: __bio_for_each_segment */
+		if ((mapped > 0) && ((blen / 8) > 0)) {
+			int blks = blen / 8;
+
+			err = zm_write_cache(znd, dm_vbuf, lba, &blen);
+			if (err) {
+				Z_ERR(znd, "%s: bio-> %" PRIx64
+				      " [%d of %d blks] -> %d",
+				      __func__, lba, blen, blks, err);
+				bio->bi_error = err;
+				bio_endio(bio);
+				goto out;
+			}
+			lba     += blks;
+			written += blks;
+			mcount  += blks;
+			mapped  -= blks;
+
+			if (mapped < 0) {
+				Z_ERR(znd, "ERROR: [2] Bad write %"
+				      PRId32 " beyond alloc'd space",
+				      mapped);
+			}
+		}
+		MutexLock(&znd->mz_io_mutex);
+		err = z_mapped_addmany(znd, s_zdm, alloc_ori, mcount, CRIT);
+		mutex_unlock(&znd->mz_io_mutex);
+		if (err) {
+			Z_ERR(znd, "%s: Journal MANY failed.", __func__);
+			err = DM_MAPIO_REQUEUE;
+			/*
+			 * FIXME:
+			 * Ending the BIO here is causing a GFP:
+			 -       DEBUG_PAGEALLOC
+			 -    in Workqueue:
+			 -        writeback bdi_writeback_workfn (flush-252:0)
+			 -    backtrace:
+			 -      __map_bio+0x7a/0x280
+			 -      __split_and_process_bio+0x2e3/0x4e0
+			 -      ? __split_and_process_bio+0x22/0x4e0
+			 -      ? generic_start_io_acct+0x5/0x210
+			 -      dm_make_request+0x6b/0x100
+			 -      generic_make_request+0xc0/0x110
+			 -      ....
+			 -
+			 - bio->bi_error = err;
+			 - bio_endio(bio);
+			 */
+			goto out;
+		}
+		increment_used_blks(znd, alloc_ori, mcount);
+
+		if (written < blks)
+			s_zdm += written;
+
+		if (written == blks && blen > 0)
+			Z_ERR(znd, "%s: blen: %d un-written blocks!!",
+			      __func__, blen);
+	} while (written < blks);
+	bio_endio(bio);
+	err = DM_MAPIO_SUBMITTED;
+
+out:
+	put_io_vcache(znd, io_vcache);
+	mutex_unlock(&znd->vcio_lock);
+
+	return err;
+}
+
+/**
+ * is_empty_page() - Scan memory range for any set bits.
+ * @pg: The start of memory to be scanned.
+ * @len: Number of bytes to check (should be long aligned)
+ * Return: 0 if any bits are set, 1 if all bits are 0.
+ */
+static int is_empty_page(void *pg, size_t len)
+{
+	unsigned long *chk = pg;
+	size_t count = len / sizeof(*chk);
+	size_t entry;
+
+	for (entry = 0; entry < count; entry++) {
+		if (chk[entry])
+			return 0;
+	}
+	return 1;
+}
+
+/**
+ * is_zero_bio() - Scan bio to see if all bytes are 0.
+ * @bio: The bio to be scanned.
+ * Return: 1 if all bits are 0. 0 if any bits in bio are set.
+ */
+static int is_zero_bio(struct bio *bio)
+{
+	int is_empty = 0;
+	struct bvec_iter iter;
+	struct bio_vec bv;
+
+	/* Scan bio to determine if it is zero'd */
+	bio_for_each_segment(bv, bio, iter) {
+		unsigned int boff;
+		void *src;
+
+		src = kmap_atomic(bv.bv_page);
+		boff = bv.bv_offset;
+		is_empty = is_empty_page(src + boff, bv.bv_len);
+		kunmap_atomic(src);
+
+		if (!is_empty)
+			break;
+	} /* end: __bio_for_each_segment */
+
+	return is_empty;
+}
+
+/**
+ * is_bio_aligned() - Test bio and bio_vec for 4k aligned pages.
+ * @bio: Bio to be tested.
+ * Return: 1 if bio is 4k aligned, 0 if not.
+ */
+static int is_bio_aligned(struct bio *bio)
+{
+	int aligned = 1;
+	struct bvec_iter iter;
+	struct bio_vec bv;
+
+	bio_for_each_segment(bv, bio, iter) {
+		if ((bv.bv_offset & 0x0FFF) || (bv.bv_len & 0x0FFF)) {
+			aligned = 0;
+			break;
+		}
+	}
+	return aligned;
+}
+
+/**
+ * zoned_map_write() - Write a bio by the fastest safe method.
+ * @znd: ZDM Instance
+ * @bio: Bio to be written
+ * @s_zdm: tLBA for mapping.
+ *
+ * Bios that are less than 4k need RMW.
+ * Bios that are single pages are deduped and written or discarded.
+ * Bios that are multiple pages with 4k aligned bvecs are written as bio(s).
+ * Biso that are multiple pages and mis-algined are copied to an algined buffer
+ * and submitted and new I/O.
+ */
+static int zoned_map_write(struct zoned *znd, struct bio *bio, u64 s_zdm)
+{
+	u32 blks     = dm_div_up(bio->bi_iter.bi_size, Z_C4K);
+	u16 ua_off   = bio->bi_iter.bi_sector & 0x0007;
+	u16 ua_size  = bio->bi_iter.bi_size & 0x0FFF;	/* in bytes */
+	int rcode    = -EIO;
+
+	if (ua_size || ua_off) {
+		u64 origin;
+
+		origin = current_mapping(znd, s_zdm, CRIT);
+		if (origin) {
+			znd->is_empty = 0;
+			rcode = zm_cow(znd, bio, s_zdm, blks, origin);
+		}
+		return rcode;
+	}
+
+	/* on RAID 4/5/6 all writes are 4k */
+	if (blks == 1) {
+		if (is_zero_bio(bio)) {
+			rcode = zoned_map_discard(znd, bio, s_zdm);
+		} else {
+			znd->is_empty = 0;
+			rcode = zm_write_bios(znd, bio, s_zdm);
+		}
+		return rcode;
+	}
+
+	/*
+	 * For larger bios test for 4k alignment.
+	 * When bios are mis-algined we must copy out the
+	 * the mis-algined pages into a new bio and submit.
+	 * [The 4k alignment requests on our queue may be ignored
+	 *  by mis-behaving layers that are not 4k safe].
+	 */
+	znd->is_empty = 0;
+	if (is_bio_aligned(bio))
+		rcode = zm_write_bios(znd, bio, s_zdm);
+	else
+		rcode = zm_write_pages(znd, bio, s_zdm);
+
+	return rcode;
+}
+
+/**
+ * zm_read_bios() - Read bios from device
+ * @znd: ZDM Instance
+ * @bio: Bio to read
+ * @s_zdm: tLBA to read from.
+ *
+ * Return DM_MAPIO_SUBMITTED or negative on error.
+ */
+static int zm_read_bios(struct zoned *znd, struct bio *bio, u64 s_zdm)
+{
+	struct bio *split = NULL;
+	int rcode = DM_MAPIO_SUBMITTED;
+	u64 lba;
+	u64 blba;
+	u32 blks;
+	int sectors;
+	int count;
+	u16 ua_off;
+	u16 ua_size;
+
+	do {
+		blks = dm_div_up(bio->bi_iter.bi_size, Z_C4K);
+		ua_off = bio->bi_iter.bi_sector & 0x0007;
+		ua_size = bio->bi_iter.bi_size & 0x0FFF;	/* in bytes */
+
+		lba = blba = current_mapping(znd, s_zdm, CRIT);
+		if (blba) {
+			for (count = 1; count < blks; count++) {
+				lba = current_mapping(znd, s_zdm + count, CRIT);
+				if (lba != (blba + count))
+					break;
+			}
+		} else {
+			for (count = 1; count < blks; count++) {
+				lba = current_mapping(znd, s_zdm + count, CRIT);
+				if (lba != 0ul)
+					break;
+			}
+		}
+		s_zdm += count;
+		sectors = (count << Z_SHFT4K);
+		if (ua_size)
+			sectors += (ua_size >> SECTOR_SHIFT) - 8;
+
+		split = zsplit_bio(znd, bio, sectors);
+		if (!split) {
+			rcode = -ENOMEM;
+			goto out;
+		}
+		if (blba) {
+			split->bi_iter.bi_sector = (blba << Z_SHFT4K) + ua_off;
+			generic_make_request(split);
+		} else {
+			zero_fill_bio(split);
+			bio_endio(split);
+		}
+	} while (split != bio);
+
+out:
+	return rcode;
+}
+
+/**
+ * zoned_bio() - Handle and incoming BIO.
+ * @znd: ZDM Instance
+ */
+static int zoned_bio(struct zoned *znd, struct bio *bio)
+{
+	bool is_write = (bio_data_dir(bio) == WRITE);
+	u64 s_zdm = (bio->bi_iter.bi_sector >> Z_SHFT4K) + znd->md_end;
+	int rcode = DM_MAPIO_SUBMITTED;
+	struct request_queue *q;
+	int force_sync_now = 0;
+
+	/* map to backing device ... NOT dm-zoned device */
+	bio->bi_bdev = znd->dev->bdev;
+
+	q = bdev_get_queue(bio->bi_bdev);
+	q->queue_flags |= QUEUE_FLAG_NOMERGES;
+
+	if (is_write && znd->meta_result) {
+		if (!(bio->bi_rw & REQ_DISCARD)) {
+			rcode = znd->meta_result;
+			Z_ERR(znd, "MAP ERR (meta): %d", rcode);
+			goto out;
+		}
+	}
+
+	/* check for REQ_FLUSH flag */
+	if (bio->bi_rw & REQ_FLUSH) {
+		bio->bi_rw &= ~(REQ_FLUSH);
+		set_bit(DO_FLUSH, &znd->flags);
+		force_sync_now = 1;
+	}
+	if (bio->bi_rw & REQ_SYNC) {
+		set_bit(DO_SYNC, &znd->flags);
+		force_sync_now = 1;
+	}
+
+	Z_DBG(znd, "%s: s:%"PRIx64" sz:%u -> %s", __func__, s_zdm,
+	      bio->bi_iter.bi_size, is_write ? "W" : "R");
+
+	if (bio->bi_iter.bi_size) {
+		if (bio->bi_rw & REQ_DISCARD) {
+			rcode = zoned_map_discard(znd, bio, s_zdm);
+		} else if (is_write) {
+			const int wait = 1;
+			u32 blks = bio->bi_iter.bi_size >> 12;
+
+			rcode = zoned_map_write(znd, bio, s_zdm);
+			if (rcode == DM_MAPIO_SUBMITTED) {
+				z_discard_partial(znd, blks, CRIT);
+				if (znd->z_gc_free < 5)
+					gc_immediate(znd, !wait, CRIT);
+			}
+		} else {
+			rcode = zm_read_bios(znd, bio, s_zdm);
+		}
+		znd->age = jiffies;
+	}
+
+	if (test_bit(DO_FLUSH, &znd->flags) ||
+	    test_bit(DO_SYNC, &znd->flags) ||
+	    test_bit(DO_JOURNAL_MOVE, &znd->flags) ||
+	    test_bit(DO_MEMPOOL, &znd->flags)) {
+		if (!test_bit(DO_METAWORK_QD, &znd->flags) &&
+		    !work_pending(&znd->meta_work)) {
+			set_bit(DO_METAWORK_QD, &znd->flags);
+			queue_work(znd->meta_wq, &znd->meta_work);
+		}
+	}
+
+	if (force_sync_now && work_pending(&znd->meta_work))
+		flush_workqueue(znd->meta_wq);
+
+out:
+	if (rcode == DM_MAPIO_REMAPPED || rcode == DM_MAPIO_SUBMITTED)
+		goto done;
+
+	if (rcode < 0 || rcode == DM_MAPIO_REQUEUE) {
+		Z_ERR(znd, "MAP ERR: %d", rcode);
+		Z_ERR(znd, "%s: s:%"PRIx64" sz:%u -> %s", __func__, s_zdm,
+		      bio->bi_iter.bi_size, is_write ? "W" : "R");
+		dump_stack();
+	} else {
+		Z_ERR(znd, "MAP ERR: %d", rcode);
+		Z_ERR(znd, "%s: s:%"PRIx64" sz:%u -> %s", __func__, s_zdm,
+		      bio->bi_iter.bi_size, is_write ? "W" : "R");
+		dump_stack();
+		rcode = -EIO;
+	}
+done:
+	return rcode;
+}
+
+/**
+ * _do_mem_purge() - conditionally trigger a reduction of cache memory
+ * @znd: ZDM Instance
+ */
+static inline int _do_mem_purge(struct zoned *znd)
+{
+	int do_work = 0;
+
+	if (atomic_read(&znd->incore) > 3) {
+		set_bit(DO_MEMPOOL, &znd->flags);
+		if (!work_pending(&znd->meta_work))
+			do_work = 1;
+	}
+	return do_work;
+}
+
+/**
+ * on_timeout_activity() - Periodic background task execution.
+ * @znd: ZDM Instance
+ * @mempurge: If memory purge should be scheduled.
+ * @delay: Delay metric for periodic GC
+ *
+ * NOTE: Executed as a worker task queued froma timer.
+ */
+static void on_timeout_activity(struct zoned *znd, int delay)
+{
+	int max_tries = 1;
+
+	if (test_bit(ZF_FREEZE, &znd->flags))
+		return;
+
+	if (is_expired_msecs(znd->age, DISCARD_IDLE_MSECS))
+		max_tries = 20;
+
+	do {
+		int count;
+
+		count = z_discard_partial(znd, Z_BLKSZ, NORMAL);
+		if (count != 1 || --max_tries < 0)
+			break;
+
+		if (test_bit(ZF_FREEZE, &znd->flags))
+			return;
+
+	} while (is_expired_msecs(znd->age, DISCARD_IDLE_MSECS));
+
+	gc_queue_with_delay(znd, delay, NORMAL);
+
+	if (!test_bit(DO_GC_NO_PURGE, &znd->flags) && _do_mem_purge(znd))
+		queue_work(znd->meta_wq, &znd->meta_work);
+}
+
+/**
+ * bg_work_task() - periodic background worker
+ * @work: context for worker thread
+ */
+static void bg_work_task(struct work_struct *work)
+{
+	struct zoned *znd;
+	const int delay = 1;
+
+	if (!work)
+		return;
+
+	znd = container_of(work, struct zoned, bg_work);
+	on_timeout_activity(znd, delay);
+}
+
+/**
+ * activity_timeout() - Handler for timer used to trigger background worker.
+ * @data: context for timer.
+ */
+static void activity_timeout(unsigned long data)
+{
+	struct zoned *znd = (struct zoned *) data;
+
+	if (!work_pending(&znd->bg_work))
+		queue_work(znd->bg_wq, &znd->bg_work);
+
+	if (!test_bit(ZF_FREEZE, &znd->flags))
+		mod_timer(&znd->timer, jiffies + msecs_to_jiffies(2500));
+}
+
+/**
+ * get_dev_size() - Report accessible size of device to upper layer.
+ * @ti: DM Target
+ *
+ * Return: Size in 512 byte sectors
+ */
+static sector_t get_dev_size(struct dm_target *ti)
+{
+	struct zoned *znd = ti->private;
+	u64 sz = i_size_read(get_bdev_bd_inode(znd));	/* size in bytes. */
+	u64 lut_resv = znd->gz_count * znd->mz_provision;
+
+	/*
+	 * NOTE: `sz` should match `ti->len` when the dm_table
+	 *       is setup correctly
+	 */
+	sz -= (lut_resv * Z_SMR_SZ_BYTES);
+
+	return to_sector(sz);
+}
+
+/**
+ * zoned_iterate_devices() - Iterate over devices call fn() at each.
+ * @ti: DM Target
+ * @fn: Function for each callout
+ * @data: Context for fn().
+ */
+static int zoned_iterate_devices(struct dm_target *ti,
+				 iterate_devices_callout_fn fn, void *data)
+{
+	struct zoned *znd = ti->private;
+	int rc;
+
+	rc = fn(ti, znd->dev, 0, get_dev_size(ti), data);
+	return rc;
+}
+
+/**
+ * zoned_io_hints() - The place to tweek queue limits for DM targets
+ * @ti: DM Target
+ * @limits: queue_limits for this DM target
+ */
+static void zoned_io_hints(struct dm_target *ti, struct queue_limits *limits)
+{
+	struct zoned *znd = ti->private;
+	u64 io_opt_sectors = limits->io_opt >> SECTOR_SHIFT;
+
+	/*
+	 * If the system-determined stacked limits are compatible with the
+	 * zoned device's blocksize (io_opt is a factor) do not override them.
+	 */
+	if (io_opt_sectors < 8 || do_div(io_opt_sectors, 8)) {
+		blk_limits_io_min(limits, 0);
+		blk_limits_io_opt(limits, 8 << SECTOR_SHIFT);
+	}
+
+	limits->logical_block_size =
+		limits->physical_block_size =
+		limits->io_min = Z_C4K;
+	if (znd->trim) {
+		limits->discard_alignment = Z_C4K;
+		limits->discard_granularity = Z_C4K;
+		limits->max_discard_sectors = 1 << 30;
+		limits->max_hw_discard_sectors = 1 << 30;
+		limits->discard_zeroes_data = 1;
+	}
+}
+
+/**
+ * zoned_status() - Report status of DM Target
+ * @ti: DM Target
+ * @type: Type of status to report.
+ * @status_flags: Flags
+ * @result: Fill in with status.
+ * @maxlen: Maximum number of bytes for result.
+ */
+static void zoned_status(struct dm_target *ti, status_type_t type,
+			 unsigned status_flags, char *result, unsigned maxlen)
+{
+	struct zoned *znd = (struct zoned *) ti->private;
+
+	switch (type) {
+	case STATUSTYPE_INFO:
+		result[0] = '\0';
+		break;
+
+	case STATUSTYPE_TABLE:
+		scnprintf(result, maxlen, "%s Z#%u", znd->dev->name,
+			 znd->zdstart);
+		break;
+	}
+}
+
+/* -------------------------------------------------------------------------- */
+/* --ProcFS Support Routines------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+#if defined(CONFIG_PROC_FS)
+
+/**
+ * struct zone_info_entry - Proc zone entry.
+ * @zone: Zone Index
+ * @info: Info (WP/Used).
+ */
+struct zone_info_entry {
+	u32 zone;
+	u32 info;
+};
+
+/**
+ * Startup writing to our proc entry
+ */
+static void *proc_wp_start(struct seq_file *seqf, loff_t *pos)
+{
+	struct zoned *znd = seqf->private;
+
+	if (*pos == 0)
+		znd->wp_proc_at = *pos;
+	return &znd->wp_proc_at;
+}
+
+/**
+ * Increment to our next 'grand zone' 4k page.
+ */
+static void *proc_wp_next(struct seq_file *seqf, void *v, loff_t *pos)
+{
+	struct zoned *znd = seqf->private;
+	u32 zone = ++znd->wp_proc_at;
+
+	return zone < znd->data_zones ? &znd->wp_proc_at : NULL;
+}
+
+/**
+ * Stop ... a place to free resources that we don't hold .. [noop].
+ */
+static void proc_wp_stop(struct seq_file *seqf, void *v)
+{
+}
+
+/**
+ * Write as many entries as possbile ....
+ */
+static int proc_wp_show(struct seq_file *seqf, void *v)
+{
+	int err = 0;
+	struct zoned *znd = seqf->private;
+	u32 zone = znd->wp_proc_at;
+	u32 out = 0;
+
+	while (zone < znd->data_zones) {
+		u32 gzno  = zone >> GZ_BITS;
+		u32 gzoff = zone & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		struct zone_info_entry entry;
+
+		entry.zone = zone;
+		entry.info = le32_to_cpu(wpg->wp_alloc[gzoff]);
+
+		err = seq_write(seqf, &entry, sizeof(entry));
+		if (err) {
+			/*
+			 * write failure is temporary ..
+			 * just return and try again
+			 */
+			err = 0;
+			goto out;
+		}
+		out++;
+		zone = ++znd->wp_proc_at;
+	}
+
+out:
+	if (err)
+		Z_ERR(znd, "%s: %llu -> %d", __func__, znd->wp_proc_at, err);
+
+	return err;
+}
+
+/**
+ * zdm_wp_ops() - Seq_file operations for retrieving WP via proc fs
+ */
+static const struct seq_operations zdm_wp_ops = {
+	.start	= proc_wp_start,
+	.next	= proc_wp_next,
+	.stop	= proc_wp_stop,
+	.show	= proc_wp_show
+};
+
+/**
+ * zdm_wp_open() - Need to migrate our private data to the seq_file
+ */
+static int zdm_wp_open(struct inode *inode, struct file *file)
+{
+	/* seq_open will populate file->private_data with a seq_file */
+	int err = seq_open(file, &zdm_wp_ops);
+
+	if (!err) {
+		struct zoned *znd = PDE_DATA(inode);
+		struct seq_file *seqf = file->private_data;
+
+		seqf->private = znd;
+	}
+	return err;
+}
+
+/**
+ * zdm_wp_fops() - File operations for retrieving WP via proc fs
+ */
+static const struct file_operations zdm_wp_fops = {
+	.open		= zdm_wp_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+};
+
+
+/**
+ * Startup writing to our proc entry
+ */
+static void *proc_used_start(struct seq_file *seqf, loff_t *pos)
+{
+	struct zoned *znd = seqf->private;
+
+	if (*pos == 0)
+		znd->wp_proc_at = *pos;
+	return &znd->wp_proc_at;
+}
+
+/**
+ * Increment to our next zone
+ */
+static void *proc_used_next(struct seq_file *seqf, void *v, loff_t *pos)
+{
+	struct zoned *znd = seqf->private;
+	u32 zone = ++znd->wp_proc_at;
+
+	return zone < znd->data_zones ? &znd->wp_proc_at : NULL;
+}
+
+/**
+ * Stop ... a place to free resources that we don't hold .. [noop].
+ */
+static void proc_used_stop(struct seq_file *seqf, void *v)
+{
+}
+
+/**
+ * proc_used_show() - Write as many 'used' entries as possbile.
+ * @seqf: seq_file I/O handler
+ * @v: An unused parameter.
+ */
+static int proc_used_show(struct seq_file *seqf, void *v)
+{
+	int err = 0;
+	struct zoned *znd = seqf->private;
+	u32 zone = znd->wp_proc_at;
+	u32 out = 0;
+
+	while (zone < znd->data_zones) {
+		u32 gzno  = zone >> GZ_BITS;
+		u32 gzoff = zone & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		struct zone_info_entry entry;
+
+		entry.zone = zone;
+		entry.info = le32_to_cpu(wpg->zf_est[gzoff]);
+
+		err = seq_write(seqf, &entry, sizeof(entry));
+		if (err) {
+			/*
+			 * write failure is temporary ..
+			 * just return and try again
+			 */
+			err = 0;
+			goto out;
+		}
+		out++;
+		zone = ++znd->wp_proc_at;
+	}
+
+out:
+	if (err)
+		Z_ERR(znd, "%s: %llu -> %d", __func__, znd->wp_proc_at, err);
+
+	return err;
+}
+
+/**
+ * zdm_used_ops() - Seq_file Ops for retrieving 'used' state via proc fs
+ */
+static const struct seq_operations zdm_used_ops = {
+	.start	= proc_used_start,
+	.next	= proc_used_next,
+	.stop	= proc_used_stop,
+	.show	= proc_used_show
+};
+
+/**
+ * zdm_used_open() - Need to migrate our private data to the seq_file
+ */
+static int zdm_used_open(struct inode *inode, struct file *file)
+{
+	/* seq_open will populate file->private_data with a seq_file */
+	int err = seq_open(file, &zdm_used_ops);
+
+	if (!err) {
+		struct zoned *znd = PDE_DATA(inode);
+		struct seq_file *seqf = file->private_data;
+
+		seqf->private = znd;
+	}
+	return err;
+}
+
+/**
+ * zdm_used_fops() - File operations for retrieving 'used' state via proc fs
+ */
+static const struct file_operations zdm_used_fops = {
+	.open		= zdm_used_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+};
+
+/**
+ * zdm_status_show() - Dump the status structure via proc fs
+ */
+static int zdm_status_show(struct seq_file *seqf, void *unused)
+{
+	struct zoned *znd = seqf->private;
+	struct zdm_ioc_status status;
+	u32 zone;
+
+	memset(&status, 0, sizeof(status));
+
+	for (zone = 0; zone < znd->data_zones; zone++) {
+		u32 gzno  = zone >> GZ_BITS;
+		u32 gzoff = zone & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 wp_at = le32_to_cpu(wpg->wp_alloc[gzoff]) & Z_WP_VALUE_MASK;
+
+		status.b_used += wp_at;
+		status.b_available += Z_BLKSZ - wp_at;
+	}
+	status.mc_entries = znd->mc_entries;
+	status.dc_entries = znd->dc_entries;
+	status.b_discard = znd->discard_count;
+
+	/*  fixed array of ->fwd_tm and ->rev_tm */
+	status.m_zones = znd->data_zones;
+
+	status.memstat = znd->memstat;
+	memcpy(status.bins, znd->bins, sizeof(status.bins));
+	status.mlut_blocks = atomic_read(&znd->incore);
+
+	return seq_write(seqf, &status, sizeof(status));
+}
+
+/**
+ * zdm_status_open() - Open seq_file from file.
+ * @inode: Our data is stuffed here, Retrieve it.
+ * @file: file objected used by seq_file.
+ */
+static int zdm_status_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, zdm_status_show, PDE_DATA(inode));
+}
+
+/**
+ * zdm_used_fops() - File operations to chain to zdm_status_open.
+ */
+static const struct file_operations zdm_status_fops = {
+	.open = zdm_status_open,
+	.read = seq_read,
+	.llseek = seq_lseek,
+	.release = single_release,
+};
+
+/**
+ * zdm_info_show() - Report some information as text.
+ * @seqf: Sequence file for writing
+ * @unused: Not used.
+ */
+static int zdm_info_show(struct seq_file *seqf, void *unused)
+{
+	struct zoned *znd = seqf->private;
+
+	seq_printf(seqf, "On device:     %s\n", _zdisk(znd));
+	seq_printf(seqf, "Data Zones:    %u\n", znd->data_zones);
+	seq_printf(seqf, "Empty Zones:   %u\n", znd->z_gc_free);
+	seq_printf(seqf, "Cached Pages:  %u\n", znd->mc_entries);
+	seq_printf(seqf, "Discard Pages: %u\n", znd->dc_entries);
+	seq_printf(seqf, "ZTL Pages:     %d\n", atomic_read(&znd->incore));
+	seq_printf(seqf, "   in ZTL:     %d\n", znd->in_zlt);
+	seq_printf(seqf, "   in LZY:     %d\n", znd->in_lzy);
+	seq_printf(seqf, "RAM in Use:    %lu\n", znd->memstat);
+	seq_printf(seqf, "Zones GC'd:    %u\n", znd->gc_events);
+	seq_printf(seqf, "GC Throttle:   %d\n", atomic_read(&znd->gc_throttle));
+#if ALLOC_DEBUG
+	seq_printf(seqf, "Max Allocs:    %u\n", znd->hw_allocs);
+#endif
+
+	return 0;
+}
+
+/**
+ * zdm_info_open() - Open seq_file from file.
+ * @inode: Our data is stuffed here, Retrieve it.
+ * @file: file objected used by seq_file.
+ */
+static int zdm_info_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, zdm_info_show, PDE_DATA(inode));
+}
+
+/**
+ * zdm_used_fops() - File operations to chain to zdm_info_open.
+ */
+static const struct file_operations zdm_info_fops = {
+	.open = zdm_info_open,
+	.read = seq_read,
+	.llseek = seq_lseek,
+	.release = single_release,
+};
+
+/**
+ * zdm_create_proc_entries() - Create proc entries for ZDM utilities
+ * @znd: ZDM Instance
+ */
+static int zdm_create_proc_entries(struct zoned *znd)
+{
+	snprintf(znd->proc_name, sizeof(znd->proc_name), "zdm_%s", _zdisk(znd));
+
+	znd->proc_fs = proc_mkdir(znd->proc_name, NULL);
+	if (!znd->proc_fs)
+		return -ENOMEM;
+
+	proc_create_data(PROC_WP, 0, znd->proc_fs, &zdm_wp_fops, znd);
+	proc_create_data(PROC_FREE, 0, znd->proc_fs, &zdm_used_fops, znd);
+	proc_create_data(PROC_DATA, 0, znd->proc_fs, &zdm_status_fops, znd);
+	proc_create_data(PROC_STATUS, 0, znd->proc_fs, &zdm_info_fops, znd);
+
+	return 0;
+}
+
+/**
+ * zdm_remove_proc_entries() - Remove proc entries
+ * @znd: ZDM Instance
+ */
+static void zdm_remove_proc_entries(struct zoned *znd)
+{
+	remove_proc_subtree(znd->proc_name, NULL);
+}
+
+#else /* !CONFIG_PROC_FS */
+
+static int zdm_create_proc_entries(struct zoned *znd)
+{
+	(void)znd;
+	return 0;
+}
+static void zdm_remove_proc_entries(struct zoned *znd)
+{
+	(void)znd;
+}
+#endif /* CONFIG_PROC_FS */
+
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+static void start_worker(struct zoned *znd)
+{
+	clear_bit(ZF_FREEZE, &znd->flags);
+	atomic_set(&znd->suspended, 0);
+	mod_timer(&znd->timer, jiffies + msecs_to_jiffies(5000));
+}
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+static void stop_worker(struct zoned *znd)
+{
+	set_bit(ZF_FREEZE, &znd->flags);
+	atomic_set(&znd->suspended, 1);
+	zoned_io_flush(znd);
+}
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+static void zoned_postsuspend(struct dm_target *ti)
+{
+	struct zoned *znd = ti->private;
+
+	stop_worker(znd);
+}
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+static void zoned_resume(struct dm_target *ti)
+{
+	/* TODO */
+}
+
+/*
+ *
+ */
+static int zoned_message(struct dm_target *ti, unsigned argc, char **argv)
+{
+	struct zoned *znd = ti->private;
+	int iter;
+
+	for (iter = 0; iter < argc; iter++)
+		Z_ERR(znd, "Message: %s not handled.", argv[argc]);
+
+	return 0;
+}
+
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+static int zoned_preresume(struct dm_target *ti)
+{
+	struct zoned *znd = ti->private;
+
+	start_worker(znd);
+	return 0;
+}
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+static struct target_type zoned_target = {
+	.name = "zoned",
+	.module = THIS_MODULE,
+	.version = {1, 0, 0},
+	.ctr = zoned_ctr,
+	.dtr = zoned_dtr,
+	.map = zoned_map,
+	.end_io = zoned_endio,
+	.postsuspend = zoned_postsuspend,
+	.preresume = zoned_preresume,
+	.resume = zoned_resume,
+	.status = zoned_status,
+	.message = zoned_message,
+	.iterate_devices = zoned_iterate_devices,
+	.io_hints = zoned_io_hints
+};
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+static int __init dm_zoned_init(void)
+{
+	int rcode = dm_register_target(&zoned_target);
+
+	if (rcode)
+		DMERR("zoned target registration failed: %d", rcode);
+
+	return rcode;
+}
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+static void __exit dm_zoned_exit(void)
+{
+	dm_unregister_target(&zoned_target);
+}
+
+module_init(dm_zoned_init);
+module_exit(dm_zoned_exit);
+
+MODULE_DESCRIPTION(DM_NAME " zoned target for Host Aware/Managed drives.");
+MODULE_AUTHOR("Shaun Tancheff <shaun.tancheff@xxxxxxxxxxx>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/md/dm-zoned.h b/drivers/md/dm-zoned.h
new file mode 100644
index 0000000..c73ddea
--- /dev/null
+++ b/drivers/md/dm-zoned.h
@@ -0,0 +1,714 @@
+/*
+ * Kernel Device Mapper for abstracting ZAC/ZBC devices as normal
+ * block devices for linux file systems.
+ *
+ * Copyright (C) 2015 Seagate Technology PLC
+ *
+ * Written by:
+ * Shaun Tancheff <shaun.tancheff@xxxxxxxxxxx>
+ *
+ * This file is licensed under  the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#ifndef _DM_ZONED_H
+#define _DM_ZONED_H
+
+#define ALLOC_DEBUG		0
+#define USE_KTHREAD		0
+#define KFIFO_SIZE		(1 << 14)
+
+#define NORMAL			0
+#define CRIT			GFP_NOIO
+
+#define DM_MSG_PREFIX		"zoned"
+
+#define PROC_WP			"wp.bin"
+#define PROC_FREE		"free.bin"
+#define PROC_DATA		"data.bin"
+#define PROC_STATUS		"status"
+
+#define ZDM_RESERVED_ZNR	0
+#define ZDM_CRC_STASH_ZNR	1 /* first 64 blocks */
+#define ZDM_RMAP_ZONE		2
+#define ZDM_SECTOR_MAP_ZNR	3
+#define ZDM_DATA_START_ZNR	4
+
+#define Z_WP_GC_FULL		(1u << 31)
+#define Z_WP_GC_ACTIVE		(1u << 30)
+#define Z_WP_GC_TARGET		(1u << 29)
+#define Z_WP_GC_READY		(1u << 28)
+#define Z_WP_GC_BITS		(0xFu << 28)
+
+#define Z_WP_GC_PENDING		(Z_WP_GC_FULL|Z_WP_GC_ACTIVE)
+#define Z_WP_NON_SEQ		(1u << 27)
+#define Z_WP_RRECALC		(1u << 26)
+#define Z_WP_RESV_02		(1u << 25)
+#define Z_WP_RESV_03		(1u << 24)
+
+#define Z_WP_VALUE_MASK		(~0u >> 8)
+#define Z_WP_FLAGS_MASK		(~0u << 24)
+#define Z_WP_STREAM_MASK	Z_WP_FLAGS_MASK
+
+#define Z_AQ_GC			(1u << 31)
+#define Z_AQ_META		(1u << 30)
+#define Z_AQ_NORMAL		(1u << 29)
+#define Z_AQ_STREAM_ID		(1u << 28)
+#define Z_AQ_STREAM_MASK	(0xFF)
+#define Z_AQ_META_STREAM	(Z_AQ_META | Z_AQ_STREAM_ID | 0xFE)
+
+#define Z_C4K			(4096ul)
+#define Z_UNSORTED		(Z_C4K / sizeof(struct map_sect_to_lba))
+#define Z_BLOCKS_PER_DM_SECTOR	(Z_C4K/512)
+#define MZ_METADATA_ZONES	(8ul)
+#define Z_SHFT4K		(3)
+
+
+#define LBA_SB_START		1
+
+#define SUPERBLOCK_LOCATION	0
+#define SUPERBLOCK_MAGIC	0x5a6f4e65ul	/* ZoNe */
+#define SUPERBLOCK_CSUM_XOR	146538381
+#define MIN_ZONED_VERSION	1
+#define Z_VERSION		1
+#define MAX_ZONED_VERSION	1
+#define INVALID_WRITESET_ROOT	SUPERBLOCK_LOCATION
+
+#define UUID_LEN		16
+
+#define Z_TYPE_SMR		2
+#define Z_TYPE_SMR_HA		1
+#define Z_VPD_INFO_BYTE		8
+
+#define MAX_CACHE_INCR		320ul
+#define CACHE_COPIES		3
+#define MAX_MZ_SUPP		64
+#define FWD_TM_KEY_BASE		4096ul
+
+#define IO_VCACHE_ORDER		8
+#define IO_VCACHE_PAGES		(1 << IO_VCACHE_ORDER)  /* 256 pages => 1MiB */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+
+enum superblock_flags_t {
+	SB_DIRTY = 1,
+};
+
+struct z_io_req_t {
+	struct dm_io_region *where;
+	struct dm_io_request *io_req;
+	struct work_struct work;
+	int result;
+};
+
+#define Z_LOWER48 (~0ul >> 16)
+#define Z_UPPER16 (~Z_LOWER48)
+
+#define STREAM_SIZE	256
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+/**
+ * enum pg_flag_enum - Map Pg flags
+ * @IS_DIRTY: Block is modified from on-disk copy.
+ * @IS_STALE: ??
+ *
+ * @IS_FWD: Is part of Forward ZLT or CRC table.
+ * @IS_REV: Is part of Reverse ZLT or CRC table.
+ * @IS_CRC: Is part of CRC table.
+ * @IS_LUT: Is part of ZTL table.
+ *
+ * @R_IN_FLIGHT: Async read in progress.
+ * @W_IN_FLIGHT: ASync write in progress.
+ * @DELAY_ADD: Spinlock was busy for zltlst, added to lazy for transit.
+ * @STICKY: ???
+ * @IS_READA: Block will pulled for Read Ahead. Cleared when used.
+ * @IS_DROPPED: Has clean/expired from zlt_lst and is waiting free().
+ *
+ * @IS_LAZY:   Had been added to 'lazy' lzy_lst.
+ * @IN_ZLT: Had been added to 'inpool' zlt_lst.
+ *
+ */
+enum pg_flag_enum {
+	IS_DIRTY,
+	IS_STALE,
+	IS_FLUSH,
+
+	IS_FWD,
+	IS_REV,
+	IS_CRC,
+	IS_LUT,
+
+	R_IN_FLIGHT,
+	W_IN_FLIGHT,
+	DELAY_ADD,
+	STICKY,
+	IS_READA,
+	IS_DROPPED,
+	IS_LAZY,
+	IN_ZLT,
+};
+
+/**
+ * enum gc_flags_enum - Garbage Collection [GC] states.
+ */
+enum gc_flags_enum {
+	DO_GC_NEW,
+	DO_GC_PREPARE,		/* -> READ or COMPLETE state */
+	DO_GC_WRITE,
+	DO_GC_META,		/* -> PREPARE state */
+	DO_GC_COMPLETE,
+};
+
+/**
+ * enum znd_flags_enum - zoned state/feature/action flags
+ */
+enum znd_flags_enum {
+	ZF_FREEZE,
+	ZF_POOL_FWD,
+	ZF_POOL_REV,
+	ZF_POOL_CRCS,
+
+	ZF_RESV_1,
+	ZF_RESV_2,
+	ZF_RESV_3,
+	ZF_RESV_4,
+
+	DO_JOURNAL_MOVE,
+	DO_MEMPOOL,
+	DO_SYNC,
+	DO_FLUSH,
+	DO_JOURNAL_LOAD,
+	DO_GC_NO_PURGE,
+	DO_METAWORK_QD,
+};
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+struct zoned;
+
+/**
+ * struct gc_state - A page of map table
+ * @znd: ZDM Instance
+ * @gc_flags: See gc_flags_enum
+ * @r_ptr: Next read in zone.
+ * @w_ptr: Next write in target zone.
+ * @nblks: Number of blocks in I/O
+ * @result: GC operation result.
+ * @z_gc: Zone undergoing compacation
+ * @tag: System wide serial number (debugging).
+ *
+ * Longer description of this structure.
+ */
+struct gc_state {
+	struct zoned *znd;
+	unsigned long gc_flags;
+
+	u32 r_ptr;
+	u32 w_ptr;
+
+	u32 nblks;		/* 1-65536 */
+	u32 z_gc;
+
+	int result;
+};
+
+/**
+ * struct map_addr - A page of map table
+ * @dm_s:    full map on dm layer
+ * @zone_id: z_id match zone_list_t.z_id
+ * @pg_idx:  entry in lut (0-1023)
+ * @lut_s:   sector table lba
+ * @lut_r:   reverse table lba
+ *
+ * Longer description of this structure.
+ */
+struct map_addr {
+	u64 dm_s;
+	u64 lut_s;
+	u64 lut_r;
+
+	u32 zone_id;
+	u32 pg_idx;
+};
+
+/**
+ * struct map_sect_to_lba - Sector to LBA mapping.
+ * @tlba: tlba
+ * @physical: blba or number of blocks
+ *
+ * Longer description of this structure.
+ */
+struct map_sect_to_lba {
+	__le64 tlba;		/* record type [16 bits] + logical sector # */
+	__le64 bval;	/* csum 16 [16 bits] + 'physical' block lba */
+} __packed;
+
+/**
+ * enum gc_flags_enum - Garbage Collection [GC] states.
+ */
+enum map_type_enum {
+	IS_MAP,
+	IS_DISCARD,
+	MAP_COUNT,
+};
+
+/**
+ * struct map_cache - A page of map table
+ * @mclist:
+ * @jdata: 4k page of data
+ * @refcount:
+ * @cached_lock:
+ * @busy_locked:
+ * @jcount:
+ * @jsorted:
+ * @jsize:
+ * @map_content: See map_type_enum
+ *
+ * Longer description of this structure.
+ */
+struct map_cache {
+	struct list_head mclist;
+	struct map_sect_to_lba *jdata;
+	atomic_t refcount;
+	struct mutex cached_lock;
+	atomic_t busy_locked;
+	u32 jcount;
+	u32 jsorted;
+	u32 jsize;
+	u32 map_content; /* IS_MAP or IS_DISC */
+};
+
+
+union map_pg_data {
+	__le32 *addr;
+	__le16 *crc;
+};
+
+
+/**
+ * struct map_pg - A page of map table
+ *
+ * @mdata:        4k page of table entries
+ * @refcount:
+ * @age:          most recent access in jiffies
+ * @lba:          logical position (use lookups to find actual)
+ * @md_lock:      lock mdata i/o
+ * @last_write:   last known position on disk
+ * @flags:        is dirty flag, is fwd, is rev, is crc, is lut
+ * @inpool:
+ *
+ * Longer description of this structure.
+ */
+struct map_pg {
+	union map_pg_data data;
+	atomic_t refcount;
+	u64 age;
+	u64 lba;
+	u64 last_write;
+	struct mutex md_lock;
+	unsigned long flags;
+	struct list_head zltlst;
+
+	/* in flight tracking */
+	struct list_head lazy;
+	struct zoned *znd;
+	struct map_pg *crc_pg;
+	int hotness;
+	int index;
+	__le16 md_crc;
+};
+
+/**
+ * struct map_crc - Map to backing crc16.
+ *
+ * @table:        Maybe via table
+ * @pg_no:        Page [index] of table entry if applicable.
+ * @pg_idx:       Offset within page (From: zoned::md_crcs when table is null)
+ *
+ * Longer description of this structure.
+ */
+struct map_crc {
+	struct map_pg **table;
+	int pg_no;
+	int pg_idx;
+};
+
+/**
+ * struct map_info - Map to backing lookup table.
+ *
+ * @table:        backing table
+ * @crc:          backing crc16 detail.
+ * @index:        index [page] of table entry. Use map_addr::pg_idx for offset.
+ * @bit_type:     IS_LUT or IS_CRC
+ * @bit_dir:      IS_FWD or IS_REV
+ *
+ * Longer description of this structure.
+ */
+struct mpinfo {
+	struct map_pg **table;
+	struct map_crc crc;
+	int index;
+	int bit_type;
+	int bit_dir;
+};
+
+
+/**
+ * struct meta_pg - A page of zone WP mapping.
+ *
+ * @wp_alloc: Bits 23-0: wp alloc location.   Bits 31-24: GC Flags, Type Flags
+ * @zf_est:   Bits 23-0: free block count.    Bits 31-24: Stream Id
+ * @wp_used:  Bits 23-0: wp written location. Bits 31-24: Ratio ReCalc flag.
+ * @lba: pinned LBA in conventional/preferred zone.
+ * @wplck: spinlock held during data updates.
+ * @flags: IS_DIRTY flag
+ *
+ * One page is used per 1024 zones on media.
+ * For an 8TB drive this uses 30 entries or about 360k RAM.
+ */
+struct meta_pg {
+	__le32 *wp_alloc;
+	__le32 *zf_est;
+	__le32 *wp_used;
+	u64 lba;
+	spinlock_t wplck;
+	unsigned long flags;
+};
+
+/**
+ * struct zdm_superblock - A page of map table
+ * @uuid:
+ * @nr_zones:
+ * @magic:
+ * @zdstart:
+ * @version:
+ * @packed_meta:
+ * @flags:
+ * @csum:
+ *
+ * Longer description of this structure.
+ */
+struct zdm_superblock {
+	u8 uuid[UUID_LEN];	/* 16 */
+	__le64 nr_zones;	/*  8 */
+	__le64 magic;		/*  8 */
+	__le32 resvd;		/*  4 */
+	__le32 zdstart;		/*  4 */
+	__le32 version;		/*  4 */
+	__le32 packed_meta;	/*  4 */
+	__le32 flags;		/*  4 */
+	__le32 csum;		/*  4 */
+} __packed;			/* 56 */
+
+#define MAX_CACHE_SYNC 400
+
+/**
+ * struct mz_superkey - A page of map table
+ * @sig0:           8 - Native endian
+ * @sig1:           8 - Little endian
+ * @sblock:        56 -
+ * @stream:      1024 -
+ * @reserved:    2982 -
+ * @gc_resv:        4 -
+ * @meta_resv:      4 -
+ * @generation:     8 -
+ * @key_crc:        2 -
+ * @magic:          8 -
+ *
+ * Longer description of this structure.
+ */
+struct mz_superkey {
+	u64 sig0;
+	__le64 sig1;
+	struct zdm_superblock sblock;
+	__le32 stream[STREAM_SIZE];
+	__le32 gc_resv;
+	__le32 meta_resv;
+	__le16 n_crcs;
+	__le16 crcs[MAX_CACHE_SYNC];
+	__le16 md_crc;
+	__le16 wp_crc[64];
+	__le16 zf_crc[64];
+	__le16 discards;
+	__le16 maps;
+	u8 reserved[1908];
+	__le32 crc32;
+	__le64 generation;
+	__le64 magic;
+} __packed;
+
+/**
+ * struct io_4k_block - Sector to LBA mapping.
+ * @data:	A 4096 byte block
+ *
+ * Longer description of this structure.
+ */
+struct io_4k_block {
+	u8 data[Z_C4K];
+};
+
+/**
+ * struct io_dm_block - Sector to LBA mapping.
+ * @data:	A 512 byte block
+ *
+ * Longer description of this structure.
+ */
+struct io_dm_block {
+	u8 data[512];
+};
+
+struct stale_tracking {
+	u32 binsz;
+	u32 count;
+	int bins[STREAM_SIZE];
+};
+
+
+/*
+ *
+ * Partition -----------------------------------------------------------------+
+ *     Table ---+                                                             |
+ *              |                                                             |
+ *   SMR Drive |^-------------------------------------------------------------^|
+ * CMR Zones     ^^^^^^^^^
+ *  meta data    |||||||||
+ *
+ *   Remaining partition is filesystem data
+ *
+ */
+
+/**
+ * struct zoned - A page of map table
+ * @ti:		dm_target entry
+ * @dev:	dm_dev entry
+ * @mclist:	list of pages of in-memory LBA mappings.
+ * @mclck:	in memory map-cache lock (spinlock)
+
+ * @zltpool:		pages of lookup table entries
+ * @zlt_lck:	zltpool: memory pool lock
+ * @lzy_pool:
+ * @lzy_lock:
+ * @fwd_tm:
+ * @rev_tm:
+
+ * @bg_work:	background worker (periodic GC/Mem purge)
+ * @bg_wq:	background work queue
+ * @stats_lock:
+ * @gc_active:	Current GC state
+ * @gc_lock:	GC Lock
+ * @gc_work:	GC Worker
+ * @gc_wq:	GC Work Queue
+ * @data_zones:	# of data zones on device
+ * @gz_count:	# of 256G mega-zones
+ * @nr_blocks:	4k blocks on backing device
+ * @md_start:	LBA at start of metadata pool
+ * @data_lba:	LBA at start of data pool
+ * @zdstart:	ZONE # at start of data pool (first Possible WP ZONE)
+ * @start_sect:	where ZDM partition starts (RAW LBA)
+ * @flags:	See: enum znd_flags_enum
+ * @gc_backlog:
+ * @gc_io_buf:
+ * @io_vcache[32]:
+ * @io_vcache_flags:
+ * @z_sballoc:
+ * @super_block:
+ * @z_mega:
+ * @meta_wq:
+ * @gc_postmap:
+ * @io_client:
+ * @io_wq:
+ * @zone_action_wq:
+ * @timer:
+ * @bins:	Memory usage accounting/reporting.
+ * @bdev_name:
+ * @memstat:
+ * @suspended:
+ * @gc_mz_pref:
+ * @mz_provision:	Number of zones per 1024 of over-provisioning.
+ * @ata_passthrough:
+ * @is_empty:		For fast discards on initial format
+ *
+ * Longer description of this structure.
+ */
+struct zoned {
+	struct dm_target *ti;
+	struct dm_dev *dev;
+
+	struct list_head zltpool;
+	struct list_head lzy_pool;
+	struct list_head mclist[MAP_COUNT];
+	spinlock_t       mclck[MAP_COUNT];
+
+	struct work_struct bg_work;
+	struct workqueue_struct *bg_wq;
+
+	spinlock_t zlt_lck;
+	spinlock_t lzy_lck;
+	spinlock_t stats_lock;
+	spinlock_t mapkey_lock; /* access LUT and CRC array of pointers */
+	spinlock_t ct_lock; /* access LUT and CRC array of pointers */
+
+	struct mutex gc_wait;
+	struct mutex pool_mtx;
+	struct mutex mz_io_mutex;
+	struct mutex vcio_lock;
+
+#if USE_KTHREAD /* experimental++ */
+	struct task_struct *bio_kthread;
+	DECLARE_KFIFO_PTR(bio_fifo, struct bio *);
+	wait_queue_head_t wait_bio;
+	wait_queue_head_t wait_fifo;
+#endif
+	struct bio_set *bio_set;
+
+	struct gc_state *gc_active;
+	spinlock_t gc_lock;
+	struct delayed_work gc_work;
+	struct workqueue_struct *gc_wq;
+
+	u64 nr_blocks;
+	u64 start_sect;
+
+	u64 md_start;
+	u64 md_end;
+	u64 data_lba;
+
+	unsigned long flags;
+
+	u64 r_base;
+	u64 s_base;
+	u64 c_base;
+	u64 c_mid;
+	u64 c_end;
+
+	u64 sk_low;   /* unused ? */
+	u64 sk_high;  /* unused ? */
+
+	struct meta_pg *wp;
+
+	struct map_pg **fwd_tm;  /* nr_blocks / 1024 */
+	struct map_pg **rev_tm;  /* nr_blocks / 1024 */
+	struct map_pg **fwd_crc; /* (nr_blocks / 1024) / 2048 */
+	struct map_pg **rev_crc; /* (nr_blocks / 1024) / 2048 */
+	__le16 *md_crcs;   /* one of crc16's for fwd, 1 for rev */
+	u32 crc_count;
+	u32 map_count;
+
+	void *z_sballoc;
+	struct mz_superkey *bmkeys;
+	struct zdm_superblock *super_block;
+
+	struct work_struct meta_work;
+	sector_t last_w;
+	u8 *cow_block;
+	u64 cow_addr;
+	u32 data_zones;
+	u32 gz_count;
+	u32 zdstart;
+	u32 z_gc_free;
+	atomic_t incore;
+	u32 discard_count;
+	u32 z_current;
+	u32 z_meta_resv;
+	u32 z_gc_resv;
+	u32 gc_events;
+	int mc_entries;
+	int dc_entries;
+	int in_zlt;
+	int in_lzy;
+	int meta_result;
+	struct stale_tracking stale;
+
+	int gc_backlog;
+	void *gc_io_buf;
+	struct mutex gc_vcio_lock;
+	struct io_4k_block *io_vcache[32];
+	unsigned long io_vcache_flags;
+	u64 age;
+	struct workqueue_struct *meta_wq;
+	struct map_cache gc_postmap;
+	struct dm_io_client *io_client;
+	struct workqueue_struct *io_wq;
+	struct workqueue_struct *zone_action_wq;
+	struct timer_list timer;
+
+	u32 bins[40];
+	char bdev_name[BDEVNAME_SIZE];
+	char proc_name[BDEVNAME_SIZE+4];
+	struct proc_dir_entry *proc_fs;
+	loff_t wp_proc_at;
+	loff_t used_proc_at;
+
+	size_t memstat;
+	atomic_t suspended;
+	atomic_t gc_throttle;
+
+#if ALLOC_DEBUG
+	atomic_t allocs;
+	int hw_allocs;
+	void **alloc_trace;
+#endif
+	u32 filled_zone;
+	u16 mz_provision;
+	unsigned bdev_is_zoned:1;
+	unsigned ata_passthrough:1;
+	unsigned issue_open_zone:1;
+	unsigned issue_close_zone:1;
+	unsigned is_empty:1;
+	unsigned trim:1;
+
+};
+
+/**
+ * struct zdm_ioc_request - Sector to LBA mapping.
+ * @result_size:
+ * @megazone_nr:
+ *
+ * Longer description of this structure.
+ */
+struct zdm_ioc_request {
+	u32 result_size;
+	u32 megazone_nr;
+};
+
+/**
+ * struct zdm_ioc_status - Sector to LBA mapping.
+ * @b_used: Number of blocks used
+ * @b_available: Number of blocks free
+ * @b_discard: Number of blocks stale
+ * @m_zones: Number of zones.
+ * @mc_entries: Mem cache blocks in use
+ * @dc_entries: Discard cache blocks in use.
+ * @mlut_blocks:
+ * @crc_blocks:
+ * @memstat: Total memory in use by ZDM via *alloc()
+ * @bins: Allocation by subsystem.
+ *
+ * This status structure is used to pass run-time information to
+ * user spaces tools (zdm-tools) for diagnostics and tuning.
+ */
+struct zdm_ioc_status {
+	u64 b_used;
+	u64 b_available;
+	u64 b_discard;
+	u64 m_zones;
+	u32 mc_entries;
+	u32 dc_entries;
+	u64 mlut_blocks;
+	u64 crc_blocks;
+	u64 memstat;
+	u32 bins[40];
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _DM_ZONED_H */
diff --git a/drivers/md/libzoned.c b/drivers/md/libzoned.c
new file mode 100644
index 0000000..f887654
--- /dev/null
+++ b/drivers/md/libzoned.c
@@ -0,0 +1,7149 @@
+/*
+ * Kernel Device Mapper for abstracting ZAC/ZBC devices as normal
+ * block devices for linux file systems.
+ *
+ * Copyright (C) 2015 Seagate Technology PLC
+ *
+ * Written by:
+ * Shaun Tancheff <shaun.tancheff@xxxxxxxxxxx>
+ *
+ * This file is licensed under  the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#define BUILD_NO		108
+
+#define EXTRA_DEBUG		0
+#define ENABLE_PG_FREE_VIA_LAZY	1
+
+#define MZ_MEMPOOL_SZ		512
+#define READ_AHEAD		16   /* number of LUT entries to read-ahead */
+#define MEMCACHE_HWMARK		5
+#define MEM_PURGE_MSECS		15000
+#define MEM_HOT_BOOST_INC	5000
+#define DISCARD_IDLE_MSECS	2000
+#define DISCARD_MAX_INGRESS	150
+#define MAX_WSET		2048
+
+
+/* acceptable 'free' count for MZ free levels */
+#define GC_PRIO_DEFAULT		0xFF00
+#define GC_PRIO_LOW		0x7FFF
+#define GC_PRIO_HIGH		0x0400
+#define GC_PRIO_CRIT		0x0040
+
+/* When less than 20 zones are free use aggressive gc in the megazone */
+#define GC_COMPACT_AGGRESSIVE	32
+
+/*
+ *  For performance tuning:
+ *   Q? smaller strips give smoother performance
+ *      a single drive I/O is 8 (or 32?) blocks?
+ *   A? Does not seem to ...
+ */
+#define GC_MAX_STRIPE		256
+#define REPORT_ORDER		7
+#define REPORT_FILL_PGS		65 /* 65 -> min # pages for 4096 descriptors */
+#define SYNC_IO_ORDER		2
+#define SYNC_IO_SZ		((1 << SYNC_IO_ORDER) * PAGE_SIZE)
+
+#define MZTEV_UNUSED		(cpu_to_le32(0xFFFFFFFFu))
+#define MZTEV_NF		(cpu_to_le32(0xFFFFFFFEu))
+
+#define Z_TABLE_MAGIC		0x123456787654321Eul
+#define Z_KEY_SIG		0xFEDCBA987654321Ful
+
+#define Z_CRC_4K		4096
+#define ZONE_SECT_BITS		19
+#define Z_BLKBITS		16
+#define Z_BLKSZ			(1ul << Z_BLKBITS)
+#define MAX_ZONES_PER_MZ	1024
+#define Z_SMR_SZ_BYTES		(Z_C4K << Z_BLKBITS)
+
+#define GC_READ			(1ul << 15)
+#define BAD_ADDR		(~0ul)
+#define MC_INVALID		(cpu_to_le64(BAD_ADDR))
+#define NOZONE			(~0u)
+
+#define GZ_BITS 10
+#define GZ_MMSK ((1u << GZ_BITS) - 1)
+
+#define CRC_BITS 11
+#define CRC_MMSK ((1u << CRC_BITS) - 1)
+
+#define MD_CRC_INIT		(cpu_to_le16(0x5249u))
+
+static int map_addr_calc(struct zoned *, u64 dm_s, struct map_addr *out);
+static int zoned_io_flush(struct zoned *znd);
+static int zoned_wp_sync(struct zoned *znd, int reset_non_empty);
+static void cache_if_dirty(struct zoned *znd, struct map_pg *pg, int wq);
+static int write_if_dirty(struct zoned *, struct map_pg *, int use_wq, int snc);
+static void gc_work_task(struct work_struct *work);
+static void meta_work_task(struct work_struct *work);
+static u64 mcache_greatest_gen(struct zoned *, int, u64 *, u64 *);
+static u64 mcache_find_gen(struct zoned *, u64 base, int, u64 *out);
+static int find_superblock(struct zoned *znd, int use_wq, int do_init);
+static int sync_mapped_pages(struct zoned *znd, int sync, int drop);
+static int unused_phy(struct zoned *znd, u64 lba, u64 orig_s, gfp_t gfp);
+static struct io_4k_block *get_io_vcache(struct zoned *znd, gfp_t gfp);
+static int put_io_vcache(struct zoned *znd, struct io_4k_block *cache);
+static struct map_pg *get_map_entry(struct zoned *, u64 lba, gfp_t gfp);
+static void put_map_entry(struct map_pg *);
+static int cache_pg(struct zoned *znd, struct map_pg *pg, gfp_t gfp,
+		    struct mpinfo *mpi);
+static int move_to_map_tables(struct zoned *znd, struct map_cache *mcache);
+static void update_stale_ratio(struct zoned *znd, u32 zone);
+static int zoned_create_disk(struct dm_target *ti, struct zoned *znd);
+static int do_init_zoned(struct dm_target *ti, struct zoned *znd);
+static int z_discard_partial(struct zoned *znd, u32 blks, gfp_t gfp);
+static u64 z_discard_range(struct zoned *znd, u64 addr, gfp_t gfp);
+static u64 z_lookup_cache(struct zoned *znd, u64 addr, int type);
+static u64 z_lookup_table(struct zoned *znd, u64 addr, gfp_t gfp);
+static u64 current_mapping(struct zoned *znd, u64 addr, gfp_t gfp);
+static int z_mapped_add_one(struct zoned *znd, u64 dm_s, u64 lba, gfp_t gfp);
+static int z_mapped_discard(struct zoned *znd, u64 tlba, u64 count, gfp_t gfp);
+static int z_mapped_addmany(struct zoned *znd, u64 dm_s, u64 lba, u64,
+			    gfp_t gfp);
+static int z_to_map_list(struct zoned *znd, u64 dm_s, u64 lba, gfp_t gfp);
+static int z_to_discard_list(struct zoned *znd, u64 dm_s, u64 blks, gfp_t gfp);
+static int discard_merge(struct zoned *znd, u64 tlba, u64 blks);
+static int z_mapped_sync(struct zoned *znd);
+static int z_mapped_init(struct zoned *znd);
+static u64 z_acquire(struct zoned *znd, u32 flags, u32 nblks, u32 *nfound);
+static __le32 sb_crc32(struct zdm_superblock *sblock);
+static int update_map_entry(struct zoned *, struct map_pg *,
+			    struct map_addr *, u64, int);
+static int read_block(struct dm_target *, enum dm_io_mem_type,
+		      void *, u64, unsigned int, int);
+static int write_block(struct dm_target *, enum dm_io_mem_type,
+		       void *, u64, unsigned int, int);
+static int writef_block(struct dm_target *ti, int rw, enum dm_io_mem_type dtype,
+			void *data, u64 lba, unsigned int count, int queue);
+static int zoned_init_disk(struct dm_target *ti, struct zoned *znd,
+			   int create, int force);
+
+#define MutexLock(m)		test_and_lock((m), __LINE__)
+#define SpinLock(s)		test_and_spin((s), __LINE__)
+
+static __always_inline void test_and_lock(struct mutex *m, int lineno)
+{
+	if (!mutex_trylock(m)) {
+		pr_debug("mutex stall at %d\n", lineno);
+		mutex_lock(m);
+	}
+}
+
+static __always_inline void test_and_spin(spinlock_t *lock, int lineno)
+{
+	if (!spin_trylock(lock)) {
+		pr_debug("spin stall at %d\n", lineno);
+		spin_lock(lock);
+	}
+}
+
+/**
+ * ref_pg() - Decrement refcount on page of ZLT
+ * @pg: Page of ZLT map
+ */
+static __always_inline void deref_pg(struct map_pg *pg)
+{
+	atomic_dec(&pg->refcount);
+}
+
+/**
+ * ref_pg() - Increment refcount on page of ZLT
+ * @pg: Page of ZLT map
+ */
+static __always_inline void ref_pg(struct map_pg *pg)
+{
+	atomic_inc(&pg->refcount);
+#if 0 /* ALLOC_DEBUG */
+	if (atomic_read(&pg->refcount) > 20) {
+		pr_err("Excessive %d who dunnit?\n",
+			atomic_read(&pg->refcount));
+		dump_stack();
+	}
+#endif
+}
+
+/**
+ * getref_pg() - Read the refcount
+ * @pg: Page of ZLT map
+ */
+static __always_inline int getref_pg(struct map_pg *pg)
+{
+	return atomic_read(&pg->refcount);
+}
+
+
+/**
+ * mcache_ref() - Increment the reference count of mcache()
+ * @mcache: Page of map cache
+ */
+static __always_inline void mcache_ref(struct map_cache *mcache)
+{
+	atomic_inc(&mcache->refcount);
+}
+
+/**
+ * mcache_deref() - Decrement the reference count of mcache()
+ * @mcache: Page of map cache
+ */
+static __always_inline void mcache_deref(struct map_cache *mcache)
+{
+	atomic_dec(&mcache->refcount);
+}
+
+/**
+ * mcache_getref() - Read the refcount
+ * @mcache: Page of map cache
+ */
+static __always_inline int mcache_getref(struct map_cache *mcache)
+{
+	return atomic_read(&mcache->refcount);
+}
+
+/**
+ * mcache_busy() - Increment the busy test of mcache()
+ * @mcache: Page of map cache
+ */
+static __always_inline void mcache_busy(struct map_cache *mcache)
+{
+	atomic_inc(&mcache->busy_locked);
+}
+
+/**
+ * mcache_unbusy() - Decrement the busy test count of mcache()
+ * @mcache: Page of map cache
+ */
+static __always_inline void mcache_unbusy(struct map_cache *mcache)
+{
+	atomic_dec(&mcache->busy_locked);
+}
+
+/**
+ * mcache_is_busy() - Test if mcache is busy
+ * @mcache: Page of map cache
+ * Return non-zero if busy otherwise 0.
+ */
+static __always_inline int mcache_is_busy(struct map_cache *mcache)
+{
+	return atomic_read(&mcache->busy_locked);
+}
+
+/**
+ * crc16_md() - 16 bit CRC on metadata blocks
+ * @data: Block of metadata.
+ * @len:  Number of bytes in block.
+ *
+ * Return: 16 bit CRC.
+ */
+static inline u16 crc16_md(void const *data, size_t len)
+{
+	const u16 init = 0xFFFF;
+	const u8 *p = data;
+
+	return crc16(init, p, len);
+}
+
+/**
+ * crc_md_le16() - 16 bit CRC on metadata blocks in little endian
+ * @data: Block of metadata.
+ * @len:  Number of bytes in block.
+ *
+ * Return: 16 bit CRC.
+ */
+static inline __le16 crc_md_le16(void const *data, size_t len)
+{
+	u16 crc = crc16_md(data, len);
+
+	return cpu_to_le16(crc);
+}
+
+/**
+ * crcpg() - 32 bit CRC [NOTE: 32c is HW assisted on Intel]
+ * @data: Block of metadata [4K bytes].
+ *
+ * Return: 32 bit CRC.
+ */
+static inline u32 crcpg(void *data)
+{
+	return crc32c(~0u, data, Z_CRC_4K) ^ SUPERBLOCK_CSUM_XOR;
+}
+
+/**
+ * le64_to_lba48() - Return the lower 48 bits of LBA
+ * @enc: 64 bit LBA + flags
+ * @flg: optional 16 bits of classification.
+ *
+ * Return: 48 bits of LBA [and flg].
+ */
+static inline u64 le64_to_lba48(__le64 enc, u16 *flg)
+{
+	const u64 lba64 = le64_to_cpu(enc);
+
+	if (flg)
+		*flg = (lba64 >> 48) & 0xFFFF;
+
+	return lba64 & Z_LOWER48;
+}
+
+/**
+ * lba48_to_le64() - Encode 48 bits of lba + 16 bits of flags.
+ * @flags: flags to encode.
+ * @lba48: LBA to encode
+ *
+ * Return: Little endian u64.
+ */
+static inline __le64 lba48_to_le64(u16 flags, u64 lba48)
+{
+	u64 high_bits = flags;
+
+	return cpu_to_le64((high_bits << 48) | (lba48 & Z_LOWER48));
+}
+
+/**
+ * sb_test_flag() - Test if flag is set in Superblock.
+ * @sb: zdm_superblock.
+ * @bit_no: superblock flag
+ *
+ * Return: non-zero if flag is set.
+ */
+static inline int sb_test_flag(struct zdm_superblock *sb, int bit_no)
+{
+	u32 flags = le32_to_cpu(sb->flags);
+
+	return (flags & (1 << bit_no)) ? 1 : 0;
+}
+
+/**
+ * sb_set_flag() - Set a flag in superblock.
+ * @sb: zdm_superblock.
+ * @bit_no: superblock flag
+ */
+static inline void sb_set_flag(struct zdm_superblock *sb, int bit_no)
+{
+	u32 flags = le32_to_cpu(sb->flags);
+
+	flags |= (1 << bit_no);
+	sb->flags = cpu_to_le32(flags);
+}
+
+/**
+ * zone_to_sector() - Calculate starting LBA of zone
+ * @zone: zone number (0 based)
+ *
+ * Return: LBA at start of zone.
+ */
+static inline u64 zone_to_sector(u64 zone)
+{
+	return zone << ZONE_SECT_BITS;
+}
+
+/**
+ * is_expired_msecs() - Determine if age + msecs is older than now.
+ * @age: jiffies at last access
+ * @msecs: msecs of extra time.
+ *
+ * Return: non-zero if block is expired.
+ */
+static inline int is_expired_msecs(u64 age, u32 msecs)
+{
+	u64 expire_at = age + msecs_to_jiffies(msecs);
+	int expired = time_after64(jiffies_64, expire_at);
+
+	return expired;
+}
+
+/**
+ * is_expired() - Determine if age is older than MEM_PURGE_MSECS.
+ * @age: jiffies at last access
+ *
+ * Return: non-zero if block is expired.
+ */
+static inline int is_expired(u64 age)
+{
+	return is_expired_msecs(age, MEM_PURGE_MSECS);
+}
+
+/**
+ * _calc_zone() - Determine zone number from addr
+ * @addr: 4k sector number
+ *
+ * Return: znum or 0xFFFFFFFF if addr is in metadata space.
+ */
+static inline u32 _calc_zone(struct zoned *znd, u64 addr)
+{
+	u32 znum = NOZONE;
+
+	if (addr < znd->data_lba)
+		return znum;
+
+	addr -= znd->data_lba;
+	znum = addr >> Z_BLKBITS;
+
+	return znum;
+}
+
+/**
+ * lazy_pool_add - Set a flag and add map page to the lazy pool
+ * @znd: ZDM Instance
+ * @expg: Map table page.
+ * @bit: Flag to set on page.
+ *
+ * Lazy pool is used for deferred adding and delayed removal.
+ */
+static __always_inline
+void lazy_pool_add(struct zoned *znd, struct map_pg *expg, int bit)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&znd->lzy_lck, flags);
+	if (!test_bit(IS_LAZY, &expg->flags)) {
+		set_bit(IS_LAZY, &expg->flags);
+		list_add(&expg->lazy, &znd->lzy_pool);
+		znd->in_lzy++;
+	}
+	set_bit(bit, &expg->flags);
+	spin_unlock_irqrestore(&znd->lzy_lck, flags);
+}
+
+/**
+ * lazy_pool_splice - Add a list of pages to the lazy pool
+ * @znd: ZDM Instance
+ * @list: List of map table page to add.
+ *
+ * Lazy pool is used for deferred adding and delayed removal.
+ */
+static __always_inline
+void lazy_pool_splice(struct zoned *znd, struct list_head *list)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&znd->lzy_lck, flags);
+	list_splice_tail(list, &znd->lzy_pool);
+	spin_unlock_irqrestore(&znd->lzy_lck, flags);
+}
+
+/**
+ * pool_add() - Add metadata block to zltlst
+ * @znd: ZDM instance
+ * @expg: current metadata block to add to zltlst list.
+ */
+static inline int pool_add(struct zoned *znd, struct map_pg *expg)
+{
+	int rcode = 0;
+
+	if (expg) {
+		unsigned long flags;
+
+		if (spin_trylock_irqsave(&znd->zlt_lck, flags)) {
+			if (test_bit(IN_ZLT, &expg->flags)) {
+				Z_ERR(znd, "Double list_add from:");
+				dump_stack();
+			} else {
+				set_bit(IN_ZLT, &expg->flags);
+				list_add(&expg->zltlst, &znd->zltpool);
+				znd->in_zlt++;
+			}
+			spin_unlock_irqrestore(&znd->zlt_lck, flags);
+			rcode = 0;
+		} else {
+			lazy_pool_add(znd, expg, DELAY_ADD);
+		}
+	}
+
+	return rcode;
+}
+
+/**
+ * to_table_entry() - Deconstrct metadata page into mpinfo
+ * @znd: ZDM instance
+ * @lba: Address (4k resolution)
+ * @expg: current metadata block in zltlst list.
+ *
+ * Return: Index into mpinfo.table.
+ */
+static int to_table_entry(struct zoned *znd, u64 lba, struct mpinfo *mpi)
+{
+	int index = -1;
+
+	if (lba >= znd->s_base && lba < znd->r_base) {
+		mpi->table	= znd->fwd_tm;
+		index		= lba - znd->s_base;
+		mpi->bit_type	= IS_LUT;
+		mpi->bit_dir	= IS_FWD;
+		mpi->crc.table	= znd->fwd_crc;
+		mpi->crc.pg_no	= index >> CRC_BITS;
+		mpi->crc.pg_idx	= index & CRC_MMSK;
+		if (index < 0 ||  index >= znd->map_count) {
+			Z_ERR(znd, "%s: FWD BAD IDX %"PRIx64" %d of %d",
+				__func__, lba, index, znd->map_count);
+			dump_stack();
+		}
+	} else if (lba >= znd->r_base && lba < znd->c_base) {
+		mpi->table	= znd->rev_tm;
+		index		= lba - znd->r_base;
+		mpi->bit_type	= IS_LUT;
+		mpi->bit_dir	= IS_REV;
+		mpi->crc.table	= znd->rev_crc;
+		mpi->crc.pg_no	= index >> CRC_BITS;
+		mpi->crc.pg_idx	= index & CRC_MMSK;
+		if (index < 0 ||  index >= znd->map_count) {
+			Z_ERR(znd, "%s: REV BAD IDX %"PRIx64" %d of %d",
+				__func__, lba, index, znd->map_count);
+			dump_stack();
+		}
+	} else if (lba >= znd->c_base && lba < znd->c_mid) {
+		mpi->table	= znd->fwd_crc;
+		index		= lba - znd->c_base;
+		mpi->bit_type	= IS_CRC;
+		mpi->bit_dir	= IS_FWD;
+		mpi->crc.table	= NULL;
+		mpi->crc.pg_no	= 0;
+		mpi->crc.pg_idx	= index & CRC_MMSK;
+		if (index < 0 ||  index >= znd->crc_count) {
+			Z_ERR(znd, "%s: CRC BAD IDX %"PRIx64" %d of %d",
+				__func__, lba, index, znd->crc_count);
+			dump_stack();
+		}
+	} else if (lba >= znd->c_mid && lba < znd->c_end) {
+		mpi->table	= znd->rev_crc;
+		index		= lba - znd->c_mid;
+		mpi->bit_type	= IS_CRC;
+		mpi->bit_dir	= IS_REV;
+		mpi->crc.table	= NULL;
+		mpi->crc.pg_no	= 1;
+		mpi->crc.pg_idx	= (1 << CRC_BITS) + (index & CRC_MMSK);
+		if (index < 0 ||  index >= znd->crc_count) {
+			Z_ERR(znd, "%s: CRC BAD IDX %"PRIx64" %d of %d",
+				__func__, lba, index, znd->crc_count);
+			dump_stack();
+		}
+	} else {
+		Z_ERR(znd, "** Corrupt lba %" PRIx64 " not in range.", lba);
+		znd->meta_result = -EIO;
+		dump_stack();
+	}
+	mpi->index = index;
+	return index;
+}
+
+/**
+ * is_ready_for_gc() - Test zone flags for GC sanity and ready flag.
+ * @znd: ZDM instance
+ * @z_id: Address (4k resolution)
+ *
+ * Return: non-zero if zone is suitable for GC.
+ */
+static inline int is_ready_for_gc(struct zoned *znd, u32 z_id)
+{
+	u32 gzno  = z_id >> GZ_BITS;
+	u32 gzoff = z_id & GZ_MMSK;
+	struct meta_pg *wpg = &znd->wp[gzno];
+	u32 wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+	u32 used = le32_to_cpu(wpg->wp_used[gzoff]) & Z_WP_VALUE_MASK;
+
+	if (((wp & Z_WP_GC_BITS) == Z_WP_GC_READY) && (used == Z_BLKSZ))
+		return 1;
+	return 0;
+}
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+/*
+ *  generic-ish n-way alloc/free
+ *  Use kmalloc for small (< 4k) allocations.
+ *  Use vmalloc for multi-page alloctions
+ *  Except:
+ *  Use multipage allocations for dm_io'd pages that a frequently hit.
+ *
+ *  NOTE: ALL allocations are zero'd before returning.
+ *	  alloc/free count is tracked for dynamic analysis.
+ */
+
+#define GET_ZPG		0x040000
+#define GET_KM		0x080000
+#define GET_VM		0x100000
+
+#define PG_01    (GET_ZPG |  1)
+#define PG_02    (GET_ZPG |  2)
+#define PG_05    (GET_ZPG |  5)
+#define PG_06    (GET_ZPG |  6)
+#define PG_08    (GET_ZPG |  8)
+#define PG_09    (GET_ZPG |  9)
+#define PG_10    (GET_ZPG | 10)
+#define PG_11    (GET_ZPG | 11)
+#define PG_13    (GET_ZPG | 13)
+#define PG_17    (GET_ZPG | 17)
+#define PG_27    (GET_ZPG | 27)
+
+#define KM_00    (GET_KM  |  0)
+#define KM_07    (GET_KM  |  7)
+#define KM_14    (GET_KM  | 14)
+#define KM_15    (GET_KM  | 15)
+#define KM_16    (GET_KM  | 16)
+#define KM_18    (GET_KM  | 18)
+#define KM_19    (GET_KM  | 19)
+#define KM_20    (GET_KM  | 20)
+#define KM_25    (GET_KM  | 25)
+#define KM_26    (GET_KM  | 26)
+#define KM_28    (GET_KM  | 28)
+#define KM_29    (GET_KM  | 29)
+#define KM_30    (GET_KM  | 30)
+
+#define VM_03    (GET_VM  |  3)
+#define VM_04    (GET_VM  |  4)
+#define VM_12    (GET_VM  | 12)
+#define VM_21    (GET_VM  | 21)
+#define VM_22    (GET_VM  | 22)
+
+#define ZDM_FREE(z, _p, sz, id) \
+	do { zdm_free((z), (_p), (sz), (id)); (_p) = NULL; } while (0)
+
+#define ZDM_ALLOC(z, sz, id, gfp)       zdm_alloc((z), (sz), (id), (gfp))
+#define ZDM_CALLOC(z, n, sz, id, gfp)   zdm_calloc((z), (n), (sz), (id), (gfp))
+
+/**
+ * zdm_free_debug() - Extra cleanup for memory debugging.
+ * @znd: ZDM instance
+ * @p: memory to be released.
+ * @sz: allocated size.
+ * @id: allocation bin.
+ *
+ * Additional alloc/free debugging and statistics handling.
+ */
+static inline void zdm_free_debug(struct zoned *znd, void *p, size_t sz, int id)
+{
+#if ALLOC_DEBUG
+	int iter;
+
+	if (atomic_read(&znd->allocs) > znd->hw_allocs)
+		znd->hw_allocs = atomic_read(&znd->allocs);
+	atomic_dec(&znd->allocs);
+	for (iter = 0; iter < znd->hw_allocs; iter++) {
+		if (p == znd->alloc_trace[iter]) {
+			znd->alloc_trace[iter] = NULL;
+			break;
+		}
+	}
+	if (iter == znd->hw_allocs) {
+		Z_ERR(znd, "Free'd something *NOT* allocated? %d", id);
+		dump_stack();
+	}
+#endif
+
+	SpinLock(&znd->stats_lock);
+	if (sz > znd->memstat)
+		Z_ERR(znd, "Free'd more mem than allocated? %d", id);
+
+	if (sz > znd->bins[id]) {
+		Z_ERR(znd, "Free'd more mem than allocated? %d", id);
+		dump_stack();
+	}
+	znd->memstat -= sz;
+	znd->bins[id] -= sz;
+	spin_unlock(&znd->stats_lock);
+}
+
+/**
+ * zdm_free() - Unified free by allocation 'code'
+ * @znd: ZDM instance
+ * @p: memory to be released.
+ * @sz: allocated size.
+ * @code: allocation size
+ *
+ * This (ugly) unified scheme helps to find leaks and monitor usage
+ *   via ioctl tools.
+ */
+static void zdm_free(struct zoned *znd, void *p, size_t sz, u32 code)
+{
+	int id    = code & 0x00FFFF;
+	int flag  = code & 0xFF0000;
+
+	if (p) {
+
+		if (znd)
+			zdm_free_debug(znd, p, sz, id);
+
+		memset(p, 0, sz); /* DEBUG */
+
+		switch (flag) {
+		case GET_ZPG:
+			free_page((unsigned long)p);
+			break;
+		case GET_KM:
+			kfree(p);
+			break;
+		case GET_VM:
+			vfree(p);
+			break;
+		default:
+			Z_ERR(znd,
+			      "zdm_free %p scheme %x not mapped.", p, code);
+			break;
+		}
+	} else {
+		Z_ERR(znd, "double zdm_free %p [%d]", p, id);
+		dump_stack();
+	}
+}
+
+/**
+ * zdm_free_debug() - Extra tracking for memory debugging.
+ * @znd: ZDM instance
+ * @p: memory to be released.
+ * @sz: allocated size.
+ * @id: allocation bin.
+ *
+ * Additional alloc/free debugging and statistics handling.
+ */
+static inline
+void zdm_alloc_debug(struct zoned *znd, void *p, size_t sz, int id)
+{
+#if ALLOC_DEBUG
+	int iter;
+	int count;
+
+	atomic_inc(&znd->allocs);
+
+	count = atomic_read(&znd->allocs);
+	for (iter = 0; iter < count; iter++) {
+		if (!znd->alloc_trace[iter]) {
+			znd->alloc_trace[iter] = p;
+			break;
+		}
+	}
+#endif
+
+	SpinLock(&znd->stats_lock);
+	znd->memstat += sz;
+	znd->bins[id] += sz;
+	spin_unlock(&znd->stats_lock);
+}
+
+
+/**
+ * zdm_alloc() - Unified alloc by 'code':
+ * @znd: ZDM instance
+ * @sz: allocated size.
+ * @code: allocation size
+ * @gfp: kernel allocation flags.
+ *
+ * There a few things (like dm_io) that seem to need pages and not just
+ *   kmalloc'd memory.
+ *
+ * This (ugly) unified scheme helps to find leaks and monitor usage
+ *   via ioctl tools.
+ */
+static void *zdm_alloc(struct zoned *znd, size_t sz, int code, gfp_t gfp)
+{
+	void *pmem = NULL;
+	int id    = code & 0x00FFFF;
+	int flag  = code & 0xFF0000;
+	gfp_t gfp_mask;
+
+#if USE_KTHREAD
+	gfp_mask = GFP_KERNEL;
+#else
+	gfp_mask = gfp ? GFP_ATOMIC : GFP_KERNEL;
+#endif
+
+	if (flag == GET_VM)
+		might_sleep();
+
+	if (gfp_mask != GFP_ATOMIC)
+		might_sleep();
+
+	switch (flag) {
+	case GET_ZPG:
+		pmem = (void *)get_zeroed_page(gfp_mask);
+		if (!pmem && gfp_mask == GFP_ATOMIC) {
+			Z_ERR(znd, "No atomic for %d, try noio.", id);
+			pmem = (void *)get_zeroed_page(GFP_NOIO);
+		}
+		break;
+	case GET_KM:
+		pmem = kzalloc(sz, gfp_mask);
+		if (!pmem && gfp_mask == GFP_ATOMIC) {
+			Z_ERR(znd, "No atomic for %d, try noio.", id);
+			pmem = kzalloc(sz, GFP_NOIO);
+		}
+		break;
+	case GET_VM:
+		WARN_ON(gfp);
+		pmem = vzalloc(sz);
+		break;
+	default:
+		Z_ERR(znd, "zdm alloc scheme for %u unknown.", code);
+		break;
+	}
+
+	if (pmem) {
+		if (znd)
+			zdm_alloc_debug(znd, pmem, sz, id);
+	} else {
+		Z_ERR(znd, "Out of memory. %d", id);
+		dump_stack();
+	}
+
+	return pmem;
+}
+
+/**
+ * zdm_calloc() - Unified alloc by 'code':
+ * @znd: ZDM instance
+ * @n: number of elements in array.
+ * @sz: allocation size of each element.
+ * @c: allocation strategy (VM, KM, PAGE, N-PAGES).
+ * @q: kernel allocation flags.
+ *
+ * calloc is just an zeroed memory array alloc.
+ * all zdm_alloc schemes are for zeroed memory so no extra memset needed.
+ */
+static void *zdm_calloc(struct zoned *znd, size_t n, size_t sz, int c, gfp_t q)
+{
+	return zdm_alloc(znd, sz * n, c, q);
+}
+
+/**
+ * get_io_vcache() - Get a pre-allocated pool of memory for IO.
+ * @znd: ZDM instance
+ * @gfp: Allocation flags if no pre-allocated pool can be found.
+ *
+ * Return: Pointer to pool memory or NULL.
+ */
+static struct io_4k_block *get_io_vcache(struct zoned *znd, gfp_t gfp)
+{
+	struct io_4k_block *cache = NULL;
+	int avail;
+
+	might_sleep();
+
+	for (avail = 0; avail < ARRAY_SIZE(znd->io_vcache); avail++) {
+		if (!test_and_set_bit(avail, &znd->io_vcache_flags)) {
+			cache = znd->io_vcache[avail];
+			if (!cache)
+				znd->io_vcache[avail] = cache =
+					ZDM_CALLOC(znd, IO_VCACHE_PAGES,
+						sizeof(*cache), VM_12, gfp);
+			if (cache)
+				break;
+		}
+	}
+	return cache;
+}
+
+/**
+ * put_io_vcache() - Get a pre-allocated pool of memory for IO.
+ * @znd: ZDM instance
+ * @cache: Allocated cache entry.
+ *
+ * Return: Pointer to pool memory or NULL.
+ */
+static int put_io_vcache(struct zoned *znd, struct io_4k_block *cache)
+{
+	int err = -ENOENT;
+	int avail;
+
+	if (cache) {
+		for (avail = 0; avail < ARRAY_SIZE(znd->io_vcache); avail++) {
+			if (cache == znd->io_vcache[avail]) {
+				WARN_ON(!test_and_clear_bit(avail,
+					&znd->io_vcache_flags));
+				err = 0;
+				break;
+			}
+		}
+	}
+	return err;
+}
+
+/**
+ * map_value() - translate a lookup table entry to a Sector #, or LBA.
+ * @znd: ZDM instance
+ * @delta: little endian map entry.
+ *
+ * Return: LBA or 0 if invalid.
+ */
+static inline u64 map_value(struct zoned *znd, __le32 delta)
+{
+	u64 mval = 0ul;
+
+	if ((delta != MZTEV_UNUSED) && (delta != MZTEV_NF))
+		mval = le32_to_cpu(delta);
+
+	return mval;
+}
+
+/**
+ * map_encode() - Encode a Sector # or LBA to a lookup table entry value.
+ * @znd: ZDM instance
+ * @to_addr: address to encode.
+ * @value: encoded value
+ *
+ * Return: 0.
+ */
+static int map_encode(struct zoned *znd, u64 to_addr, __le32 *value)
+{
+	int err = 0;
+
+	*value = MZTEV_UNUSED;
+	if (to_addr != BAD_ADDR)
+		*value = cpu_to_le32((u32)to_addr);
+
+	return err;
+}
+
+/**
+ * release_memcache() - Free all the memcache blocks.
+ * @znd: ZDM instance
+ *
+ * Return: 0.
+ */
+static int release_memcache(struct zoned *znd)
+{
+	int no;
+
+	for (no = 0; no < MAP_COUNT; no++) {
+		struct list_head *head = &(znd->mclist[no]);
+		struct map_cache *mcache;
+		struct map_cache *_mc;
+
+		if (list_empty(head))
+			return 0;
+
+		SpinLock(&znd->mclck[no]);
+		list_for_each_entry_safe(mcache, _mc, head, mclist) {
+			list_del(&mcache->mclist);
+			ZDM_FREE(znd, mcache->jdata, Z_C4K, PG_08);
+			ZDM_FREE(znd, mcache, sizeof(*mcache), KM_07);
+		}
+		spin_unlock(&znd->mclck[no]);
+
+	}
+	return 0;
+}
+
+/**
+ * warn_bad_lba() - Warn if a give LBA is not valid (Esp if beyond a WP)
+ * @znd: ZDM instance
+ * @lba48: 48 bit lba.
+ *
+ * Return: non-zero if lba is not valid.
+ */
+static inline int warn_bad_lba(struct zoned *znd, u64 lba48)
+{
+#define FMT_ERR "LBA %" PRIx64 " is not valid: Z# %u, off:%x wp:%x"
+	int rcode = 0;
+	u32 zone;
+
+	if (lba48 < znd->data_lba)
+		return rcode;
+
+	zone = _calc_zone(znd, lba48);
+	if (zone < znd->data_zones) { /* FIXME: MAYBE? md_end */
+		u32 gzoff = zone & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[zone >> GZ_BITS];
+		u32 wp_at = le32_to_cpu(wpg->wp_alloc[gzoff]) & Z_WP_VALUE_MASK;
+		u16 off = (lba48 - znd->data_lba) % Z_BLKSZ;
+
+		if (off >= wp_at) {
+			rcode = 1;
+			Z_ERR(znd, FMT_ERR, lba48, zone, off, wp_at);
+			dump_stack();
+		}
+	} else {
+		rcode = 1;
+		Z_ERR(znd, "LBA is not valid - Z# %u, count %u",
+		      zone, znd->data_zones);
+	}
+
+	return rcode;
+}
+
+/**
+ * mapped_free() - Release a page of lookup table entries.
+ * @znd: ZDM instance
+ * @mapped: mapped page struct to free.
+ */
+static void mapped_free(struct zoned *znd, struct map_pg *mapped)
+{
+	if (mapped) {
+		MutexLock(&mapped->md_lock);
+		WARN_ON(test_bit(IS_DIRTY, &mapped->flags));
+		if (mapped->data.addr) {
+			ZDM_FREE(znd, mapped->data.addr, Z_C4K, PG_27);
+			atomic_dec(&znd->incore);
+		}
+		mutex_unlock(&mapped->md_lock);
+		ZDM_FREE(znd, mapped, sizeof(*mapped), KM_20);
+	}
+}
+
+/**
+ * flush_map() - write dirty map entres to disk.
+ * @znd: ZDM instance
+ * @map: Array of mapped pages.
+ * @count: number of elements in range.
+ * Return: non-zero on error.
+ */
+static int flush_map(struct zoned *znd, struct map_pg **map, u32 count)
+{
+	const int use_wq = 1;
+	const int sync = 1;
+	u32 ii;
+	int err = 0;
+
+	if (!map)
+		return err;
+
+	for (ii = 0; ii < count; ii++) {
+		if (map[ii] && map[ii]->data.addr) {
+			cache_if_dirty(znd, map[ii], use_wq);
+			err |= write_if_dirty(znd, map[ii], use_wq, sync);
+		}
+	}
+
+	return err;
+}
+
+/**
+ * zoned_io_flush() - flush all pending IO.
+ * @znd: ZDM instance
+ */
+static int zoned_io_flush(struct zoned *znd)
+{
+	int err = 0;
+
+	set_bit(ZF_FREEZE, &znd->flags);
+	atomic_inc(&znd->gc_throttle);
+
+	mod_delayed_work(znd->gc_wq, &znd->gc_work, 0);
+	flush_delayed_work(&znd->gc_work);
+
+	clear_bit(DO_GC_NO_PURGE, &znd->flags);
+	set_bit(DO_JOURNAL_MOVE, &znd->flags);
+	set_bit(DO_MEMPOOL, &znd->flags);
+	set_bit(DO_SYNC, &znd->flags);
+	set_bit(DO_FLUSH, &znd->flags);
+	queue_work(znd->meta_wq, &znd->meta_work);
+	flush_workqueue(znd->meta_wq);
+	flush_workqueue(znd->bg_wq);
+
+	mod_delayed_work(znd->gc_wq, &znd->gc_work, 0);
+	flush_delayed_work(&znd->gc_work);
+	atomic_dec(&znd->gc_throttle);
+
+	spin_lock(&znd->lzy_lck);
+	INIT_LIST_HEAD(&znd->lzy_pool);
+	spin_unlock(&znd->lzy_lck);
+
+	spin_lock(&znd->zlt_lck);
+	INIT_LIST_HEAD(&znd->zltpool);
+	spin_unlock(&znd->zlt_lck);
+
+	err = flush_map(znd, znd->fwd_tm, znd->map_count);
+	if (err)
+		goto out;
+
+	err = flush_map(znd, znd->rev_tm, znd->map_count);
+	if (err)
+		goto out;
+
+	err = flush_map(znd, znd->fwd_crc, znd->crc_count);
+	if (err)
+		goto out;
+
+	err = flush_map(znd, znd->rev_crc, znd->crc_count);
+	if (err)
+		goto out;
+
+	set_bit(DO_SYNC, &znd->flags);
+	set_bit(DO_FLUSH, &znd->flags);
+	queue_work(znd->meta_wq, &znd->meta_work);
+	flush_workqueue(znd->meta_wq);
+
+out:
+	return err;
+}
+
+/**
+ * release_table_pages() - flush and free all table map entries.
+ * @znd: ZDM instance
+ */
+static void release_table_pages(struct zoned *znd)
+{
+	int entry;
+
+	if (znd->fwd_tm) {
+		for (entry = 0; entry < znd->map_count; entry++) {
+			mapped_free(znd, znd->fwd_tm[entry]);
+			znd->fwd_tm[entry] = NULL;
+		}
+	}
+	if (znd->rev_tm) {
+		for (entry = 0; entry < znd->map_count; entry++) {
+			mapped_free(znd, znd->rev_tm[entry]);
+			znd->rev_tm[entry] = NULL;
+		}
+	}
+	if (znd->fwd_crc) {
+		for (entry = 0; entry < znd->crc_count; entry++) {
+			mapped_free(znd, znd->fwd_crc[entry]);
+			znd->fwd_crc[entry] = NULL;
+		}
+	}
+	if (znd->rev_crc) {
+		for (entry = 0; entry < znd->crc_count; entry++) {
+			mapped_free(znd, znd->rev_crc[entry]);
+			znd->rev_crc[entry] = NULL;
+		}
+	}
+}
+
+/**
+ * _release_wp() - free all WP alloc/usage/used data.
+ * @znd: ZDM instance
+ * @wp: Object to free.
+ */
+static void _release_wp(struct zoned *znd, struct meta_pg *wp)
+{
+	u32 gzno;
+
+	for (gzno = 0; gzno < znd->gz_count; gzno++) {
+		struct meta_pg *wpg = &wp[gzno];
+
+		if (wpg->wp_alloc)
+			ZDM_FREE(znd, wpg->wp_alloc, Z_C4K, PG_06);
+		if (wpg->zf_est)
+			ZDM_FREE(znd, wpg->zf_est, Z_C4K, PG_06);
+		if (wpg->wp_used)
+			ZDM_FREE(znd, wpg->wp_used, Z_C4K, PG_06);
+	}
+	ZDM_FREE(znd, wp, znd->gz_count * sizeof(*wp), VM_21);
+	znd->wp = NULL;
+}
+
+/**
+ * zoned_destroy() - Teardown a zoned device mapper instance.
+ * @znd: ZDM instance
+ */
+static void zoned_destroy(struct zoned *znd)
+{
+	int purge;
+	size_t ppgsz = sizeof(struct map_pg *);
+	size_t mapsz = ppgsz * znd->map_count;
+	size_t crcsz = ppgsz * znd->crc_count;
+
+	del_timer_sync(&znd->timer);
+
+	if (zoned_io_flush(znd))
+		Z_ERR(znd, "sync/flush failure");
+
+	release_table_pages(znd);
+	release_memcache(znd);
+
+	if (znd->dev) {
+		dm_put_device(znd->ti, znd->dev);
+		znd->dev = NULL;
+	}
+	if (znd->io_wq) {
+		destroy_workqueue(znd->io_wq);
+		znd->io_wq = NULL;
+	}
+	if (znd->zone_action_wq) {
+		destroy_workqueue(znd->zone_action_wq);
+		znd->zone_action_wq = NULL;
+	}
+	if (znd->bg_wq) {
+		destroy_workqueue(znd->bg_wq);
+		znd->bg_wq = NULL;
+	}
+	if (znd->gc_wq) {
+		destroy_workqueue(znd->gc_wq);
+		znd->gc_wq = NULL;
+	}
+	if (znd->meta_wq) {
+		destroy_workqueue(znd->meta_wq);
+		znd->meta_wq = NULL;
+	}
+	if (znd->io_client)
+		dm_io_client_destroy(znd->io_client);
+	if (znd->wp)
+		_release_wp(znd, znd->wp);
+	if (znd->md_crcs)
+		ZDM_FREE(znd, znd->md_crcs, Z_C4K * 2, VM_22);
+	if (znd->gc_io_buf)
+		ZDM_FREE(znd, znd->gc_io_buf, Z_C4K * GC_MAX_STRIPE, VM_04);
+	if (znd->gc_postmap.jdata) {
+		size_t sz = Z_BLKSZ * sizeof(*znd->gc_postmap.jdata);
+
+		ZDM_FREE(znd, znd->gc_postmap.jdata, sz, VM_03);
+	}
+	if (znd->fwd_tm)
+		ZDM_FREE(znd, znd->fwd_tm, mapsz, VM_21);
+	if (znd->rev_tm)
+		ZDM_FREE(znd, znd->rev_tm, mapsz, VM_22);
+	if (znd->fwd_crc)
+		ZDM_FREE(znd, znd->fwd_crc, crcsz, VM_21);
+	if (znd->rev_crc)
+		ZDM_FREE(znd, znd->rev_crc, crcsz, VM_22);
+
+	for (purge = 0; purge < ARRAY_SIZE(znd->io_vcache); purge++) {
+		size_t vcsz = IO_VCACHE_PAGES * sizeof(struct io_4k_block *);
+
+		if (znd->io_vcache[purge]) {
+			if (test_and_clear_bit(purge, &znd->io_vcache_flags))
+				Z_ERR(znd, "sync cache entry %d still in use!",
+				      purge);
+			ZDM_FREE(znd, znd->io_vcache[purge], vcsz, VM_12);
+		}
+	}
+	if (znd->z_sballoc)
+		ZDM_FREE(znd, znd->z_sballoc, Z_C4K, PG_05);
+
+	ZDM_FREE(NULL, znd, sizeof(*znd), KM_00);
+}
+
+
+/**
+ * _init_streams() - Setup initial conditions for streams and reserved zones.
+ * @znd: ZDM instance
+ */
+static void _init_streams(struct zoned *znd)
+{
+	u64 stream;
+
+	for (stream = 0; stream < ARRAY_SIZE(znd->bmkeys->stream); stream++)
+		znd->bmkeys->stream[stream] = cpu_to_le32(NOZONE);
+
+	znd->z_meta_resv = cpu_to_le32(znd->data_zones - 2);
+	znd->z_gc_resv   = cpu_to_le32(znd->data_zones - 1);
+	znd->z_gc_free   = znd->data_zones - 2;
+}
+
+/**
+ * _init_mdcrcs() - Setup initial values for empty CRC blocks.
+ * @znd: ZDM instance
+ */
+static void _init_mdcrcs(struct zoned *znd)
+{
+	int idx;
+
+	for (idx = 0; idx < Z_C4K; idx++)
+		znd->md_crcs[idx] = MD_CRC_INIT;
+}
+
+/**
+ * _init_wp() - Setup initial usage for empty data zones.
+ * @znd: ZDM instance
+ */
+static void _init_wp(struct zoned *znd, u32 gzno, struct meta_pg *wpg)
+{
+	u32 gzcount = 1 << GZ_BITS;
+	u32 iter;
+
+	if (znd->data_zones < ((gzno+1) << GZ_BITS))
+		gzcount = znd->data_zones & GZ_MMSK;
+
+	/* mark as empty */
+	for (iter = 0; iter < gzcount; iter++)
+		wpg->zf_est[iter] = cpu_to_le32(Z_BLKSZ);
+
+	/* mark as n/a -- full */
+	gzcount = 1 << GZ_BITS;
+	for (; iter < gzcount; iter++) {
+		wpg->wp_alloc[iter] = cpu_to_le32(~0u);
+		wpg->wp_used[iter] = cpu_to_le32(~0u);
+	}
+}
+
+/**
+ * _alloc_wp() - Allocate needed WP (meta_pg) objects
+ * @znd: ZDM instance
+ */
+struct meta_pg *_alloc_wp(struct zoned *znd)
+{
+	struct meta_pg *wp;
+	u32 gzno;
+
+	wp = ZDM_CALLOC(znd, znd->gz_count, sizeof(*znd->wp), VM_21, NORMAL);
+	if (!wp)
+		goto out;
+	for (gzno = 0; gzno < znd->gz_count; gzno++) {
+		struct meta_pg *wpg = &wp[gzno];
+
+		spin_lock_init(&wpg->wplck);
+		wpg->lba      = 2048ul + (gzno * 2);
+		wpg->wp_alloc = ZDM_ALLOC(znd, Z_C4K, PG_06, NORMAL);
+		wpg->zf_est   = ZDM_ALLOC(znd, Z_C4K, PG_06, NORMAL);
+		wpg->wp_used  = ZDM_ALLOC(znd, Z_C4K, PG_06, NORMAL);
+		if (!wpg->wp_alloc || !wpg->zf_est || !wpg->wp_used) {
+			_release_wp(znd, wp);
+			wp = NULL;
+			goto out;
+		}
+		_init_wp(znd, gzno, wpg);
+		set_bit(IS_DIRTY, &wpg->flags);
+		clear_bit(IS_FLUSH, &wpg->flags);
+	}
+
+out:
+	return wp;
+}
+
+/**
+ * alloc_map_tables() - Allocate map table entries.
+ * @znd: ZDM Target
+ * @mapct: Number of map entries needed.
+ * @crcct: Number of CRC entries needed.
+ */
+static int alloc_map_tables(struct zoned *znd, u64 mapct, u64 crcct)
+{
+	const size_t ptrsz = sizeof(struct map_pg *);
+	int rcode = 0;
+
+	znd->fwd_tm  = ZDM_CALLOC(znd, mapct, ptrsz, VM_21, NORMAL);
+	znd->rev_tm  = ZDM_CALLOC(znd, mapct, ptrsz, VM_22, NORMAL);
+	znd->fwd_crc = ZDM_CALLOC(znd, crcct, ptrsz, VM_21, NORMAL);
+	znd->rev_crc = ZDM_CALLOC(znd, crcct, ptrsz, VM_22, NORMAL);
+	if (!znd->fwd_tm || !znd->rev_tm || !znd->fwd_crc || !znd->rev_crc)
+		rcode = -ENOMEM;
+
+	return rcode;
+}
+
+/**
+ * do_init_zoned() - Initialize a zoned device mapper instance
+ * @ti: DM Target Info
+ * @znd: ZDM Target
+ *
+ * Return: 0 on success.
+ *
+ * Setup the zone pointer table and do a one time calculation of some
+ * basic limits.
+ *
+ *  While start of partition may not be zone aligned
+ *  md_start, data_lba and md_end are all zone aligned.
+ *  From start of partition [or device] to md_start some conv/pref
+ *  space is required for superblocks, memcache, zone pointers, crcs
+ *  and optionally pinned forward lookup blocks.
+ *
+ *    0 < znd->md_start <= znd->data_lba <= znd->md_end
+ *
+ *  incoming [FS sectors] are linearly mapped after md_end.
+ *  And blocks following data_lba are serialzied into zones either with
+ *  explicit stream id support from BIOs [FUTURE], or implictly by LBA
+ *  or type of data.
+ */
+static int do_init_zoned(struct dm_target *ti, struct zoned *znd)
+{
+	u64 size = i_size_read(get_bdev_bd_inode(znd));
+	u64 bdev_nr_sect4k = size / Z_C4K;
+	u64 data_zones = (bdev_nr_sect4k >> Z_BLKBITS) - znd->zdstart;
+	u64 mzcount = dm_div_up(data_zones, MAX_ZONES_PER_MZ);
+	u64 cache = MAX_CACHE_INCR * CACHE_COPIES * MAX_MZ_SUPP;
+	u64 z0_lba = dm_round_up(znd->start_sect, Z_BLKSZ);
+	u64 mapct = dm_div_up(data_zones << Z_BLKBITS, 1024);
+	u64 crcct = dm_div_up(mapct, 2048);
+	int type;
+	u32 mz_min = 0; /* cache */
+	int rcode = 0;
+
+	INIT_LIST_HEAD(&znd->zltpool);
+	INIT_LIST_HEAD(&znd->lzy_pool);
+
+	for (type = 0; type < MAP_COUNT; type++) {
+		INIT_LIST_HEAD(&znd->mclist[type]);
+		spin_lock_init(&znd->mclck[type]);
+	}
+	spin_lock_init(&znd->zlt_lck);
+	spin_lock_init(&znd->lzy_lck);
+	spin_lock_init(&znd->gc_lock);
+	spin_lock_init(&znd->stats_lock);
+	spin_lock_init(&znd->mapkey_lock);
+	spin_lock_init(&znd->ct_lock);
+
+	mutex_init(&znd->pool_mtx);
+	mutex_init(&znd->gc_wait);
+	mutex_init(&znd->gc_postmap.cached_lock);
+	mutex_init(&znd->gc_vcio_lock);
+	mutex_init(&znd->vcio_lock);
+	mutex_init(&znd->mz_io_mutex);
+
+	znd->data_zones = data_zones;
+	znd->gz_count = mzcount;
+	znd->crc_count = crcct;
+	znd->map_count = mapct;
+
+	/* z0_lba - lba of first full zone in disk addr space      */
+
+	znd->md_start = z0_lba - znd->start_sect;
+	if (znd->md_start < cache) {
+		znd->md_start += Z_BLKSZ;
+		mz_min++;
+	}
+
+	/* md_start - lba of first full zone in partition addr space */
+	znd->s_base = znd->md_start;
+	mz_min += mzcount;
+	if (mz_min < znd->zdstart)
+		set_bit(ZF_POOL_FWD, &znd->flags);
+
+	znd->r_base = znd->s_base + (mzcount << Z_BLKBITS);
+	mz_min += mzcount;
+	if (mz_min < znd->zdstart)
+		set_bit(ZF_POOL_REV, &znd->flags);
+
+	znd->c_base = znd->r_base + (mzcount << Z_BLKBITS);
+	znd->c_mid  = znd->c_base + (mzcount * 0x20);
+	znd->c_end  = znd->c_mid + (mzcount * 0x20);
+	mz_min++;
+
+	if (mz_min < znd->zdstart) {
+		Z_ERR(znd, "Conv space for CRCs: Seting ZF_POOL_CRCS");
+		set_bit(ZF_POOL_CRCS, &znd->flags);
+	}
+
+	if (test_bit(ZF_POOL_FWD, &znd->flags)) {
+		znd->sk_low = znd->sk_high = 0;
+	} else {
+		znd->sk_low = znd->s_base;
+		znd->sk_high = znd->sk_low + (mzcount * 0x40);
+	}
+
+	/* logical *ending* lba for meta [bumped up to next zone alignment] */
+	znd->md_end = znd->c_base + (1 << Z_BLKBITS);
+
+	/* actual starting lba for data pool */
+	znd->data_lba = znd->md_end;
+	if (!test_bit(ZF_POOL_CRCS, &znd->flags))
+		znd->data_lba = znd->c_base;
+	if (!test_bit(ZF_POOL_REV, &znd->flags))
+		znd->data_lba = znd->r_base;
+	if (!test_bit(ZF_POOL_FWD, &znd->flags))
+		znd->data_lba = znd->s_base;
+
+	/* NOTE: md_end == data_lba => all meta is in conventional zones. */
+	Z_INFO(znd, "ZDM #%u", BUILD_NO);
+	Z_INFO(znd, "Start Sect: %" PRIx64 " ZDStart# %u md- start %" PRIx64
+		   " end %" PRIx64 " Data LBA %" PRIx64,
+		znd->start_sect, znd->zdstart, znd->md_start,
+		znd->md_end, znd->data_lba);
+
+	Z_INFO(znd, "%s: size:%" PRIu64 " zones: %u, gzs %u, resvd %d",
+	       __func__, size, znd->data_zones, znd->gz_count,
+	       znd->gz_count * znd->mz_provision);
+
+#if ALLOC_DEBUG
+	znd->alloc_trace = vzalloc(sizeof(*znd->alloc_trace) * 65536);
+	if (!znd->alloc_trace) {
+		ti->error = "couldn't allocate in-memory mem debug trace";
+		rcode = -ENOMEM;
+		goto out;
+	}
+#endif
+
+	znd->z_sballoc = ZDM_ALLOC(znd, Z_C4K, PG_05, NORMAL);
+	if (!znd->z_sballoc) {
+		ti->error = "couldn't allocate in-memory superblock";
+		rcode = -ENOMEM;
+		goto out;
+	}
+
+	rcode = alloc_map_tables(znd, mapct, crcct);
+	if (rcode)
+		goto out;
+
+	znd->gc_postmap.jdata = ZDM_CALLOC(znd, Z_BLKSZ,
+		sizeof(*znd->gc_postmap.jdata), VM_03, NORMAL);
+	znd->md_crcs = ZDM_ALLOC(znd, Z_C4K * 2, VM_22, NORMAL);
+	znd->gc_io_buf = ZDM_CALLOC(znd, GC_MAX_STRIPE, Z_C4K, VM_04, NORMAL);
+	znd->wp = _alloc_wp(znd);
+	znd->io_vcache[0] = ZDM_CALLOC(znd, IO_VCACHE_PAGES,
+				sizeof(struct io_4k_block), VM_12, NORMAL);
+	znd->io_vcache[1] = ZDM_CALLOC(znd, IO_VCACHE_PAGES,
+				sizeof(struct io_4k_block), VM_12, NORMAL);
+
+	if (!znd->gc_postmap.jdata || !znd->md_crcs || !znd->gc_io_buf ||
+	    !znd->wp || !znd->io_vcache[0] || !znd->io_vcache[1]) {
+		rcode = -ENOMEM;
+		goto out;
+	}
+	znd->gc_postmap.jsize = Z_BLKSZ;
+	_init_mdcrcs(znd);
+
+	znd->io_client = dm_io_client_create();
+	if (!znd->io_client) {
+		rcode = -ENOMEM;
+		goto out;
+	}
+
+	znd->meta_wq = create_singlethread_workqueue("znd_meta_wq");
+	if (!znd->meta_wq) {
+		ti->error = "couldn't start metadata worker thread";
+		rcode = -ENOMEM;
+		goto out;
+	}
+
+	znd->gc_wq = create_singlethread_workqueue("znd_gc_wq");
+	if (!znd->gc_wq) {
+		ti->error = "couldn't start GC workqueue.";
+		rcode = -ENOMEM;
+		goto out;
+	}
+
+	znd->bg_wq = create_singlethread_workqueue("znd_bg_wq");
+	if (!znd->bg_wq) {
+		ti->error = "couldn't start background workqueue.";
+		rcode = -ENOMEM;
+		goto out;
+	}
+
+	znd->io_wq = create_singlethread_workqueue("kzoned_dm_io_wq");
+	if (!znd->io_wq) {
+		ti->error = "couldn't start DM I/O thread";
+		rcode = -ENOMEM;
+		goto out;
+	}
+
+	znd->zone_action_wq = create_singlethread_workqueue("zone_act_wq");
+	if (!znd->zone_action_wq) {
+		ti->error = "couldn't start zone action worker";
+		rcode = -ENOMEM;
+		goto out;
+	}
+
+#if USE_KTHREAD
+	init_waitqueue_head(&znd->wait_bio);
+	init_waitqueue_head(&znd->wait_fifo);
+#endif
+
+	INIT_WORK(&znd->meta_work, meta_work_task);
+	INIT_WORK(&znd->bg_work, bg_work_task);
+	INIT_DELAYED_WORK(&znd->gc_work, gc_work_task);
+	setup_timer(&znd->timer, activity_timeout, (unsigned long)znd);
+	znd->last_w = BAD_ADDR;
+	set_bit(DO_SYNC, &znd->flags);
+
+out:
+	return rcode;
+}
+
+/**
+ * check_metadata_version() - Test ZDM version for compatibility.
+ * @sblock: Super block
+ *
+ * Return 0 if valud, or -EINVAL if version is not recognized.
+ */
+static int check_metadata_version(struct zdm_superblock *sblock)
+{
+	u32 metadata_version = le32_to_cpu(sblock->version);
+
+	if (metadata_version < MIN_ZONED_VERSION
+	    || metadata_version > MAX_ZONED_VERSION) {
+		DMERR("Unsupported metadata version %u found.",
+		      metadata_version);
+		DMERR("Only versions between %u and %u supported.",
+		      MIN_ZONED_VERSION, MAX_ZONED_VERSION);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * sb_crc32() - CRC check for superblock.
+ * @sblock: Superblock to check.
+ */
+static __le32 sb_crc32(struct zdm_superblock *sblock)
+{
+	const __le32 was = sblock->csum;
+	u32 crc;
+
+	sblock->csum = 0;
+	crc = crc32c(~(u32) 0u, sblock, sizeof(*sblock)) ^ SUPERBLOCK_CSUM_XOR;
+
+	sblock->csum = was;
+	return cpu_to_le32(crc);
+}
+
+/**
+ * sb_check() - Check the superblock to see if it is valid and not corrupt.
+ * @sblock: Superblock to check.
+ */
+static int sb_check(struct zdm_superblock *sblock)
+{
+	__le32 csum_le;
+
+	if (le64_to_cpu(sblock->magic) != SUPERBLOCK_MAGIC) {
+		DMERR("sb_check failed: magic %" PRIx64 ": wanted %lx",
+		      le64_to_cpu(sblock->magic), SUPERBLOCK_MAGIC);
+		return -EILSEQ;
+	}
+
+	csum_le = sb_crc32(sblock);
+	if (csum_le != sblock->csum) {
+		DMERR("sb_check failed: csum %u: wanted %u",
+		      csum_le, sblock->csum);
+		return -EILSEQ;
+	}
+
+	return check_metadata_version(sblock);
+}
+
+/**
+ * zoned_create_disk() - Initialize the on-disk format of a zoned device mapper.
+ * @ti: DM Target Instance
+ * @znd: ZDM Instance
+ */
+static int zoned_create_disk(struct dm_target *ti, struct zoned *znd)
+{
+	const int reset_non_empty = 1;
+	struct zdm_superblock *sblock = znd->super_block;
+	int err;
+
+	memset(sblock, 0, sizeof(*sblock));
+	generate_random_uuid(sblock->uuid);
+	sblock->magic = cpu_to_le64(SUPERBLOCK_MAGIC);
+	sblock->version = cpu_to_le32(Z_VERSION);
+	sblock->zdstart = cpu_to_le32(znd->zdstart);
+
+	err = zoned_wp_sync(znd, reset_non_empty);
+
+	return err;
+}
+
+
+/**
+ * zoned_repair() - Attempt easy on-line fixes.
+ * @znd: ZDM Instance
+ *
+ * Repair an otherwise good device mapper instance that was not cleanly removed.
+ */
+static int zoned_repair(struct zoned *znd)
+{
+	Z_INFO(znd, "Is Dirty .. zoned_repair consistency fixer TODO!!!.");
+	return -ENOMEM;
+}
+
+
+/**
+ * zoned_init_disk() - Init from exising or re-create DM Target (ZDM)
+ * @ti: DM Target Instance
+ * @znd: ZDM Instance
+ * @create: Create if not found.
+ * @force: Force create even if it looks like a ZDM was here.
+ *
+ * Locate the existing SB on disk and re-load or create the device-mapper
+ * instance based on the existing disk state.
+ */
+static int zoned_init_disk(struct dm_target *ti, struct zoned *znd,
+			   int create, int force)
+{
+	struct mz_superkey *key_blk = znd->z_sballoc;
+
+	int jinit = 1;
+	int n4kblks = 1;
+	int use_wq = 1;
+	int rc = 0;
+	u32 zdstart = znd->zdstart;
+
+	memset(key_blk, 0, sizeof(*key_blk));
+
+	znd->super_block = &key_blk->sblock;
+
+	znd->bmkeys = key_blk;
+	znd->bmkeys->sig0 = Z_KEY_SIG;
+	znd->bmkeys->sig1 = cpu_to_le64(Z_KEY_SIG);
+	znd->bmkeys->magic = cpu_to_le64(Z_TABLE_MAGIC);
+
+	_init_streams(znd);
+
+	znd->stale.binsz = STREAM_SIZE;
+	if (znd->data_zones < STREAM_SIZE)
+		znd->stale.binsz = znd->data_zones;
+	else if ((znd->data_zones / STREAM_SIZE) > STREAM_SIZE)
+		znd->stale.binsz = dm_div_up(znd->data_zones, STREAM_SIZE);
+	znd->stale.count = dm_div_up(znd->data_zones, znd->stale.binsz);
+
+	Z_ERR(znd, "Bin: Sz %u, count %u", znd->stale.binsz, znd->stale.count);
+
+	if (create && force) {
+		Z_ERR(znd, "Force Creating a clean instance.");
+	} else if (find_superblock(znd, use_wq, 1)) {
+		u64 sb_lba = 0;
+		u64 generation;
+
+/* FIXME: Logic [zdstart] not well tested or ... logicial */
+
+		Z_INFO(znd, "Found existing superblock");
+		if (zdstart != znd->zdstart) {
+			if (force) {
+				Z_ERR(znd, "  (force) zdstart: %u <- %u",
+					zdstart, znd->zdstart);
+			} else {
+				znd->zdstart = zdstart;
+				jinit = 0;
+			}
+		}
+
+		generation = mcache_greatest_gen(znd, use_wq, &sb_lba, NULL);
+		Z_DBG(znd, "Generation: %" PRIu64 " @ %" PRIx64,
+			generation, sb_lba);
+
+		rc = read_block(ti, DM_IO_KMEM, key_blk, sb_lba,
+				n4kblks, use_wq);
+		if (rc) {
+			ti->error = "Superblock read error.";
+			return rc;
+		}
+	}
+
+	rc = sb_check(znd->super_block);
+	if (rc) {
+		jinit = 0;
+		if (create) {
+			DMWARN("Check failed .. creating superblock.");
+			zoned_create_disk(ti, znd);
+			znd->super_block->nr_zones =
+				cpu_to_le64(znd->data_zones);
+			DMWARN("in-memory superblock created.");
+			znd->is_empty = 1;
+		} else {
+			ti->error = "Superblock check failed.";
+			return rc;
+		}
+	}
+
+	if (sb_test_flag(znd->super_block, SB_DIRTY)) {
+		int repair_check = zoned_repair(znd);
+
+		if (!force) {
+			/* if repair failed -- don't load from disk */
+			if (repair_check)
+				jinit = 0;
+		} else if (repair_check && jinit) {
+			Z_ERR(znd, "repair failed, force enabled loading ...");
+		}
+	}
+
+	if (jinit) {
+		Z_ERR(znd, "INIT: Reloading DM Zoned metadata from DISK");
+		znd->zdstart = le32_to_cpu(znd->super_block->zdstart);
+		set_bit(DO_JOURNAL_LOAD, &znd->flags);
+		queue_work(znd->meta_wq, &znd->meta_work);
+		Z_ERR(znd, "Waiting for load to complete.");
+		flush_workqueue(znd->meta_wq);
+	}
+
+	Z_ERR(znd, "ZONED: Build No %d marking superblock dirty.", BUILD_NO);
+
+	/* write the 'dirty' flag back to disk. */
+	sb_set_flag(znd->super_block, SB_DIRTY);
+	znd->super_block->csum = sb_crc32(znd->super_block);
+
+	return 0;
+}
+
+/**
+ * compare_tlba() - Compare on tlba48 ignoring high 16 bits.
+ * @x1: Page of map cache
+ * @x2: Page of map cache
+ *
+ * Return: -1 less than, 1 greater than, 0 if equal.
+ */
+static int compare_tlba(const void *x1, const void *x2)
+{
+	const struct map_sect_to_lba *r1 = x1;
+	const struct map_sect_to_lba *r2 = x2;
+	const u64 v1 = le64_to_lba48(r1->tlba, NULL);
+	const u64 v2 = le64_to_lba48(r2->tlba, NULL);
+
+	return (v1 < v2) ? -1 : ((v1 > v2) ? 1 : 0);
+}
+
+/**
+ * _lsearch_tlba() - Do a linear search over a page of map_cache entries.
+ * @mcache: Page of map cache entries to search.
+ * @dm_s: tlba being sought.
+ *
+ * Return: 0 to jcount - 1 if found. -1 if not found
+ */
+static int _lsearch_tlba(struct map_cache *mcache, u64 dm_s)
+{
+	int at = -1;
+	int jentry;
+
+	for (jentry = mcache->jcount; jentry > 0; jentry--) {
+		u64 logi = le64_to_lba48(mcache->jdata[jentry].tlba, NULL);
+
+		if (logi == dm_s) {
+			at = jentry - 1;
+			goto out;
+		}
+	}
+
+out:
+	return at;
+}
+
+/**
+ * _lsearch_extent() - Do a linear search over a page of discard entries.
+ * @mcache: Page of map cache entries to search.
+ * @dm_s: tlba being sought.
+ *
+ * Return: 0 to jcount - 1 if found. -1 if not found
+ *
+ * NOTE: In this case the match is if the tlba is include in the extent
+ */
+static int _lsearch_extent(struct map_cache *mcache, u64 dm_s)
+{
+	int at = -1;
+	int jentry;
+
+	for (jentry = mcache->jcount; jentry > 0; jentry--) {
+		u64 addr = le64_to_lba48(mcache->jdata[jentry].tlba, NULL);
+		u64 blocks = le64_to_lba48(mcache->jdata[jentry].bval, NULL);
+
+		if ((dm_s >= addr) && dm_s < (addr+blocks)) {
+			at = jentry - 1;
+			goto out;
+		}
+	}
+
+out:
+	return at;
+}
+
+/**
+ * _bsrch_tlba() - Do a binary search over a page of map_cache entries.
+ * @mcache: Page of map cache entries to search.
+ * @tlba: tlba being sought.
+ *
+ * Return: 0 to jcount - 1 if found. -1 if not found
+ */
+static int _bsrch_tlba(struct map_cache *mcache, u64 tlba)
+{
+	int at = -1;
+	void *base = &mcache->jdata[1];
+	void *map;
+	struct map_sect_to_lba find;
+
+	find.tlba = lba48_to_le64(0, tlba);
+
+	if (mcache->jcount < 0 || mcache->jcount > mcache->jsize)
+		return at;
+
+	map = bsearch(&find, base, mcache->jcount, sizeof(find), compare_tlba);
+	if (map)
+		at = (map - base) / sizeof(find);
+
+	return at;
+}
+
+/**
+ * compare_extent() - Compare on tlba48 ignoring high 16 bits.
+ * @x1: Page of map cache
+ * @x2: Page of map cache
+ *
+ * Return: -1 less than, 1 greater than, 0 if equal.
+ */
+static int compare_ext(const void *x1, const void *x2)
+{
+	const struct map_sect_to_lba *r1 = x1;
+	const struct map_sect_to_lba *r2 = x2;
+	const u64 key = le64_to_lba48(r1->tlba, NULL);
+	const u64 val = le64_to_lba48(r2->tlba, NULL);
+	const u64 ext = le64_to_lba48(r2->bval, NULL);
+
+	if (key < val)
+		return -1;
+	if (key < (val+ext))
+		return 0;
+	return 1;
+}
+
+
+/**
+ * _bsrch_extent() - Do a binary search over a page of map_cache extents.
+ * @mcache: Page of map cache extent entries to search.
+ * @tlba: tlba being sought.
+ *
+ * Return: 0 to jcount - 1 if found. -1 if not found
+ */
+static int _bsrch_extent(struct map_cache *mcache, u64 tlba)
+{
+	int at = -1;
+	void *base = &mcache->jdata[1];
+	void *map;
+	struct map_sect_to_lba find;
+
+	find.tlba = lba48_to_le64(0, tlba);
+	map = bsearch(&find, base, mcache->jcount, sizeof(find), compare_ext);
+	if (map)
+		at = (map - base) / sizeof(find);
+
+#if 1
+	{
+		int cmp = _lsearch_extent(mcache, tlba);
+
+		if (cmp != at) {
+			pr_err("_bsrch_extent is failing got %d need %d.\n",
+				at, cmp);
+			at = cmp;
+		}
+	}
+#endif
+
+	return at;
+}
+
+/**
+ * _bsrch_tlba() - Mutex Lock and binary search.
+ * @mcache: Page of map cache entries to search.
+ * @tlba: tlba being sought.
+ *
+ * Return: 0 to jcount - 1 if found. -1 if not found
+ */
+static int _bsrch_tlba_lck(struct map_cache *mcache, u64 tlba)
+{
+	int at = -1;
+
+	MutexLock(&mcache->cached_lock);
+	at = _bsrch_tlba(mcache, tlba);
+	mutex_unlock(&mcache->cached_lock);
+
+	return at;
+}
+
+/**
+ * _bsrch_extent_lck() - Mutex Lock and binary search.
+ * @mcache: Page of map cache extent entries to search.
+ * @tlba: tlba being sought.
+ *
+ * Return: 0 to jcount - 1 if found. -1 if not found
+ */
+static int _bsrch_extent_lck(struct map_cache *mcache, u64 tlba)
+{
+	int at = -1;
+
+	MutexLock(&mcache->cached_lock);
+	at = _bsrch_extent(mcache, tlba);
+	mutex_unlock(&mcache->cached_lock);
+
+	return at;
+}
+
+/**
+ * mcache_bsearch() - Do a binary search over a page of map_cache entries.
+ * @mcache: Page of map cache entries to search.
+ * @tlba: tlba being sought.
+ */
+static inline int mcache_bsearch(struct map_cache *mcache, u64 tlba)
+{
+	int at = (mcache->map_content == IS_MAP)
+		? _bsrch_tlba(mcache, tlba)
+		: _bsrch_extent(mcache, tlba);
+
+	return at;
+}
+
+
+/**
+ * mcache_lsearch() - Do a linear search over a page of map/discard entries.
+ * @mcache: Page of map cache entries to search.
+ * @dm_s: tlba being sought.
+ */
+static inline int mcache_lsearch(struct map_cache *mcache, u64 tlba)
+{
+	int at = (mcache->map_content == IS_MAP)
+		? _lsearch_tlba(mcache, tlba)
+		: _lsearch_extent(mcache, tlba);
+
+	return at;
+}
+
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+static u64 z_lookup_key_range(struct zoned *znd, u64 addr)
+{
+	u64 found = 0ul;
+
+	if (test_bit(ZF_POOL_FWD, &znd->flags))
+		return found;
+
+	if ((znd->sk_low <= addr) && (addr < znd->sk_high))
+		found = FWD_TM_KEY_BASE + (addr - znd->sk_low);
+
+	return found;
+}
+
+/**
+ * increment_used_blks() - Update the 'used' WP when data hits disk.
+ * @znd: ZDM Instance
+ * @lba: blba if bio completed.
+ * @blks: number of blocks of bio completed.
+ *
+ * Called from a BIO end_io function so should not sleep or deadlock
+ * The 'critical' piece here is ensuring that the wp is advanced to 0x10000
+ * Secondarily is triggering filled_zone which ultimatly sets the
+ *   Z_WP_GC_READY in wp_alloc. While important this flag could be set
+ *   during other non-critical passes over wp_alloc and wp_used such
+ *   as during update_stale_ratio().
+ */
+static void increment_used_blks(struct zoned *znd, u64 lba, u32 blks)
+{
+	u32 zone = _calc_zone(znd, lba);
+	u32 wwp = ((lba - znd->data_lba) - (zone << Z_BLKBITS)) + blks;
+
+	if (zone < znd->data_zones) {
+		u32 gzno  = zone >> GZ_BITS;
+		u32 gzoff = zone & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 used, uflags;
+
+		used = le32_to_cpu(wpg->wp_used[gzoff]);
+		uflags = used & Z_WP_FLAGS_MASK;
+		used &= Z_WP_VALUE_MASK;
+
+		if (wwp > used) {
+			wpg->wp_used[gzoff] = cpu_to_le32(wwp | uflags);
+			/* signal zone closure */
+			if (wwp == Z_BLKSZ)
+				znd->filled_zone = zone;
+		}
+	}
+}
+
+/**
+ * current_mapping() - Lookup a logical sector address to find the disk LBA
+ * @znd: ZDM instance
+ * @nodisk: Optional ignore the discard cache
+ * @addr: Logical LBA of page.
+ * @gfp: Memory allocation rule
+ *
+ * Return: Disk LBA or 0 if not found.
+ */
+static u64 _current_mapping(struct zoned *znd, int nodisc, u64 addr, gfp_t gfp)
+{
+	u64 found = 0ul;
+
+	if (addr < znd->data_lba)
+		found = addr;
+	if (!found)
+		found = z_lookup_key_range(znd, addr);
+	if (!found && !nodisc) {
+		found = z_lookup_cache(znd, addr, IS_DISCARD);
+		if (found) {
+			found = 0;
+			goto out;
+		}
+	}
+	if (!found)
+		found = z_lookup_cache(znd, addr, IS_MAP);
+	if (!found)
+		found = z_lookup_table(znd, addr, gfp);
+
+out:
+	return found;
+}
+
+/**
+ * current_mapping() - Lookup a logical sector address to find the disk LBA
+ * @znd: ZDM instance
+ * @addr: Logical LBA of page.
+ * @gfp: Memory allocation rule
+ *
+ * NOTE: Discard cache is checked.
+ *
+ * Return: Disk LBA or 0 if not found.
+ */
+static u64 current_mapping(struct zoned *znd, u64 addr, gfp_t gfp)
+{
+	const int nodisc = 0;
+
+	return _current_mapping(znd, nodisc, addr, gfp);
+}
+
+/**
+ * z_mapped_add_one() - Add an entry to the map cache block mapping.
+ * @znd: ZDM Instance
+ * @tlba: Address being discarded.
+ * @blks: number of blocks being discard.
+ * @gfp: Current memory allocation scheme.
+ *
+ * Add a new extent entry.
+ */
+static int z_mapped_add_one(struct zoned *znd, u64 dm_s, u64 lba, gfp_t gfp)
+{
+	int err = 0;
+
+	if (dm_s < znd->data_lba)
+		return err;
+
+#if 0 /* FIXME: Do the RIGHT thing ... whatever that is. */
+
+	/*
+	 * location of the SLT key sectors need to be
+	 * stashed into the sector lookup table block map
+	 * Does dm_s point in the sector lookup table block map
+	 */
+	if (!test_bit(ZF_POOL_FWD, &znd->flags)) {
+		if ((znd->sk_low <= dm_s) && (dm_s < znd->sk_high))
+			lba = FWD_TM_KEY_BASE + (dm_s - znd->sk_low);
+	}
+#endif
+
+	/*
+	 * If this mapping over-writes a discard entry then we need to punch a
+	 * hole in the discard entry.
+	 *  - Find the discard entry.
+	 *  - Translate a chunk of the discard into map entries.
+	 *  - Break [or trim] the discard entry.
+	 */
+	if (lba) {
+		int is_locked = mutex_is_locked(&znd->mz_io_mutex);
+
+		if (is_locked)
+			mutex_unlock(&znd->mz_io_mutex);
+		z_discard_range(znd, dm_s, gfp);
+		if (is_locked)
+			MutexLock(&znd->mz_io_mutex);
+	}
+
+	do {
+		err = z_to_map_list(znd, dm_s, lba, gfp);
+	} while (-EBUSY == err);
+
+	return err;
+}
+
+/**
+ * z_discard_small() - Translate discard extents to map cache block mapping.
+ * @znd: ZDM Instance
+ * @tlba: Address being discarded.
+ * @blks: number of blocks being discard.
+ * @gfp: Current memory allocation scheme.
+ *
+ * Add a new extent entry.
+ */
+static int z_discard_small(struct zoned *znd, u64 tlba, u64 count, gfp_t gfp)
+{
+	const int nodisc = 1;
+	int err = 0;
+	u64 ii;
+
+	for (ii = 0; ii < count; ii++) {
+		u64 lba = _current_mapping(znd, nodisc, tlba+ii, gfp);
+
+		if (lba)
+			do {
+				err = z_to_map_list(znd, tlba+ii, 0ul, gfp);
+			} while (-EBUSY == err);
+		if (err)
+			break;
+	}
+	return err;
+}
+
+/**
+ * z_discard_large() - Add a discard extent to the mapping cache
+ * @znd: ZDM Instance
+ * @tlba: Address being discarded.
+ * @blks: number of blocks being discard.
+ * @gfp: Current memory allocation scheme.
+ *
+ * Add a new extent entry.
+ */
+static int z_discard_large(struct zoned *znd, u64 tlba, u64 blks, gfp_t gfp)
+{
+	int rcode = 0;
+
+	/* for a large discard ... add it to the discard queue */
+	do {
+		rcode = z_to_discard_list(znd, tlba, blks, gfp);
+	} while (-EBUSY == rcode);
+
+	return rcode;
+}
+
+/**
+ * mapped_discard() - Add a discard extent to the mapping cache
+ * @znd: ZDM Instance
+ * @tlba: Address being discarded.
+ * @blks: number of blocks being discard.
+ * @merge: Attempt to merge with an existing extent.
+ * @gfp: Current memory allocation scheme.
+ *
+ * First attempt to merge with an existing entry.
+ * Otherwise attempt to push small entires into the map_cache 1:1 mapping
+ * Otherwise add a new extent entry.
+ */
+static
+int mapped_discard(struct zoned *znd, u64 tlba, u64 blks, int merge, gfp_t gfp)
+{
+	int err = 0;
+
+	if (merge && discard_merge(znd, tlba, blks))
+		goto done;
+
+	if (blks < DISCARD_MAX_INGRESS || znd->dc_entries > 39) {
+		/* io_mutex is avoid adding map cache entries during SYNC */
+		MutexLock(&znd->mz_io_mutex);
+		err = z_discard_small(znd, tlba, blks, gfp);
+		mutex_unlock(&znd->mz_io_mutex);
+		goto done;
+	}
+
+	err = z_discard_large(znd, tlba, blks, gfp);
+
+done:
+	return err;
+}
+
+/**
+ * z_mapped_discard() - Add a discard extent to the mapping cache
+ * @znd: ZDM Instance
+ * @tlba: Address being discarded.
+ * @blks: number of blocks being discard.
+ * @gfp: Current memory allocation scheme.
+ */
+static int z_mapped_discard(struct zoned *znd, u64 tlba, u64 blks, gfp_t gfp)
+{
+	const int merge = 1;
+
+	return mapped_discard(znd, tlba, blks, merge, gfp);
+}
+
+/**
+ * mcache_alloc() - Allocate a new chunk of map cache
+ * @znd: ZDM Instance
+ * @gfp: Current memory allocation scheme.
+ */
+static struct map_cache *mcache_alloc(struct zoned *znd, gfp_t gfp)
+{
+	struct map_cache *mcache;
+
+	mcache = ZDM_ALLOC(znd, sizeof(*mcache), KM_07, gfp);
+	if (mcache) {
+		mutex_init(&mcache->cached_lock);
+		mcache->jcount = 0;
+		mcache->jsorted = 0;
+		mcache->jdata = ZDM_CALLOC(znd, Z_UNSORTED,
+			sizeof(*mcache->jdata), PG_08, gfp);
+
+		if (mcache->jdata) {
+			u64 logical = Z_LOWER48;
+			u64 physical = Z_LOWER48;
+
+			mcache->jdata[0].tlba = cpu_to_le64(logical);
+			mcache->jdata[0].bval = cpu_to_le64(physical);
+			mcache->jsize = Z_UNSORTED - 1;
+
+		} else {
+			Z_ERR(znd, "%s: in memory journal is out of space.",
+			      __func__);
+			ZDM_FREE(znd, mcache, sizeof(*mcache), KM_07);
+			mcache = NULL;
+		}
+	}
+	return mcache;
+}
+
+/**
+ * mcache_first_get() - Get the first chunk from the list.
+ * @znd: ZDM Instance
+ * @mcache: Map cache chunk to add.
+ * @type: Type flag of mcache entry (MAP or DISCARD extent).
+ *
+ * The mcache chunk will have it's refcount elevated in the xchg.
+ * Caller should use xchg_next to navigate through the list or
+ * deref_cache when exiting from the list walk early.
+ */
+static struct map_cache *mcache_first_get(struct zoned *znd, int type)
+{
+	unsigned long flags;
+	struct map_cache *mc;
+
+	spin_lock_irqsave(&znd->mclck[type], flags);
+	mc = list_first_entry_or_null(&znd->mclist[type], typeof(*mc), mclist);
+	if (mc && (&mc->mclist != &znd->mclist[type]))
+		mcache_ref(mc);
+	else
+		mc = NULL;
+	spin_unlock_irqrestore(&znd->mclck[type], flags);
+
+	return mc;
+}
+
+/**
+ * mcache_put_get_next() - Get the next map cache page and put current.
+ * @znd: ZDM Instance
+ * @mcache: Map cache chunk to add.
+ * @type: Type flag of mcache entry (MAP or DISCARD extent).
+ *
+ * The current mcache chunk will have an elevated refcount,
+ * that will dropped, while the new the returned chunk will
+ * have it's refcount elevated in the xchg.
+ */
+static inline struct map_cache *mcache_put_get_next(struct zoned *znd,
+						    struct map_cache *mcache,
+						    int type)
+{
+	unsigned long flags;
+	struct map_cache *next;
+
+	spin_lock_irqsave(&znd->mclck[type], flags);
+	next = list_next_entry(mcache, mclist);
+	if (next && (&next->mclist != &znd->mclist[type]))
+		mcache_ref(next);
+	else
+		next = NULL;
+	mcache_deref(mcache);
+	spin_unlock_irqrestore(&znd->mclck[type], flags);
+
+	return next;
+}
+
+/**
+ * mclist_add() - Add a new chunk to the map_cache pool.
+ * @znd: ZDM Instance
+ * @mcache: Map cache chunk to add.
+ * @type: Type flag of mcache entry (MAP or DISCARD extent).
+ */
+static inline void mclist_add(struct zoned *znd,
+			      struct map_cache *mcache, int type)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&znd->mclck[type], flags);
+	list_add(&mcache->mclist, &znd->mclist[type]);
+	spin_unlock_irqrestore(&znd->mclck[type], flags);
+}
+
+/**
+ * mclist_del() - Release an (empty) chunk of map cache.
+ * @znd: ZDM Instance
+ * @mcache: Map cache chunk to add.
+ * @type: Type flag of mcache entry (MAP or DISCARD extent).
+ */
+static inline int mclist_del(struct zoned *znd,
+			     struct map_cache *mcache, int type)
+{
+	unsigned long flags;
+	int deleted = 0;
+
+	if (spin_trylock_irqsave(&znd->mclck[type], flags)) {
+		MutexLock(&mcache->cached_lock);
+		if (mcache->jcount == 0 && mcache_getref(mcache) == 1) {
+			list_del(&mcache->mclist);
+			deleted = 1;
+		}
+		mutex_unlock(&mcache->cached_lock);
+		spin_unlock_irqrestore(&znd->mclck[type], flags);
+	}
+	return deleted;
+}
+
+/**
+ * mcache_put() - Dereference an mcache page
+ * @mcache: Map cache page.
+ */
+static inline void mcache_put(struct map_cache *mcache)
+{
+	mcache_deref(mcache);
+}
+
+/**
+ * memcache_sort() - Sort an unsorted map cache page
+ * @znd: ZDM instance
+ * @mcache: Map cache page.
+ *
+ * Sort a map cache entry if is sorted.
+ * Lock using mutex if not already locked.
+ */
+static void memcache_sort(struct zoned *znd, struct map_cache *mcache)
+{
+	mcache_ref(mcache);
+	if (mcache->jsorted < mcache->jcount) {
+		sort(&mcache->jdata[1], mcache->jcount,
+		     sizeof(*mcache->jdata), compare_tlba, NULL);
+		mcache->jsorted = mcache->jcount;
+	}
+	mcache_deref(mcache);
+}
+
+/**
+ * memcache_lock_and_sort() - Attempt to sort an unsorted map cache page
+ * @znd: ZDM instance
+ * @mcache: Map cache page.
+ *
+ * Sort a map cache entry if is not busy being purged to the ZLT.
+ *
+ * Return: -EBUSY if busy, or 0 if sorted.
+ */
+static int memcache_lock_and_sort(struct zoned *znd, struct map_cache *mcache)
+{
+	if (mcache_is_busy(mcache))
+		return -EBUSY;
+
+	if (mcache->jsorted == mcache->jcount)
+		return 0;
+
+	mutex_lock_nested(&mcache->cached_lock, SINGLE_DEPTH_NESTING);
+	memcache_sort(znd, mcache);
+	mutex_unlock(&mcache->cached_lock);
+	return 0;
+}
+
+/**
+ * z_lookup_cache() - Scan mcache entries for addr
+ * @znd: ZDM Instance
+ * @addr: Address [tLBA] to find.
+ * @type: mcache type (MAP or DISCARD cache).
+ */
+static u64 z_lookup_cache(struct zoned *znd, u64 addr, int type)
+{
+	struct map_cache *mcache;
+	u64 found = 0ul;
+
+	mcache = mcache_first_get(znd, type);
+	while (mcache) {
+		int at;
+		int err;
+
+		/* sort, if needed. only err is -EBUSY so do a linear find. */
+		err = memcache_lock_and_sort(znd, mcache);
+		if (!err)
+			at = mcache_bsearch(mcache, addr);
+		else
+			at = mcache_lsearch(mcache, addr);
+
+		if (at != -1) {
+			struct map_sect_to_lba *data = &mcache->jdata[at + 1];
+
+			found = le64_to_lba48(data->bval, NULL);
+		}
+		if (found) {
+			mcache_put(mcache);
+			goto out;
+		}
+
+		mcache = mcache_put_get_next(znd, mcache, type);
+	}
+out:
+	return found;
+}
+
+/**
+ * mcache_insert - Insertion sort (for mostly sorted data)
+ * @znd: ZDM Instance
+ * @mcache: Map Cache block
+ * @tlba: upper stack sector
+ * @blba: lower mapped lba
+ */
+static void mcache_insert(struct zoned *znd, struct map_cache *mcache,
+			 u64 tlba, u64 blba)
+{
+	u16 fido = ++mcache->jcount;
+	u16 idx;
+
+	WARN_ON(mcache->jcount > mcache->jsize);
+
+	for (idx = fido; idx > 1; --idx) {
+		u16 prev = idx - 1;
+		u64 a0 = le64_to_lba48(mcache->jdata[prev].tlba, NULL);
+
+		if (a0 < tlba) {
+			mcache->jdata[idx].tlba = lba48_to_le64(0, tlba);
+			mcache->jdata[idx].bval = lba48_to_le64(0, blba);
+			if ((mcache->jsorted + 1) == mcache->jcount)
+				mcache->jsorted = mcache->jcount;
+			break;
+		}
+		/* move UP .. tlba < a0 */
+		mcache->jdata[idx].tlba = mcache->jdata[prev].tlba;
+		mcache->jdata[idx].bval = mcache->jdata[prev].bval;
+	}
+	if (idx == 1) {
+		mcache->jdata[idx].tlba = lba48_to_le64(0, tlba);
+		mcache->jdata[idx].bval = lba48_to_le64(0, blba);
+		mcache->jsorted = mcache->jcount;
+	}
+}
+
+
+/**
+ * mc_delete_entry - Overwrite a map_cache entry (for mostly sorted data)
+ * @mcache: Map Cache block
+ * @entry: Entry to remove. [1 .. count]
+ */
+void mc_delete_entry(struct map_cache *mcache, int entry)
+{
+	int iter;
+
+	MutexLock(&mcache->cached_lock);
+
+	mcache->jdata[entry].tlba = lba48_to_le64(0, 0ul);
+	mcache->jdata[entry].bval = lba48_to_le64(0, 0ul);
+
+	for (iter = entry; iter < mcache->jcount; iter++) {
+		int next = iter + 1;
+
+		mcache->jdata[iter].tlba = mcache->jdata[next].tlba;
+		mcache->jdata[iter].bval = mcache->jdata[next].bval;
+	}
+	if (mcache->jcount > 0) {
+		if (mcache->jsorted == mcache->jcount)
+			mcache->jsorted--;
+		mcache->jcount--;
+	} else {
+		pr_err("mcache delete ... discard beyound empty\n");
+	}
+	mutex_unlock(&mcache->cached_lock);
+}
+
+/**
+ * _discard_split_entry() - Split a discard into two parts.
+ * @znd: ZDM Instance
+ * @mcahce: mcache holding the current entry
+ * @at: location of entry in mcache (0 .. count - 1)
+ * @addr: address to punch a hole at
+ * @gfp: current memory allocation rules.
+ *
+ * Discard 'extents' need to broken up when a sub range is being allocated
+ * for writing. Ideally the punch only requires the extent to be shrunk.
+ */
+static int _discard_split_entry(struct zoned *znd, struct map_cache *mcache,
+				int at, u64 addr, gfp_t gfp)
+{
+	const int merge = 0;
+	int err = 0;
+	struct map_sect_to_lba *entry = &mcache->jdata[at + 1];
+	u64 tlba   = le64_to_lba48(entry->tlba, NULL);
+	u64 blocks = le64_to_lba48(entry->bval, NULL);
+	u64 dblks  = blocks;
+	const u64 chunk = addr - tlba;
+
+	/*
+	 * Now chunk is a number of blocks until addr starts.
+	 * If the chunk is small push the discard immediately.
+	 * Otherwise punch a hole and see if we need to create
+	 * a new entry with what remains.
+	 */
+	if (chunk > 0 && chunk < DISCARD_MAX_INGRESS) {
+
+		err = mapped_discard(znd, tlba, chunk, merge, gfp);
+		if (err)
+			goto out;
+
+		blocks -= chunk;
+		tlba += chunk;
+
+		entry->tlba = lba48_to_le64(0, tlba);
+		entry->bval = lba48_to_le64(0, blocks);
+
+		if (blocks == 0ul) {
+			mc_delete_entry(mcache, at + 1);
+			goto out;
+		}
+		dblks = blocks;
+	} else {
+		dblks = blocks - chunk; /* nblks from addr to end */
+	}
+
+	/*
+	 * When chunk >= DISCARD_MAX_INGRESS
+	 * then the current entry needs to be ended at 'chunk'.
+	 * Note that tlba != addr in this case and 'chunk' will
+	 *	be until we get around to committing to truncating
+	 *	the entry extent. Specifcily we need to ensure that
+	 *	the hole doens't expose blocks covered by the current
+	 *	extent.
+	 */
+	if (dblks > DISCARD_MAX_INGRESS)
+		dblks = DISCARD_MAX_INGRESS - 1;
+
+	err = mapped_discard(znd, addr, dblks, merge, gfp);
+	if (err)
+		goto out;
+
+	/*
+	 * Now we have moved the hole into the map cache we are free
+	 * to update the entry.
+	 * If tlba == addr the we have continued to discard from the
+	 * from of this extent.
+	 *
+	 * If tlba != addr then we need discarded from the middle or
+	 * the end. Either there is a hole and we need a second extent
+	 * entry or we ran to the end of the extent and we can just
+	 * truncate the extent.
+	 */
+
+	if (tlba == addr) {
+		blocks -= dblks;
+		tlba += dblks;
+		entry->tlba = lba48_to_le64(0, tlba);
+		entry->bval = lba48_to_le64(0, blocks);
+		if (blocks == 0ul)
+			mc_delete_entry(mcache, at + 1);
+	} else {
+		u64 start = addr + dblks;
+
+		/*
+		 * Because we are doing insertion sorting
+		 * and tlba < addr we can safely add a new
+		 * discard extent entry.
+		 * As the current entry is 'stable' we don't need to
+		 * re-acquire it.
+		 */
+		blocks -= (chunk + dblks);
+		if (blocks > 0ul) {
+			err = mapped_discard(znd, start, blocks, merge, gfp);
+			if (err)
+				goto out;
+		}
+		entry->bval = lba48_to_le64(0, chunk);
+	}
+
+out:
+	return err;
+}
+
+
+/**
+ * z_discard_range() - Migrate a discard range to memcache/ZLT
+ * @znd: ZDM Instance
+ * @addr: tLBA target for breaking up a discard range.
+ * @gfp: Memory allocation scheme
+ *
+ * When an address is being written to a check is made to see if the tLBA
+ * overlaps with an exisitng discard pool. If so the discard pool is
+ * split and multiple smaller discard entries are pushed into the memcache.
+ *
+ * The overall purpose of the discard pool is to reduce the amount of intake
+ * on the memcache to avoid starving the I/O requests.
+ */
+static u64 z_discard_range(struct zoned *znd, u64 addr, gfp_t gfp)
+{
+	struct map_cache *mcache;
+	u64 found = 0ul;
+	int type = IS_DISCARD;
+
+	mcache = mcache_first_get(znd, type);
+	while (mcache) {
+		int at;
+		int err;
+
+		/* sort, if needed. only err is -EBUSY so do a linear find. */
+		err = memcache_lock_and_sort(znd, mcache);
+		if (!err)
+			at = mcache_bsearch(mcache, addr);
+		else
+			at = mcache_lsearch(mcache, addr);
+
+		if (at != -1) {
+			/* break apart the entry */
+			err = _discard_split_entry(znd, mcache, at, addr, gfp);
+			if (err) {
+				mcache_put(mcache);
+				goto out;
+			}
+			found = 1;
+		}
+
+		if (found) {
+			mcache_put(mcache);
+			goto out;
+		}
+
+		mcache = mcache_put_get_next(znd, mcache, type);
+	}
+out:
+	return found;
+}
+
+/**
+ * z_discard_partial() - Migrate a discard range to memcache/ZLT
+ * @znd: ZDM Instance
+ * @minblks: Target number of blocks to cull.
+ * @gfp: Memory allocation scheme
+ *
+ * When an address is being written to a check is made to see if the tLBA
+ * overlaps with an exisitng discard pool. If so the discard pool is
+ * split and multiple smaller discard entries are pushed into the memcache.
+ *
+ * The overall purpose of the discard pool is to reduce the amount of intake
+ * on the memcache to avoid starving the I/O requests.
+ */
+static int z_discard_partial(struct zoned *znd, u32 minblks, gfp_t gfp)
+{
+	struct map_cache *mcache;
+	int completions = 0;
+	int type = IS_DISCARD;
+	int lumax = minblks;
+
+	mcache = mcache_first_get(znd, type);
+	while (mcache) {
+		int at = -1;
+		int err;
+
+		if (mcache->jcount > 1)
+			at = 0;
+
+		if (at != -1) {
+			const int nodisc = 1;
+			s32 e = at + 1;
+			u64 tlba = le64_to_lba48(mcache->jdata[e].tlba, NULL);
+			u64 blks = le64_to_lba48(mcache->jdata[e].bval, NULL);
+			u64 chunk = DISCARD_MAX_INGRESS - 1;
+			u32 dcount = 0;
+			u32 dstop = minblks;
+
+			while (blks > 0 && dcount < dstop && --lumax > 0) {
+				u64 lba = _current_mapping(znd, nodisc,
+							   tlba, gfp);
+				if (lba) {
+					do {
+						err = z_to_map_list(znd, tlba,
+								    0ul, gfp);
+					} while (-EBUSY == err);
+					if (err)
+						break;
+					dcount++;
+				}
+				tlba++;
+				blks--;
+			}
+
+			if (chunk > blks)
+				chunk = blks;
+
+			if (chunk == 0) {
+				mc_delete_entry(mcache, e);
+				completions++;
+			} else if (chunk < DISCARD_MAX_INGRESS) {
+				err = mapped_discard(znd, tlba, chunk, 0, gfp);
+				if (err) {
+					mcache_put(mcache);
+					goto out;
+				}
+				blks -= chunk;
+				tlba += chunk;
+
+				mcache->jdata[e].tlba = lba48_to_le64(0, tlba);
+				mcache->jdata[e].bval = lba48_to_le64(0, blks);
+				if (blks == 0ul)
+					mc_delete_entry(mcache, e);
+			}
+			completions++;
+		}
+		if (completions) {
+			int deleted = 0;
+
+			if (completions > 1 &&
+			    znd->dc_entries > MEMCACHE_HWMARK &&
+			    mcache->jcount == 0 &&
+			    mcache_getref(mcache) == 1)
+				deleted = mclist_del(znd, mcache, type);
+
+			mcache_put(mcache);
+
+			if (deleted) {
+				ZDM_FREE(znd, mcache->jdata, Z_C4K, PG_08);
+				ZDM_FREE(znd, mcache, sizeof(*mcache), KM_07);
+				mcache = NULL;
+				znd->dc_entries--;
+			}
+			goto out;
+		}
+		mcache = mcache_put_get_next(znd, mcache, type);
+	}
+out:
+	return completions;
+}
+
+/**
+ * lba_in_zone() - Scan a map cache entry for any targets pointing to zone.
+ * @znd: ZDM instance
+ * @mcache: Map cache page.
+ * @zone: Zone (that was gc'd).
+ *
+ * Scan the map cache page to see if any entries are pointing to a zone.
+ *
+ * Return: 0 if none are found. 1 if any are found.
+ */
+static int lba_in_zone(struct zoned *znd, struct map_cache *mcache, u32 zone)
+{
+	int jentry;
+
+	if (zone >= znd->data_zones)
+		goto out;
+
+	for (jentry = mcache->jcount; jentry > 0; jentry--) {
+		u64 lba = le64_to_lba48(mcache->jdata[jentry].bval, NULL);
+
+		if (lba && _calc_zone(znd, lba) == zone)
+			return 1;
+	}
+out:
+	return 0;
+}
+
+/**
+ * gc_verify_cache() - Test map cache to see if any entries point to zone.
+ * @znd: ZDM instance
+ * @zone: Zone (that was gc'd).
+ *
+ * Scan the map cache to see if any entries are pointing to a zone that
+ * was just GC'd.
+ * If any are found it indicates a bug.
+ *
+ * Return: 0 on success or 1 on error and sets meta_result to ENOSPC
+ *         to effectivly freeze ZDM.
+ */
+static int gc_verify_cache(struct zoned *znd, u32 zone)
+{
+	struct map_cache *mcache = NULL;
+	int err = 0;
+	int type = IS_MAP;
+
+	mcache = mcache_first_get(znd, type);
+	while (mcache) {
+		MutexLock(&mcache->cached_lock);
+		if (lba_in_zone(znd, mcache, zone)) {
+			Z_ERR(znd, "GC: **ERR** %" PRIx32
+			      " LBA in cache <= Corrupt", zone);
+			err = 1;
+			znd->meta_result = -ENOSPC;
+		}
+		mutex_unlock(&mcache->cached_lock);
+		mcache = mcache_put_get_next(znd, mcache, type);
+	}
+
+	return err;
+}
+
+
+/**
+ * __cached_to_tables() - Migrate map cache entries to ZLT
+ * @znd: ZDM instance
+ * @type: Which type (MAP) of cache entries to migrate.
+ * @zone: zone to force migration for partial memcache block
+ *
+ * Scan the memcache and move any full blocks to lookup tables
+ * If a (the) partial memcache block contains lbas that map to zone force
+ * early migration of the memcache block to ensure it is properly accounted
+ * for and migrated during and upcoming GC pass.
+ *
+ * Return: 0 on success or -errno value
+ */
+static int __cached_to_tables(struct zoned *znd, int type, u32 zone)
+{
+	struct map_cache *mcache = NULL;
+	int err = 0;
+	int try_free = 0;
+
+	mcache = mcache_first_get(znd, type);
+	while (mcache) {
+		int deleted = 0;
+		struct map_cache *jskip;
+
+		mcache_busy(mcache);
+		MutexLock(&mcache->cached_lock);
+		if (mcache->jcount == mcache->jsize) {
+			memcache_sort(znd, mcache);
+			err = move_to_map_tables(znd, mcache);
+			if (!err && (mcache->jcount == 0))
+				try_free++;
+		} else {
+			if (lba_in_zone(znd, mcache, zone)) {
+				Z_ERR(znd,
+					"Moving %d Runts because z: %u",
+					mcache->jcount, zone);
+
+				memcache_sort(znd, mcache);
+				err = move_to_map_tables(znd, mcache);
+			}
+		}
+		mutex_unlock(&mcache->cached_lock);
+		mcache_unbusy(mcache);
+		if (err) {
+			mcache_put(mcache);
+			Z_ERR(znd, "%s: Sector map failed.", __func__);
+			goto out;
+		}
+
+		mcache_ref(mcache);
+		jskip = mcache_put_get_next(znd, mcache, type);
+
+		if (try_free > 0 && znd->mc_entries > MEMCACHE_HWMARK &&
+		    mcache->jcount == 0 && mcache_getref(mcache) == 1) {
+			deleted = mclist_del(znd, mcache, type);
+			try_free--;
+		}
+
+		mcache_deref(mcache);
+
+		if (deleted) {
+			ZDM_FREE(znd, mcache->jdata, Z_C4K, PG_08);
+			ZDM_FREE(znd, mcache, sizeof(*mcache), KM_07);
+			mcache = NULL;
+			znd->mc_entries--;
+		}
+		mcache = jskip;
+	}
+out:
+
+	return err;
+}
+
+
+/**
+ * _cached_to_tables() - Migrate memcache entries to lookup tables
+ * @znd: ZDM instance
+ * @zone: zone to force migration for partial memcache block
+ *
+ * Scan the memcache and move any full blocks to lookup tables
+ * If a (the) partial memcache block contains lbas that map to zone force
+ * early migration of the memcache block to ensure it is properly accounted
+ * for and migrated during and upcoming GC pass.
+ *
+ * Return: 0 on success or -errno value
+ */
+static int _cached_to_tables(struct zoned *znd, u32 zone)
+{
+	int err;
+
+	err = __cached_to_tables(znd, IS_MAP, zone);
+	return err;
+}
+
+
+/**
+ * z_flush_bdev() - Request backing device flushed to disk.
+ * @znd: ZDM instance
+ *
+ * Return: 0 on success or -errno value
+ */
+static int z_flush_bdev(struct zoned *znd, gfp_t gfp)
+{
+	int err;
+	sector_t bi_done;
+
+	err = blkdev_issue_flush(znd->dev->bdev, gfp, &bi_done);
+	if (err)
+		Z_ERR(znd, "%s: flush failing sector %lu!", __func__, bi_done);
+
+	return err;
+}
+
+/**
+ * pg_delete - Free a map_pg with spin locks held.
+ * @znd: ZDM Instance
+ * @expg: Page being released.
+ *
+ * Forced inline as it is 'optional' and because it is called with
+ * spin locks enabled and only from a single caller.
+ */
+static __always_inline int pg_delete(struct zoned *znd, struct map_pg *expg)
+{
+	struct map_pg **zlt;
+	spinlock_t *lock;
+	int req_flush = 0;
+	int dropped = 0;
+	int index = expg->index;
+	int is_lut = test_bit(IS_LUT, &expg->flags);
+	int is_fwd = test_bit(IS_FWD, &expg->flags);
+	u32 count = is_lut ? znd->map_count : znd->crc_count;
+
+	if (is_lut) {
+		zlt = is_fwd ? znd->fwd_tm : znd->rev_tm;
+		lock = &znd->mapkey_lock;
+	} else {
+		zlt = is_fwd ? znd->fwd_crc : znd->rev_crc;
+		lock = &znd->ct_lock;
+	}
+
+	if (!spin_trylock(lock))
+		return req_flush;
+
+	if (index > -1 && index < count && expg == zlt[index] &&
+	    test_bit(IS_DROPPED, &expg->flags)) {
+		zlt[index] = NULL;
+		MutexLock(&expg->md_lock);
+		if (test_and_clear_bit(IS_LAZY, &expg->flags)) {
+			clear_bit(IS_DROPPED, &expg->flags);
+			clear_bit(DELAY_ADD, &expg->flags);
+			if (expg->data.addr) {
+				void *pg = expg->data.addr;
+
+				expg->data.addr = NULL;
+				ZDM_FREE(znd, pg, Z_C4K, PG_27);
+				atomic_dec(&znd->incore);
+			} else {
+				Z_ERR(znd, "** No data pg? %llx", expg->lba);
+			}
+			list_del(&expg->lazy);
+			znd->in_lzy--;
+			req_flush = !test_bit(IS_FLUSH, &expg->flags);
+			dropped = 1;
+		} else {
+			Z_ERR(znd, "Detected double list del.");
+		}
+		mutex_unlock(&expg->md_lock);
+		if (dropped)
+			ZDM_FREE(znd, expg, sizeof(*expg), KM_20);
+	}
+	spin_unlock(lock);
+	return req_flush;
+}
+
+/**
+ * manage_lazy_activity() - Migrate delayed 'add' entries to the 'ZTL'
+ * @znd: ZDM Instance
+ *
+ * The lzy list is used to perform less critical activities that could
+ * be done via the ZTL primary list but gives a second chance when
+ *   - Adding if the spin lock would lock.
+ *   - Deleting ... if the cache entry turns out to be 'hotter' than
+ *     the default we can catch it and make it 'hotter' before the
+ *     hotness indicator is lost.
+ */
+static int manage_lazy_activity(struct zoned *znd)
+{
+	struct map_pg *expg;
+	struct map_pg *_tpg;
+	int want_flush = 0;
+	const u32 msecs = MEM_PURGE_MSECS;
+
+	spin_lock(&znd->lzy_lck);
+	expg = list_first_entry_or_null(&znd->lzy_pool, typeof(*expg), lazy);
+	if (!expg || (&expg->lazy == &znd->lzy_pool))
+		goto out;
+
+	_tpg = list_next_entry(expg, lazy);
+	while (&expg->lazy != &znd->lzy_pool) {
+		/*
+		 * this should never happen:
+		 */
+		if (test_bit(IS_DIRTY, &expg->flags))
+			set_bit(DELAY_ADD, &expg->flags);
+		/*
+		 * Migrage pg to zltlst list
+		 */
+		if (test_bit(DELAY_ADD, &expg->flags)) {
+			if (spin_trylock(&znd->zlt_lck)) {
+				if (!test_bit(IN_ZLT, &expg->flags)) {
+					list_del(&expg->lazy);
+					znd->in_lzy--;
+					clear_bit(IS_LAZY, &expg->flags);
+					clear_bit(IS_DROPPED, &expg->flags);
+					clear_bit(DELAY_ADD, &expg->flags);
+					set_bit(IN_ZLT, &expg->flags);
+					list_add(&expg->zltlst, &znd->zltpool);
+					znd->in_zlt++;
+				} else {
+					Z_ERR(znd, "** ZLT double add? %llx",
+						expg->lba);
+				}
+				spin_unlock(&znd->zlt_lck);
+			}
+#if ENABLE_PG_FREE_VIA_LAZY
+		} else {
+			/*
+			 * Delete page
+			 */
+			if (!test_bit(IN_ZLT, &expg->flags) &&
+			     test_bit(IS_DROPPED, &expg->flags)) {
+				if (is_expired_msecs(expg->age, msecs))
+					want_flush |= pg_delete(znd, expg);
+			}
+#endif
+		}
+		expg = _tpg;
+		_tpg = list_next_entry(expg, lazy);
+	}
+
+out:
+	spin_unlock(&znd->lzy_lck);
+	return want_flush;
+}
+
+
+
+/**
+ * mark_clean_flush_zlt() - Mark all non-dirty ZLT blocks as 'FLUSH'
+ * @znd: ZDM instance
+ *
+ * After a FLUSH/FUA these blocks are on disk and redundant FLUSH
+ * can be skipped if the block is later ejected.
+ */
+static void mark_clean_flush_zlt(struct zoned *znd)
+{
+	struct map_pg *expg = NULL;
+	struct map_pg *_tpg;
+
+	spin_lock(&znd->zlt_lck);
+	if (list_empty(&znd->zltpool))
+		goto out;
+
+	expg = list_last_entry(&znd->zltpool, typeof(*expg), zltlst);
+	if (!expg || &expg->zltlst == (&znd->zltpool))
+		goto out;
+
+	_tpg = list_prev_entry(expg, zltlst);
+	while (&expg->zltlst != &znd->zltpool) {
+		ref_pg(expg);
+		if (!test_bit(IS_DIRTY, &expg->flags))
+			set_bit(IS_FLUSH, &expg->flags);
+		deref_pg(expg);
+		expg = _tpg;
+		_tpg = list_prev_entry(expg, zltlst);
+	}
+
+out:
+	spin_unlock(&znd->zlt_lck);
+}
+
+/**
+ * mark_clean_flush_lzy() - Mark all non-dirty ZLT blocks as 'FLUSH'
+ * @znd: ZDM instance
+ *
+ * After a FLUSH/FUA these blocks are on disk and redundant FLUSH
+ * can be skipped if the block is later ejected.
+ */
+static void mark_clean_flush_lzy(struct zoned *znd)
+{
+	struct map_pg *expg = NULL;
+	struct map_pg *_tpg;
+
+	spin_lock(&znd->lzy_lck);
+	expg = list_first_entry_or_null(&znd->lzy_pool, typeof(*expg), lazy);
+	if (!expg || (&expg->lazy == &znd->lzy_pool))
+		goto out;
+
+	_tpg = list_next_entry(expg, lazy);
+	while (&expg->lazy != &znd->lzy_pool) {
+		ref_pg(expg);
+		if (!test_bit(IS_DIRTY, &expg->flags))
+			set_bit(IS_FLUSH, &expg->flags);
+		deref_pg(expg);
+		expg = _tpg;
+		_tpg = list_next_entry(expg, lazy);
+	}
+
+out:
+	spin_unlock(&znd->lzy_lck);
+}
+
+/**
+ * mark_clean_flush() - Mark all non-dirty ZLT/LZY blocks as 'FLUSH'
+ * @znd: ZDM instance
+ *
+ * After a FLUSH/FUA these blocks are on disk and redundant FLUSH
+ * can be skipped if the block is later ejected.
+ */
+static void mark_clean_flush(struct zoned *znd)
+{
+	mark_clean_flush_zlt(znd);
+	mark_clean_flush_lzy(znd);
+}
+
+/**
+ * do_sync_metadata() - Write ZDM state to disk.
+ * @znd: ZDM instance
+ *
+ * Return: 0 on success or -errno value
+ */
+static int do_sync_metadata(struct zoned *znd, int sync, int drop)
+{
+	int err = 0;
+	int want_flush;
+
+	want_flush = manage_lazy_activity(znd);
+	if (want_flush)
+		set_bit(DO_FLUSH, &znd->flags);
+
+	/* if drop is non-zero, DO_FLUSH may be set on return */
+	err = sync_mapped_pages(znd, sync, drop);
+	if (err) {
+		Z_ERR(znd, "Uh oh: sync_mapped_pages -> %d", err);
+		goto out;
+	}
+
+	/*
+	 * If we are lucky then this sync will get us to a 'clean'
+	 * state and the follow on bdev flush is redunant and skipped
+	 *
+	 * If not we will suffer a performance stall because we were
+	 * ejected blocks.
+	 *
+	 * TODO: On Sync/Flush/FUA we can mark all of our clean ZLT
+	 *       as flushed and we can bypass elevating the drop count
+	 *       to trigger a flush for such already flushed blocks.
+	 */
+	want_flush = test_bit(DO_FLUSH, &znd->flags);
+	err = z_mapped_sync(znd);
+	if (err) {
+		Z_ERR(znd, "Uh oh. z_mapped_sync -> %d", err);
+		goto out;
+	}
+
+	if (test_and_clear_bit(DO_FLUSH, &znd->flags)) {
+		err = z_flush_bdev(znd, GFP_KERNEL);
+		if (err) {
+			Z_ERR(znd, "Uh oh. flush_bdev failed. -> %d", err);
+			goto out;
+		}
+		want_flush = 1;
+	}
+	if (want_flush)
+		mark_clean_flush(znd);
+
+out:
+	return err;
+}
+
+/**
+ * do_init_from_journal() - Restore ZDM state from disk.
+ * @znd: ZDM instance
+ *
+ * Return: 0 on success or -errno value
+ */
+static int do_init_from_journal(struct zoned *znd)
+{
+	int err = 0;
+
+	if (test_and_clear_bit(DO_JOURNAL_LOAD, &znd->flags))
+		err = z_mapped_init(znd);
+
+	return err;
+}
+
+/**
+ * do_journal_to_table() - Migrate memcache entries to lookup tables
+ * @znd: ZDM instance
+ *
+ * Return: 0 on success or -errno value
+ */
+static int do_journal_to_table(struct zoned *znd)
+{
+	int err = 0;
+
+	if (test_and_clear_bit(DO_JOURNAL_MOVE, &znd->flags) ||
+	    test_bit(DO_SYNC, &znd->flags))
+		err = _cached_to_tables(znd, znd->data_zones);
+
+	return err;
+}
+
+/**
+ * do_sync_to_disk() - Write ZDM state to disk.
+ * @znd: ZDM instance
+ *
+ * Return: 0 on success or -errno value
+ */
+static int do_sync_to_disk(struct zoned *znd)
+{
+	int err = 0;
+	int drop = 0;
+	int sync = 0;
+
+	if (test_and_clear_bit(DO_SYNC, &znd->flags) ||
+	    test_bit(DO_FLUSH, &znd->flags))
+		sync = 1;
+
+	if (test_and_clear_bit(DO_MEMPOOL, &znd->flags)) {
+		int pool_size = MZ_MEMPOOL_SZ * 4;
+
+#if ENABLE_PG_FREE_VIA_LAZY
+		/**
+		 * Trust our cache miss algo
+		 */
+		pool_size = MZ_MEMPOOL_SZ * 3;
+#endif
+
+		if (is_expired_msecs(znd->age, MEM_PURGE_MSECS * 2))
+			pool_size = 3;
+		else if (is_expired_msecs(znd->age, MEM_PURGE_MSECS))
+			pool_size = MZ_MEMPOOL_SZ;
+
+		if (atomic_read(&znd->incore) > pool_size)
+			drop = atomic_read(&znd->incore) - pool_size;
+	}
+	if (sync || drop)
+		err = do_sync_metadata(znd, sync, drop);
+
+	return err;
+}
+
+/**
+ * meta_work_task() - Worker thread from metadata activity.
+ * @work: Work struct containing ZDM instance.
+ */
+static void meta_work_task(struct work_struct *work)
+{
+	int err = 0;
+	int locked = 0;
+	struct zoned *znd;
+
+	if (!work)
+		return;
+
+	znd = container_of(work, struct zoned, meta_work);
+	if (!znd)
+		return;
+
+	err = do_init_from_journal(znd);
+
+	if (test_bit(DO_JOURNAL_MOVE, &znd->flags) ||
+	    test_bit(DO_FLUSH, &znd->flags) ||
+	    test_bit(DO_SYNC, &znd->flags)) {
+		MutexLock(&znd->mz_io_mutex);
+		locked = 1;
+	}
+
+	/*
+	 * Reduce memory pressure on journal list of arrays
+	 * by pushing them into the sector map lookup tables
+	 */
+	if (!err)
+		err = do_journal_to_table(znd);
+
+	/* force a consistent set of meta data out to disk */
+	if (!err)
+		err = do_sync_to_disk(znd);
+
+	if (locked)
+		mutex_unlock(&znd->mz_io_mutex);
+
+	znd->age = jiffies_64;
+	if (err < 0)
+		znd->meta_result = err;
+
+	clear_bit(DO_METAWORK_QD, &znd->flags);
+}
+
+/**
+ * _update_disc() - Update an existing map cache entry.
+ * @znd: ZDM instance
+ * @mcache: Map cache block
+ * @at: Data entry in map cache block
+ * @dm_s: tBLA mapping from
+ * @lba: New lba
+ * @gfp: Allocation flags to use.
+ *
+ * If overwriting an entry will trigger an 'unused' to be pushed recording
+ * the old block being stale.
+ */
+static int _update_entry(struct zoned *znd, struct map_cache *mcache, int at,
+			 u64 dm_s, u64 lba, gfp_t gfp)
+{
+	struct map_sect_to_lba *data;
+	u64 lba_was;
+	int err = 0;
+
+	data = &mcache->jdata[at + 1];
+	lba_was = le64_to_lba48(data->bval, NULL);
+	lba &= Z_LOWER48;
+	data->bval = lba48_to_le64(0, lba);
+
+	if (lba != lba_was) {
+		Z_DBG(znd, "Remap %" PRIx64 " -> %" PRIx64
+			   " (was %" PRIx64 "->%" PRIx64 ")",
+		      dm_s, lba, le64_to_lba48(data->tlba, NULL), lba_was);
+		err = unused_phy(znd, lba_was, 0, gfp);
+		if (err == 1)
+			err = 0;
+	}
+
+	return err;
+}
+
+/**
+ * _update_disc() - Update an existing discard cache entry
+ * @znd: ZDM instance
+ * @mcache: Map cache block
+ * @at: Data entry in map cache block
+ * @dm_s: tBLA mapping from
+ * @blks: number of blocks
+ */
+static int _update_disc(struct zoned *znd, struct map_cache *mcache, int at,
+			u64 dm_s, u64 blks)
+{
+	struct map_sect_to_lba *data;
+	u64 oldsz;
+	int err = 0;
+
+	data = &mcache->jdata[at + 1];
+	oldsz = le64_to_lba48(data->bval, NULL);
+	data->bval = lba48_to_le64(0, blks);
+
+	return err;
+}
+
+/**
+ * mc_update() - Update an existing cache entry
+ * @znd: ZDM instance
+ * @mcache: Map cache block
+ * @at: Data entry in map cache block
+ * @dm_s: tBLA mapping from
+ * @lba: Value (lba or number of blocks)
+ * @type: List (type) to be adding to (MAP or DISCARD)
+ * @gfp: Allocation (kmalloc) flags
+ */
+static int mc_update(struct zoned *znd, struct map_cache *mcache, int at,
+		     u64 dm_s, u64 lba, int type, gfp_t gfp)
+{
+	int rcode;
+
+	if (type == IS_MAP)
+		rcode = _update_entry(znd, mcache, at, dm_s, lba, gfp);
+	else
+		rcode = _update_disc(znd, mcache, at, dm_s, lba);
+
+	return rcode;
+}
+
+/**
+ * _mapped_list() - Add a entry to the discard or map cache
+ * @znd: ZDM instance
+ * @dm_s: tBLA mapping from
+ * @lba: Value (lba or number of blocks)
+ * @type: List (type) to be adding to (MAP or DISCARD)
+ * @gfp: Allocation (kmalloc) flags
+ */
+static
+int _mapped_list(struct zoned *znd, u64 dm_s, u64 lba, int type, gfp_t gfp)
+{
+	struct map_cache *mcache = NULL;
+	struct map_cache *mc_add = NULL;
+	int handled = 0;
+	int list_count = 0;
+	int err = 0;
+	int add_to_list = 0;
+
+	mcache = mcache_first_get(znd, type);
+	while (mcache) {
+		int at;
+
+		MutexLock(&mcache->cached_lock);
+		memcache_sort(znd, mcache);
+		at = _bsrch_tlba(mcache, dm_s);
+		if (at != -1) {
+			mcache_ref(mcache);
+			err = mc_update(znd, mcache, at, dm_s, lba, type, gfp);
+			mcache_deref(mcache);
+			handled = 1;
+		} else if (!mc_add) {
+			if (mcache->jcount < mcache->jsize) {
+				mc_add = mcache;
+				mcache_ref(mc_add);
+			}
+		}
+		mutex_unlock(&mcache->cached_lock);
+		if (handled) {
+			if (mc_add)
+				mcache_deref(mc_add);
+			mcache_put(mcache);
+			goto out;
+		}
+		mcache = mcache_put_get_next(znd, mcache, type);
+		list_count++;
+	}
+
+	/* ------------------------------------------------------------------ */
+	/* ------------------------------------------------------------------ */
+
+	if (!mc_add) {
+		mc_add = mcache_alloc(znd, gfp);
+		if (!mc_add) {
+			Z_ERR(znd, "%s: in memory journal is out of space.",
+			      __func__);
+			err = -ENOMEM;
+			goto out;
+		}
+		mcache_ref(mc_add);
+		if (type == IS_MAP) {
+			if (list_count > MEMCACHE_HWMARK)
+				set_bit(DO_JOURNAL_MOVE, &znd->flags);
+			znd->mc_entries = list_count + 1;
+		} else {
+			znd->dc_entries = list_count + 1;
+		}
+		add_to_list = 1;
+	}
+
+	/* ------------------------------------------------------------------ */
+	/* ------------------------------------------------------------------ */
+
+	if (mc_add) {
+		MutexLock(&mc_add->cached_lock);
+
+		if (!mc_add->jdata) {
+			mc_add->jdata = ZDM_CALLOC(znd, Z_UNSORTED,
+				sizeof(*mc_add->jdata), PG_08, gfp);
+		}
+		if (!mc_add->jdata) {
+			Z_ERR(znd, "%s: in memory journal is out of space.",
+			      __func__);
+			err = -ENOMEM;
+			goto out;
+		}
+
+		if (mc_add->jcount < mc_add->jsize) {
+			mcache_insert(znd, mc_add, dm_s, lba);
+			if (add_to_list)
+				mclist_add(znd, mc_add, type);
+		} else {
+			err = -EBUSY;
+		}
+		mutex_unlock(&mc_add->cached_lock);
+		mcache_put(mc_add);
+	}
+out:
+	return err;
+}
+
+/**
+ * z_to_map_list() - Add a discard extent entry to the discard cache
+ * @znd: ZDM instance
+ * @dm_s: tBLA mapping from
+ * @blks: Number of blocks in the extent.
+ * @gfp: Allocation (kmalloc) flags
+ */
+static int z_to_discard_list(struct zoned *znd, u64 dm_s, u64 blks, gfp_t gfp)
+{
+	return _mapped_list(znd, dm_s, blks, IS_DISCARD, gfp);
+}
+
+/**
+ * z_to_map_list() - Add a map cache entry to the map cache
+ * @znd: ZDM instance
+ * @dm_s: tBLA mapping from
+ * @lba: bLBA mapping to
+ * @gfp: Allocation (kmalloc) flags
+ */
+static int z_to_map_list(struct zoned *znd, u64 dm_s, u64 lba, gfp_t gfp)
+{
+	return _mapped_list(znd, dm_s, lba, IS_MAP, gfp);
+}
+
+/**
+ * discard_merge() - Merge a discard request with a existing entry.
+ * @znd: ZDM Instance
+ * @tlba: Starting address
+ * @blks: Number of blocks in discard.
+ * @gfp: Current memory allocation scheme.
+ *
+ */
+static int discard_merge(struct zoned *znd, u64 tlba, u64 blks)
+{
+	struct map_cache *mcache;
+	u64 ends = tlba - 1;
+	u64 next = tlba + blks;
+	int type = IS_DISCARD;
+	int merged = 0;
+
+	mcache = mcache_first_get(znd, type);
+	while (mcache) {
+		int at;
+		int err;
+
+		/*
+		 * 1) Modify existing discard entry, if it exists
+		 * 2) Otherwise: If an existing entry *starts* where this
+		 *    entry *ends* extend that entry.
+		 * 3) Otherwise: If an existing entry ends where this
+		 *    entry starts, extend *that* entry.
+		 */
+		err = memcache_lock_and_sort(znd, mcache);
+		if (err == -EBUSY)
+			at = _lsearch_tlba(mcache, tlba);
+		else
+			at = _bsrch_tlba_lck(mcache, tlba);
+
+		if (at != -1) {
+			mcache_ref(mcache);
+			err = _update_disc(znd, mcache, at, tlba, blks);
+			mcache_deref(mcache);
+			merged = 1;
+			at = -1;
+		}
+
+		/* Existing entry stars where this discard ends */
+		if (!merged) {
+			if (err == -EBUSY)
+				at = _lsearch_tlba(mcache, next);
+			else
+				at = _bsrch_tlba_lck(mcache, next);
+		}
+		at = _bsrch_tlba(mcache, ends);
+		if (at != -1) {
+			struct map_sect_to_lba *data;
+			u64 oldsz;
+
+			mcache_ref(mcache);
+			data = &mcache->jdata[at + 1];
+			data->tlba = lba48_to_le64(0, tlba);
+			oldsz = le64_to_lba48(data->bval, NULL);
+			data->bval = lba48_to_le64(0, oldsz + blks);
+			mcache_deref(mcache);
+			merged = 1;
+			at = -1;
+		}
+
+		/*
+		 * Find a discard that includes 'ends', if so
+		 * determine if the containing discard should be extended
+		 * or if this discard is fully contained within the current
+		 * discard entry.
+		 */
+		if (!merged) {
+			if (err == -EBUSY)
+				at = _lsearch_extent(mcache, ends);
+			else
+				at = _bsrch_extent_lck(mcache, ends);
+		}
+		if (at != -1) {
+			struct map_sect_to_lba *data;
+			u64 oldsz;
+			u64 origin;
+			u64 extend;
+
+			mcache_ref(mcache);
+			data = &mcache->jdata[at + 1];
+			origin = le64_to_lba48(data->tlba, NULL);
+			oldsz = le64_to_lba48(data->bval, NULL);
+			extend = (tlba - origin) + blks;
+			if (extend > oldsz)
+				data->bval = lba48_to_le64(0, extend);
+			mcache_deref(mcache);
+			merged = 1;
+			at = -1;
+		}
+		if (merged) {
+			mcache_put(mcache);
+			goto done;
+		}
+		mcache = mcache_put_get_next(znd, mcache, type);
+	}
+done:
+	return merged;
+}
+
+/**
+ * next_generation() - Increment generation number for superblock.
+ * @znd: ZDM instance
+ */
+static inline u64 next_generation(struct zoned *znd)
+{
+	u64 generation = le64_to_cpu(znd->bmkeys->generation);
+
+	if (generation == 0)
+		generation = 2;
+
+	generation++;
+	if (generation == 0)
+		generation++;
+
+	return generation;
+}
+
+/**
+ * z_mapped_sync() - Write cache entries and bump superblock generation.
+ * @znd: ZDM instance
+ */
+static int z_mapped_sync(struct zoned *znd)
+{
+	struct dm_target *ti = znd->ti;
+	struct map_cache *mcache;
+	int nblks = 1;
+	int use_wq = 0;
+	int rc = 1;
+	int discards = 0;
+	int maps = 0;
+	int jwrote = 0;
+	int cached = 0;
+	int idx = 0;
+	int no;
+	int need_sync_io = 1;
+	u64 lba = LBA_SB_START;
+	u64 generation = next_generation(znd);
+	u64 modulo = CACHE_COPIES;
+	u64 incr = MAX_CACHE_INCR;
+	struct io_4k_block *io_vcache;
+
+	MutexLock(&znd->vcio_lock);
+	io_vcache = get_io_vcache(znd, NORMAL);
+	if (!io_vcache) {
+		Z_ERR(znd, "%s: FAILED to get IO CACHE.", __func__);
+		rc = -ENOMEM;
+		goto out;
+	}
+
+	lba = 0x2048ul;
+	for (idx = 0; idx < znd->gz_count; idx++) {
+		struct meta_pg *wpg = &znd->wp[idx];
+
+		if (test_bit(IS_DIRTY, &wpg->flags)) {
+			cached = 0;
+			memcpy(io_vcache[cached].data, wpg->wp_alloc, Z_C4K);
+			znd->bmkeys->wp_crc[idx] =
+				crc_md_le16(io_vcache[cached].data, Z_CRC_4K);
+			cached++;
+			memcpy(io_vcache[cached].data, wpg->zf_est, Z_C4K);
+			znd->bmkeys->zf_crc[idx] =
+				crc_md_le16(io_vcache[cached].data, Z_CRC_4K);
+			cached++;
+			rc = write_block(ti, DM_IO_VMA, io_vcache, lba,
+				cached, use_wq);
+			if (rc)
+				goto out;
+			clear_bit(IS_DIRTY, &wpg->flags);
+
+			Z_DBG(znd, "%d# -- WP: %04x | ZF: %04x",
+			      idx, znd->bmkeys->wp_crc[idx],
+				   znd->bmkeys->zf_crc[idx]);
+		}
+		lba += 2;
+	}
+
+	lba = (generation % modulo) * incr;
+	if (lba == 0)
+		lba++;
+
+	znd->bmkeys->generation = cpu_to_le64(generation);
+	znd->bmkeys->gc_resv = cpu_to_le32(znd->z_gc_resv);
+	znd->bmkeys->meta_resv = cpu_to_le32(znd->z_meta_resv);
+
+	idx = 0;
+	for (no = 0; no < MAP_COUNT; no++) {
+		mcache = mcache_first_get(znd, no);
+		while (mcache) {
+			u64 phy = le64_to_lba48(mcache->jdata[0].bval, NULL);
+			u16 jcount = mcache->jcount & 0xFFFF;
+
+			if (jcount) {
+				mcache->jdata[0].bval =
+					lba48_to_le64(jcount, phy);
+				znd->bmkeys->crcs[idx] =
+					crc_md_le16(mcache->jdata, Z_CRC_4K);
+				idx++;
+				memcpy(io_vcache[cached].data,
+					mcache->jdata, Z_C4K);
+				cached++;
+
+				if (no == IS_DISCARD)
+					discards++;
+				else
+					maps++;
+			}
+
+			if (cached == IO_VCACHE_PAGES) {
+				rc = write_block(ti, DM_IO_VMA, io_vcache, lba,
+						 cached, use_wq);
+				if (rc) {
+					Z_ERR(znd, "%s: cache-> %" PRIu64
+					      " [%d blks] %p -> %d",
+					      __func__, lba, nblks,
+					      mcache->jdata, rc);
+					mcache_put(mcache);
+					goto out;
+				}
+				lba    += cached;
+				jwrote += cached;
+				cached  = 0;
+			}
+			mcache = mcache_put_get_next(znd, mcache, no);
+		}
+	}
+	jwrote += cached;
+	if (discards > 40)
+		Z_ERR(znd, "**WARNING** large discard cache %d", discards);
+	if (maps > 40)
+		Z_ERR(znd, "**WARNING** large map cache %d", maps);
+
+	znd->bmkeys->md_crc = crc_md_le16(znd->md_crcs, 2 * Z_CRC_4K);
+	znd->bmkeys->n_crcs = cpu_to_le16(jwrote);
+	znd->bmkeys->discards = cpu_to_le16(discards);
+	znd->bmkeys->maps = cpu_to_le16(maps);
+	znd->bmkeys->crc32 = 0;
+	znd->bmkeys->crc32 = cpu_to_le32(crc32c(~0u, znd->bmkeys, Z_CRC_4K));
+	if (cached < (IO_VCACHE_PAGES - 3)) {
+		memcpy(io_vcache[cached].data, znd->bmkeys, Z_C4K);
+		cached++;
+		memcpy(io_vcache[cached].data, znd->md_crcs, Z_C4K * 2);
+		cached += 2;
+		need_sync_io = 0;
+	}
+
+	do {
+		if (cached > 0) {
+			int rw = WRITE;
+
+			if (!need_sync_io &&
+			    test_and_clear_bit(DO_FLUSH, &znd->flags))
+				rw |= (REQ_FLUSH | REQ_FUA);
+			rc = writef_block(ti, rw, DM_IO_VMA, io_vcache, lba,
+					  cached, use_wq);
+			if (rc) {
+				Z_ERR(znd, "%s: mcache-> %" PRIu64
+				      " [%d blks] %p -> %d",
+				      __func__, lba, cached, io_vcache, rc);
+				goto out;
+			}
+			lba += cached;
+		}
+
+		cached = 0;
+		if (need_sync_io) {
+			memcpy(io_vcache[cached].data, znd->bmkeys, Z_C4K);
+			cached++;
+			memcpy(io_vcache[cached].data, znd->md_crcs, Z_C4K * 2);
+			cached += 2;
+			need_sync_io = 0;
+		}
+	} while (cached > 0);
+
+out:
+	put_io_vcache(znd, io_vcache);
+	mutex_unlock(&znd->vcio_lock);
+	return rc;
+}
+
+/**
+ * is_key_page() - Probe block for magic and crc to see if it is recognized.
+ * @_data: ZDM instance
+ */
+static inline int is_key_page(void *_data)
+{
+	int is_key = 0;
+	struct mz_superkey *data = _data;
+
+	/* Starts with Z_KEY_SIG and ends with magic */
+
+	if (le64_to_cpu(data->sig1) == Z_KEY_SIG &&
+	    le64_to_cpu(data->magic) == Z_TABLE_MAGIC) {
+		__le32 orig = data->crc32;
+		__le32 crc_check;
+
+		data->crc32 = 0;
+		crc_check = cpu_to_le32(crc32c(~0u, data, Z_CRC_4K));
+		data->crc32 = orig;
+		if (crc_check == orig)
+			is_key = 1;
+	}
+	return is_key;
+}
+
+/**
+ * zoned_personality() - Update zdstart value from superblock
+ * @znd: ZDM instance
+ * @sblock: Lba to start scanning for superblock.
+ */
+static inline
+void zoned_personality(struct zoned *znd, struct zdm_superblock *sblock)
+{
+	znd->zdstart = le32_to_cpu(sblock->zdstart);
+}
+
+/**
+ * find_superblock_at() - Find superblock following lba
+ * @znd: ZDM instance
+ * @lba: Lba to start scanning for superblock.
+ * @use_wq: If a work queue is needed to scanning.
+ * @do_init: Set zdstart from found superblock.
+ */
+static
+int find_superblock_at(struct zoned *znd, u64 lba, int use_wq, int do_init)
+{
+	struct dm_target *ti = znd->ti;
+	int found = 0;
+	int nblks = 1;
+	int rc = -ENOMEM;
+	u32 count = 0;
+	u64 *data = ZDM_ALLOC(znd, Z_C4K, PG_10, NORMAL);
+
+	if (!data) {
+		Z_ERR(znd, "No memory for finding generation ..");
+		return 0;
+	}
+	if (lba == 0)
+		lba++;
+	do {
+		rc = read_block(ti, DM_IO_KMEM, data, lba, nblks, use_wq);
+		if (rc) {
+			Z_ERR(znd, "%s: read @%" PRIu64 " [%d blks] %p -> %d",
+			      __func__, lba, nblks, data, rc);
+			goto out;
+		}
+		if (is_key_page(data)) {
+			struct mz_superkey *kblk = (struct mz_superkey *) data;
+			struct zdm_superblock *sblock = &kblk->sblock;
+			int err = sb_check(sblock);
+
+			if (!err) {
+				found = 1;
+				if (do_init)
+					zoned_personality(znd, sblock);
+			}
+			goto out;
+		}
+		if (data[0] == 0 && data[1] == 0) {
+			/* No SB here. */
+			Z_ERR(znd, "FGen: Invalid block %" PRIx64 "?", lba);
+			goto out;
+		}
+		lba++;
+		count++;
+		if (count > MAX_CACHE_SYNC) {
+			Z_ERR(znd, "FSB: Too deep to be useful.");
+			goto out;
+		}
+	} while (!found);
+
+out:
+	ZDM_FREE(znd, data, Z_C4K, PG_10);
+	return found;
+}
+
+/**
+ * find_superblock() - Find (any) superblock
+ * @znd: ZDM instance
+ * @use_wq: If a work queue is needed to scanning.
+ * @do_init: Set/Retrieve zdstart from found super block.
+ */
+static int find_superblock(struct zoned *znd, int use_wq, int do_init)
+{
+	int found = 0;
+	int iter = 0;
+	u64 lba = LBA_SB_START;
+	u64 last = MAX_CACHE_INCR * CACHE_COPIES;
+
+	do {
+		found = find_superblock_at(znd, lba, use_wq, do_init);
+		if (found)
+			break;
+		iter++;
+		lba = MAX_CACHE_INCR * iter;
+	} while (lba < last);
+
+	return found;
+}
+
+/**
+ * mcache_find_gen() - Find the super block following lba and get gen#
+ * @znd: ZDM instance
+ * @lba: LBA to start scanning for the super block.
+ * @use_wq: If a work queue is needed to scanning.
+ * @sb_lba: LBA where the super block was found.
+ */
+static u64 mcache_find_gen(struct zoned *znd, u64 lba, int use_wq, u64 *sb_lba)
+{
+	struct dm_target *ti = znd->ti;
+	u64 generation = 0;
+	int nblks = 1;
+	int rc = 1;
+	int done = 0;
+	u32 count = 0;
+	u64 *data = ZDM_ALLOC(znd, Z_C4K, PG_11, NORMAL);
+
+	if (!data) {
+		Z_ERR(znd, "No memory for finding generation ..");
+		return 0;
+	}
+	do {
+		rc = read_block(ti, DM_IO_KMEM, data, lba, nblks, use_wq);
+		if (rc) {
+			Z_ERR(znd, "%s: mcache-> %" PRIu64
+				    " [%d blks] %p -> %d",
+			      __func__, lba, nblks, data, rc);
+			goto out;
+		}
+		if (is_key_page(data)) {
+			struct mz_superkey *kblk = (struct mz_superkey *) data;
+
+			generation = le64_to_cpu(kblk->generation);
+			done = 1;
+			if (sb_lba)
+				*sb_lba = lba;
+			goto out;
+		}
+		lba++;
+		count++;
+		if (count > MAX_CACHE_SYNC) {
+			Z_ERR(znd, "FGen: Too deep to be useful.");
+			goto out;
+		}
+	} while (!done);
+
+out:
+	ZDM_FREE(znd, data, Z_C4K, PG_11);
+	return generation;
+}
+
+/**
+ * cmp_gen() - compare two u64 numbers considering rollover
+ * @left: a u64
+ * @right: a u64
+ * Return: -1, 0, 1 if left < right, equal, or > respectively.
+ */
+static inline int cmp_gen(u64 left, u64 right)
+{
+	int result = 0;
+
+	if (left != right) {
+		u64 delta = (left > right) ? left - right : right - left;
+
+		result = (left > right) ? -1 : 1;
+		if (delta > 0xFFFFFFFF) {
+			if (left == BAD_ADDR)
+				result = 1;
+		} else {
+			if (right > left)
+				result = 1;
+		}
+	}
+
+	return result;
+}
+
+/**
+ * mcache_greatest_gen() - Pick the lba where the super block should start.
+ * @znd: ZDM instance
+ * @use_wq: If a workqueue is needed for IO.
+ * @sb: LBA of super block itself.
+ * @at_lba: LBA where sync data starts (in front of the super block).
+ */
+static
+u64 mcache_greatest_gen(struct zoned *znd, int use_wq, u64 *sb, u64 *at_lba)
+{
+	u64 lba = LBA_SB_START;
+	u64 gen_no[CACHE_COPIES] = { 0ul, 0ul, 0ul };
+	u64 gen_lba[CACHE_COPIES] = { 0ul, 0ul, 0ul };
+	u64 gen_sb[CACHE_COPIES] = { 0ul, 0ul, 0ul };
+	u64 incr = MAX_CACHE_INCR;
+	int locations = ARRAY_SIZE(gen_lba);
+	int pick = 0;
+	int idx;
+
+	for (idx = 0; idx < locations; idx++) {
+		u64 *pAt = &gen_sb[idx];
+
+		gen_lba[idx] = lba;
+		gen_no[idx] = mcache_find_gen(znd, lba, use_wq, pAt);
+		if (gen_no[idx])
+			pick = idx;
+		lba = idx * incr;
+	}
+
+	for (idx = 0; idx < locations; idx++) {
+		if (cmp_gen(gen_no[pick], gen_no[idx]) > 0)
+			pick = idx;
+	}
+
+	if (gen_no[pick]) {
+		if (at_lba)
+			*at_lba = gen_lba[pick];
+		if (sb)
+			*sb = gen_sb[pick];
+	}
+
+	return gen_no[pick];
+}
+
+/**
+ * count_stale_blocks() - Number of stale blocks covered by meta_pg.
+ * @znd: ZDM instance
+ * @gzno: Meta page # to scan.
+ * @wpg: Meta page to scan.
+ */
+static u64 count_stale_blocks(struct zoned *znd, u32 gzno, struct meta_pg *wpg)
+{
+	u32 gzcount = 1 << GZ_BITS;
+	u32 iter;
+	u64 stale = 0;
+
+	if ((gzno << GZ_BITS) > znd->data_zones)
+		gzcount = znd->data_zones & GZ_MMSK;
+
+	/* mark as empty */
+	for (iter = 0; iter < gzcount; iter++) {
+		u32 wp = le32_to_cpu(wpg->wp_alloc[iter]) & Z_WP_VALUE_MASK;
+		u32 nf = le32_to_cpu(wpg->zf_est[iter]) & Z_WP_VALUE_MASK;
+
+		if (wp > (Z_BLKSZ - nf))
+			stale += (wp - (Z_BLKSZ - nf));
+	}
+
+	return stale;
+}
+
+static int do_load_cache(struct zoned *znd, int type, u64 lba, int idx, int wq)
+{
+	u16 count;
+	__le16 crc;
+	int rc = 0;
+	int blks = 1;
+	struct map_cache *mcache = mcache_alloc(znd, NORMAL);
+
+	if (!mcache) {
+		rc = -ENOMEM;
+		goto out;
+	}
+
+	rc = read_block(znd->ti, DM_IO_KMEM, mcache->jdata, lba, blks, wq);
+	if (rc) {
+		Z_ERR(znd, "%s: mcache-> %" PRIu64
+			   " [%d blks] %p -> %d",
+		      __func__, lba, blks, mcache->jdata, rc);
+
+		ZDM_FREE(znd, mcache->jdata, Z_C4K, PG_08);
+		ZDM_FREE(znd, mcache, sizeof(*mcache), KM_07);
+		goto out;
+	}
+	crc = crc_md_le16(mcache->jdata, Z_CRC_4K);
+	if (crc != znd->bmkeys->crcs[idx]) {
+		rc = -EIO;
+		Z_ERR(znd, "%s: bad crc %" PRIu64,  __func__, lba);
+		goto out;
+	}
+	(void)le64_to_lba48(mcache->jdata[0].bval, &count);
+	mcache->jcount = count;
+	mclist_add(znd, mcache, type);
+
+out:
+	return rc;
+}
+
+static int do_load_map_cache(struct zoned *znd, u64 lba, int idx, int wq)
+{
+	return do_load_cache(znd, IS_MAP, lba, idx, wq);
+}
+
+static int do_load_discard_cache(struct zoned *znd, u64 lba, int idx, int wq)
+{
+	return do_load_cache(znd, IS_DISCARD, lba, idx, wq);
+}
+
+
+
+/**
+ * z_mapped_init() - Re-Load an existing ZDM instance from the block device.
+ * @znd: ZDM instance
+ *
+ * FIXME: Discard extent read-back does not match z_mapped_sync writing
+ */
+static int z_mapped_init(struct zoned *znd)
+{
+	struct dm_target *ti = znd->ti;
+	int nblks = 1;
+	int wq = 0;
+	int rc = 1;
+	int idx = 0;
+	int jcount = 0;
+	u64 sblba = 0;
+	u64 lba = 0;
+	u64 generation;
+	__le32 crc_chk;
+	struct io_4k_block *io_vcache;
+
+	MutexLock(&znd->vcio_lock);
+	io_vcache = get_io_vcache(znd, NORMAL);
+
+	if (!io_vcache) {
+		Z_ERR(znd, "%s: FAILED to get SYNC CACHE.", __func__);
+		rc = -ENOMEM;
+		goto out;
+	}
+
+	generation = mcache_greatest_gen(znd, wq, &sblba, &lba);
+	if (generation == 0) {
+		rc = -ENODATA;
+		goto out;
+	}
+
+	if (lba == 0)
+		lba++;
+
+	/* read superblock */
+	rc = read_block(ti, DM_IO_VMA, io_vcache, sblba, nblks, wq);
+	if (rc)
+		goto out;
+
+	memcpy(znd->bmkeys, io_vcache, sizeof(*znd->bmkeys));
+
+	/* read in map cache */
+	for (idx = 0; idx < le16_to_cpu(znd->bmkeys->maps); idx++) {
+		rc = do_load_map_cache(znd, lba++, jcount++, wq);
+		if (rc)
+			goto out;
+	}
+
+	/* read in discard cache */
+	for (idx = 0; idx < le16_to_cpu(znd->bmkeys->discards); idx++) {
+		rc = do_load_discard_cache(znd, lba++, jcount++, wq);
+		if (rc)
+			goto out;
+	}
+
+	/* skip re-read of superblock */
+	if (lba == sblba)
+		lba++;
+
+	/* read in CRC pgs */
+	rc = read_block(ti, DM_IO_VMA, znd->md_crcs, lba, 2, wq);
+	if (rc)
+		goto out;
+
+	crc_chk = znd->bmkeys->crc32;
+	znd->bmkeys->crc32 = 0;
+	znd->bmkeys->crc32 = cpu_to_le32(crc32c(~0u, znd->bmkeys, Z_CRC_4K));
+
+	if (crc_chk != znd->bmkeys->crc32) {
+		Z_ERR(znd, "Bad Block Map KEYS!");
+		Z_ERR(znd, "Key CRC: Ex: %04x vs %04x <- calculated",
+		      le32_to_cpu(crc_chk),
+		      le32_to_cpu(znd->bmkeys->crc32));
+		rc = -EIO;
+		goto out;
+	}
+
+	if (jcount != le16_to_cpu(znd->bmkeys->n_crcs)) {
+		Z_ERR(znd, " ... mcache entries: found = %u, expected = %u",
+		      jcount, le16_to_cpu(znd->bmkeys->n_crcs));
+		rc = -EIO;
+		goto out;
+	}
+
+	crc_chk = crc_md_le16(znd->md_crcs, Z_CRC_4K * 2);
+	if (crc_chk != znd->bmkeys->md_crc) {
+		Z_ERR(znd, "CRC of CRC PGs: Ex %04x vs %04x  <- calculated",
+		      le16_to_cpu(znd->bmkeys->md_crc),
+		      le16_to_cpu(crc_chk));
+		rc = -EIO;
+		goto out;
+	}
+
+	/*
+	 * Read write pointers / free counters.
+	 */
+	lba = 0x2048ul;
+	znd->discard_count = 0;
+	for (idx = 0; idx < znd->gz_count; idx++) {
+		struct meta_pg *wpg = &znd->wp[idx];
+		__le16 crc_wp;
+		__le16 crc_zf;
+
+		rc = read_block(ti, DM_IO_KMEM, wpg->wp_alloc, lba, 1, wq);
+		if (rc)
+			goto out;
+		crc_wp = crc_md_le16(wpg->wp_alloc, Z_CRC_4K);
+		if (znd->bmkeys->wp_crc[idx] != crc_wp)
+			Z_ERR(znd, "WP @ %d does not match written.", idx);
+
+		rc = read_block(ti, DM_IO_KMEM, wpg->zf_est, lba + 1, 1, wq);
+		if (rc)
+			goto out;
+		crc_zf = crc_md_le16(wpg->zf_est, Z_CRC_4K);
+		if (znd->bmkeys->zf_crc[idx] != crc_zf)
+			Z_ERR(znd, "ZF @ %d does not match written.", idx);
+
+		Z_DBG(znd, "%d# -- WP: %04x [%04x] | ZF: %04x [%04x]",
+		      idx, znd->bmkeys->wp_crc[idx], crc_wp,
+			   znd->bmkeys->zf_crc[idx], crc_zf);
+
+
+		if (znd->bmkeys->wp_crc[idx] == crc_wp &&
+		    znd->bmkeys->zf_crc[idx] == crc_zf)
+			znd->discard_count += count_stale_blocks(znd, idx, wpg);
+
+		lba += 2;
+	}
+	znd->z_gc_resv   = le32_to_cpu(znd->bmkeys->gc_resv);
+	znd->z_meta_resv = le32_to_cpu(znd->bmkeys->meta_resv);
+
+out:
+	put_io_vcache(znd, io_vcache);
+	mutex_unlock(&znd->vcio_lock);
+	return rc;
+}
+
+/**
+ * z_mapped_addmany() - Add multiple entries into the map_cache
+ * @znd: ZDM instance
+ * @dm_s: tLBA
+ * @lba: lba on backing device.
+ * @c: Number of (contiguous) map entries to add.
+ * @fp: Allocation flags (for _ALLOC)
+ */
+static int z_mapped_addmany(struct zoned *znd, u64 dm_s, u64 lba,
+			    u64 count, gfp_t gfp)
+{
+	int rc = 0;
+	sector_t blk;
+
+	for (blk = 0; blk < count; blk++) {
+		rc = z_mapped_add_one(znd, dm_s + blk, lba + blk, gfp);
+		if (rc)
+			goto out;
+	}
+
+out:
+	return rc;
+}
+
+/**
+ * alloc_pg() - Allocate a map page
+ * @znd: ZDM instance
+ * @entry: entry (in mpi table) to update on allocation.
+ * @lba: LBA associated with the page of ZLT
+ * @mpi: Map page information (lookup table entry, bit flags, etc)
+ * @ahead: Flag to set READA flag on page
+ * @gfp: Allocation flags (for _ALLOC)
+ */
+static struct map_pg *alloc_pg(struct zoned *znd, int entry, u64 lba,
+			       struct mpinfo *mpi, int ahead, gfp_t gfp)
+{
+	spinlock_t *lock = &znd->ct_lock;
+	struct map_pg *found = ZDM_ALLOC(znd, sizeof(*found), KM_20, gfp);
+
+	if (found) {
+		if (mpi->bit_type == IS_LUT)
+			lock = &znd->mapkey_lock;
+
+		found->lba = lba;
+		mutex_init(&found->md_lock);
+		set_bit(mpi->bit_dir, &found->flags);
+		set_bit(mpi->bit_type, &found->flags);
+		found->age = jiffies_64;
+		found->index = entry;
+		INIT_LIST_HEAD(&found->zltlst);
+		INIT_LIST_HEAD(&found->lazy);
+		found->znd = znd;
+		found->crc_pg = NULL; /* */
+		ref_pg(found);
+		if (ahead)
+			set_bit(IS_READA, &found->flags);
+
+		/* allocation done. check and see if there as a
+		 * concurrent race
+		 */
+		spin_lock(lock);
+		if (mpi->table[entry] == NULL)
+			mpi->table[entry] = found;
+		else
+			ZDM_FREE(znd, found, sizeof(*found), KM_20);
+		spin_unlock(lock);
+	} else {
+		Z_ERR(znd, "NO MEM for mapped_t !!!");
+	}
+	return found;
+}
+
+/**
+ * _maybe_undrop() - If a page is on its way out of cache pull it back.
+ * @znd: ZDM instance
+ * @pg: Page to claim
+ *
+ * When a table page is being dropped from the cache it may transition
+ * through the lazy pool. It a page is caught in the lazy pool it is
+ * deemed to be 'warm'. Not hot enough to be frequently hit but clearly
+ * too warm to be dropped quickly. Give it a boost to keep in in cache
+ * longer.
+ */
+static __always_inline int _maybe_undrop(struct zoned *znd, struct map_pg *pg)
+{
+	int undrop = 0;
+
+	if (test_bit(IS_DROPPED, &pg->flags)) {
+		spin_lock(&znd->lzy_lck);
+		if (test_bit(IS_DROPPED, &pg->flags) &&
+		    test_bit(IS_LAZY, &pg->flags)) {
+			clear_bit(IS_DROPPED, &pg->flags);
+			set_bit(DELAY_ADD, &pg->flags);
+		}
+		spin_unlock(&znd->lzy_lck);
+		if (pg->hotness < MEM_PURGE_MSECS)
+			pg->hotness += MEM_HOT_BOOST_INC;
+		pg->age = jiffies_64 + msecs_to_jiffies(pg->hotness);
+		undrop = 1;
+	}
+	ref_pg(pg);
+	return undrop;
+}
+
+/**
+ * _load_backing_pages() - Cache a backing page
+ * @znd: ZDM instance
+ * @lba: Logical LBA of page.
+ * @gfp: Memory allocation rule
+ *
+ * When metadata is pooled with data the FWD table lookup can
+ * be recursive (the page needed to resolve the FWD entry is
+ * itself on disk). The recursion is never deep but it can
+ * be avoided or mitigated by keep such 'key' pages in cache.
+ */
+static int _load_backing_pages(struct zoned *znd, u64 lba, gfp_t gfp)
+{
+	const int raflg = 1;
+	int rc = 0;
+	int entry;
+	struct mpinfo mpi;
+	struct map_addr maddr;
+	struct map_pg *found;
+
+	if (lba < znd->data_lba)
+		goto out;
+
+	map_addr_calc(znd, lba, &maddr);
+	entry = to_table_entry(znd, maddr.lut_s, &mpi);
+	if (entry > -1) {
+		spinlock_t *lock = &znd->ct_lock;
+
+		if (mpi.bit_type == IS_LUT)
+			lock = &znd->mapkey_lock;
+
+		spin_lock(lock);
+		found = mpi.table[entry];
+		if (found)
+			_maybe_undrop(znd, found);
+		spin_unlock(lock);
+		if (!found)
+			found = alloc_pg(znd, entry, lba, &mpi, raflg, gfp);
+		if (found) {
+			if (!found->data.addr) {
+				rc = cache_pg(znd, found, gfp, &mpi);
+				if (rc < 0 && rc != -EBUSY)
+					znd->meta_result = rc;
+			}
+			deref_pg(found);
+
+			if (getref_pg(found) != 0)
+				Z_ERR(znd, "Backing page with elevated ref: %u",
+				      getref_pg(found));
+		}
+	}
+
+out:
+	return rc;
+}
+
+/**
+ * _load_crc_page() - Cache a page of CRC
+ * @znd: ZDM instance
+ * @lba: Logical LBA of page.
+ * @gfp: Memory allocation rule
+ *
+ * When a table page is cached the page containing its CRC is also pulled
+ * into cache. Rather than defer it to cache_pg() it's brought into the
+ * cache here.
+ */
+static int _load_crc_page(struct zoned *znd, struct mpinfo *mpi, gfp_t gfp)
+{
+	const int raflg = 0;
+	int rc = 0;
+	int entry;
+	struct map_pg *found;
+	spinlock_t *lock = &znd->ct_lock;
+	u64 base = (mpi->bit_dir == IS_REV) ? znd->c_mid : znd->c_base;
+
+	base += mpi->crc.pg_no;
+	entry = to_table_entry(znd, base, mpi);
+
+	if (mpi->bit_type != IS_CRC)
+		return rc;
+	if (entry >= znd->crc_count)
+		return rc;
+
+	spin_lock(lock);
+	found = mpi->table[entry];
+	if (found)
+		_maybe_undrop(znd, found);
+	spin_unlock(lock);
+	if (!found)
+		found = alloc_pg(znd, entry, base, mpi, raflg, gfp);
+
+	if (found) {
+		if (!found->data.crc) {
+			rc = cache_pg(znd, found, gfp, mpi);
+			if (rc < 0 && rc != -EBUSY)
+				znd->meta_result = rc;
+		}
+		deref_pg(found);
+	}
+	return rc;
+}
+
+/**
+ * put_map_entry() - Decrement refcount of mapped page.
+ * @pg: mapped page
+ */
+static inline void put_map_entry(struct map_pg *pg)
+{
+	if (pg)
+		deref_pg(pg);
+}
+
+/**
+ * get_map_entry() - Find a page of LUT or CRC table map.
+ * @znd: ZDM instance
+ * @lba: Logical LBA of page.
+ * @gfp: Memory allocation rule
+ *
+ * Return: struct map_pg * or NULL on error.
+ *
+ * Page will be loaded from disk it if is not already in core memory.
+ */
+static struct map_pg *get_map_entry(struct zoned *znd, u64 lba, gfp_t gfp)
+{
+	struct mpinfo mpi;
+	struct map_pg *ahead[READ_AHEAD];
+	int entry;
+	int iter;
+	int range;
+	u32 count;
+	spinlock_t *lock;
+	int empty_val;
+
+	memset(&ahead, 0, sizeof(ahead));
+
+	entry = to_table_entry(znd, lba, &mpi);
+	if (entry < 0)
+		return NULL;
+
+	if (mpi.bit_type == IS_LUT) {
+		lock = &znd->mapkey_lock;
+		count = znd->map_count;
+		empty_val = 0xff;
+
+		if (mpi.bit_dir == IS_FWD) {
+			/*
+			 * pre-load any backing pages
+			 * to unwind recursive page lookups.
+			 */
+			_load_backing_pages(znd, lba, gfp);
+		}
+		_load_crc_page(znd, &mpi, gfp);
+
+		range = entry + ARRAY_SIZE(ahead);
+		if (range > count)
+			range = count;
+	} else {
+		lock = &znd->ct_lock;
+		count = znd->crc_count;
+		empty_val = 0;
+
+		/* CRC's cover 2k pages .. so only pull one extra */
+		range = entry + 1;
+		if (range > count)
+			range = count;
+	}
+
+	iter = 0;
+	while (entry < range) {
+		int want_cached = 1;
+		struct map_pg *found;
+
+		entry = to_table_entry(znd, lba, &mpi);
+		if (entry < 0)
+			break;
+
+		spin_lock(lock);
+		found = mpi.table[entry];
+		if (found &&
+		    _maybe_undrop(znd, found) &&
+		    test_bit(IS_READA, &found->flags)
+		    && iter > 0)
+			want_cached = 0;
+
+		if (found) {
+			if (want_cached)
+				found->age = jiffies_64
+					   + msecs_to_jiffies(found->hotness);
+			else
+				found->age +=
+					msecs_to_jiffies(MEM_HOT_BOOST_INC);
+		}
+		spin_unlock(lock);
+
+		if (want_cached && !found)
+			found = alloc_pg(znd, entry, lba, &mpi, iter, gfp);
+
+		if (found) {
+			if (want_cached)
+				ahead[iter] = found;
+			else
+				deref_pg(found);
+		}
+		iter++;
+		entry++;
+		lba++;
+	}
+
+	/*
+	 *  Each entry in ahead has an elevated refcount.
+	 * Only allow the target of get_map_entry() to remain elevated.
+	 */
+	for (iter = 0; iter < ARRAY_SIZE(ahead); iter++) {
+		struct map_pg *pg = ahead[iter];
+
+		if (pg) {
+			if (!pg->data.addr) {
+				int rc;
+
+				to_table_entry(znd, pg->lba, &mpi);
+				rc = cache_pg(znd, pg, gfp, &mpi);
+
+				if (rc < 0 && rc != -EBUSY) {
+					znd->meta_result = rc;
+					ahead[iter] = NULL;
+				}
+			}
+			if (iter > 0)
+				deref_pg(pg);
+		}
+	}
+
+	return ahead[0];
+}
+
+/**
+ * metadata_dirty_fling() - Force a ZLT block into cache and flag it dirty.
+ * @znd: ZDM Instance
+ * @dm_s: Current lba to consider.
+ *
+ * Used when data and ZDM's metadata are co-mingled. If dm_s is a block
+ * of ZDM's metadata it needs to be relocated. Since we re-locate
+ * blocks that are in dirty and in the cache ... if this block is
+ * metadata, force it into the cache and flag it as dirty.
+ */
+static int metadata_dirty_fling(struct zoned *znd, u64 dm_s)
+{
+	struct map_pg *pg = NULL;
+	int is_flung = 0;
+
+	/*
+	 * When co
+	 * nothing in the GC reverse map should point to
+	 * a block *before* a data pool block
+	 */
+	if (dm_s < znd->data_lba)
+		return is_flung;
+
+	if (dm_s >= znd->r_base && dm_s < znd->c_end) {
+		pg = get_map_entry(znd, dm_s, NORMAL);
+		if (!pg)
+			Z_ERR(znd, "Failed to fling: %" PRIx64, dm_s);
+	}
+	if (pg) {
+		MutexLock(&pg->md_lock);
+		pg->age = jiffies_64 + msecs_to_jiffies(pg->hotness);
+		clear_bit(IS_READA, &pg->flags);
+		set_bit(IS_DIRTY, &pg->flags);
+		clear_bit(IS_FLUSH, &pg->flags);
+		is_flung = 1;
+		mutex_unlock(&pg->md_lock);
+
+		if (pg->lba != dm_s)
+			Z_ERR(znd, "Excess churn? lba %"PRIx64
+				   " [last: %"PRIx64"]", dm_s, pg->lba);
+		put_map_entry(pg);
+	}
+	return is_flung;
+}
+
+/**
+ * z_do_copy_more() - GC transition to read more blocks.
+ * @gc_state: GC State to be updated.
+ */
+static inline void z_do_copy_more(struct gc_state *gc_entry)
+{
+	unsigned long flags;
+	struct zoned *znd = gc_entry->znd;
+
+	spin_lock_irqsave(&znd->gc_lock, flags);
+	set_bit(DO_GC_PREPARE, &gc_entry->gc_flags);
+	spin_unlock_irqrestore(&znd->gc_lock, flags);
+}
+
+/**
+ * gc_post_add() - Add a tLBA and current bLBA origin.
+ * @znd: ZDM Instance
+ * @dm_s: tLBA
+ * @lba: bLBA
+ *
+ * Return: 1 if tLBA is added, 0 if block was stale.
+ *
+ * Stale block checks are performed before tLBA is added.
+ * Add a non-stale block to the list of blocks for moving and
+ * metadata updating.
+ */
+static int gc_post_add(struct zoned *znd, u64 addr, u64 lba)
+{
+	struct map_cache *post = &znd->gc_postmap;
+	int handled = 0;
+
+	if (z_lookup_cache(znd, addr, IS_DISCARD))
+		return handled;
+
+	if (metadata_dirty_fling(znd, addr))
+		return handled;
+
+	if (post->jcount < post->jsize) {
+		u16 idx = ++post->jcount;
+
+		WARN_ON(post->jcount > post->jsize);
+
+		post->jdata[idx].tlba = lba48_to_le64(0, addr);
+		post->jdata[idx].bval = lba48_to_le64(0, lba);
+		handled = 1;
+	} else {
+		Z_ERR(znd, "*CRIT* post overflow L:%" PRIx64 "-> S:%" PRIx64,
+		      lba, addr);
+	}
+	return handled;
+}
+
+
+/**
+ * _fwd_to_cache() - Helper to pull the forward map block into cache.
+ * @znd: ZDM Instance
+ * @addr: Address (from reverse map table entry).
+ */
+static __always_inline int _fwd_to_cache(struct zoned *znd, u64 addr)
+{
+	int err = 0;
+	struct map_pg *pg;
+	struct map_addr maddr;
+
+	if (addr < znd->nr_blocks) {
+		map_addr_calc(znd, addr, &maddr);
+		pg = get_map_entry(znd, maddr.lut_s, NORMAL);
+		if (!pg)
+			err = -ENOMEM;
+		put_map_entry(pg);
+	} else {
+		Z_ERR(znd, "Invalid rmap entry: %" PRIx64, addr);
+	}
+
+	return err;
+}
+
+/**
+ * z_zone_gc_metadata_to_ram() - Load affected metadata blocks to ram.
+ * @gc_entry: Compaction event in progress
+ *
+ * Return: 0, otherwise errno.
+ *
+ * Use the reverse ZLT to find the forward ZLT entries that need to be
+ * remapped in this zone.
+ * When complete the znd->gc_postmap have a map of all the non-stale
+ * blocks remaining in the zone.
+ */
+static int z_zone_gc_metadata_to_ram(struct gc_state *gc_entry)
+{
+	struct zoned *znd = gc_entry->znd;
+	u64 from_lba = (gc_entry->z_gc << Z_BLKBITS) + znd->md_end;
+	struct map_pg *rev_pg;
+	struct map_addr origin;
+	int count;
+	int rcode = 0;
+
+	/* pull all of the affected struct map_pg and crc pages into memory: */
+	for (count = 0; count < Z_BLKSZ; count++) {
+		__le32 ORencoded;
+		u64 blba = from_lba + count;
+
+		map_addr_calc(znd, blba, &origin);
+		rev_pg = get_map_entry(znd, origin.lut_r, NORMAL);
+		if (rev_pg && rev_pg->data.addr) {
+			ref_pg(rev_pg);
+			MutexLock(&rev_pg->md_lock);
+			ORencoded = rev_pg->data.addr[origin.pg_idx];
+			mutex_unlock(&rev_pg->md_lock);
+			if (ORencoded != MZTEV_UNUSED) {
+				u64 tlba = map_value(znd, ORencoded);
+				int err = _fwd_to_cache(znd, tlba);
+
+				if (err)
+					rcode = err;
+				gc_post_add(znd, tlba, blba);
+			}
+			deref_pg(rev_pg);
+		}
+		put_map_entry(rev_pg);
+	}
+
+	return rcode;
+}
+
+/**
+ * append_blks() - Read (more) blocks into buffer.
+ * @znd: ZDM Instance
+ * @lba: Starting blba
+ * @io_buf: Buffer to read into
+ * @count: Number of blocks to read.
+ */
+static int append_blks(struct zoned *znd, u64 lba,
+		       struct io_4k_block *io_buf, int count)
+{
+	int rcode = 0;
+	int rc;
+	u32 chunk;
+	struct io_4k_block *io_vcache;
+
+	MutexLock(&znd->gc_vcio_lock);
+	io_vcache = get_io_vcache(znd, NORMAL);
+	if (!io_vcache) {
+		Z_ERR(znd, "%s: FAILED to get SYNC CACHE.", __func__);
+		rc = -ENOMEM;
+		goto out;
+	}
+
+	for (chunk = 0; chunk < count; chunk += IO_VCACHE_PAGES) {
+		u32 nblks = count - chunk;
+
+		if (nblks > IO_VCACHE_PAGES)
+			nblks = IO_VCACHE_PAGES;
+
+		rc = read_block(znd->ti, DM_IO_VMA, io_vcache, lba, nblks, 0);
+		if (rc) {
+			Z_ERR(znd, "Reading error ... disable zone: %u",
+				(u32)(lba >> 16));
+			rcode = -EIO;
+			goto out;
+		}
+		memcpy(&io_buf[chunk], io_vcache, nblks * Z_C4K);
+		lba += nblks;
+	}
+out:
+	put_io_vcache(znd, io_vcache);
+	mutex_unlock(&znd->gc_vcio_lock);
+	return rcode;
+}
+
+/**
+ * z_zone_gc_read() - Read (up to) a buffer worth of data from zone.
+ * @gc_entry: Active GC state
+ */
+static int z_zone_gc_read(struct gc_state *gc_entry)
+{
+	struct zoned *znd = gc_entry->znd;
+	struct io_4k_block *io_buf = znd->gc_io_buf;
+	struct map_cache *post = &znd->gc_postmap;
+	unsigned long flags;
+	u64 start_lba;
+	int nblks;
+	int rcode = 0;
+	int fill = 0;
+	int jstart;
+	int jentry;
+
+	spin_lock_irqsave(&znd->gc_lock, flags);
+	jstart = gc_entry->r_ptr;
+	spin_unlock_irqrestore(&znd->gc_lock, flags);
+
+	if (!jstart)
+		jstart++;
+
+	MutexLock(&post->cached_lock);
+
+	/* A discard may have puched holes in the postmap. re-sync lba */
+	jentry = jstart;
+	while (jentry <= post->jcount && (Z_LOWER48 ==
+			le64_to_lba48(post->jdata[jentry].bval, NULL))) {
+		jentry++;
+	}
+	/* nothing left to move */
+	if (jentry > post->jcount)
+		goto out_finished;
+
+	/* skip over any discarded blocks */
+	if (jstart != jentry)
+		jstart = jentry;
+
+	start_lba = le64_to_lba48(post->jdata[jentry].bval, NULL);
+	post->jdata[jentry].bval = lba48_to_le64(GC_READ, start_lba);
+	nblks = 1;
+	jentry++;
+
+	while (jentry <= post->jcount && (nblks+fill) < GC_MAX_STRIPE) {
+		u64 dm_s = le64_to_lba48(post->jdata[jentry].tlba, NULL);
+		u64 lba = le64_to_lba48(post->jdata[jentry].bval, NULL);
+
+		if (Z_LOWER48 == dm_s || Z_LOWER48 == lba) {
+			jentry++;
+			continue;
+		}
+
+		post->jdata[jentry].bval = lba48_to_le64(GC_READ, lba);
+
+		/* if the block is contiguous add it to the read */
+		if (lba == (start_lba + nblks)) {
+			nblks++;
+		} else {
+			if (nblks) {
+				int err;
+
+				err = append_blks(znd, start_lba,
+						  &io_buf[fill], nblks);
+				if (err) {
+					rcode = err;
+					goto out;
+				}
+				fill += nblks;
+			}
+			start_lba = lba;
+			nblks = 1;
+		}
+		jentry++;
+	}
+
+	/* Issue a copy of 'nblks' blocks */
+	if (nblks > 0) {
+		int err;
+
+		err = append_blks(znd, start_lba, &io_buf[fill], nblks);
+		if (err) {
+			rcode = err;
+			goto out;
+		}
+		fill += nblks;
+	}
+
+out_finished:
+	Z_DBG(znd, "Read %d blocks from %d", fill, gc_entry->r_ptr);
+
+	spin_lock_irqsave(&znd->gc_lock, flags);
+	gc_entry->nblks = fill;
+	gc_entry->r_ptr = jentry;
+	if (fill > 0)
+		set_bit(DO_GC_WRITE, &gc_entry->gc_flags);
+	else
+		set_bit(DO_GC_COMPLETE, &gc_entry->gc_flags);
+
+	spin_unlock_irqrestore(&znd->gc_lock, flags);
+
+out:
+	mutex_unlock(&post->cached_lock);
+
+	return rcode;
+}
+
+/**
+ * z_zone_gc_write() - Write (up to) a buffer worth of data to WP.
+ * @gc_entry: Active GC state
+ * @stream_id: Stream Id to prefer for allocation.
+ */
+static int z_zone_gc_write(struct gc_state *gc_entry, u32 stream_id)
+{
+	struct zoned *znd = gc_entry->znd;
+	struct dm_target *ti = znd->ti;
+	struct io_4k_block *io_buf = znd->gc_io_buf;
+	struct map_cache *post = &znd->gc_postmap;
+	unsigned long flags;
+	u32 aq_flags = Z_AQ_GC | Z_AQ_STREAM_ID | stream_id;
+	u64 lba;
+	u32 nblks;
+	u32 out = 0;
+	int err = 0;
+	int jentry;
+
+	spin_lock_irqsave(&znd->gc_lock, flags);
+	jentry = gc_entry->w_ptr;
+	nblks = gc_entry->nblks;
+	spin_unlock_irqrestore(&znd->gc_lock, flags);
+
+	if (!jentry)
+		jentry++;
+
+	MutexLock(&post->cached_lock);
+	while (nblks > 0) {
+		u32 nfound = 0;
+		u32 added = 0;
+
+		/*
+		 * When lba is zero blocks were not allocated.
+		 * Retry with the smaller request
+		 */
+		lba = z_acquire(znd, aq_flags, nblks, &nfound);
+		if (!lba) {
+			if (nfound) {
+				u32 avail = nfound;
+
+				nfound = 0;
+				lba = z_acquire(znd, aq_flags, avail, &nfound);
+			}
+		}
+
+		if (!lba) {
+			err = -ENOSPC;
+			goto out;
+		}
+
+		err = write_block(ti, DM_IO_VMA, &io_buf[out], lba, nfound, 0);
+		if (err) {
+			Z_ERR(znd, "Write %d blocks to %"PRIx64". ERROR: %d",
+			      nfound, lba, err);
+			goto out;
+		}
+		out += nfound;
+
+		while ((jentry <= post->jcount) && (added < nfound)) {
+			u16 rflg;
+			u64 orig = le64_to_lba48(
+					post->jdata[jentry].bval, &rflg);
+			u64 dm_s = le64_to_lba48(
+					post->jdata[jentry].tlba, NULL);
+
+			if ((Z_LOWER48 == dm_s || Z_LOWER48 == orig)) {
+				jentry++;
+
+				if (rflg & GC_READ) {
+					Z_ERR(znd, "ERROR: %" PRIx64
+					      " read and not written %" PRIx64,
+					      orig, dm_s);
+					lba++;
+					added++;
+				}
+				continue;
+			}
+			rflg &= ~GC_READ;
+			post->jdata[jentry].bval = lba48_to_le64(rflg, lba);
+			lba++;
+			added++;
+			jentry++;
+		}
+		nblks -= nfound;
+	}
+	Z_DBG(znd, "Write %d blocks from %d", gc_entry->nblks, gc_entry->w_ptr);
+	set_bit(DO_GC_META, &gc_entry->gc_flags);
+
+out:
+	spin_lock_irqsave(&znd->gc_lock, flags);
+	gc_entry->nblks = 0;
+	gc_entry->w_ptr = jentry;
+	spin_unlock_irqrestore(&znd->gc_lock, flags);
+	mutex_unlock(&post->cached_lock);
+
+	return err;
+}
+
+/**
+ * gc_finalize() - Final sanity check on GC'd block map.
+ * @gc_entry: Active GC state
+ *
+ * gc_postmap is expected to be empty (all blocks original
+ * scheduled to be moved to a new zone have been accounted for...
+ */
+static int gc_finalize(struct gc_state *gc_entry)
+{
+	int err = 0;
+	struct zoned *znd = gc_entry->znd;
+	struct map_cache *post = &znd->gc_postmap;
+	int jentry;
+
+	MutexLock(&post->cached_lock);
+	for (jentry = post->jcount; jentry > 0; jentry--) {
+		u64 dm_s = le64_to_lba48(post->jdata[jentry].tlba, NULL);
+		u64 lba = le64_to_lba48(post->jdata[jentry].bval, NULL);
+
+		if (dm_s != Z_LOWER48 || lba != Z_LOWER48) {
+			Z_ERR(znd, "GC: Failed to move %" PRIx64
+				   " from %"PRIx64" [%d]",
+			      dm_s, lba, jentry);
+			err = -EIO;
+		}
+	}
+	mutex_unlock(&post->cached_lock);
+	post->jcount = jentry;
+	post->jsorted = 0;
+
+	return err;
+}
+
+/**
+ * clear_gc_target_flag() - Clear any zone tagged as a GC target.
+ * @znd: ZDM Instance
+ *
+ * FIXME: Can we reduce the weight of this ?
+ *	Ex. execute as zones are closed and specify the zone to clear
+ *	    at GC completion/cleanup.
+ */
+static void clear_gc_target_flag(struct zoned *znd)
+{
+	int z_id;
+
+	for (z_id = 0; z_id < znd->data_zones; z_id++) {
+		u32 gzno  = z_id >> GZ_BITS;
+		u32 gzoff = z_id & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 wp;
+
+		SpinLock(&wpg->wplck);
+		wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+		if (wp & Z_WP_GC_TARGET) {
+			wp &= ~Z_WP_GC_TARGET;
+			wpg->wp_alloc[gzoff] = cpu_to_le32(wp);
+		}
+		set_bit(IS_DIRTY, &wpg->flags);
+		clear_bit(IS_FLUSH, &wpg->flags);
+		spin_unlock(&wpg->wplck);
+	}
+}
+
+/**
+ * z_zone_gc_metadata_update() - Update ZLT as needed.
+ * @gc_entry: Active GC state
+ *
+ * Dispose or account for all blocks originally scheduled to be
+ * moved. Update ZLT (via map cache) for all moved blocks.
+ */
+static int z_zone_gc_metadata_update(struct gc_state *gc_entry)
+{
+	struct zoned *znd = gc_entry->znd;
+	struct map_cache *post = &znd->gc_postmap;
+	u32 used = post->jcount;
+	int err = 0;
+	int jentry;
+
+	for (jentry = post->jcount; jentry > 0; jentry--) {
+		int discard = 0;
+		int mapping = 0;
+		struct map_pg *mapped = NULL;
+		u64 dm_s = le64_to_lba48(post->jdata[jentry].tlba, NULL);
+		u64 lba = le64_to_lba48(post->jdata[jentry].bval, NULL);
+
+		if ((znd->r_base <= dm_s) && dm_s < (znd->r_base + Z_BLKSZ)) {
+			u64 off = dm_s - znd->r_base;
+
+			mapped = znd->rev_tm[off];
+			mapping = 1;
+		} else if ((znd->s_base <= dm_s) &&
+			   (dm_s < (znd->s_base + Z_BLKSZ))) {
+			u64 off = dm_s - znd->s_base;
+
+			mapped = znd->fwd_tm[off];
+			mapping = 1;
+		}
+
+		if (mapping && !mapped)
+			Z_ERR(znd, "MD: dm_s: %" PRIx64 " -> lba: %" PRIx64
+				   " no mapping in ram.", dm_s, lba);
+
+		if (mapped) {
+			u32 in_z;
+
+			ref_pg(mapped);
+			MutexLock(&mapped->md_lock);
+			in_z = _calc_zone(znd, mapped->last_write);
+			if (in_z != gc_entry->z_gc) {
+				Z_ERR(znd, "MD: %" PRIx64
+				      " Discarded - %" PRIx64
+				      " already flown to: %x",
+				      dm_s, mapped->last_write, in_z);
+				discard = 1;
+			} else if (mapped->data.addr &&
+				   test_bit(IS_DIRTY, &mapped->flags)) {
+				Z_ERR(znd,
+				      "MD: %" PRIx64 " Discarded - %"PRIx64
+				      " is in-flight",
+				      dm_s, mapped->last_write);
+				discard = 2;
+			}
+			if (!discard)
+				mapped->last_write = lba;
+			mutex_unlock(&mapped->md_lock);
+			deref_pg(mapped);
+		}
+
+		MutexLock(&post->cached_lock);
+		if (discard == 1) {
+			Z_ERR(znd, "Dropped: %" PRIx64 " ->  %"PRIx64,
+			      le64_to_cpu(post->jdata[jentry].tlba),
+			      le64_to_cpu(post->jdata[jentry].bval));
+
+			post->jdata[jentry].tlba = MC_INVALID;
+			post->jdata[jentry].bval = MC_INVALID;
+		}
+		if (post->jdata[jentry].tlba  == MC_INVALID &&
+		    post->jdata[jentry].bval == MC_INVALID) {
+			used--;
+		} else if (lba) {
+			increment_used_blks(znd, lba, 1);
+		}
+		mutex_unlock(&post->cached_lock);
+	}
+	err = move_to_map_tables(znd, post);
+	if (err)
+		Z_ERR(znd, "Move to tables post GC failure");
+
+	clear_gc_target_flag(znd);
+
+	return err;
+}
+
+/**
+ * _blkalloc() - Attempt to reserve blocks at z_at in ZDM znd
+ * @znd:	 ZDM instance.
+ * @z_at:	 Zone to write data to
+ * @flags:	 Acquisition type.
+ * @nblks:	 Number of blocks desired.
+ * @nfound:	 Number of blocks allocated or available.
+ *
+ * Attempt allocation of @nblks within fron the current WP of z_at
+ * When nblks are not available 0 is returned and @nfound is the
+ * contains the number of blocks *available* but not *allocated*.
+ * When nblks are available the starting LBA in 4k space is returned and
+ * nblks are allocated *allocated* and *nfound is the number of blocks
+ * remaining in zone z_at from the LBA returned.
+ *
+ * Return: LBA if request is met, otherwise 0. nfound will contain the
+ *	 available blocks remaining.
+ */
+static sector_t _blkalloc(struct zoned *znd, u32 z_at, u32 flags,
+			  u32 nblks, u32 *nfound)
+{
+#define ALLOC_STICKY (Z_WP_GC_TARGET|Z_WP_NON_SEQ|Z_WP_RRECALC)
+
+	sector_t found = 0;
+	u32 avail = 0;
+	int do_open_zone = 0;
+	u32 gzno  = z_at >> GZ_BITS;
+	u32 gzoff = z_at & GZ_MMSK;
+	struct meta_pg *wpg = &znd->wp[gzno];
+	u32 wp;
+	u32 wptr;
+	u32 gc_tflg;
+
+	if (gzno >= znd->gz_count || z_at >= znd->data_zones) {
+		Z_ERR(znd, "Invalid zone for allocation: %u", z_at);
+		dump_stack();
+		return 0ul;
+	}
+
+	SpinLock(&wpg->wplck);
+	wp      = le32_to_cpu(wpg->wp_alloc[gzoff]);
+	gc_tflg = wp & ALLOC_STICKY;
+	wptr    = wp & ~ALLOC_STICKY;
+	if (wptr < Z_BLKSZ)
+		avail = Z_BLKSZ - wptr;
+
+#if 0 /* DEBUG START: Testing zm_write_pages() */
+	if (avail > 7)
+		avail = 7;
+#endif /* DEBUG END: Testing zm_write_pages() */
+
+	*nfound = avail;
+	if (nblks <= avail) {
+		u64 lba = ((u64)z_at << Z_BLKBITS) + znd->data_lba;
+		u32 zf_est = le32_to_cpu(wpg->zf_est[gzoff]) & Z_WP_VALUE_MASK;
+
+		found = lba + wptr;
+		*nfound = nblks;
+		if (wptr == 0)
+			do_open_zone = 1;
+
+		wptr += nblks;
+		zf_est -= nblks;
+		if (wptr == Z_BLKSZ)
+			znd->discard_count += zf_est;
+
+		wptr |= gc_tflg;
+		if (flags & Z_AQ_GC)
+			wptr |= Z_WP_GC_TARGET;
+
+		if (flags & Z_AQ_STREAM_ID)
+			zf_est |= (flags & Z_AQ_STREAM_MASK) << 24;
+		else
+			zf_est |= le32_to_cpu(wpg->zf_est[gzoff])
+				& Z_WP_STREAM_MASK;
+
+		wpg->wp_alloc[gzoff] = cpu_to_le32(wptr);
+		wpg->zf_est[gzoff] = cpu_to_le32(zf_est);
+		set_bit(IS_DIRTY, &wpg->flags);
+		clear_bit(IS_FLUSH, &wpg->flags);
+	}
+	spin_unlock(&wpg->wplck);
+
+	if (do_open_zone)
+		dmz_open_zone(znd, z_at);
+
+	return found;
+}
+
+/**
+ * update_stale_ratio() - Update the stale ratio for the finished bin.
+ * @znd: ZDM instance
+ * @zone: Zone that needs update.
+ */
+static void update_stale_ratio(struct zoned *znd, u32 zone)
+{
+	u64 total_stale = 0;
+	u64 free_zones = 1;
+	u32 bin = zone / znd->stale.binsz;
+	u32 z_id = bin * znd->stale.binsz;
+	u32 s_end = z_id + znd->stale.binsz;
+
+	if (s_end > znd->data_zones)
+		s_end = znd->data_zones;
+
+	for (; z_id < s_end; z_id++) {
+		u32 gzno  = z_id >> GZ_BITS;
+		u32 gzoff = z_id & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 stale = le32_to_cpu(wpg->zf_est[gzoff]) & Z_WP_VALUE_MASK;
+		u32 wp = le32_to_cpu(wpg->wp_alloc[gzoff]) & Z_WP_VALUE_MASK;
+		u32 wflg = le32_to_cpu(wpg->wp_alloc[gzoff]);
+
+		if (wflg & Z_WP_RRECALC) {
+			SpinLock(&wpg->wplck);
+			wflg = le32_to_cpu(wpg->wp_alloc[gzoff])
+			     & ~Z_WP_RRECALC;
+			wpg->wp_alloc[gzoff] = cpu_to_le32(wflg);
+			spin_unlock(&wpg->wplck);
+		}
+
+		if (wp == Z_BLKSZ)
+			total_stale += stale;
+		else
+			free_zones++;
+	}
+
+	total_stale /= free_zones;
+	znd->stale.bins[bin] = (total_stale > ~0u) ? ~0u : total_stale;
+}
+
+/**
+ * z_zone_compact_queue() - Queue zone compaction.
+ * @znd: ZDM instance
+ * @z_gc: Zone to queue.
+ * @delay: Delay queue metric
+ * @gfp: Allocation scheme.
+ *
+ * Return: 1 on success, 0 if not queued/busy, negative on error.
+ */
+static
+int z_zone_compact_queue(struct zoned *znd, u32 z_gc, int delay, gfp_t gfp)
+{
+	unsigned long flags;
+	int do_queue = 0;
+	int err = 0;
+	struct gc_state *gc_entry;
+
+	gc_entry = ZDM_ALLOC(znd, sizeof(*gc_entry), KM_16, gfp);
+	if (!gc_entry) {
+		Z_ERR(znd, "No Memory for compact!!");
+		return -ENOMEM;
+	}
+	gc_entry->znd = znd;
+	gc_entry->z_gc = z_gc;
+	set_bit(DO_GC_NEW, &gc_entry->gc_flags);
+	znd->gc_backlog++;
+
+	spin_lock_irqsave(&znd->gc_lock, flags);
+	if (!znd->gc_active) {
+		znd->gc_active = gc_entry;
+		do_queue = 1;
+	} else {
+		Z_ERR(znd, "GC: Tried to queue but already active");
+	}
+	spin_unlock_irqrestore(&znd->gc_lock, flags);
+
+	if (do_queue) {
+		unsigned long tval = msecs_to_jiffies(delay);
+
+		if (queue_delayed_work(znd->gc_wq, &znd->gc_work, tval))
+			err = 1;
+	} else {
+		ZDM_FREE(znd, gc_entry, sizeof(*gc_entry), KM_16);
+		znd->gc_backlog--;
+		Z_ERR(znd, "GC: FAILED to queue work");
+	}
+
+	return err;
+}
+
+/**
+ * zone_zfest() - Queue zone compaction.
+ * @znd: ZDM instance
+ * @z_id: Zone to queue.
+ */
+static u32 zone_zfest(struct zoned *znd, u32 z_id)
+{
+	u32 gzno  = z_id >> GZ_BITS;
+	u32 gzoff = z_id & GZ_MMSK;
+	struct meta_pg *wpg = &znd->wp[gzno];
+
+	return le32_to_cpu(wpg->zf_est[gzoff]) & Z_WP_VALUE_MASK;
+}
+
+/**
+ * gc_request_queued() - Called periodically to initiate GC
+ *
+ * @znd: ZDM instance
+ * @bin: Bin with stale zones to scan for GC
+ * @delay: Metric for delay queuing.
+ * @gfp: Default memory allocation scheme.
+ *
+ */
+static int gc_request_queued(struct zoned *znd, int bin, int delay, gfp_t gfp)
+{
+	unsigned long flags;
+	int queued = 0;
+	u32 top_roi = NOZONE;
+	u32 stale = 0;
+	u32 z_gc = bin * znd->stale.binsz;
+	u32 s_end = z_gc + znd->stale.binsz;
+
+	if (znd->meta_result)
+		goto out;
+
+	if (test_bit(ZF_FREEZE, &znd->flags)) {
+		Z_ERR(znd, "Is frozen -- GC paused.");
+		goto out;
+	}
+
+	spin_lock_irqsave(&znd->gc_lock, flags);
+	if (znd->gc_active)
+		queued = 1;
+	spin_unlock_irqrestore(&znd->gc_lock, flags);
+	if (queued)
+		goto out;
+
+	if (s_end > znd->data_zones)
+		s_end = znd->data_zones;
+
+	/* scan for most stale zone in STREAM [top_roi] */
+	for (; z_gc < s_end; z_gc++) {
+		u32 gzno  = z_gc >> GZ_BITS;
+		u32 gzoff = z_gc & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 wp_v = le32_to_cpu(wpg->wp_alloc[gzoff]);
+		u32 nfree = le32_to_cpu(wpg->zf_est[gzoff]) & Z_WP_VALUE_MASK;
+		u32 wp_f = wp_v & Z_WP_FLAGS_MASK;
+
+		wp_v &= Z_WP_VALUE_MASK;
+		if (wp_v == 0)
+			continue;
+		if ((wp_f & Z_WP_GC_PENDING) != 0)
+			continue;
+
+		if (wp_v == Z_BLKSZ) {
+			stale += nfree;
+			if ((wp_f & Z_WP_GC_BITS) == Z_WP_GC_READY) {
+				if (top_roi == NOZONE)
+					top_roi = z_gc;
+				else if (nfree > zone_zfest(znd, top_roi))
+					top_roi = z_gc;
+			}
+		}
+	}
+
+	if (!delay && top_roi == NOZONE)
+		Z_ERR(znd, "No GC candidate in bin: %u -> %u", z_gc, s_end);
+
+	/* determine the cut-off for GC based on MZ overall staleness */
+	if (top_roi != NOZONE) {
+		u32 state_metric = GC_PRIO_DEFAULT;
+		u32 n_empty = znd->z_gc_free;
+		int pctfree = n_empty * 100 / znd->data_zones;
+
+		/*
+		 * -> at less than 5 zones free switch to critical
+		 * -> at less than 5% zones free switch to HIGH
+		 * -> at less than 25% free switch to LOW
+		 * -> high level is 'cherry picking' near empty zones
+		 */
+		if (znd->z_gc_free < 5)
+			state_metric = GC_PRIO_CRIT;
+		else if (pctfree < 5)
+			state_metric = GC_PRIO_HIGH;
+		else if (pctfree < 25)
+			state_metric = GC_PRIO_LOW;
+
+		if (zone_zfest(znd, top_roi) > state_metric) {
+			int rc;
+
+			rc = z_zone_compact_queue(znd, top_roi, delay * 5, gfp);
+			if (rc == 1)
+				queued = 1;
+			else if (rc < 0)
+				Z_ERR(znd, "GC: Z#%u !Q: ERR: %d", top_roi, rc);
+		}
+
+		if (!delay && !queued)
+			Z_ERR(znd, "GC: Z#%u !Q .. M: %u E: %u PCT: %d ZF: %u",
+				   top_roi, state_metric, n_empty, pctfree,
+				   zone_zfest(znd, top_roi));
+	}
+out:
+	return queued;
+}
+
+/**
+ * z_zone_gc_compact() - Primary compaction worker.
+ * @gc_entry: GC State
+ */
+static int z_zone_gc_compact(struct gc_state *gc_entry)
+{
+	unsigned long flags;
+	int err = 0;
+	struct zoned *znd = gc_entry->znd;
+	u32 z_gc = gc_entry->z_gc;
+	u32 gzno  = z_gc >> GZ_BITS;
+	u32 gzoff = z_gc & GZ_MMSK;
+	struct meta_pg *wpg = &znd->wp[gzno];
+
+	znd->age = jiffies_64;
+
+	if (test_bit(DO_GC_NEW, &gc_entry->gc_flags)) {
+		err = z_flush_bdev(znd, GFP_KERNEL);
+		if (err) {
+			gc_entry->result = err;
+			goto out;
+		}
+	}
+
+	/* If a SYNC is in progress and we can delay then postpone*/
+	if (mutex_is_locked(&znd->mz_io_mutex) &&
+	    atomic_read(&znd->gc_throttle) == 0)
+		return -EAGAIN;
+
+	if (test_and_clear_bit(DO_GC_NEW, &gc_entry->gc_flags)) {
+		u32 wp;
+
+		SpinLock(&wpg->wplck);
+		wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+		wp |= Z_WP_GC_FULL;
+		wpg->wp_alloc[gzoff] = cpu_to_le32(wp);
+		set_bit(IS_DIRTY, &wpg->flags);
+		clear_bit(IS_FLUSH, &wpg->flags);
+		spin_unlock(&wpg->wplck);
+
+		err = _cached_to_tables(znd, gc_entry->z_gc);
+		if (err) {
+			Z_ERR(znd, "Failed to purge journal: %d", err);
+			gc_entry->result = err;
+			goto out;
+		}
+
+		if (znd->gc_postmap.jcount > 0) {
+			Z_ERR(znd, "*** Unexpected data in postmap!!");
+			znd->gc_postmap.jcount = 0;
+		}
+
+		err = z_zone_gc_metadata_to_ram(gc_entry);
+		if (err) {
+			Z_ERR(znd,
+			      "Pre-load metadata to memory failed!! %d", err);
+			gc_entry->result = err;
+			goto out;
+		}
+
+		do {
+			err = memcache_lock_and_sort(znd, &znd->gc_postmap);
+		} while (err == -EBUSY);
+
+		set_bit(DO_GC_PREPARE, &gc_entry->gc_flags);
+
+		if (znd->gc_throttle.counter == 0)
+			return -EAGAIN;
+	}
+
+next_in_queue:
+	znd->age = jiffies_64;
+
+	if (test_and_clear_bit(DO_GC_PREPARE, &gc_entry->gc_flags)) {
+
+		MutexLock(&znd->mz_io_mutex);
+		mutex_unlock(&znd->mz_io_mutex);
+
+		err = z_zone_gc_read(gc_entry);
+		if (err < 0) {
+			Z_ERR(znd, "z_zone_gc_chunk issue failure: %d", err);
+			gc_entry->result = err;
+			goto out;
+		}
+		if (znd->gc_throttle.counter == 0)
+			return -EAGAIN;
+	}
+
+	if (test_and_clear_bit(DO_GC_WRITE, &gc_entry->gc_flags)) {
+		u32 sid = le32_to_cpu(wpg->zf_est[gzoff]) & Z_WP_STREAM_MASK;
+
+		MutexLock(&znd->mz_io_mutex);
+		mutex_unlock(&znd->mz_io_mutex);
+
+		err = z_zone_gc_write(gc_entry, sid >> 24);
+		if (err) {
+			Z_ERR(znd, "z_zone_gc_write issue failure: %d", err);
+			gc_entry->result = err;
+			goto out;
+		}
+		if (znd->gc_throttle.counter == 0)
+			return -EAGAIN;
+	}
+
+	if (test_and_clear_bit(DO_GC_META, &gc_entry->gc_flags)) {
+		z_do_copy_more(gc_entry);
+		goto next_in_queue;
+	}
+
+	znd->age = jiffies_64;
+	if (test_and_clear_bit(DO_GC_COMPLETE, &gc_entry->gc_flags)) {
+		u32 non_seq;
+		u32 reclaimed;
+
+		err = z_zone_gc_metadata_update(gc_entry);
+		gc_entry->result = err;
+		if (err) {
+			Z_ERR(znd, "Metadata error ... disable zone: %u",
+			      gc_entry->z_gc);
+		}
+		err = gc_finalize(gc_entry);
+		if (err) {
+			Z_ERR(znd, "GC: Failed to finalize: %d", err);
+			gc_entry->result = err;
+			goto out;
+		}
+
+		gc_verify_cache(znd, gc_entry->z_gc);
+
+		err = _cached_to_tables(znd, gc_entry->z_gc);
+		if (err) {
+			Z_ERR(znd, "Failed to purge journal: %d", err);
+			gc_entry->result = err;
+			goto out;
+		}
+
+		err = z_flush_bdev(znd, GFP_KERNEL);
+		if (err) {
+			gc_entry->result = err;
+			goto out;
+		}
+
+		/* Release the zones for writing */
+		dmz_reset_wp(znd, gc_entry->z_gc);
+
+		SpinLock(&wpg->wplck);
+		non_seq = le32_to_cpu(wpg->wp_alloc[gzoff]) & Z_WP_NON_SEQ;
+		reclaimed = le32_to_cpu(wpg->zf_est[gzoff]) & Z_WP_VALUE_MASK;
+		wpg->wp_alloc[gzoff] = cpu_to_le32(non_seq);
+		wpg->wp_used[gzoff] = cpu_to_le32(0u);
+		wpg->zf_est[gzoff] = cpu_to_le32(Z_BLKSZ);
+		znd->discard_count -= reclaimed;
+		znd->z_gc_free++;
+		if (znd->z_gc_resv & Z_WP_GC_ACTIVE)
+			znd->z_gc_resv = gc_entry->z_gc;
+		else if (znd->z_meta_resv & Z_WP_GC_ACTIVE)
+			znd->z_meta_resv = gc_entry->z_gc;
+		set_bit(IS_DIRTY, &wpg->flags);
+		clear_bit(IS_FLUSH, &wpg->flags);
+		spin_unlock(&wpg->wplck);
+
+		znd->gc_events++;
+		update_stale_ratio(znd, gc_entry->z_gc);
+		spin_lock_irqsave(&znd->gc_lock, flags);
+		znd->gc_backlog--;
+		znd->gc_active = NULL;
+		spin_unlock_irqrestore(&znd->gc_lock, flags);
+
+		ZDM_FREE(znd, gc_entry, sizeof(*gc_entry), KM_16);
+
+		set_bit(DO_JOURNAL_MOVE, &znd->flags);
+		set_bit(DO_MEMPOOL, &znd->flags);
+		set_bit(DO_SYNC, &znd->flags);
+	}
+out:
+	return 0;
+}
+
+/**
+ * gc_work_task() - Worker thread for GC activity.
+ * @work: Work struct holding the ZDM instance to do work on ...
+ */
+static void gc_work_task(struct work_struct *work)
+{
+	struct gc_state *gc_entry = NULL;
+	unsigned long flags;
+	struct zoned *znd;
+	int err;
+
+	if (!work)
+		return;
+
+	znd = container_of(to_delayed_work(work), struct zoned, gc_work);
+	if (!znd)
+		return;
+
+	spin_lock_irqsave(&znd->gc_lock, flags);
+	if (znd->gc_active)
+		gc_entry = znd->gc_active;
+	spin_unlock_irqrestore(&znd->gc_lock, flags);
+
+	if (!gc_entry) {
+		Z_ERR(znd, "ERROR: gc_active not set!");
+		return;
+	}
+
+	err = z_zone_gc_compact(gc_entry);
+	if (-EAGAIN == err) {
+		unsigned long tval = msecs_to_jiffies(10);
+
+		queue_delayed_work(znd->gc_wq, &znd->gc_work, tval);
+	} else {
+		const int delay = 10;
+
+		on_timeout_activity(znd, delay);
+	}
+}
+
+/**
+ * is_reserved() - Check to see if a zone is 'special'
+ * @znd: ZDM Instance
+ * @z_pref: Zone to be tested.
+ */
+static inline int is_reserved(struct zoned *znd, const u32 z_pref)
+{
+	const u32 gc   = znd->z_gc_resv   & Z_WP_VALUE_MASK;
+	const u32 meta = znd->z_meta_resv & Z_WP_VALUE_MASK;
+
+	return (gc == z_pref || meta == z_pref) ? 1 : 0;
+}
+
+/**
+ * gc_can_cherrypick() - Queue a GC for zone in this bin if ... it will be easy
+ * @znd: ZDM Instance
+ * @bin: The bin (0 to 255)
+ * @delay: Delay metric
+ * @gfp: Allocation flags to use.
+ */
+static int gc_can_cherrypick(struct zoned *znd, u32 bin, int delay, gfp_t gfp)
+{
+	u32 z_id = bin * znd->stale.binsz;
+	u32 s_end = z_id + znd->stale.binsz;
+
+	if (s_end > znd->data_zones)
+		s_end = znd->data_zones;
+
+	for (; z_id < s_end; z_id++) {
+		u32 gzno  = z_id >> GZ_BITS;
+		u32 gzoff = z_id & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+		u32 nfree = le32_to_cpu(wpg->zf_est[gzoff]) & Z_WP_VALUE_MASK;
+
+		if (wp & Z_WP_RRECALC)
+			update_stale_ratio(znd, z_id);
+
+		if (((wp & Z_WP_GC_BITS) == Z_WP_GC_READY) &&
+		    ((wp & Z_WP_VALUE_MASK) == Z_BLKSZ) &&
+		    (nfree == Z_BLKSZ)) {
+			if (z_zone_compact_queue(znd, z_id, delay, gfp))
+				return 1;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * gc_queue_with_delay() - Scan to see if a GC can/should be queued.
+ * @znd: ZDM Instance
+ * @delay: Delay metric
+ * @gfp: Allocation flags to use.
+ *
+ * Return 1 if gc in progress or queued. 0 otherwise.
+ */
+static int gc_queue_with_delay(struct zoned *znd, int delay, gfp_t gfp)
+{
+	int gc_idle = 0;
+	unsigned long flags;
+
+	spin_lock_irqsave(&znd->gc_lock, flags);
+	gc_idle = !znd->gc_active;
+	spin_unlock_irqrestore(&znd->gc_lock, flags);
+
+	if (gc_idle) {
+		int bin = 0;
+		int ratio = 0;
+		u32 iter;
+
+		/* Find highest ratio stream */
+		for (iter = 0; iter < znd->stale.count; iter++)
+			if (znd->stale.bins[iter] > ratio)
+				ratio = znd->stale.bins[iter], bin = iter;
+
+		/* Cherrypick a zone in the stream */
+		if (gc_idle && gc_can_cherrypick(znd, bin, delay, gfp))
+			gc_idle = 0;
+
+		/* Otherwise cherrypick *something* */
+		for (iter = 0; gc_idle && (iter < znd->stale.count); iter++)
+			if (gc_idle && (bin != iter) &&
+			    gc_can_cherrypick(znd, iter, delay, gfp))
+				gc_idle = 0;
+
+		/* Otherwise compact a zone in the stream */
+		if (gc_idle && gc_request_queued(znd, bin, delay, gfp))
+			gc_idle = 0;
+
+		/* Otherwise compact *something* */
+		for (iter = 0; gc_idle && (iter < znd->stale.count); iter++)
+			if (gc_idle && gc_request_queued(znd, iter, delay, gfp))
+				gc_idle = 0;
+	}
+	return !gc_idle;
+}
+
+/**
+ * gc_immediate() - Free up some space as soon as possible.
+ * @znd: ZDM Instance
+ * @gfp: Allocation flags to use.
+ */
+static int gc_immediate(struct zoned *znd, int wait, gfp_t gfp)
+{
+	const int delay = 0;
+	int can_retry = 0;
+	int lock_owner = 0;
+	int queued;
+
+	/*
+	 * Lock on the first entry. On subsequent entries
+	 * only lock of a 'wait' was requested.
+	 *
+	 * Note: We really should be using a wait queue here but this
+	 *       mutex hack should do as a temporary workaround.
+	 */
+	atomic_inc(&znd->gc_throttle);
+	if (atomic_read(&znd->gc_throttle) == 1) {
+		mutex_lock(&znd->gc_wait);
+		lock_owner = 1;
+	} else if (atomic_read(&znd->gc_throttle) > 1) {
+		if (wait) {
+			mutex_lock(&znd->gc_wait);
+			can_retry = 1;
+			mutex_unlock(&znd->gc_wait);
+		}
+		goto out;
+	}
+	flush_delayed_work(&znd->gc_work);
+	queued = gc_queue_with_delay(znd, delay, gfp);
+	if (!queued) {
+		Z_ERR(znd, " ... GC immediate .. failed to queue GC!!.");
+		z_discard_partial(znd, DISCARD_MAX_INGRESS, GFP_KERNEL);
+		can_retry = 1;
+		goto out;
+	}
+	can_retry = flush_delayed_work(&znd->gc_work);
+
+out:
+	if (lock_owner)
+		mutex_unlock(&znd->gc_wait);
+	atomic_dec(&znd->gc_throttle);
+
+	return can_retry;
+}
+
+/**
+ * set_current() - Make zone the preferred zone for allocation.
+ * @znd: ZDM Instance
+ * @flags: BLock allocation scheme (including stream id)
+ * @zone: The zone to make preferred.
+ *
+ * Once a zone is opened for allocation, future allocations will prefer
+ * the same zone, until the zone is full.
+ * Each stream id has it's own preferred zone.
+ *
+ * NOTE: z_current is being deprecated if favor of assuming a default
+ *       stream id when nothing is provided.
+ */
+static inline void set_current(struct zoned *znd, u32 flags, u32 zone)
+{
+	if (flags & Z_AQ_STREAM_ID) {
+		u32 stream_id = flags & Z_AQ_STREAM_MASK;
+
+		znd->bmkeys->stream[stream_id] = cpu_to_le32(zone);
+	}
+	znd->z_current = zone;
+	if (znd->z_gc_free > 0)
+		znd->z_gc_free--;
+	else
+		Z_ERR(znd, "Dec z_gc_free below 0?");
+}
+
+/**
+ * next_open_zone() - Grab the next available zone
+ * @znd: ZDM Instance
+ * @z_at: Zone to start scanning from (presumable just filled).
+ *
+ * Return: NOZONE if no zone exists with space for writing.
+ *
+ * Scan through the available zones for an empty zone.
+ * If no empty zone is available the a zone that is not full is
+ * used instead.
+ */
+static u32 next_open_zone(struct zoned *znd, u32 z_at)
+{
+	u32 zone = NOZONE;
+	u32 z_id;
+
+	if (znd->data_zones < z_at)
+		z_at = znd->data_zones;
+
+	/* scan higher lba zones */
+	for (z_id = z_at; z_id < znd->data_zones; z_id++) {
+		u32 gzno  = z_id >> GZ_BITS;
+		u32 gzoff = z_id & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+
+		if ((wp & Z_WP_VALUE_MASK) == 0) {
+			u32 check = gzno << GZ_BITS | gzoff;
+
+			if (!is_reserved(znd, check)) {
+				zone = check;
+				goto out;
+			}
+		}
+	}
+
+	/* scan lower lba zones */
+	for (z_id = 0; z_id < z_at; z_id++) {
+		u32 gzno  = z_id >> GZ_BITS;
+		u32 gzoff = z_id & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+
+		if ((wp & Z_WP_VALUE_MASK) == 0) {
+			u32 check = gzno << GZ_BITS | gzoff;
+
+			if (!is_reserved(znd, check)) {
+				zone = check;
+				goto out;
+			}
+		}
+	}
+
+	/* No empty zones .. start co-mingling streams */
+	for (z_id = 0; z_id < znd->data_zones; z_id++) {
+		u32 gzno  = z_id >> GZ_BITS;
+		u32 gzoff = z_id & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+
+		if ((wp & Z_WP_VALUE_MASK) < Z_BLKSZ) {
+			u32 check = gzno << GZ_BITS | gzoff;
+
+			if (!is_reserved(znd, check)) {
+				zone = check;
+				goto out;
+			}
+		}
+	}
+
+out:
+	return zone;
+}
+
+/**
+ * zone_filled_cleanup() - Update wp_alloc GC Readu flags based on wp_used.
+ * @znd: ZDM Instance
+ */
+static void zone_filled_cleanup(struct zoned *znd)
+{
+	if (znd->filled_zone != NOZONE) {
+		u32 zone = znd->filled_zone;
+		u32 gzno;
+		u32 gzoff;
+		u32 wp;
+		u32 used;
+		struct meta_pg *wpg;
+
+		znd->filled_zone = NOZONE;
+
+		gzno = zone >> GZ_BITS;
+		gzoff = zone & GZ_MMSK;
+		wpg = &znd->wp[gzno];
+
+		SpinLock(&wpg->wplck);
+		wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+		used = le32_to_cpu(wpg->wp_used[gzoff]) & Z_WP_VALUE_MASK;
+		if (used == Z_BLKSZ) {
+			if (Z_BLKSZ == (wp & Z_WP_VALUE_MASK)) {
+				wpg->wp_alloc[gzoff] = cpu_to_le32(wp
+						     | Z_WP_GC_READY);
+				set_bit(IS_DIRTY, &wpg->flags);
+				clear_bit(IS_FLUSH, &wpg->flags);
+			} else {
+				Z_ERR(znd, "Zone %u seems bogus.", zone);
+			}
+		}
+		spin_unlock(&wpg->wplck);
+
+		dmz_close_zone(znd, zone);
+		update_stale_ratio(znd, zone);
+	}
+}
+
+/**
+ * z_acquire() - Allocate blocks for writing
+ * @znd: ZDM Instance
+ * @flags: Alloc strategy and stream id.
+ * @nblks: Number of blocks desired.
+ * @nfound: Number of blocks available.
+ *
+ * Return: Lba for writing.
+ */
+static u64 z_acquire(struct zoned *znd, u32 flags, u32 nblks, u32 *nfound)
+{
+	sector_t found = 0;
+	u32 z_pref = znd->z_current;
+	u32 stream_id = 0;
+	u32 z_find;
+	const int wait = 1;
+	gfp_t gfp = (flags & Z_AQ_NORMAL) ? GFP_ATOMIC : GFP_KERNEL;
+
+	zone_filled_cleanup(znd);
+
+	if (flags & Z_AQ_STREAM_ID) {
+		stream_id = flags & Z_AQ_STREAM_MASK;
+		z_pref = le32_to_cpu(znd->bmkeys->stream[stream_id]);
+	}
+	if (z_pref >= znd->data_zones) {
+		z_pref = next_open_zone(znd, znd->z_current);
+		if (z_pref < znd->data_zones)
+			set_current(znd, flags, z_pref);
+	}
+
+	if (z_pref < znd->data_zones) {
+		found = _blkalloc(znd, z_pref, flags, nblks, nfound);
+		if (found || *nfound)
+			goto out;
+	}
+
+	if (znd->z_gc_free < 5) {
+		Z_DBG(znd, "... alloc - gc low on free space.");
+		gc_immediate(znd, !wait, gfp);
+	}
+
+retry:
+	z_find = next_open_zone(znd, znd->z_current);
+	if (z_find < znd->data_zones) {
+		found = _blkalloc(znd, z_find, flags, nblks, nfound);
+		if (found || *nfound) {
+			set_current(znd, flags, z_find);
+			goto out;
+		}
+	}
+
+	if (flags & Z_AQ_GC) {
+		u32 gresv = znd->z_gc_resv & Z_WP_VALUE_MASK;
+
+		Z_ERR(znd, "Using GC Reserve (%u)", gresv);
+		found = _blkalloc(znd, gresv, flags, nblks, nfound);
+		znd->z_gc_resv |= Z_WP_GC_ACTIVE;
+	}
+
+	if (flags & Z_AQ_META) {
+		int can_retry = gc_immediate(znd, wait, gfp);
+		u32 mresv = znd->z_meta_resv & Z_WP_VALUE_MASK;
+
+		Z_DBG(znd, "GC: Need META.");
+		if (can_retry)
+			goto retry;
+
+		Z_ERR(znd, "Using META Reserve (%u)", znd->z_meta_resv);
+		found = _blkalloc(znd, mresv, flags, nblks, nfound);
+	}
+
+out:
+	if (!found && (*nfound == 0)) {
+		Z_DBG(znd, "... alloc - out of space?");
+		if (gc_immediate(znd, wait, gfp))
+			goto retry;
+
+		Z_ERR(znd, "%s: -> Out of space.", __func__);
+	}
+	return found;
+}
+
+/**
+ * compare_lba() - Compare map page on lba.
+ * @x1: map page
+ * @x2: map page
+ *
+ * Return -1, 0, or 1 if x1 < x2, equal, or >, respectivly.
+ */
+static int compare_lba(const void *x1, const void *x2)
+{
+	const struct map_pg *v1 = *(const struct map_pg **)x1;
+	const struct map_pg *v2 = *(const struct map_pg **)x2;
+	int cmp = (v1->lba < v2->lba) ? -1 : ((v1->lba > v2->lba) ? 1 : 0);
+
+	return cmp;
+}
+
+/**
+ * is_dirty() - Test map page if is of bit_type and dirty.
+ * @expg: map page
+ * @bit_type: map page flag to test for...
+ *
+ * Return 1 if page had dirty and bit_type flags set.
+ *
+ * Note: bit_type is IS_CRC and IS_LUT most typically.
+ */
+static __always_inline int is_dirty(struct map_pg *expg, int bit_type)
+{
+	return (test_bit(bit_type, &expg->flags) &&
+		test_bit(IS_DIRTY, &expg->flags));
+}
+
+
+/**
+ * is_old_and_clean() - Test map page if it expired and can be dropped.
+ * @expg: map page
+ * @bit_type: map page flag to test for...
+ *
+ * Return 1 if page is clean and old.
+ */
+static __always_inline int is_old_and_clean(struct map_pg *expg, int bit_type)
+{
+	int is_match = 0;
+
+	if (!test_bit(IS_DIRTY, &expg->flags) && is_expired(expg->age)) {
+		if (getref_pg(expg) == 1)
+			is_match = 1;
+		else
+			pr_err("%llu: dirty, exp, and elev: %d\n",
+				expg->lba, getref_pg(expg));
+	}
+
+	(void)bit_type;
+
+	return is_match;
+}
+
+/**
+ * _pool_write() - Sort/Write and array of ZLT pages.
+ * @znd: ZDM Instance
+ * @wset: Array of pages to be written.
+ * @count: Number of entries.
+ *
+ *  NOTE: On entry all map_pg entries have elevated refcount from _pool_fill().
+ *	write_if_dirty() will dec the refcount when the block hits disk.
+ */
+static int _pool_write(struct zoned *znd, struct map_pg **wset, int count)
+{
+	const int use_wq = 0;
+	int iter;
+	struct map_pg *expg;
+	int err = 0;
+
+	/* write dirty table pages */
+	if (count <= 0)
+		goto out;
+
+	sort(wset, count, sizeof(*wset), compare_lba, NULL);
+
+	for (iter = 0; iter < count; iter++) {
+		expg = wset[iter];
+		if (expg) {
+			if (iter && expg->lba == wset[iter-1]->lba) {
+				wset[iter] = NULL;
+				Z_ERR(znd, "** DUPE ** ignored.");
+			} else
+				cache_if_dirty(znd, expg, use_wq);
+		}
+	}
+
+	for (iter = 0; iter < count; iter++) {
+		const int sync = 1;
+		/* REDUX: For Async WB:  int sync = (iter == last) ? 1 : 0; */
+
+		expg = wset[iter];
+		if (expg) {
+			err = write_if_dirty(znd, expg, use_wq, sync);
+			if (err) {
+				Z_ERR(znd, "Write failed: %d", err);
+				goto out;
+			}
+		}
+	}
+	err = count;
+
+out:
+	return err;
+}
+
+/**
+ * z_metadata_lba() - Alloc a block of metadata.
+ * @znd: ZDM Instance
+ * @map: Block of metadata.
+ * @num: Number of blocks allocated.
+ *
+ * Return: lba or 0 on failure.
+ *
+ * When map->lba is less than data_lba the metadata is pinned to it's logical
+ * location.
+ * When map->lba lands in data space it is dynmaically allocated and intermixed
+ * within the datapool.
+ *
+ * [FUTURE: Pick a stream_id for floating metadata]
+ */
+static u64 z_metadata_lba(struct zoned *znd, struct map_pg *map, u32 *num)
+{
+	u32 nblks = 1;
+	u64 lba = map->lba;
+
+	if (lba < znd->data_lba) {
+		*num = 1;
+		return map->lba;
+	}
+	return z_acquire(znd, Z_AQ_META_STREAM, nblks, num);
+}
+
+/**
+ * pg_crc() -
+ * @znd: ZDM Instance
+ * @pg:
+ * @md_crc: 16 bit crc of page.
+ *
+ * callback from dm_io notify.. cannot hold mutex here
+ */
+static void pg_crc(struct zoned *znd, struct map_pg *pg, __le16 md_crc)
+{
+	struct mpinfo mpi;
+
+	to_table_entry(znd, pg->lba, &mpi);
+	if (pg->crc_pg) {
+		struct map_pg *crc_pg = pg->crc_pg;
+		int entry = mpi.crc.pg_idx;
+
+		if (crc_pg && crc_pg->data.crc) {
+			ref_pg(crc_pg);
+			crc_pg->data.crc[entry] = md_crc;
+			clear_bit(IS_READA, &crc_pg->flags);
+			set_bit(IS_DIRTY, &crc_pg->flags);
+			clear_bit(IS_FLUSH, &crc_pg->flags);
+			crc_pg->age = jiffies_64
+				    + msecs_to_jiffies(crc_pg->hotness);
+			deref_pg(crc_pg);
+		}
+		put_map_entry(crc_pg);
+		if (getref_pg(crc_pg) == 0)
+			pg->crc_pg = NULL;
+	} else if (!test_bit(IS_LUT, &pg->flags)) {
+		znd->md_crcs[mpi.crc.pg_idx] = md_crc;
+	} else {
+		Z_ERR(znd, "unexpected state.");
+		dump_stack();
+	}
+}
+
+
+/**
+ * pg_written() - Handle accouting related to lookup table page writes
+ * @pg: The page of lookup table [or CRC] that was written.
+ * @error: non-zero if an error occurred.
+ *
+ * callback from dm_io notify.. cannot hold mutex here, cannot sleep.
+ *
+ */
+static int pg_written(struct map_pg *pg, unsigned long error)
+{
+	int rcode = 0;
+	struct zoned *znd = pg->znd;
+	__le16 md_crc;
+
+	if (error) {
+		Z_ERR(znd, "write_page: %" PRIx64 " -> %" PRIx64
+			   " ERR: %ld", pg->lba, pg->last_write, error);
+		rcode = -EIO;
+		goto out;
+	}
+
+	/*
+	 * Re-calculate CRC on current memory page. If unchanged then on-disk
+	 * is stable and in-memory is not dirty. Otherwise in memory changed
+	 * during write back so leave the dirty flag set. For the purpose of
+	 * the CRC table we assume that in-memory == on-disk although this
+	 * is not strictly true as the page could have updated post disk write.
+	 */
+
+	md_crc = crc_md_le16(pg->data.addr, Z_CRC_4K);
+	pg->age = jiffies_64 + msecs_to_jiffies(pg->hotness);
+	if (md_crc == pg->md_crc)
+		clear_bit(IS_DIRTY, &pg->flags);
+
+	clear_bit(W_IN_FLIGHT, &pg->flags);
+	pg_crc(znd, pg, md_crc);
+
+	if (pg->last_write < znd->data_lba)
+		goto out;
+
+/*
+ * NOTE: If we reach here it's a problem.
+ *       TODO: mutex free path for adding a map entry ...
+ */
+
+	/* written lba was allocated from data-pool */
+	rcode = z_mapped_addmany(znd, pg->lba, pg->last_write, 1, CRIT);
+	if (rcode) {
+		Z_ERR(znd, "%s: Journal MANY failed.", __func__);
+		goto out;
+	}
+	increment_used_blks(znd, pg->last_write, 1);
+
+out:
+	return rcode;
+}
+
+
+/**
+ * on_pg_written() - A block of map table was written.
+ * @error: Any error code that occurred during the I/O.
+ * @context: The map_pg that was queued/written.
+ */
+static void on_pg_written(unsigned long error, void *context)
+{
+	struct map_pg *pg = context;
+	int rcode;
+
+	rcode = pg_written(pg, error);
+	deref_pg(pg);
+	if (rcode < 0)
+		pg->znd->meta_result = rcode;
+}
+
+/**
+ * queue_pg() - Queue a map table page for writeback
+ * @znd: ZDM Instance
+ * @pg: The target page to ensure the cover CRC blocks is cached.
+ * @lba: The address to write the block to.
+ */
+static int queue_pg(struct zoned *znd, struct map_pg *pg, u64 lba)
+{
+	const int use_wq = 0;
+	sector_t block = lba << Z_SHFT4K;
+	unsigned int nDMsect = 1 << Z_SHFT4K;
+	int rc;
+
+	pg->znd = znd;
+	MutexLock(&pg->md_lock);
+	pg->md_crc = crc_md_le16(pg->data.addr, Z_CRC_4K);
+	pg->last_write = lba; /* presumably */
+	mutex_unlock(&pg->md_lock);
+
+	rc = znd_async_io(znd, DM_IO_KMEM, pg->data.addr, block, nDMsect, WRITE,
+			  use_wq, on_pg_written, pg);
+	if (rc) {
+		Z_ERR(znd, "queue error: %d Q: %" PRIx64 " [%u dm sect] (Q:%d)",
+		      rc, lba, nDMsect, use_wq);
+		dump_stack();
+	}
+
+	return rc;
+}
+
+/**
+ * cache_if_dirty() - Load a page of CRC's into memory.
+ * @znd: ZDM Instance
+ * @pg: The target page to ensure the cover CRC blocks is cached.
+ * @wp: If a queue is needed for I/O.
+ *
+ * The purpose of loading is to ensure the pages are in memory with the
+ * async_io (write) completes the CRC accounting doesn't cause a sleep
+ * and violate the callback() API rules.
+ */
+static void cache_if_dirty(struct zoned *znd, struct map_pg *pg, int wq)
+{
+	if (test_bit(IS_DIRTY, &pg->flags) && test_bit(IS_LUT, &pg->flags) &&
+	    pg->data.addr) {
+		u64 base = znd->c_base;
+		struct map_pg *crc_pg;
+		struct mpinfo mpi;
+
+		if (test_bit(IS_REV, &pg->flags))
+			base = znd->c_mid;
+
+		to_table_entry(znd, pg->lba, &mpi);
+		base += mpi.crc.pg_no;
+		crc_pg = get_map_entry(znd, base, NORMAL);
+		if (!crc_pg)
+			Z_ERR(znd, "Out of memory. No CRC Pg");
+		pg->crc_pg = crc_pg;
+	}
+}
+
+/**
+ * write_if_dirty() - Write old pages (flagged as DIRTY) pages of table map.
+ * @znd: ZDM instance.
+ * @pg: A page of table map data.
+ * @wq: Use worker queue for sync writes.
+ * @snc: Performa a Sync or Async write.
+ *
+ * Return: 0 on success or -errno value
+ */
+static int write_if_dirty(struct zoned *znd, struct map_pg *pg, int wq, int snc)
+{
+	int rcode = 0;
+
+	if (!pg)
+		return rcode;
+
+	if (test_bit(IS_DIRTY, &pg->flags) && pg->data.addr) {
+		u64 dm_s = pg->lba;
+		u32 nf;
+		u64 lba = z_metadata_lba(znd, pg, &nf);
+
+		Z_DBG(znd, "Write if dirty: %" PRIx64" -> %" PRIx64, dm_s, lba);
+
+		if (lba && nf) {
+			int rcwrt;
+			int count = 1;
+			__le16 md_crc;
+
+			set_bit(W_IN_FLIGHT, &pg->flags);
+
+			if (!snc) {
+				rcode = queue_pg(znd, pg, lba);
+				goto out_queued;
+			}
+
+			md_crc = crc_md_le16(pg->data.addr, Z_CRC_4K);
+			rcwrt = write_block(znd->ti, DM_IO_KMEM,
+					    pg->data.addr, lba, count, wq);
+			pg->age = jiffies_64 + msecs_to_jiffies(pg->hotness);
+			pg->last_write = lba;
+			pg_crc(znd, pg, md_crc);
+
+			if (rcwrt) {
+				Z_ERR(znd, "write_page: %" PRIx64 " -> %" PRIx64
+					   " ERR: %d", pg->lba, lba, rcwrt);
+				rcode = rcwrt;
+				goto out;
+			}
+
+			if (crc_md_le16(pg->data.addr, Z_CRC_4K) == md_crc)
+				clear_bit(IS_DIRTY, &pg->flags);
+			clear_bit(W_IN_FLIGHT, &pg->flags);
+			if (lba < znd->data_lba)
+				goto out;
+
+			rcwrt = z_mapped_addmany(znd, dm_s, lba, nf, NORMAL);
+			if (rcwrt) {
+				Z_ERR(znd, "%s: Journal MANY failed.",
+				      __func__);
+				rcode = rcwrt;
+				goto out;
+			}
+			increment_used_blks(znd, lba, nf);
+		} else {
+			Z_ERR(znd, "%s: Out of space for metadata?", __func__);
+			rcode = -ENOSPC;
+			goto out;
+		}
+	}
+
+out:
+	deref_pg(pg);
+
+out_queued:
+	if (rcode < 0)
+		znd->meta_result = rcode;
+
+	return rcode;
+}
+
+/**
+ * _sync_dirty() - Write all *dirty* ZLT blocks to disk (journal->SYNC->home)
+ * @znd: ZDM instance
+ * @bit_type: MAP blocks then CRC blocks.
+ * @sync: If true write dirty blocks to disk
+ * @drop: Number of ZLT blocks to free.
+ *
+ * Return: 0 on success or -errno value
+ */
+static int _sync_dirty(struct zoned *znd, int bit_type, int sync, int drop)
+{
+	int err = 0;
+	int entries = 0;
+	int want_flush = 0;
+	struct map_pg *expg = NULL;
+	struct map_pg *_tpg;
+	struct map_pg **wset = NULL;
+	int dlstsz = 0;
+	LIST_HEAD(droplist);
+
+	wset = ZDM_CALLOC(znd, sizeof(*wset), MAX_WSET, KM_19, NORMAL);
+
+	spin_lock(&znd->zlt_lck);
+	if (list_empty(&znd->zltpool))
+		goto writeback;
+
+	expg = list_last_entry(&znd->zltpool, typeof(*expg), zltlst);
+	if (!expg || &expg->zltlst == (&znd->zltpool))
+		goto writeback;
+
+	_tpg = list_prev_entry(expg, zltlst);
+	while (&expg->zltlst != &znd->zltpool) {
+		ref_pg(expg);
+
+		if (sync && (entries < MAX_WSET) && is_dirty(expg, bit_type)) {
+			wset[entries++] = expg;
+		} else if ((drop > 0) && is_old_and_clean(expg, bit_type)) {
+			struct map_pg **zlt;
+			spinlock_t *lock;
+			int index = expg->index;
+			int is_lut = test_bit(IS_LUT, &expg->flags);
+			int is_fwd = test_bit(IS_FWD, &expg->flags);
+			u32 count = is_lut ? znd->map_count : znd->crc_count;
+
+			if (is_lut) {
+				zlt = is_fwd ? znd->fwd_tm : znd->rev_tm;
+				lock = &znd->mapkey_lock;
+			} else {
+				zlt = is_fwd ? znd->fwd_crc : znd->rev_crc;
+				lock = &znd->ct_lock;
+			}
+			spin_lock(lock);
+			if (index > -1 && index < count &&
+			    getref_pg(expg) == 1 && expg == zlt[index]) {
+				zlt[expg->index] = NULL;
+				list_del(&expg->zltlst);
+				znd->in_zlt--;
+				clear_bit(IN_ZLT, &expg->flags);
+				drop--;
+				expg->age = jiffies_64;
+					  + msecs_to_jiffies(expg->hotness);
+
+#if ENABLE_PG_FREE_VIA_LAZY
+				if (test_bit(IS_LAZY, &expg->flags))
+					Z_ERR(znd, "** Pg is lazy && zlt %llx",
+						expg->lba);
+
+				if (!test_bit(IS_LAZY, &expg->flags)) {
+					list_add(&expg->lazy, &droplist);
+					znd->in_lzy++;
+					set_bit(IS_LAZY, &expg->flags);
+					set_bit(IS_DROPPED, &expg->flags);
+					dlstsz++;
+				}
+				zlt[expg->index] = expg; /* so undrop works */
+				deref_pg(expg);
+				if (getref_pg(expg) > 0)
+					Z_ERR(znd, "Moving elv ref: %u",
+					      getref_pg(expg));
+#else
+				MutexLock(&expg->md_lock);
+				if (expg->data.addr) {
+					void *pg = expg->data.addr;
+
+					expg->data.addr = NULL;
+					ZDM_FREE(znd, pg, Z_C4K, PG_27);
+					atomic_dec(&znd->incore);
+					if (!test_bit(IS_FLUSH, &expg->flags))
+						want_flush++;
+				}
+				mutex_unlock(&expg->md_lock);
+				deref_pg(expg);
+				if (getref_pg(expg) > 0)
+					Z_ERR(znd, "Dropped elv ref: %u",
+					      getref_pg(expg));
+				ZDM_FREE(znd, expg, sizeof(*expg), KM_20);
+#endif
+			}
+			spin_unlock(lock);
+		} else {
+			deref_pg(expg);
+		}
+		if (entries == MAX_WSET)
+			break;
+
+		expg = _tpg;
+		_tpg = list_prev_entry(expg, zltlst);
+	}
+
+writeback:
+	spin_unlock(&znd->zlt_lck);
+
+	if (entries > 0) {
+		err = _pool_write(znd, wset, entries);
+		if (err < 0)
+			goto out;
+		if (entries == MAX_WSET)
+			err = -EBUSY;
+	}
+
+out:
+	if (want_flush)
+		set_bit(DO_FLUSH, &znd->flags);
+	if (!list_empty(&droplist))
+		lazy_pool_splice(znd, &droplist);
+
+	if (wset)
+		ZDM_FREE(znd, wset, sizeof(*wset) * MAX_WSET, KM_19);
+
+	return err;
+}
+
+/**
+ * sync_dirty() - Write all *dirty* ZLT blocks to disk (journal->SYNC->home)
+ * @znd: ZDM instance
+ * @bit_type: MAP blocks then CRC blocks.
+ * @sync: Write dirty blocks
+ * @drop: IN: # of pages to free.
+ *
+ * Return: 0 on success or -errno value
+ */
+static int sync_dirty(struct zoned *znd, int bit_type, int sync, int drop)
+{
+	int err;
+
+	MutexLock(&znd->pool_mtx);
+	do {
+		err = _sync_dirty(znd, bit_type, sync, drop);
+		drop = 0;
+	} while (err == -EBUSY);
+
+	if (err > 0)
+		err = 0;
+	mutex_unlock(&znd->pool_mtx);
+
+	return err;
+}
+
+/**
+ * sync_mapped_pages() - Migrate lookup tables and crc pages to disk
+ * @znd: ZDM instance
+ * @sync: If dirty blocks need to be written.
+ * @drop: Number of blocks to drop.
+ *
+ * Return: 0 on success or -errno value
+ */
+static int sync_mapped_pages(struct zoned *znd, int sync, int drop)
+{
+	int err;
+	int remove = drop ? 1 : 0;
+
+	err = sync_dirty(znd, IS_LUT, sync, drop);
+
+	/* on error return */
+	if (err < 0)
+		return err;
+
+	/* TBD: purge CRC's on ref-count? */
+	err = sync_dirty(znd, IS_CRC, sync, remove);
+
+	return err;
+}
+
+/**
+ * dm_s is a logical sector that maps 1:1 to the whole disk in 4k blocks
+ * Here the logical LBA and field are calculated for the lookup table
+ * where the physical LBA can be read from disk.
+ */
+static int map_addr_aligned(struct zoned *znd, u64 dm_s, struct map_addr *out)
+{
+	u64 block	= dm_s >> 10;
+
+	out->zone_id	= block >> 6;
+	out->lut_s	= block + znd->s_base;
+	out->lut_r	= block + znd->r_base;
+	out->pg_idx	= dm_s & 0x3FF;
+
+	return 0;
+}
+
+/* -------------------------------------------------------------------------- */
+/* -------------------------------------------------------------------------- */
+
+/**
+ * map_addr_calc()
+ * @znd: ZDM instance
+ * @origin: address to calc
+ * @out: address, zone, crc, lut addr
+ *
+ * dm_s is a logical sector that maps 1:1 to the whole disk in 4k blocks
+ * Here the logical LBA and field are calculated for the lookup table
+ * where the physical LBA can be read from disk.
+ */
+static int map_addr_calc(struct zoned *znd, u64 origin, struct map_addr *out)
+{
+	u64 offset = ((origin < znd->md_end) ? znd->md_start : znd->md_end);
+
+	out->dm_s = origin;
+	return map_addr_aligned(znd, origin - offset, out);
+}
+
+/**
+ * read_pg() - Read a page of LUT/CRC from disk.
+ * @znd: ZDM instance
+ * @pg: Page to fill
+ * @gfp: Memory allocation rule
+ * @mpi: Backing page locations.
+ *
+ * Load a page of the sector lookup table that maps to pg->lba
+ * If pg->lba is not on disk return 0
+ *
+ * Return: 1 if page exists, 0 if unmodified, else -errno on error.
+ */
+static int read_pg(struct zoned *znd, struct map_pg *pg, u64 lba48, gfp_t gfp,
+		   struct mpinfo *mpi)
+{
+	int rcode = 0;
+	const int count = 1;
+	const int wq = 1;
+	int rd;
+	__le16 check;
+	__le16 expect = 0;
+
+	/**
+	 * This table entry may be on-disk, if so it needs to
+	 * be loaded.
+	 * If not it needs to be initialized to 0xFF
+	 *
+	 * When ! ZF_POOL_FWD and mapped block is FWD && LUT
+	 *   the mapped->lba may be pinned in sk_ pool
+	 *   if so the sk_ pool entry may need to be pulled
+	 *   to resvole the current address of mapped->lba
+	 */
+
+/* TODO: Async reads */
+
+	if (warn_bad_lba(znd, lba48))
+		Z_ERR(znd, "Bad PAGE %" PRIx64, lba48);
+
+	rd = read_block(znd->ti, DM_IO_KMEM, pg->data.addr, lba48, count, wq);
+	if (rd) {
+		Z_ERR(znd, "%s: read_block: ERROR: %d", __func__, rd);
+		rcode = -EIO;
+		goto out;
+	}
+
+	/*
+	 * Now check block crc
+	 */
+	check = crc_md_le16(pg->data.addr, Z_CRC_4K);
+	if (test_bit(IS_LUT, &pg->flags)) {
+		struct map_pg *crc_pg;
+		u64 base = znd->c_base;
+
+		if (test_bit(IS_REV, &pg->flags))
+			base = znd->c_mid;
+
+		crc_pg = get_map_entry(znd, mpi->crc.pg_no + base, gfp);
+		if (crc_pg) {
+			ref_pg(crc_pg);
+			if (crc_pg->data.crc) {
+				MutexLock(&crc_pg->md_lock);
+				expect = crc_pg->data.crc[mpi->crc.pg_idx];
+				mutex_unlock(&crc_pg->md_lock);
+				crc_pg->age = jiffies_64
+					    + msecs_to_jiffies(crc_pg->hotness);
+			}
+			deref_pg(crc_pg);
+		}
+		put_map_entry(crc_pg);
+	} else {
+		expect = znd->md_crcs[mpi->crc.pg_idx];
+	}
+
+	if (check != expect) {
+		znd->meta_result = -ENOSPC;
+
+		Z_ERR(znd,
+		      "Corrupt metadata: %" PRIx64 " from %" PRIx64
+		      " [%04x != %04x]",
+		      pg->lba, lba48,
+		      le16_to_cpu(check),
+		      le16_to_cpu(expect));
+	}
+	rcode = 1;
+
+out:
+	return rcode;
+}
+
+/**
+ * cache_pg() - Load a page of LUT/CRC into memory from disk, or default values.
+ * @znd: ZDM instance
+ * @pg: Page to fill
+ * @gfp: Memory allocation rule
+ * @mpi: Backing page locations.
+ *
+ * Return: 1 if page loaded from disk, 0 if empty, else -errno on error.
+ */
+static int cache_pg(struct zoned *znd, struct map_pg *pg, gfp_t gfp,
+		    struct mpinfo *mpi)
+{
+	int rc = 0;
+	int empty_val = test_bit(IS_LUT, &pg->flags) ? 0xff : 0;
+	u64 lba48 = current_mapping(znd, pg->lba, gfp);
+
+	ref_pg(pg);
+	MutexLock(&pg->md_lock);
+	if (!pg->data.addr) {
+		pg->data.addr = ZDM_ALLOC(znd, Z_C4K, PG_27, gfp);
+		if (pg->data.addr) {
+			if (lba48)
+				rc = read_pg(znd, pg, lba48, gfp, mpi);
+			else
+				memset(pg->data.addr, empty_val, Z_C4K);
+
+			if (rc < 0) {
+				Z_ERR(znd, "%s: read_pg from %" PRIx64
+					   " [to? %x] error: %d", __func__,
+				      pg->lba, empty_val, rc);
+				ZDM_FREE(znd, pg->data.addr, Z_C4K, PG_27);
+			}
+		} else {
+			Z_ERR(znd, "%s: Out of memory.", __func__);
+			rc = -ENOMEM;
+		}
+		if (pg->data.addr) {
+			set_bit(IS_FLUSH, &pg->flags);
+			atomic_inc(&znd->incore);
+			pg->age = jiffies_64 + msecs_to_jiffies(pg->hotness);
+			rc = pool_add(znd, pg);
+			Z_DBG(znd, "Page loaded: lba: %" PRIx64, pg->lba);
+		}
+	}
+	mutex_unlock(&pg->md_lock);
+
+	deref_pg(pg);
+	return rc;
+}
+
+/**
+ * z_lookup_table() - resolve a sector mapping via ZLT mapping
+ * @znd: ZDM Instance
+ * @addr: Address to resolve (via FWD map).
+ * @gfp: Current allocation flags.
+ */
+static u64 z_lookup_table(struct zoned *znd, u64 addr, gfp_t gfp)
+{
+	struct map_addr maddr;
+	struct map_pg *pg;
+	u64 old_phy = 0;
+
+	map_addr_calc(znd, addr, &maddr);
+	pg = get_map_entry(znd, maddr.lut_s, gfp);
+	if (pg) {
+		ref_pg(pg);
+		if (pg->data.addr) {
+			__le32 delta;
+
+			MutexLock(&pg->md_lock);
+			delta = pg->data.addr[maddr.pg_idx];
+			mutex_unlock(&pg->md_lock);
+			old_phy = map_value(znd, delta);
+			pg->age = jiffies_64 + msecs_to_jiffies(pg->hotness);
+			clear_bit(IS_READA, &pg->flags);
+		}
+		deref_pg(pg);
+		put_map_entry(pg);
+	}
+	return old_phy;
+}
+
+/**
+ * update_map_entry() - Migrate memcache to lookup table map entries.
+ * @znd: ZDM instance
+ * @mapped: memcache block.
+ * @maddr: map_addr
+ * @to_addr: LBA or sector #.
+ * @is_fwd: flag forward or reverse lookup table.
+ *
+ * when is_fwd is 0:
+ *  - maddr->dm_s is a sector -> lba.
+ *	 in this case the old lba is discarded and scheduled for cleanup
+ *	 by updating the reverse map lba tables noting that this location
+ *	 is now unused.
+ * when is_fwd is 0:
+ *  - maddr->dm_s is an lba, lba -> dm_s
+ *
+ * Return: non-zero on error.
+ */
+static int update_map_entry(struct zoned *znd, struct map_pg *pg,
+			    struct map_addr *maddr, u64 to_addr, int is_fwd)
+{
+	int err = -ENOMEM;
+
+	if (pg && pg->data.addr) {
+		u64 index = maddr->pg_idx;
+		__le32 delta;
+		__le32 value;
+		int was_updated = 0;
+
+		ref_pg(pg);
+		MutexLock(&pg->md_lock);
+		delta = pg->data.addr[index];
+		err = map_encode(znd, to_addr, &value);
+		if (!err) {
+			/*
+			 * if the value is modified update the table and
+			 * place it on top of the active [zltlst] list
+			 * this will keep the chunk of lookup table in
+			 * memory.
+			 */
+			if (pg->data.addr[index] != value) {
+				pg->data.addr[index] = value;
+				pg->age = jiffies_64
+					    + msecs_to_jiffies(pg->hotness);
+				set_bit(IS_DIRTY, &pg->flags);
+				clear_bit(IS_FLUSH, &pg->flags);
+				clear_bit(IS_READA, &pg->flags);
+				was_updated = 1;
+			} else if (value != MZTEV_UNUSED) {
+				Z_ERR(znd, "*ERR* %" PRIx64
+					   " -> data.addr[index] (%x) == (%x)",
+				      maddr->dm_s, pg->data.addr[index],
+				      value);
+				dump_stack();
+			}
+		} else {
+			Z_ERR(znd, "*ERR* Mapping: %" PRIx64 " to %" PRIx64,
+			      to_addr, maddr->dm_s);
+		}
+		mutex_unlock(&pg->md_lock);
+
+		if (was_updated && is_fwd && (delta != MZTEV_UNUSED)) {
+			u64 old_phy = map_value(znd, delta);
+
+			/*
+			 * add to discard list of the controlling mzone
+			 * for the 'delta' physical block
+			 */
+			Z_DBG(znd, "%s: unused_phy: %" PRIu64
+			      " (new lba: %" PRIu64 ")",
+			      __func__, old_phy, to_addr);
+
+			WARN_ON(old_phy >= znd->nr_blocks);
+
+			err = unused_phy(znd, old_phy, 0, NORMAL);
+			if (err)
+				err = -ENOSPC;
+		}
+		deref_pg(pg);
+	}
+	return err;
+}
+
+/**
+ * move_to_map_tables() - Migrate memcache to lookup table map entries.
+ * @znd: ZDM instance
+ * @mcache: memcache block.
+ *
+ * Return: non-zero on error.
+ */
+static int move_to_map_tables(struct zoned *znd, struct map_cache *mcache)
+{
+	struct map_pg *smtbl = NULL;
+	struct map_pg *rmtbl = NULL;
+	struct map_addr maddr = { .dm_s = 0ul };
+	struct map_addr rev = { .dm_s = 0ul };
+	u64 lut_s = BAD_ADDR;
+	u64 lut_r = BAD_ADDR;
+	int jentry;
+	int err = 0;
+	int is_fwd = 1;
+
+	/* the journal being move must remain stable so sorting
+	 * is disabled. If a sort is desired due to an unsorted
+	 * page the search devolves to a linear lookup.
+	 */
+	mcache_busy(mcache);
+	for (jentry = mcache->jcount; jentry > 0;) {
+		u64 dm_s = le64_to_lba48(mcache->jdata[jentry].tlba, NULL);
+		u64 lba = le64_to_lba48(mcache->jdata[jentry].bval, NULL);
+
+		if (dm_s == Z_LOWER48 || lba == Z_LOWER48) {
+			mcache->jcount = --jentry;
+			continue;
+		}
+
+		map_addr_calc(znd, dm_s, &maddr);
+		if (lut_s != maddr.lut_s) {
+			put_map_entry(smtbl);
+			if (smtbl)
+				deref_pg(smtbl);
+			smtbl = get_map_entry(znd, maddr.lut_s, NORMAL);
+			if (!smtbl) {
+				err = -ENOMEM;
+				goto out;
+			}
+			ref_pg(smtbl);
+			lut_s = smtbl->lba;
+		}
+
+		is_fwd = 1;
+		if (lba == 0ul)
+			lba = BAD_ADDR;
+		err = update_map_entry(znd, smtbl, &maddr, lba, is_fwd);
+		if (err < 0)
+			goto out;
+
+		/*
+		 * In this case the reverse was handled as part of
+		 * discarding the forward map entry -- if it was in use.
+		 */
+		if (lba != BAD_ADDR) {
+			map_addr_calc(znd, lba, &rev);
+			if (lut_r != rev.lut_r) {
+				put_map_entry(rmtbl);
+				if (rmtbl)
+					deref_pg(rmtbl);
+				rmtbl = get_map_entry(znd, rev.lut_r, NORMAL);
+				if (!rmtbl) {
+					err = -ENOMEM;
+					goto out;
+				}
+				ref_pg(rmtbl);
+				lut_r = rmtbl->lba;
+			}
+			is_fwd = 0;
+			err = update_map_entry(znd, rmtbl, &rev, dm_s, is_fwd);
+			if (err == 1)
+				err = 0;
+		}
+
+		if (err < 0)
+			goto out;
+
+		mcache->jdata[jentry].tlba = MC_INVALID;
+		mcache->jdata[jentry].bval = MC_INVALID;
+		if (mcache->jsorted == mcache->jcount)
+			mcache->jsorted--;
+		mcache->jcount = --jentry;
+	}
+out:
+	if (smtbl)
+		deref_pg(smtbl);
+	if (rmtbl)
+		deref_pg(rmtbl);
+	put_map_entry(smtbl);
+	put_map_entry(rmtbl);
+	set_bit(DO_MEMPOOL, &znd->flags);
+	mcache_unbusy(mcache);
+
+	return err;
+}
+
+/**
+ * unused_phy() - Mark a block as unused.
+ * @znd: ZDM instance
+ * @lba: Logical LBA of block.
+ * @orig_s: Sector being marked.
+ * @gfp: Memory allocation rule
+ *
+ * Return: non-zero on error.
+ *
+ * Add an unused block to the list of blocks to be discarded during
+ * garbage collection.
+ */
+static int unused_phy(struct zoned *znd, u64 lba, u64 orig_s, gfp_t gfp)
+{
+	int err = 0;
+	struct map_pg *pg;
+	struct map_addr reverse;
+	int z_off;
+
+	if (lba < znd->data_lba)
+		return 0;
+
+	map_addr_calc(znd, lba, &reverse);
+	z_off = reverse.zone_id % 1024;
+	pg = get_map_entry(znd, reverse.lut_r, gfp);
+	if (!pg) {
+		err = -EIO;
+		Z_ERR(znd, "unused_phy: Reverse Map Entry not found.");
+		goto out;
+	}
+
+	if (!pg->data.addr) {
+		Z_ERR(znd, "Catastrophic missing LUT page.");
+		dump_stack();
+		err = -EIO;
+		goto out;
+	}
+	ref_pg(pg);
+
+	/*
+	 * if the value is modified update the table and
+	 * place it on top of the active [zltlst] list
+	 */
+	if (pg->data.addr[reverse.pg_idx] != MZTEV_UNUSED) {
+		u32 gzno  = reverse.zone_id >> GZ_BITS;
+		u32 gzoff = reverse.zone_id & GZ_MMSK;
+		struct meta_pg *wpg = &znd->wp[gzno];
+		u32 wp;
+		u32 zf;
+		u32 stream_id;
+
+		if (orig_s) {
+			__le32 enc = pg->data.addr[reverse.pg_idx];
+			u64 dm_s = map_value(znd, enc);
+			int drop_discard = 0;
+
+			if (dm_s < znd->data_lba) {
+				drop_discard = 1;
+				Z_ERR(znd, "Discard invalid target %"
+				      PRIx64" - Is ZDM Meta %"PRIx64" vs %"
+				      PRIx64, lba, orig_s, dm_s);
+			}
+			if (orig_s != dm_s) {
+				drop_discard = 1;
+				Z_ERR(znd,
+				      "Discard %" PRIx64
+				      " mismatched src: %"PRIx64 " vs %" PRIx64,
+				      lba, orig_s, dm_s);
+			}
+			if (drop_discard)
+				goto out_unlock;
+		}
+		MutexLock(&pg->md_lock);
+		pg->data.addr[reverse.pg_idx] = MZTEV_UNUSED;
+		mutex_unlock(&pg->md_lock);
+		pg->age = jiffies_64 + msecs_to_jiffies(pg->hotness);
+		set_bit(IS_DIRTY, &pg->flags);
+		clear_bit(IS_READA, &pg->flags);
+		clear_bit(IS_FLUSH, &pg->flags);
+
+		SpinLock(&wpg->wplck);
+		wp = le32_to_cpu(wpg->wp_alloc[gzoff]);
+		zf = le32_to_cpu(wpg->zf_est[gzoff]) & Z_WP_VALUE_MASK;
+		stream_id = le32_to_cpu(wpg->zf_est[gzoff]) & Z_WP_STREAM_MASK;
+		if (wp > 0 && zf < Z_BLKSZ) {
+			zf++;
+			wpg->zf_est[gzoff] = cpu_to_le32(zf | stream_id);
+			wpg->wp_alloc[gzoff] = cpu_to_le32(wp | Z_WP_RRECALC);
+			set_bit(IS_DIRTY, &wpg->flags);
+			clear_bit(IS_FLUSH, &wpg->flags);
+		}
+		spin_unlock(&wpg->wplck);
+		if ((wp & Z_WP_VALUE_MASK) == Z_BLKSZ)
+			znd->discard_count++;
+	} else {
+		Z_DBG(znd, "lba: %" PRIx64 " alread reported as free?", lba);
+	}
+
+out_unlock:
+	deref_pg(pg);
+
+out:
+	put_map_entry(pg);
+
+	return err;
+}
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Filesystems]     [Linux SCSI]     [Linux RAID]     [Git]     [Kernel Newbies]     [Linux Newbie]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Samba]     [Device Mapper]

  Powered by Linux