Hi everyone, We talked a bit about the proposed "KeyFile" backend a couple months back. I've started putting together a basic implementation and wanted to give people and update about what things are currently looking like. We're calling it NewStore for now unless/until someone comes up with a better name (KeyFileStore is way too confusing). (*) You can peruse the incomplete code at https://github.com/liewegas/ceph/tree/wip-newstore/src/os/newstore This is a bit of a brain dump. Please ask questions if anything isn't clear. Also keep in mind I'm still at the stage where I'm trying to get it into a semi-working state as quickly as possible so the implementation is pretty rough. Basic design: We use a KeyValueDB (leveldb, rocksdb, ...) for all of our metadata. Object data is stored in files with simple names (%d) in a simple directory structure (one level deep, default 1M files per dir). The main piece of metadata we store is a mapping from object name (ghobject_t) to onode_t, which looks like this: struct onode_t { uint64_t size; ///< object size map<string, bufferptr> attrs; ///< attrs map<uint64_t, fragment_t> data_map; ///< data (offset to fragment mapping) i.e., it's what we used to rely on xattrs on the inode for. Here, we'll only lean on the file system for file data and it's block management. fragment_t looks like struct fragment_t { uint32_t offset; ///< offset in file to first byte of this fragment uint32_t length; ///< length of fragment/extent fid_t fid; ///< file backing this fragment and fid_t is struct fid_t { uint32_t fset, fno; // identify the file name: fragments/%d/%d To start we'll keep the mapping pretty simple (just one fragment_t) but later we can go for varying degrees of complexity. We lean on the kvdb for our transactions. If we are creating new objects, we write data into a new file/fid, [aio_]fsync, and then commit the transaction. If we are doing an overwrite, we include a write-ahead log (wal) item in our transaction, and then apply it afterwards. For example, a 4k overwrite would make whatever metadata changes are included, and a wal item that says "then overwrite this 4k in this fid with this data". i.e., the worst case is more or less what FileStore is doing now with its journal, except here we're using the kvdb (and its journal) for that. On restart we can queue up and apply any unapplied wal items. An alternative approach here that we discussed a bit yesterday would be to write the small overwrites into the kvdb adjacent to the onode. Actually writing them back to the file could be deferred until later, maybe when there are many small writes to be done together. But right now the write behavior is very simple, and handles just 3 cases: https://github.com/liewegas/ceph/blob/wip-newstore/src/os/newstore/NewStore.cc#L1339 1. New object: create a new file and write there. 2. Append: append to an existing fid. We store the size in the onode so we can be a bit sloppy and in the failure case (where we write some extra data to the file but don't commit the onode) just ignore any trailing file data. 3. Anything else: generate a WAL item. 4. Maybe later, for some small [over]writes, we instead put the new data next to the onode. There is no omap yet. I think we should do basically what DBObjectMap did (with a layer of indirection to allow clone etc), but we need to rejigger it so that the initial pointer into that structure is embedded in the onode. We may want to do some other optimization to avoid extra indirection in the common case. Leaving this for later, though... We are designing for the case where the workload is already sharded across collections. Each collection gets an in-memory Collection, which has its own RWLock and its own onode_map (SharedLRU cache). A split will basically amount to registering the new collection in the kvdb and clearing the in-memory onode cache. There is a TransContext structure that is used to track the progress of a transaction. It'll list which fd's need to get synced pre-commit, which onodes need to get written back in the transaction, and any WAL items to include and queue up after the transaction commits. Right now the queue_transaction path does most of the work synchronously just to get things working. Looking ahead I think what it needs to do is: - assemble the transaction - start any aio writes (we could use O_DIRECT here if the new hints include WONTNEED?) - start any aio fsync's - queue kvdb transaction - fire onreadable[_sync] notifications (I suspect we'll want to do this unconditionally; maybe we avoid using them entirely?) On transaction commit, - fire commit notifications - queue WAL operations to a finisher The WAL ops will be linked to the TransContext so that if you want to do a read on the onode you can block until it completes. If we keep the (currently simple) locking then we can use the Collection rwlock to block new writes while we want for previous ones to apply. Or we can get more granular with the read vs write locks, but I'm not sure it'll be any use until we make major changes in the OSD (like dispatching parallel reads within a PG). Clone is annoying; if the FS doesn't support it natively (anything not btrfs) I think we should just do a sync read and then write for simplicity. A few other thoughts: - For a fast kvdb, we may want to do the transaction commit synchronously. For disk backends I think we'll want it async, though, to avoid blocking the caller. - The fid_t has a inode number stashed in it. The idea is to use open_by_handle to avoid traversing the (shallow) directory and go straight to the inode. On XFS this means we traverse the inode btree to verify it is in fast a valid ino, which isn't totally ideal but probably what we have to live with. Note that open_by_handle will work on any other (NFS-exportable) filesystem as well so this is in no way XFS-specific. This is implemented yet, but when we do, we'll probably want to verify we got the right file by putting some id in an xattr; that way you could safely copy the whole thing to another filesystem and it could gracefully fall back to opening using the file names. - I think we could build a variation on this implementation on top of an NVMe device instead of a file system. It could pretty trivially lay out writes in the address space as a linear sweep across the virutal address space. If the NVMe address space is big enough, maybe we could even avoid thinking about reusing addresses for deleted object? We'd just send a discard and then forget about it. Not sure if the address space is really that big, though... If not, we'd need to do make a simple allocator (blah). sage * This follows in the Messenger's naming footsteps, which went like this: MPIMessenger, NewMessenger, NewerMessenger, SimpleMessenger (which ended up being anything but simple). -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
