Hi sage. Thanks for your comments! I created pads in order to brainstorm design option about #1, #2 first. #1 http://pad.ceph.com/p/deduplication_how_dedup_manifists #2 http://pad.ceph.com/p/deduplication_how_do_we_store_chunk Thanks. 2017-03-16 22:42 GMT+09:00 Sage Weil <sweil@xxxxxxxxxx>: > Hi Myoungwon, > > This is quite a patch! Sorry for the slow reply. > > On Tue, 14 Mar 2017, myoungwon oh wrote: >> Hi Sage >> >> >> I addressed all of your concerns (I applied CAS pool and dedup >> metadata in object_info_t) and created public repository in order to >> show the prototype implementation >> (https://github.com/myoungwon/ceph/commit/13597f62405d1c5a4977d630e69331407ef3a07a, >> support non-aligned I/O, but for (K)RBD). This code is based on Jewel >> and is not cleaned well but you can see the basic flow (start_flush(), >> maybe_handle_cache_detail() ). It would be nice if you give me some >> comments. >> >> I have some queries mentioned below on which your feedback is highly required. >> >> 1. dedup metadata in object_info_t >> >> You mentioned that it would be nice to make tuple in object_info_t >> such as map<offset, tuple<length, cas object, pool>> But, I made >> dedup_chunk_info_t in object_info_t because I need one more parameter >> (chunk_state) and for extensibility. > > Yes, we definitely want an extensible approach to the state in > object_info_t that will support > > - a simple redirect ("the object is in that other pool") > - a dedup object ("the object consists of these N lumps, each one > referencing an object named X_i in pool Y_i") > - an external system (extenral archive, like a backup system, external > object store, whatever) > > I think we should try to come up with a general notion, like "redirect" or > "object map" or something that covers other options... not just dedup! > >> This is because to avoid read and >> fingerprinting during flush time. chunk_state represents three states >> in writeback mode. First is CLEAN (data and fingerprint are not >> modified). Second is MODIFIED (data is modified but fingerprint is not >> calculated). Third is CALCULATED (data is modified and fingerprint is >> also calculated). When data is stored in cache tier, chunk_state will >> be defined. Therefore, reading data and fingerprinting can be removed >> during flush. > > I'm not following this, though. I think "clean" would just mean we are > storing the normal object in the pool. "modified" would mean that the > FLAG_DIRTY is set. And "calculated" would mean we have successfully > chunked the object, stored or taken refs on the chunks, and written the > chunk map into object_info_t? > > >> 2. Single Rados Operation >> >> You mentioned a Rados operation which can concurrently read the >> reference count and write data. Do you want that API in objecter >> class? (for example, objector->read_ref_and_write()) > > We may not need to make it a first-class rados operation. For example, > cls_refcount could probably be extended with a write_or_get operation. > But it might also be advantageous to make it a native op. The main thing > I'm worried about here is that we probably want to make the refs > reliable and autitable, which means backpointers (so you can look at a > chunk and see which dedup objects are using it). That means that a > popular sequence of bytes might have a huge number of references, and that > will need to scale gracefully. Or, we just use counters, accept that > failure conditions could make us leak dedup chunks, and make all of our > failure paths fail-safe. > >> 3. Write sequence for performance. >> >> Current write sequence (proxy mode) is >> >> a. Read metadata (promote_object) >> b. Send data to OSD (in CAS pool) and send dedup metadata to OSD (in >> original pool) >> c. If data and metadata are stored then, proxy osd will issue message >> to decrease the reference count (for previous chunk) to OSD (in CAS >> pool) and update local object metadata (via simple_opc_submit) >> d. If reference count is successful, send Ack to client >> >> As you can see, the number of operations increased due to reference >> count and metadata updates. This can degrade performance. My question >> is that can we send ack to client at (c) above? (But I am worried >> about inconsistent reference count state.) > > I'm worried that if we focus on inline dedup immediately we'll end up with > something that is less general and more fragile. It's also harder. > Instead, we can consider the inline and async dedup separately. Async: > > writeback: > a. normal write into object. ack client. > ... > b. dedup agent: read object (from cache), chunk > c. dedup agent: write/refcount chunks > d. replace object with dedup manifest > > This could happen with or without a delay. I don't think it makes sense > to consider "promote" here at all; it sounds like you're assuming the > initial dedup tier is a cache tier, and we should try not to assume that > (even though it might be possible). Instead, I think a "basic" setup > would probably be > > 1. base pool (all ssd; contains all metadata for all objects, and absorbs > writes). > 2. dedup pool(s) contain refcounted chunks > > If we want to do inline dedup, it would be some complex code that combines > all of the steps above into one, at the expense of client latency. > > > In any case, it's awesome that you have a working prototype. However, > it's not going to be practical to take a huge patch(set) like this and > merge it all at once. It's too much code to review, too complex, and too > hard to test. Also, it's changing 5000 in ReplicatedPG.cc (since renamed > PrimaryLogPG.cc), which is slated for a big refactor right after luminous. > > The way to approach this to get it upstream is to break this down into > different logical components and design/review/test/merge each of them > indepdendently. Having a prototype is useful in that it will be easier to > answer a lot of the questions we'll have deciding how each part should > work and what it needs to be able to handle, but don't expect that most > of that code will end up in the final version! > > I'm guessing we can break this down into a few logical components: > > 1) How do we store chunks. We know we want refcounted objects for each > chunk. We don't know how we'll manage the refcounts, whether we want/need > backpointers, whether we are willing to tolerate "leaking" references in > failure cases (so that we fail to clean up all chunks if we e.g. delete > all data), whether we want to implement it as a rados class or a native > rados op, whether we want to support EC, compression, etc. This whole > discussion one is a great place to start because it is self-contained and > doesn't break anything else. > > 2) How do we do the dedup manifists (and redirects) in object_info_t. We > want the solution to include or be compatible with simpler tiering, like > having the object_info_t simply be a pointer to a different (colder) pool. > In fact, I think this is the thing to do first becuase it will make us > fix/solve all the basic problems with flush and promote. And extending > this to include dedup (object is composed of many little bits in other > pools) is then a matter of making that 'manifest' (or whatever we call it) > a generic and extensible description. Remember we also want to support > pushing objects into external systems (say, glacier, or some other > external object store like a backup system). > > 3) How do we chunk. You have some classes that handle aligned chunking. > We'll probably eventually want content-based chunking (based on Rabin > fingerprinting or whatever the new hotness is). Real users will probably > want adjustable policies based on what they know of the content they're > storing, and the system will probably want to support multiple CAS pools > based on which policy is being used (as that determines chunk sizes > etc and whether we'll actually have any dedup happening). > > 4) How to drive the dedup process itself. An async agent that's part of > the exiting tier_agent? An external process? Something inline in the > write path? This is the hardest question to answer, and the one that is > most likely to collide with other planned OSD work. It can also come > last, IMO! We can start with a simple offline agent and perhaps > eventually do something more clever or efficient. > > In any case, I think #1 and #2 are the key discussions we should have now. > I suggest starting a pad and email thread for each (pad.ceph.com) so we > can brainstorm design options, weight trade-offs, and come to some > consensus. (I had some thoughts, for example, on a hybrid scheme > somewhere between explicit backpointers and a simple refcount that could > consume fixed overhead but still provide information that would enable a > moderately efficient scrub/audit.) > > Thanks! > sage > > > > >> Write sequence (writeback mode) is >> >> a. Read object data and do fingerprinting (if data is not calculated). >> b. Send reference count decrement message (for previous chunk) to osd >> (in CAS pool) and updates local object metadata >> c. Send copy_from message to osd (in CAS pool) and send copy_from >> message (in order to copy the dedup metadata) to a osd (in original >> pool) >> >> Writeback mode also increase the number of operation. Can we reduce? >> >> >> >> 4. Performance. >> >> Performance is improved compared to previous results. But It still >> seems to be improving. (512KB block, Seq. workload, fio, KRBD, single >> thread, target_max_objects = 4) >> >> Major concerns are first is fingerprint overhead and second is >> writeback performance in cache tier. When the chunk size is large >> (>512KB), SHA1 takes more than 3ms. (This can be reduced if we use >> small chunk.) >> >> Regarding writeback performance, Flush need two more operations than >> proxy mode. First is "marking clean state". Second is "reading dedup >> metadata and data from storage". Therefore, actual read and write >> occur. These cause that flush completion is delayed. >> >> Small chunk performance in the writeback mode is significantly >> degraded because single flush thread handles multiple copy_from >> message. It seems that we should improve basic flushing performance. >> >> >> Write performance (MB/s) >> >> Dedup ratio 0 60 100 >> >> Proxy 55 64 73 >> >> Writeback 48 50 50 >> >> Original 120 120 122 >> >> >> >> Read performance (MB/s) >> >> Dedup ratio 0 60 100 >> >> Proxy 117 130 141 >> >> Writeback 198 197 200 >> >> Original 280 276 285 >> >> >> >> >> 5. Command to enable dedup >> >> Ceph osd pool create sds-hot 1024 >> Ceph osd pool create sds-cas 1024 >> Ceph osd tier add_cas rbd sds-hot sds-cas >> Ceph osd tier sds-hot (proxy or writeback) >> Ceph osd tier dedup_block rbd sds-hot sds-cas (chunk size. e.g. 65536, 131072..) >> Ceph osd tier set-overlay rbd sds-hot >> >> >> >> Thanks >> Myoungwon Oh >> (omwmw@xxxxxx) >> >> 2017-02-07 23:50 GMT+09:00 Sage Weil <sweil@xxxxxxxxxx>: >> > On Tue, 7 Feb 2017, myoungwon oh wrote: >> >> Hi sage. >> >> >> >> I uploaded the document which describe my overall appoach. >> >> please see it and give me feedback. >> >> slide: https://www.slideshare.net/secret/JZcy3yYEDIHPyg >> > >> > This approach looks pretty close to what we have been planning. A few >> > comments: >> > >> > 1) I think it may be better to view the tier/pool that has the object >> > metadata as the "base" pool, and the CAS pool with the refcounted >> > object chunks as as tier below that. >> > >> > 2) I think we can use an object class or a handful of new native rados >> > operations to make the CAS pool read/write operations more efficient. In >> > your slides you describe a process something like >> > >> > rados(getattr) >> > if exists >> > rados(increment ref count) >> > else >> > rados(write object and set ref count to 1) >> > >> > This could be collapsed into a single optimistic operation that sends the >> > data and a command that says "create or increment ref count" so that the >> > conditional behavior is handled at the OSD. This will be more efficient >> > for small chunks. (For large chunks, or in cases where we have some >> > confidence that the chunk probably already exists, the pessimistic >> > approach might still make sense.) Either way, we should probably support >> > both. >> > >> > 3) We'd like to generalize the first pool behavior so that it is just a >> > special case of the new tiering functionality. The idea is that an >> > object_info_t can have a 'manifest' that described where and how the >> > object is really stored instead of the object data itself (much like it >> > can already be a whiteout, etc.). In the simplest case, the manifest >> > would just say "this object is stored in pool X" (simple tiering). In >> > this case, the manifest would a structure like >> > >> > map<offset, tuple<length, cas object, pool>> >> > >> > I think it'll be worth the effort to build a general struture here that we >> > can use for basic tiering (not just dedup). >> > >> > sage >> > >> > >> > >> >> >> >> thanks >> >> >> >> >> >> 2017-01-31 23:24 GMT+09:00 Sage Weil <sage@xxxxxxxxxxxx>: >> >> > On Thu, 26 Jan 2017, myoungwon oh wrote: >> >> >> I have two questions. >> >> >> >> >> >> 1. I would like to ask about CAS location. current our implementation store >> >> >> content address object in storage tier.However, If we store the CAO in the >> >> >> cache tier, we can get a performance advantage. Do you think we can create >> >> >> CAO in cachetier? or create a separate storage pool for CAS? >> >> > >> >> > It depends on the design. If the you are naming the objects at the >> >> > librados client side, then you can use the rados cluster itself >> >> > unmodified (with or without a cache tier). This is roughly how I have >> >> > anticipated implementing the CAS storage portion. If you are doing the >> >> > chunking hashing and within the OSD itself, then you can't do the CAS >> >> > at the first tier because the requests won't be directed at the right OSD. >> >> > >> >> >> 2. The results below are performance result for our current implementation. >> >> >> experiment setup: >> >> >> PROXY (inline dedup), WRITEBACK (lazy dedup, target_max_bytes: 50MB), >> >> >> ORIGINAL(without dedup feature and cache tier), >> >> >> fio, 512K block, seq. I/O, single thread >> >> >> >> >> >> One thing to note is that the writeback case is slower than the proxy. >> >> >> We think there are three problems as follows. >> >> >> >> >> >> A. The current implementation creates a fingerprint by reading the entire >> >> >> object when flushing. Therefore, there is a problem that read and write are >> >> >> mixed. >> >> > >> >> > I expect this is a small factor compared to the fact that in writeback >> >> > mode you have to *write* to the cache tier, which is 3x replicated, >> >> > whereas in proxy mode those writes don't happen at all. >> >> > >> >> >> B. When client request read, the promote_object function reads the object >> >> >> and writes it back to the cache tier, which also causes a mix of read and >> >> >> write. >> >> > >> >> > This can be mitigated by setting the min_read_recency_for_promote pool >> >> > property to something >1. Then reads will be proxied unless the object >> >> > appears to be hot (because it has been touched over multiple >> >> > hitset intervals). >> >> > >> >> >> C. When flushing, the unchanged part is rewritten because flush operation >> >> >> perform per-object based. >> >> > >> >> > Yes. >> >> > >> >> > Is there a description of your overall approach somewhere? >> >> > >> >> > sage >> >> > >> >> > >> >> >> >> >> >> Do I have something wrong? or Could you give me a suggestion to improve >> >> >> performance? >> >> >> >> >> >> >> >> >> a. Write performance (KB/s) >> >> >> >> >> >> dedup_ratio 0 20 40 60 80 100 >> >> >> >> >> >> PROXY 45586 47804 51120 52844 56167 55302 >> >> >> >> >> >> WRITEBACK 13151 11078 9531 13010 9518 8319 >> >> >> >> >> >> ORIGINAL 121209 124786 122140 121195 122540 132363 >> >> >> >> >> >> >> >> >> b. Read performance (KB/s) >> >> >> >> >> >> dedup_ratio 0 20 40 60 80 100 >> >> >> >> >> >> PROXY 112231 118994 118070 120071 117884 132748 >> >> >> >> >> >> WRITEBACK 34040 29109 19104 26677 24756 21695 >> >> >> >> >> >> ORIGINAL 285482 284398 278063 277989 271793 285094 >> >> >> >> >> >> >> >> >> thanks, >> >> >> Myoungwon Oh >> >> >> -- >> >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> >> >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> >> >> >> >> >> >> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
