I added comments in the pad. I will make pads in order to discuss #3 and #4 if you agree with #1, #2. thanks. 2017-03-25 4:32 GMT+09:00 Sage Weil <sweil@xxxxxxxxxx>: > On Mon, 20 Mar 2017, myoungwon oh wrote: >> Hi sage. >> >> Thanks for your comments! >> I created pads in order to brainstorm design option about #1, #2 first. >> >> #1 http://pad.ceph.com/p/deduplication_how_dedup_manifists >> #2 http://pad.ceph.com/p/deduplication_how_do_we_store_chunk > > I made some comments in the pad! > > sage > >> >> >> Thanks. >> >> 2017-03-16 22:42 GMT+09:00 Sage Weil <sweil@xxxxxxxxxx>: >> > Hi Myoungwon, >> > >> > This is quite a patch! Sorry for the slow reply. >> > >> > On Tue, 14 Mar 2017, myoungwon oh wrote: >> >> Hi Sage >> >> >> >> >> >> I addressed all of your concerns (I applied CAS pool and dedup >> >> metadata in object_info_t) and created public repository in order to >> >> show the prototype implementation >> >> (https://github.com/myoungwon/ceph/commit/13597f62405d1c5a4977d630e69331407ef3a07a, >> >> support non-aligned I/O, but for (K)RBD). This code is based on Jewel >> >> and is not cleaned well but you can see the basic flow (start_flush(), >> >> maybe_handle_cache_detail() ). It would be nice if you give me some >> >> comments. >> >> >> >> I have some queries mentioned below on which your feedback is highly required. >> >> >> >> 1. dedup metadata in object_info_t >> >> >> >> You mentioned that it would be nice to make tuple in object_info_t >> >> such as map<offset, tuple<length, cas object, pool>> But, I made >> >> dedup_chunk_info_t in object_info_t because I need one more parameter >> >> (chunk_state) and for extensibility. >> > >> > Yes, we definitely want an extensible approach to the state in >> > object_info_t that will support >> > >> > - a simple redirect ("the object is in that other pool") >> > - a dedup object ("the object consists of these N lumps, each one >> > referencing an object named X_i in pool Y_i") >> > - an external system (extenral archive, like a backup system, external >> > object store, whatever) >> > >> > I think we should try to come up with a general notion, like "redirect" or >> > "object map" or something that covers other options... not just dedup! >> > >> >> This is because to avoid read and >> >> fingerprinting during flush time. chunk_state represents three states >> >> in writeback mode. First is CLEAN (data and fingerprint are not >> >> modified). Second is MODIFIED (data is modified but fingerprint is not >> >> calculated). Third is CALCULATED (data is modified and fingerprint is >> >> also calculated). When data is stored in cache tier, chunk_state will >> >> be defined. Therefore, reading data and fingerprinting can be removed >> >> during flush. >> > >> > I'm not following this, though. I think "clean" would just mean we are >> > storing the normal object in the pool. "modified" would mean that the >> > FLAG_DIRTY is set. And "calculated" would mean we have successfully >> > chunked the object, stored or taken refs on the chunks, and written the >> > chunk map into object_info_t? >> > >> > >> >> 2. Single Rados Operation >> >> >> >> You mentioned a Rados operation which can concurrently read the >> >> reference count and write data. Do you want that API in objecter >> >> class? (for example, objector->read_ref_and_write()) >> > >> > We may not need to make it a first-class rados operation. For example, >> > cls_refcount could probably be extended with a write_or_get operation. >> > But it might also be advantageous to make it a native op. The main thing >> > I'm worried about here is that we probably want to make the refs >> > reliable and autitable, which means backpointers (so you can look at a >> > chunk and see which dedup objects are using it). That means that a >> > popular sequence of bytes might have a huge number of references, and that >> > will need to scale gracefully. Or, we just use counters, accept that >> > failure conditions could make us leak dedup chunks, and make all of our >> > failure paths fail-safe. >> > >> >> 3. Write sequence for performance. >> >> >> >> Current write sequence (proxy mode) is >> >> >> >> a. Read metadata (promote_object) >> >> b. Send data to OSD (in CAS pool) and send dedup metadata to OSD (in >> >> original pool) >> >> c. If data and metadata are stored then, proxy osd will issue message >> >> to decrease the reference count (for previous chunk) to OSD (in CAS >> >> pool) and update local object metadata (via simple_opc_submit) >> >> d. If reference count is successful, send Ack to client >> >> >> >> As you can see, the number of operations increased due to reference >> >> count and metadata updates. This can degrade performance. My question >> >> is that can we send ack to client at (c) above? (But I am worried >> >> about inconsistent reference count state.) >> > >> > I'm worried that if we focus on inline dedup immediately we'll end up with >> > something that is less general and more fragile. It's also harder. >> > Instead, we can consider the inline and async dedup separately. Async: >> > >> > writeback: >> > a. normal write into object. ack client. >> > ... >> > b. dedup agent: read object (from cache), chunk >> > c. dedup agent: write/refcount chunks >> > d. replace object with dedup manifest >> > >> > This could happen with or without a delay. I don't think it makes sense >> > to consider "promote" here at all; it sounds like you're assuming the >> > initial dedup tier is a cache tier, and we should try not to assume that >> > (even though it might be possible). Instead, I think a "basic" setup >> > would probably be >> > >> > 1. base pool (all ssd; contains all metadata for all objects, and absorbs >> > writes). >> > 2. dedup pool(s) contain refcounted chunks >> > >> > If we want to do inline dedup, it would be some complex code that combines >> > all of the steps above into one, at the expense of client latency. >> > >> > >> > In any case, it's awesome that you have a working prototype. However, >> > it's not going to be practical to take a huge patch(set) like this and >> > merge it all at once. It's too much code to review, too complex, and too >> > hard to test. Also, it's changing 5000 in ReplicatedPG.cc (since renamed >> > PrimaryLogPG.cc), which is slated for a big refactor right after luminous. >> > >> > The way to approach this to get it upstream is to break this down into >> > different logical components and design/review/test/merge each of them >> > indepdendently. Having a prototype is useful in that it will be easier to >> > answer a lot of the questions we'll have deciding how each part should >> > work and what it needs to be able to handle, but don't expect that most >> > of that code will end up in the final version! >> > >> > I'm guessing we can break this down into a few logical components: >> > >> > 1) How do we store chunks. We know we want refcounted objects for each >> > chunk. We don't know how we'll manage the refcounts, whether we want/need >> > backpointers, whether we are willing to tolerate "leaking" references in >> > failure cases (so that we fail to clean up all chunks if we e.g. delete >> > all data), whether we want to implement it as a rados class or a native >> > rados op, whether we want to support EC, compression, etc. This whole >> > discussion one is a great place to start because it is self-contained and >> > doesn't break anything else. >> > >> > 2) How do we do the dedup manifists (and redirects) in object_info_t. We >> > want the solution to include or be compatible with simpler tiering, like >> > having the object_info_t simply be a pointer to a different (colder) pool. >> > In fact, I think this is the thing to do first becuase it will make us >> > fix/solve all the basic problems with flush and promote. And extending >> > this to include dedup (object is composed of many little bits in other >> > pools) is then a matter of making that 'manifest' (or whatever we call it) >> > a generic and extensible description. Remember we also want to support >> > pushing objects into external systems (say, glacier, or some other >> > external object store like a backup system). >> > >> > 3) How do we chunk. You have some classes that handle aligned chunking. >> > We'll probably eventually want content-based chunking (based on Rabin >> > fingerprinting or whatever the new hotness is). Real users will probably >> > want adjustable policies based on what they know of the content they're >> > storing, and the system will probably want to support multiple CAS pools >> > based on which policy is being used (as that determines chunk sizes >> > etc and whether we'll actually have any dedup happening). >> > >> > 4) How to drive the dedup process itself. An async agent that's part of >> > the exiting tier_agent? An external process? Something inline in the >> > write path? This is the hardest question to answer, and the one that is >> > most likely to collide with other planned OSD work. It can also come >> > last, IMO! We can start with a simple offline agent and perhaps >> > eventually do something more clever or efficient. >> > >> > In any case, I think #1 and #2 are the key discussions we should have now. >> > I suggest starting a pad and email thread for each (pad.ceph.com) so we >> > can brainstorm design options, weight trade-offs, and come to some >> > consensus. (I had some thoughts, for example, on a hybrid scheme >> > somewhere between explicit backpointers and a simple refcount that could >> > consume fixed overhead but still provide information that would enable a >> > moderately efficient scrub/audit.) >> > >> > Thanks! >> > sage >> > >> > >> > >> > >> >> Write sequence (writeback mode) is >> >> >> >> a. Read object data and do fingerprinting (if data is not calculated). >> >> b. Send reference count decrement message (for previous chunk) to osd >> >> (in CAS pool) and updates local object metadata >> >> c. Send copy_from message to osd (in CAS pool) and send copy_from >> >> message (in order to copy the dedup metadata) to a osd (in original >> >> pool) >> >> >> >> Writeback mode also increase the number of operation. Can we reduce? >> >> >> >> >> >> >> >> 4. Performance. >> >> >> >> Performance is improved compared to previous results. But It still >> >> seems to be improving. (512KB block, Seq. workload, fio, KRBD, single >> >> thread, target_max_objects = 4) >> >> >> >> Major concerns are first is fingerprint overhead and second is >> >> writeback performance in cache tier. When the chunk size is large >> >> (>512KB), SHA1 takes more than 3ms. (This can be reduced if we use >> >> small chunk.) >> >> >> >> Regarding writeback performance, Flush need two more operations than >> >> proxy mode. First is "marking clean state". Second is "reading dedup >> >> metadata and data from storage". Therefore, actual read and write >> >> occur. These cause that flush completion is delayed. >> >> >> >> Small chunk performance in the writeback mode is significantly >> >> degraded because single flush thread handles multiple copy_from >> >> message. It seems that we should improve basic flushing performance. >> >> >> >> >> >> Write performance (MB/s) >> >> >> >> Dedup ratio 0 60 100 >> >> >> >> Proxy 55 64 73 >> >> >> >> Writeback 48 50 50 >> >> >> >> Original 120 120 122 >> >> >> >> >> >> >> >> Read performance (MB/s) >> >> >> >> Dedup ratio 0 60 100 >> >> >> >> Proxy 117 130 141 >> >> >> >> Writeback 198 197 200 >> >> >> >> Original 280 276 285 >> >> >> >> >> >> >> >> >> >> 5. Command to enable dedup >> >> >> >> Ceph osd pool create sds-hot 1024 >> >> Ceph osd pool create sds-cas 1024 >> >> Ceph osd tier add_cas rbd sds-hot sds-cas >> >> Ceph osd tier sds-hot (proxy or writeback) >> >> Ceph osd tier dedup_block rbd sds-hot sds-cas (chunk size. e.g. 65536, 131072..) >> >> Ceph osd tier set-overlay rbd sds-hot >> >> >> >> >> >> >> >> Thanks >> >> Myoungwon Oh >> >> (omwmw@xxxxxx) >> >> >> >> 2017-02-07 23:50 GMT+09:00 Sage Weil <sweil@xxxxxxxxxx>: >> >> > On Tue, 7 Feb 2017, myoungwon oh wrote: >> >> >> Hi sage. >> >> >> >> >> >> I uploaded the document which describe my overall appoach. >> >> >> please see it and give me feedback. >> >> >> slide: https://www.slideshare.net/secret/JZcy3yYEDIHPyg >> >> > >> >> > This approach looks pretty close to what we have been planning. A few >> >> > comments: >> >> > >> >> > 1) I think it may be better to view the tier/pool that has the object >> >> > metadata as the "base" pool, and the CAS pool with the refcounted >> >> > object chunks as as tier below that. >> >> > >> >> > 2) I think we can use an object class or a handful of new native rados >> >> > operations to make the CAS pool read/write operations more efficient. In >> >> > your slides you describe a process something like >> >> > >> >> > rados(getattr) >> >> > if exists >> >> > rados(increment ref count) >> >> > else >> >> > rados(write object and set ref count to 1) >> >> > >> >> > This could be collapsed into a single optimistic operation that sends the >> >> > data and a command that says "create or increment ref count" so that the >> >> > conditional behavior is handled at the OSD. This will be more efficient >> >> > for small chunks. (For large chunks, or in cases where we have some >> >> > confidence that the chunk probably already exists, the pessimistic >> >> > approach might still make sense.) Either way, we should probably support >> >> > both. >> >> > >> >> > 3) We'd like to generalize the first pool behavior so that it is just a >> >> > special case of the new tiering functionality. The idea is that an >> >> > object_info_t can have a 'manifest' that described where and how the >> >> > object is really stored instead of the object data itself (much like it >> >> > can already be a whiteout, etc.). In the simplest case, the manifest >> >> > would just say "this object is stored in pool X" (simple tiering). In >> >> > this case, the manifest would a structure like >> >> > >> >> > map<offset, tuple<length, cas object, pool>> >> >> > >> >> > I think it'll be worth the effort to build a general struture here that we >> >> > can use for basic tiering (not just dedup). >> >> > >> >> > sage >> >> > >> >> > >> >> > >> >> >> >> >> >> thanks >> >> >> >> >> >> >> >> >> 2017-01-31 23:24 GMT+09:00 Sage Weil <sage@xxxxxxxxxxxx>: >> >> >> > On Thu, 26 Jan 2017, myoungwon oh wrote: >> >> >> >> I have two questions. >> >> >> >> >> >> >> >> 1. I would like to ask about CAS location. current our implementation store >> >> >> >> content address object in storage tier.However, If we store the CAO in the >> >> >> >> cache tier, we can get a performance advantage. Do you think we can create >> >> >> >> CAO in cachetier? or create a separate storage pool for CAS? >> >> >> > >> >> >> > It depends on the design. If the you are naming the objects at the >> >> >> > librados client side, then you can use the rados cluster itself >> >> >> > unmodified (with or without a cache tier). This is roughly how I have >> >> >> > anticipated implementing the CAS storage portion. If you are doing the >> >> >> > chunking hashing and within the OSD itself, then you can't do the CAS >> >> >> > at the first tier because the requests won't be directed at the right OSD. >> >> >> > >> >> >> >> 2. The results below are performance result for our current implementation. >> >> >> >> experiment setup: >> >> >> >> PROXY (inline dedup), WRITEBACK (lazy dedup, target_max_bytes: 50MB), >> >> >> >> ORIGINAL(without dedup feature and cache tier), >> >> >> >> fio, 512K block, seq. I/O, single thread >> >> >> >> >> >> >> >> One thing to note is that the writeback case is slower than the proxy. >> >> >> >> We think there are three problems as follows. >> >> >> >> >> >> >> >> A. The current implementation creates a fingerprint by reading the entire >> >> >> >> object when flushing. Therefore, there is a problem that read and write are >> >> >> >> mixed. >> >> >> > >> >> >> > I expect this is a small factor compared to the fact that in writeback >> >> >> > mode you have to *write* to the cache tier, which is 3x replicated, >> >> >> > whereas in proxy mode those writes don't happen at all. >> >> >> > >> >> >> >> B. When client request read, the promote_object function reads the object >> >> >> >> and writes it back to the cache tier, which also causes a mix of read and >> >> >> >> write. >> >> >> > >> >> >> > This can be mitigated by setting the min_read_recency_for_promote pool >> >> >> > property to something >1. Then reads will be proxied unless the object >> >> >> > appears to be hot (because it has been touched over multiple >> >> >> > hitset intervals). >> >> >> > >> >> >> >> C. When flushing, the unchanged part is rewritten because flush operation >> >> >> >> perform per-object based. >> >> >> > >> >> >> > Yes. >> >> >> > >> >> >> > Is there a description of your overall approach somewhere? >> >> >> > >> >> >> > sage >> >> >> > >> >> >> > >> >> >> >> >> >> >> >> Do I have something wrong? or Could you give me a suggestion to improve >> >> >> >> performance? >> >> >> >> >> >> >> >> >> >> >> >> a. Write performance (KB/s) >> >> >> >> >> >> >> >> dedup_ratio 0 20 40 60 80 100 >> >> >> >> >> >> >> >> PROXY 45586 47804 51120 52844 56167 55302 >> >> >> >> >> >> >> >> WRITEBACK 13151 11078 9531 13010 9518 8319 >> >> >> >> >> >> >> >> ORIGINAL 121209 124786 122140 121195 122540 132363 >> >> >> >> >> >> >> >> >> >> >> >> b. Read performance (KB/s) >> >> >> >> >> >> >> >> dedup_ratio 0 20 40 60 80 100 >> >> >> >> >> >> >> >> PROXY 112231 118994 118070 120071 117884 132748 >> >> >> >> >> >> >> >> WRITEBACK 34040 29109 19104 26677 24756 21695 >> >> >> >> >> >> >> >> ORIGINAL 285482 284398 278063 277989 271793 285094 >> >> >> >> >> >> >> >> >> >> >> >> thanks, >> >> >> >> Myoungwon Oh >> >> >> >> -- >> >> >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> >> >> >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> >> >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
