Hi, 2012-12-12 08:08, Vyacheslav Dubeyko <slava@xxxxxxxxxxx>: > Hi Andreas, > > On Tue, 2012-12-11 at 14:54 +0100, Andreas Rohner wrote: >> Hi Vyacheslav, >> >> Thanks for your response. >> >> > > 2. Is there some fundamental difficulty that makes it hard to > implement >> > > for a log-structured fs? >> > >> > I think that the most fundamental possible issue can be a possible >> > performance degradation. But first of all, from my point of view, it >> > needs to discuss what the online defrag is and how it is possible to >> > implement it. What do you mean personally by online defrag? And how do >> > you imagine online defrag mechanism for NILFS2 in particular? When you >> > describe your understanding then it will be possible to discuss about >> > difficulties, I think. :-) >> >> One way would be to just write out heavily fragmented files sequentially >> and atomically switch to the new blocks. But as you suggested this >> simple approach would probably result in performance degradation, >> because it would eat up free segments and the segments of the old blocks >> would contain more unusable free space, that has to be cleaned first. >> This could result in an undesirable situation where most of the segments >> are 60% full and for every clean segment the cleaner has to read in 4 >> half full segments. I think the difficult part is to find a suitable >> heuristic to decide if it is beneficial to defragment a file or not. My >> aim would be to produce as many clean or nearly clean segments as >> possible in the process. I would try to implement and test different >> heuristics and algorithms with differently aged file systems and compare >> the results. >> > > I think that this task hides many difficult questions. How does it > define what files fragmented or not? How does it measure the > fragmentation degree? What fragmentation degree should be a basis for > defragmentation activity? When does it need to detect fragmentation and > how to keep this knowledge? How does it make defragmentation without > performance degradation? These questions are of special interest if we through in the type of media in the discussion. Fragmentation is not a big deal on NAND-based media (SSD:s, memory cards, USB-sticks, etc). Defragmentation activity might even shorten the lifetime for such media due to the limited amount write/erase cycles. Brgds /S-G > As I understand, when we are talking about defragmentation then we > expect a performance enhancement as a result. But defragmenter activity > can be a background reason of performance degradation. Not every > workload or I/O pattern can be a reason of significant fragmentation. > > Also, it is a very important to choose a point of defragmentation. I > mean that it is possible to try to prevent fragmentation or to correct > fragmentation after flushing on the volume. It is possible to have a > some hybrid technique, I think. An I/O pattern or file type can be a > basis for such decision, I think. > > As I understand, F2FS [1] has some defragmenting approaches. I think > that it needs to discuss more deeply about technique of detecting > fragmented files and fragmentation degree. But maybe hot data tracking > patch [2,3] will be a basis for such discussion. > > I think that it can be a useful some materials about NILFS2. I began a > design document for NILFS2 [4] but unfortunately it is not ended yet. It > was published a review of NILFS2 [5] not so recently. > > It exists some defragmentation-related papers but I haven't > comprehensive list. I can mention about "The Effects of Filesystem > Fragmentation" [6]. Maybe it can be useful "A Five-Year Study of > File-System Metadata" [7] and "A File Is Not a File: Understanding the > I/O Behavior of Apple Desktop Applications" [8] papers. > > So, I feel necessity to think more deeply about online defragment task > and about what you said. But, anyway, it is a beginning of > discussion. :-) > > [1] http://lwn.net/Articles/518988/ > [2] http://lwn.net/Articles/525425/ > [3] http://lwn.net/Articles/400029/ > [4] http://dubeyko.com/development/FileSystems/NILFS/nilfs2-design.pdf > [5] http://lwn.net/Articles/522507/ > [6] > http://www.google.ru/url?sa=t&rct=j&q=the%20effects%20of%20filesystem%20fragmentation&source=web&cd=2&ved=0CD0QFjAB&url=http%3A%2F%2Fwww.kernel.org%2Fdoc%2Fols%2F2006%2Fols2006v1-pages-193-208.pdf&ei=6CnIUJeHHYqB4gS6l4GoCQ&usg=AFQjCNFLhxtq89VLzE_fLuX7CDDpk_1Krw&bvm=bv.1355272958,d.bGE&cad=rjt > [7] > http://www.google.ru/url?sa=t&rct=j&q=a%20five-year%20study%20of%20file-system%20metadata&source=web&cd=1&ved=0CC0QFjAA&url=http%3A%2F%2Fresearch.microsoft.com%2Fpubs%2F72896%2Ffast07-final.pdf&ei=syvIULepFoWE4ASDmYDIBA&usg=AFQjCNE5mFDPqgEvGYe32RkNyVa2oxVxkw&bvm=bv.1355272958,d.bGE&cad=rjt > [8] > http://www.google.ru/url?sa=t&rct=j&q=a%20file%20is%20not%20a%20file%3A%20understanding%20the%20i%2Fo%20behavior%20of%20apple%20desktop%20applications&source=web&cd=1&ved=0CC0QFjAA&url=http%3A%2F%2Fresearch.cs.wisc.edu%2Fwind%2FPublications%2Fibench-1c-sosp11.pdf&ei=NSzIUP3zCKqK4ASU5oCACw&usg=AFQjCNEFr-bw1Ke382_rQBYGQwI88MPkKg&bvm=bv.1355272958,d.bGE&cad=rjt > > With the best regards, > Vyacheslav Dubeyko. > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nilfs" > in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
