On Fri, May 15, 2015 at 06:14:24PM -0700, Jaegeuk Kim wrote: > On Thu, May 14, 2015 at 09:50:44AM -0700, Tom Marshall wrote: > > Please keep in mind that I'm also working on transparent > > compression. I'm watching this thread closely so that I can > > implement a compression library alongside the crypto library. If > > there is any interest or benefit, I would be glad to work together > > so that the two can be done cooperatively at the same time. > > I can't imagine quickly how compression code can be shared with crypto. > The basic approach for compression would be that X pages can be compressed into > small number of pages, Y, which can be a X to Y mapping. > But, this per-file encryption supports only 1 to 1 4KB mapping, so that it could > be quite a simple implementation. No, I don't intend to share actual code with crypto -- at least not much. I'm more interested in looking at how the crypto layer is implemented to give me clues about how to implement a compression layer. > Could you elaborate on your approach or design? Or, codes? > Whatever, IMO, it needs to implement it by any filesystem first. I don't really have any working code yet. I will probably get to that in the coming few weeks. Right now I'm still working with the ugly VFS stacking implementation that I posted initially. The thing that I have done is dismissed the standard compression framing formats. zlib (and gzip) are designed for streaming and it is quite difficult to implement random access on it. See the example code in the zlib source, zran.c. It's not really tenable because 32kb of prior data is required to be kept as priming information. Even doing fully encapsulated blocks with Z_FINISH, there is still no way to skip over data without decompressing it first to build an index. lz4 is somewhat better in that blocks are self contained. But block lengths must be read sequentially. This means that reading an arbitrary position in a file requires a proportional number of reads to find the desired block. So, I am working with a simple framing format that I threw together. The header has a compression method (zlib or lz4), block size, original input size, and a block map. -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
