So I've been playing around a bit and I have a basic strategy laid out. Please let me know if I'm on the right track. Compressed file attributes ========================== The filesystem is responsible for detecting whether a file is compressed and hooking into the compression lib. This may be done with an inode flag, xattr, or any other applicable method. No other special attributes are necessary. Compressed file format ====================== Compressed files shall have header, block map, and data sections. Header: byte[4] magic 'zzzz' (not strictly needed) byte param1 method and flags bits 0..3 = compression method (1=zlib, 2=lz4, etc.) bits 4..7 = flags (none defined yet) byte blocksize log2 of blocksize (max 31) le48 orig_size original uncompressed file size Block map: Vector of le16 (if blocksize <= 16) or le32 (if blocksize > 16). Each entry is the compressed size of the block. Zero indicates that the block is stored uncompressed, in case compression expanded the block. Data: Compressed data. Compression library =================== I'm just fleshing this out and trying to make it work. The basic strategy is to wrap the underlying readpage/readpages. Right now I have the following: /* * For convenience. */ typedef int (*readpage_t)(struct page *); /* * Indicate whether compression is enabled. It may be desirable to disable * compression for test, backup, or maintenance activities. Controlled by * a sysfs file. */ extern int zfile_enabled(void); /* * Initialize internal data structures for a given inode. This will * result in reading the file header and block map, so the inode must * be fully populated and ready to accept readpage requests. */ extern int zinode_init(struct inode *inode, readpage_t lower_readpage); /* * Wrapper for filesystem's readpage. Consults the block map and reads * the appropriate compressed pages for the requested block, decompresses * them, and populates the page with uncompressed data. */ extern int zinode_readpage(readpage_t lower_readpage, struct page *page); /* * As above, but for multiple pages. */ extern int zinode_readpages(readpage_t lower_readpage, struct address_space *mapping, struct list_head *pages, unsigned nr_pages); Questions and issues ==================== Should there be any padding for the data blocks? For example, if writing is to be supported, padding the compressed data to the filesystem block size would allow for easy rewriting of individual blocks without disturbing the surrounding blocks. Perhaps padding could be indicated by a flag. Note readpage_t does not take a file ptr, as it's not available at inode creation time. ext4 only uses it to lookup the inode ptr. Is this true across all filesystems, or do some actually require the file ptr in their readpage? The compression code must be able to read pages from the underlying filesystem. This involves using the pagecache. But the uncompressed data is what ultimately should end up in the pagecache. This is where I'm currently stuck. How do I implement the code such that the underlying compressed data may be read (using the pagecache or not) while not disturbing the pagecache for the uncompressed data? I'm wondering if I need to create an internal address_space to pass down into the underlying readpage? Or is there another way to do this? -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
