Hello everyone, Here we provide some basic backgrounds in addition to the manpages at http://www.gluster.org/community/documentation/index.php?title=Features/disk-encryption Comments, questions, any your experience are welcome. Thanks, Edward. Transparent encryption and authentication in distributed systems with non-trusted servers Distributed systems impose tighter requirements to at-rest encryption. This is because your encrypted data will be stored on servers, which are de facto untrusted. In particular, your private encrypted data can be subjected to analysis and tampering, which eventually will lead to its revealing, if it is not properly protected. Specifically, usually it is not enough to just encrypt data. In distributed systems serious protection of your personal data is possible only in conjunction with a special process, which is called authentication. GlusterFS provides such enhanced service: In GlusterFS encryption is enhanced with authentication. Currently we provide protection from "silent tampering". This is a kind of tampering which is hard to detect, because it doesn't break POSIX compliance. Specifically, we protect encryption-specific file's metadata which includes unique file's object id (GFID), cipher algorithm id, cipher block size and other attributes used by the encryption process. Restrictions 1. We encrypt only file content. The feature of transparent encryption doesn't protect file names: they are neither encrypted, nor verified. Protection of file names is not so critical as protection of encryption-specific file's metadata: any attacks based on tampering file names will break POSIX compliance and result in massive corruption, which is easy to detect. 2. The feature of transparent encryption is not supported in NFS mounts of GlusterFS volumes: NFS's file handles introduce security issues, which are hard to resolve. 3. The feature of transparent encryption is incompatible with GlusterFS performance translators (quick-read, write-behind and open-behind). I. Trusted and non-trusted machines Suppose you are a user of this feature (transparent encryption and authentication in GlusterFS). You qualify every machine as either trusted, or non-trusted. Examples: I.1 Your personal laptop, which is under your supervision is trusted machine. I.2 Remote GlusterFS servers, which are not under your supervision are non-trusted machines. They are managed by a good admin, but you don't trust him your private data. I.3 Clouds are important example of a set of non-trusted machines: you don't know what is going on there at all. II. Trusted and non-trusted objects Every machine contains objects (in the RAM, disks, registers, etc). All objects of every non-trusted machine are non-trusted by definition. Trusted machine contains both type of objects (trusted and non-trusted). Sources of non-trusted objects on your trusted machine: . non-trusted media; . non-trusted network; . social engineering; . etc. Sources of trusted objects on your trusted machine: . trusted media; . trusted network; . process of verification of non-trusted objects (authentication). Examples: II.1 email that you have received without any checks is non-trusted object; II.2 email with properly verified digital signature is trusted object; II.3 Your secret key properly generated on your trusted machine, or retrieved from trusted media is trusted object. II.4 You wanted to look at user accounts on your trusted local machine. The string "/etc/passwd" that you have passed to the open(2) is trusted object. II.5 Someone you don't know asked you to check if you have a file "/foo/bar" on your trusted local machine. The string "/foo/bar", that you have passed to readdir(2) is non-trusted object. II.6 Encrypted content of any regular file received from the non-trusted GlusterFS servers before processing by the crypt translator is non-trusted object. II.7 Decrypted content of your regular file after successful processing by the crypt translator is trusted object. II.8 List of file names provided by ls(1) for your mounted encrypted GlusterFS volume is non-trusted object (see subsection 1 of the Restrictions above). Status of some objects on your trusted machine can be changed from "non-trusted" to "trusted" by a special process, which is called authentication. Authentication includes creating/checking a special MAC (Message Authentication Code) for every object that you will want to verify after its storing on the non-trusted machines. III. Encryption and authentication in GlusterFS In GlusterFS a special translator (encryption/crypt) is responsible for both, encryption and authentication. The crypt translator works only on trusted client machines. Data Encryption We encrypt file content by the AES cipher algorithm with XTS cipher mode. This mode provides "weak data authentication". Tampering of ciphertext created in this mode will lead to unpredictable changes in the plain text, i.e. in data corruption, which is easy to detect. Data encryption is performed with unique per-file cipher key generated by master volume key and "salted" by the unique trusted object id (GFID). Metadata Authentication In the feature of transparent encryption the unique object id (GFID) is an important encryption-specific attribute, which needs protection, since it is stored on the non-trusted servers. We protect GFID by creating/checking MACs. Every such MAC is "salted" by the trusted absolute file name. This "salt" is needed to prevent a special kind of tampering, which extends the scope of per-file data cipher key (there are known attacks based on such extending). Every hardlink adds a respective MAC to the file. When the file is renamed, the respective MAC gets updated. Whenever the file is opened, we check all the per-name MACs up to the first match. No matches means failed verification: GlusterFS will refuse to open such file. Otherwise, the status of file's GFID will be changed to "trusted". Encryption-specific file attributes including the array of per-name MACs are stored on the untrusted server as file's xattrs with special key "trusted.glusterfs.crypt.att.cfmt". Let's create a file in our encrypted volume mounted at /mnt/glusterfs: # pwd /mnt/glusterfs # touch file # getfattr -n trusted.glusterfs.crypt.att.cfmt -d -e hex file trusted.glusterfs.crypt.att.cfmt=0x00fe824979347d90af1c8d87d67faae50d243d84d641 The first byte in the string contains version (format) of the string (0 in this example). In this format per-name MAC is 8 bytes long, and array of all MACs locates at the end of the string. In this example file has only one name and the string respectively contains only one MAC (aae50d243d84d641). Let's now create a hardlink: # ln file file-link # getfattr -n trusted.glusterfs.crypt.att.cfmt -d -e hex file trusted.glusterfs.crypt.att.cfmt=0x00fe824979347d90af1c8d87d67faae50d243d84d641c66d01a59a6a2e8c The file has acquired the second name ("file-link"), and the string respectively has been supplemented with the second MAC (c66d01a59a6a2e8c). # mv file file-renamed # getfattr -n trusted.glusterfs.crypt.att.cfmt -d -e hex file-renamed trusted.glusterfs.crypt.att.cfmt=0x00fe824979347d90af1c8d87d67fb46f237c94cb87dec66d01a59a6a2e8c We changed the file name, and the respective MAC was updated (the new value is b46f237c94cb87de) # rm -f file-renamed # getfattr -n trusted.glusterfs.crypt.att.cfmt -d -e hex file-link trusted.glusterfs.crypt.att.cfmt=0x00fe824979347d90af1c8d87d67f c66d01a59a6a2e8c We removed the hardlink, and respective MAC (b46f237c94cb87de) was removed. 13 bytes right after the format id (fe824979347d90af1c8d87d67f in our example) contain the following attributes, which are encrypted in a special AEAD mode: . data cipher algorithm id; . data cipher mode id; . encoded block size; . encryption translator id; . encoded size of the data cipher key; GFID is verified only for file operations, which invoke crypto transforms (cipher, authentication, etc). In particular, we need to vertify GFID during ->open(), ->read(), ->write(), ->truncate(), ->link(), etc. file operations. The example of a file operation, which doesn't require to verify GFID is ->readdir() (see subsection 1 of the Restrictions above). Currently GFID verification procedure is encapsulated in FOP->open() of the crypt translator. So the crypt translator mandatory calls the FOP->open() whenever the trusted GFID is required and is not in the cache (e.g. during FOP->truncate()). IV. Why we don't support the feature of transparent encryption in NFS mounts of GlusterFS volumes In NFS mounts of GlusterFS volumes file operations usually don't have file names. They manipulate with file handles instead (which actually are GFIDs). Respectively, we have to be sure that every file handle in the cache of the client machine is trusted. This is not simple to implement with a guarantee that future changes in GlusterFS code won't add a security hole, which will lead to appearing of non-verified file handles in the cache of the client machine. _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-devel
