Re: [RFC PATCH v10 00/48] ceph+fscrypt: full support

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 2/14/22 8:08 PM, Xiubo Li wrote:

On 2/14/22 7:33 PM, Jeff Layton wrote:
On Mon, 2022-02-14 at 17:37 +0800, Xiubo Li wrote:
Hi Jeff,

I am using the 'wip-fscrypt' branch to test other issue and hit:

cp: cannot access './dir___683': No buffer space available
cp: cannot access './dir___686': No buffer space available
cp: cannot access './dir___687': No buffer space available
cp: cannot access './dir___688': No buffer space available
cp: cannot access './dir___689': No buffer space available
cp: cannot access './dir___693': No buffer space available

...

[root@lxbceph1 kcephfs]# diff ./dir___997 /data/backup/kernel/dir___997
diff: ./dir___997: No buffer space available


The dmesg logs:

<7>[ 1256.918228] ceph:  do_getattr inode 0000000089964a71 mask AsXsFs
mode 040755
<7>[ 1256.918232] ceph:  __ceph_caps_issued_mask ino 0x100000009be cap
0000000014f1c64b issued pAsLsXsFs (mask AsXsFs)
<7>[ 1256.918237] ceph:  __touch_cap 0000000089964a71 cap
0000000014f1c64b mds0
<7>[ 1256.918250] ceph:  readdir 0000000089964a71 file 00000000065cb689
pos 0
<7>[ 1256.918254] ceph:  readdir off 0 -> '.'
<7>[ 1256.918258] ceph:  readdir off 1 -> '..'
<4>[ 1256.918262] fscrypt (ceph, inode 1099511630270): Error -105
getting encryption context
<7>[ 1256.918269] ceph:  readdir 0000000089964a71 file 00000000065cb689
pos 2
<4>[ 1256.918273] fscrypt (ceph, inode 1099511630270): Error -105
getting encryption context
<7>[ 1256.918288] ceph:  release inode 0000000089964a71 dir file
00000000065cb689
<7>[ 1256.918310] ceph:  __ceph_caps_issued_mask ino 0x1 cap
00000000aa2afb8b issued pAsLsXsFs (mask Fs)
<7>[ 1257.574593] ceph:  mdsc delayed_work

I did nothing about the fscrypt after mounting the kclient, just create
2000 directories and then made some snapshots on the root dir and then
try to copy the root directory to the backup.

- Xiubo

That means that ceph_crypt_get_context returned -ENODATA, which it can

It should be -ENOBUFS.

I am not sure it relates to the snapshot stuff. I will try without the snapshot later.

Without snapshot, I also can see this error.

-- Xiubo


I can debug it later, maybe in next week.

-- Xiubo

do for several different reasons. We probably need to add in some
debugging there to see which one it is...

TBH, I've done absolutely no testing with snapshots, so it's quite
possible there is some interaction there that is causing problems.

-- Jeff

On 1/12/22 3:15 AM, Jeff Layton wrote:
This patchset represents a (mostly) complete rough draft of fscrypt
support for cephfs. The context, filename and symlink support is more or less the same as the versions posted before, and comprise the first half
of the patches.

The new bits here are the size handling changes and support for content encryption, in buffered, direct and synchronous codepaths. Much of this
code is still very rough and needs a lot of cleanup work.

fscrypt support relies on some MDS changes that are being tracked here:

      https://github.com/ceph/ceph/pull/43588

In particular, this PR adds some new opaque fields in the inode that we use to store fscrypt-specific information, like the context and the real
size of a file. That is slated to be merged for the upcoming Quincy
release (which is sometime this northern spring).

There are still some notable bugs:

1/ we've identified a few more potential races in truncate handling
which will probably necessitate a protocol change, as well as changes to
the MDS and kclient patchsets. The good news is that we think we have
an approach that will resolve this.

2/ the kclient doesn't handle reading sparse regions in OSD objects
properly yet. The client can end up writing to a non-zero offset in a
non-existent object. Then, if the client tries to read the written
region back later, it'll get back zeroes and give you garbage when you
try to decrypt them.

It turns out that the OSD already supports a SPARSE_READ operation, so
I'm working on implementing that in the kclient to make it not try to
decrypt the sparse regions.

Still, I was able to run xfstests on this set yesterday. Bug #2 above
prevented all of the tests from passing, but it didn't oops! I call that progress! Given that, I figured this is a good time to post what I have
so far.

Note that the buffered I/O changes in this set are not suitable for
merge and will likely end up being discarded. We need to plumb the
encryption in at the netfs layer, so that we can store encrypted data
in fscache.

The non-buffered codepaths will likely also need substantial changes
before merging. It may be simpler to just move that into the netfs layer
too as cifs will need something similar anyway.

My goal is to get most of this into v5.18, but v5.19 might be more
realistiv. Hopefully I'll have a non-RFC patchset to send in a few
weeks.

Special thanks to Xiubo who came through with the MDS patches. Also,
thanks to everyone (especially Eric Biggers) for all of the previous
reviews. It's much appreciated!

Jeff Layton (43):
    vfs: export new_inode_pseudo
    fscrypt: export fscrypt_base64url_encode and fscrypt_base64url_decode     fscrypt: export fscrypt_fname_encrypt and fscrypt_fname_encrypted_size
    fscrypt: add fscrypt_context_for_new_inode
    ceph: preallocate inode for ops that may create one
    ceph: crypto context handling for ceph
    ceph: parse new fscrypt_auth and fscrypt_file fields in inode traces
    ceph: add fscrypt_* handling to caps.c
    ceph: add ability to set fscrypt_auth via setattr
    ceph: implement -o test_dummy_encryption mount option
    ceph: decode alternate_name in lease info
    ceph: add fscrypt ioctls
    ceph: make ceph_msdc_build_path use ref-walk
    ceph: add encrypted fname handling to ceph_mdsc_build_path
    ceph: send altname in MClientRequest
    ceph: encode encrypted name in dentry release
    ceph: properly set DCACHE_NOKEY_NAME flag in lookup
    ceph: make d_revalidate call fscrypt revalidator for encrypted
      dentries
    ceph: add helpers for converting names for userland presentation
    ceph: add fscrypt support to ceph_fill_trace
    ceph: add support to readdir for encrypted filenames
    ceph: create symlinks with encrypted and base64-encoded targets
    ceph: make ceph_get_name decrypt filenames
    ceph: add a new ceph.fscrypt.auth vxattr
    ceph: add some fscrypt guardrails
    libceph: add CEPH_OSD_OP_ASSERT_VER support
    ceph: size handling for encrypted inodes in cap updates
    ceph: fscrypt_file field handling in MClientRequest messages
    ceph: get file size from fscrypt_file when present in inode traces
    ceph: handle fscrypt fields in cap messages from MDS
    ceph: add infrastructure for file encryption and decryption
    libceph: allow ceph_osdc_new_request to accept a multi-op read
    ceph: disable fallocate for encrypted inodes
    ceph: disable copy offload on encrypted inodes
    ceph: don't use special DIO path for encrypted inodes
    ceph: set encryption context on open
    ceph: align data in pages in ceph_sync_write
    ceph: add read/modify/write to ceph_sync_write
    ceph: plumb in decryption during sync reads
    ceph: set i_blkbits to crypto block size for encrypted inodes
    ceph: add fscrypt decryption support to ceph_netfs_issue_op
    ceph: add encryption support to writepage
    ceph: fscrypt support for writepages

Luis Henriques (1):
    ceph: don't allow changing layout on encrypted files/directories

Xiubo Li (4):
    ceph: add __ceph_get_caps helper support
    ceph: add __ceph_sync_read helper support
    ceph: add object version support for sync read
    ceph: add truncate size handling support for fscrypt

   fs/ceph/Makefile                |   1 +
   fs/ceph/acl.c                   |   4 +-
   fs/ceph/addr.c                  | 128 +++++--
   fs/ceph/caps.c                  | 211 ++++++++++--
   fs/ceph/crypto.c                | 374 +++++++++++++++++++++
   fs/ceph/crypto.h                | 237 +++++++++++++
   fs/ceph/dir.c                   | 209 +++++++++---
   fs/ceph/export.c                |  44 ++-
   fs/ceph/file.c                  | 476 +++++++++++++++++++++-----
   fs/ceph/inode.c                 | 576 +++++++++++++++++++++++++++++---
   fs/ceph/ioctl.c                 |  87 +++++
   fs/ceph/mds_client.c            | 349 ++++++++++++++++---
   fs/ceph/mds_client.h            |  24 +-
   fs/ceph/super.c                 |  90 ++++-
   fs/ceph/super.h                 |  43 ++-
   fs/ceph/xattr.c                 |  29 ++
   fs/crypto/fname.c               |  44 ++-
   fs/crypto/fscrypt_private.h     |   9 +-
   fs/crypto/hooks.c               |   6 +-
   fs/crypto/policy.c              |  35 +-
   fs/inode.c                      |   1 +
   include/linux/ceph/ceph_fs.h    |  21 +-
   include/linux/ceph/osd_client.h |   6 +-
   include/linux/ceph/rados.h      |   4 +
   include/linux/fscrypt.h         |  10 +
   net/ceph/osd_client.c           |  32 +-
   26 files changed, 2700 insertions(+), 350 deletions(-)
   create mode 100644 fs/ceph/crypto.c
   create mode 100644 fs/ceph/crypto.h





[Index of Archives]     [CEPH Users]     [Ceph Large]     [Ceph Dev]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux