On 1/25/2023 3:41 PM, David Howells wrote:
Hi Tom,
Steve suggested I should ask you about this.
I have IWarp RDMA mostly working with my iteratorisation patches - certainly
better than without them, but I think that's mostly due to the patch that
Stefan Metzmacher so dislikes ("cifs: Fix problem with encrypted RDMA data
read").
The encryption problem is real, and Metze is correct. The client
shouldnt be requesting, and the server shouldn't be responding,
with unencrypted messages on encrypted shares.
The problem is, the proper fix is complicated.
- We've reported the issue to Microsoft, but they have not yet
said how the Windows client and server are intended to behave,
and they have not yet revealed how the protocol document will
be changed. At this time, the Linux implementation conforms,
dangerously, with the published spec.
- There is some unexplained behavior in the client when the
connection is lost after failing to decrypt the unencrypted
response. In my earlier look at the traces, for some reason
it reconnects and retries without requesting RDMA. This
succeeds, because the "inline" requests and responses are
encrypted and decrypted successfully.
It's interesting that this occurs on a compounded fallocate call.
That might be a clue, too.
What are you trying to test? Since encrypted SMBDirect traffic
is known to have an issue, I guess I'd suggest turning off
encryption-by-default on the share.
I'll poke Microsoft again on the protocol ticket.
Tom.
However, fallocate doesn't work:
# rdma link add siw0 type siw netdev enp6s0 # andromeda, softIWarp
# mount //192.168.6.1/test /xfstest.test -o user=shares,pass=foobar,rdma
# fallocate -l 1M /xfstest.test/hello
fallocate: fallocate failed: Resource temporarily unavailable
Because smb3_simple_fallocate_write_range() calls SMB2_write(), which calls
cifs_send_recv() then compound_send_recv() and thence to smb_send_rqst().
smb_send_rqst() encrypts the buffer it is given and smbd_send() attempts to
shovel it to the server using Direct Data Placement - which I think might fail
because the data is encrypted.
In one run of the above commands, the data in the kvec array looked like:
fe534d42400001000000000009000a0000000000000000001600000000000000a01300000200
0000000000000000000000000000000000000000000000000000000000000000000000000000
before the smb_send_rqst() gets to ->init_transform_rq() and like:
98eddc1bc31da7c55c00341e4dc769fa4c8b2b0ecdacbad33eb31855ec162fa2458b8437edc7
88ee0a033c84aa857b65ab31ce553594d412719cc3daf925e873e80062ec16b97c855721a42d
after. The encrypted data is seen on the wire in DDP/RDMA packets.
Any thoughts as to how to fix this?
Does it need to pass a flag down to suppress the encryption or suppress the
use of direct data placement? Or should it perhaps go through something like
->write_iter()?
Note also that it encrypts the buffer in place and then
smb3_simple_fallocate_write_range() reuses the buffer multiple times without
clearing it.
I've pushed my cifs iteratorisation patches to:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=iov-cifs
I can post them by email a bit later.
David
