On 8/18/23 11:14 AM, Olga Kornievskaia wrote:
On Fri, Aug 11, 2023 at 10:28 AM Olga Kornievskaia <aglo@xxxxxxxxx> wrote:
On Fri, Aug 11, 2023 at 10:23 AM Jeff Layton <jlayton@xxxxxxxxxx> wrote:
Chuck and I were chatting yesterday about what it will take to make the
inter_copy_offload_enable module option on by default, and I'd like to
start working toward that end.
I think what we want to aim for is to eventually deprecate the module
option and have this "just work" when the conditions are right.
It looks like main obstacle is this (from RFC7862 section 4.9):
NFSv4 clients and servers supporting the inter-server COPY operations
described in this section are REQUIRED to implement the mechanism
described in Section 4.9.1.1 and to support rejecting COPY_NOTIFY
requests that do not use the RPC security protocol (RPCSEC_GSS)
[RFC7861] with privacy. If the server-to-server copy protocol is
based on ONC RPC, the servers are also REQUIRED to implement
[RFC7861], including the RPCSEC_GSSv3 "copy_to_auth",
"copy_from_auth", and "copy_confirm_auth" structured privileges.
This requirement to implement is not a requirement to use; for
example, a server may, depending on configuration, also allow
COPY_NOTIFY requests that use only AUTH_SYS.
This spec wording that it's required to implement but not required to
use makes me ask why is it a requirement at all. Anyway...
Yes, the wording is confusing. My impression is if an inter-server copy
implementation is to be secure it needs to implement the RPCSEC_GSSv3
with structured privileges. However the use of AUTH_SYS is also allowed
if security is not a concern for a given configuration.
In fact, from section 4.9.1.2 Inter-Server Copy via ONC RPC without
RPCSEC_GSS:
ONC RPC security flavors other than RPCSEC_GSS MAY be used with the
server-side copy offload operations described in this section. In
particular, host-based ONC RPC security flavors such as AUTH_NONE and
AUTH_SYS MAY be used. If a host-based security flavor is used, a
minimal level of protection for the server-to-server copy protocol is
possible.
The biggest issue is that there is a lack of a strong security method
to allow the source server and destination server to identify
themselves to each other. A further complication is that in a
multihomed environment the destination server might not contact the
source server from the same network address specified by the client
in the COPY_NOTIFY. The cnr_stateid returned from the COPY_NOTIFY
can be used to uniquely identify the destination server to the source
server. The use of the cnr_stateid provides initial authentication
of the destination server but cannot defend against man-in-the-middle
attacks after authentication or against an eavesdropper that observes
the opaque stateid on the wire. Other secure communication
techniques (e.g., IPsec) are necessary to block these attacks.
Servers SHOULD reject COPY_NOTIFY requests that do not use RPCSEC_GSS
with privacy, thus ensuring that the cnr_stateid in the COPY_NOTIFY
reply is encrypted. For the same reason, clients SHOULD send COPY
requests to the destination using RPCSEC_GSS with privacy.
It seems like if krb5p is used as the security service between the client
and the source (for COPY_NOTIFY) as well as the destination server (for COPY),
and some form of encryption is used between the destination and source server
then this would provide a minimal protection of man-in-the-middle attack.
Since the spec suggests IPsec is used between destination and source
server, I think RPC-with-TLS would accomplish the same purpose.
The use of krb5p between the client and the source and destination server
would not add much overhead to the overall copy operation. However the
encryption of read data between the destination and the source server
would add significant overhead and might defeat the performance benefit
of inter-server copy.
Also, is allowing the user to decide the security flavor between the
source and destination server the same as letting the user to decide
which security flavor to use for a regular copy between the client and
server?
-Dai
If a server requires the use of an RPCSEC_GSSv3 copy_to_auth,
copy_from_auth, or copy_confirm_auth privilege and it is not used,
the server will reject the request with NFS4ERR_PARTNER_NO_AUTH.
We don't (yet) have GSSv3 support, so we'd need to implement that in
order to make this work right with krb5. Has anyone started looking at
GSSv3?
Andy Adamson way back when implemented a draft gssv3 implementation
and I believe we still have those patches. Anna periodically have been
rebasing them but no more than that. I believe there might have been
even some patches for the copy piece but I believe those might be
lost. I'd have to dig around in my oldest laptop.
I'd like to address some other questions later as I'm out of the office today.
Incidentally, has anyone tried doing this with sec=krb5 in the current
code?
I'm not sure I fully understand your question but yes the COPY would
work over a sec=krb5* mount. What is not there is fulfillment of the
requirement to make sure that the client does COPY_NOTIFY over
sec=krb5p gssv3 regardless what mount flavor was used originally.
Does it actually work? I don't see any place where we return
nfserr_partner_no_auth,
That's because initial implementation followed the spec wording that
it is allowed to use auth_sys and not enforce gssv3.
so I wonder if we need to fix up the s2s COPY
authentication and error handling?
Yes the server would need to be change to enforce several things with
regards to the COPY_NOTIFY and inter-server copy processing in
general.
Another question: The v4.2 spec was written before the RPC over TLS
spec. Should we aim to allow this to work by default if the client and
both servers are using xprtsec=mtls and are secured by the same CA?
Yes and no. The fact that COPY_NOTIFY needs to be done over krb5p to
insure privacy/integrity of passing the structured privilege. But then
in order to use the structured privilege a new operation is used
GSSv3_create which makes use of that. this operation must done with
gss privacy. TLS is not a GSS protocol so underneath the only choice
is krb5(p). You COULD layer gssv3 over TLS but I'm not sure what would
be the point of that.
So I think the real answer is:"no" we can't use TLS here. Or need to
update the spec with a new way of doing "inter" copy security over
TLS.
1/ the client and servers are all using GSSv3 with krb5p (or some other
encryption)
...or...
2/ the client and servers are all using mtls with certificates signed by
the same CA
...I expect we'll probably be able to accomodate #2 before #1.
Beyond that, we could allow for module or export option that still
allows s2s copy to work and relaxes the above restrictions (to allow
people to use it over plaintext with AUTH_SYS on "secure" networks).
Anything I've overlooked here, or other thoughts?
Cheers,
--
Jeff Layton <jlayton@xxxxxxxxxx>
