Hi Al,
Thanks, that is clear. I think that discussing the assumption of honesty among the parties involved in benchmarking would be a useful addition to the Security Considerations section in the draft.
Mališa
On 15/12/2020 14:45, "MORTON, ALFRED C (AL)" <acm@xxxxxxxxxxxxxxxx> wrote:
Hi Mališa,
thanks for your review, please see below for one reply to your question (acm].
Al
> -----Original Message-----
> From: bmwg [mailto:bmwg-bounces@xxxxxxxx] On Behalf Of Mališa Vucinic via
> Datatracker
> Sent: Tuesday, December 15, 2020 6:30 AM
> To: secdir@xxxxxxxx
> Cc: last-call@xxxxxxxx; bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-
> frame.all@xxxxxxxx
> Subject: [bmwg] Secdir telechat review of draft-ietf-bmwg-b2b-frame-03
>
> Reviewer: Mališa Vučinić
> Review result: Ready
>
> I reviewed this document as part of the Security Directorate's ongoing
> effort
> to review all IETF documents being processed by the IESG. These comments
> were
> written primarily for the benefit of the Security Area Directors. Document
> authors, document editors, and WG chairs should treat these comments just
> like
> any other IETF Last Call comments.
>
> Thank you for this well-written document, it was a pleasure to read and I
> think
> it is ready to proceed. Since the document updates RFC2544 benchmarking
> procedure for estimating the buffer time of a Device Under Test (DUT), it
> does
> not raise any security issues. Security Considerations section is quite
> clear
> and it stresses that these tests are performed in a lab environment.
>
> I do have a question regarding the last paragraph of the Security
> Considerations on special capabilities of DUTs for benchmarking purposes.
> Currently, the sentence reads: "Special capabilities SHOULD NOT exist in
> the
> DUT/SUT specifically for benchmarking purposes." Why is this a SHOULD NOT
> and
> not a MUST NOT? Could you give an example when such special capabilities
> in a
> DUT are appropriate?
[acm]
We can only make a strong recommendation in this area. As testers/benchmarkers are often independent from the DUT developers and conduct testing external to the DUT, we assume honesty among other parties but we cannot require it. If someone constructed a DUT that recognized test conditions and operated differently to perform better somehow, our tests would measure the intended "better" performance. It takes a special/additional test effort to prove that a DUT has "designed to the test" (consider Volkswagen and fuel efficiency testing [0]).
We simply do not have any authority in this matter, but we can let all parties know that gaming the test can be discovered and reported (albeit with more testing that we do not describe).
[0] https://www.consumerreports.org/fuel-economy-efficiency/volkswagen-used-special-software-to-exaggerate-fuel-economy/
>
>
>
> _______________________________________________
> bmwg mailing list
> bmwg@xxxxxxxx
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!
> !BhdT!1JFeLsENzMU-ew89jxmJKxfp4wj5Zo3AZ6V8iULU3hWAentH1dymqJmDOvw7$
--
last-call mailing list
last-call@xxxxxxxx
https://www.ietf.org/mailman/listinfo/last-call
