Thanks Al for formatting Scott's comment.
Hi Scott,
Thank you for reviewing the draft and providing your comments. Please find below our response inline to your comments.
Best Regards,
Bhuvan
On Tue, Jan 30, 2018 at 08:02 PM, "MORTON, ALFRED C (AL)" <acmorton@xxxxxxx> wrote:
Hi Scott,
Thanks for your review and insightful comments!
Something modified your original formatting, so
I've tried to restore it below. Authors, please
reply to Scott's comments and add changes to your
working version of the draft.
regards,
Al
doc shepherd
-=-=-=-=-=-=-=-=-=-=-=-
Reviewer: Scott Bradner
Review result: Has Issues
This ID specifies the methodology to be used in testing the performance of SDN
controllers. It is a standard BMWG methodology document and thus, cannot have
any impact, operational or otherwise, on operational networks - see RFC 6815
The following are some comments on the document itself:
section 4.1 - Leaf-Spine used but not defined
[Bhuvan] We will add the definition in the terminology document
section 4.4 - using old software versions seems to add complexity with little
benefit
[Bhuvan] We agree with your observation. We have added this verification to ensure backward compatibility. In fact we have given this example (older vs current) for better clarity.
section 4.7 - would be useful to say that the variance over the
repeated tests should be reported in the "test reporting" section
- e.g., in section 5.1.1 I would add a row that reports the variance
- same for all tests
[Bhuvan] We will add a row to report variance across all tests
section 5.1.1 - is the topology discovery process slow enough that a 3 second
granularity of the measurement will show differences between systems?
[Bhuvan] We have considered 3 seconds for the worst case scenario
section 5.1.2 - procedure step 2 - what inserts the timestamp in the response message
(R1)?
[Bhuvan] The intent was to find the difference between the transmitted time of request and the received time of the corresponding response. I think we need to rephrase the text as follows. '2. Record every request transmit (T1) time and the corresponding response (R1) received time at the forwarding plane test emulator interface (I1) for every successful message exchange'
[Bhuvan] We will report the % of unsuccessful exchanges as wellsection 5.1.2 - the ID says "successful messages exchanged" - might it be
useful to report the % of unsuccessful exchanges?
[Bhuvan] We will reword the objective section to reflect the rate.
section 5.1.6 - the title & objective description do not match
- the title says "rate" but the objective is a count
- the test seems to try to get the rate section 5.1.7
- same issue as section 5.1.6
[Bhuvan] The intent is to generate unique src/dst address combinations to populate the forwarding table and measure its capacity. Could you please clarify what you mean by unchanging in Step 1. ?
section 5.2.3 - procedure step 1
- this reads as if the addresses are unique but unchanging
- if that is not what is meant then it should specifically say that
the addresses are changing
section 5.3.1 - it might be useful to establish specific "invalid messages"
since I could imagine that the devices could handle different types in
different ways that could have an impact on speed of handling
[Bhuvan] We have clarified the invalid message in the Note section. Do we need to add the info in the test report as well?
section 5.3.2 - "a huge number" is somewhat undefined
do you mean to say TCP SYN messages rather than TCP SYNC messages?
[Bhuvan] Yes. We will address this comment in the revised draft.
section 6 - RFC 2026 is referenced in the introduction but not listed in the
references
[Bhuvan] We will include RFC 2026 in the reference
section 9 - I have now retired so no affiliation should be listed
[Bhuvan] We will remove the affiliation
-----Original Message-----
From: bmwg [mailto:bmwg-bounces@xxxxxxxx] On Behalf Of Scott Bradner
Sent: Monday, January 29, 2018 9:17 PM
To: ops-dir@xxxxxxxx
Cc: ietf@xxxxxxxx; bmwg@xxxxxxxx; draft-ietf-bmwg-sdn-controller-
benchmark-meth.all@xxxxxxxx
Subject: [bmwg] Opsdir last call review of draft-ietf-bmwg-sdn-
controller-benchmark-meth-07
Reviewer: Scott Bradner
Review result: Has Issues
This ID specifies the methodology to be used in testing the performance
of SDN
controllers. It is a standard BMWG methodology document and thus, cannot
have
any impact, operational or otherwise, on operational networks - see RFC
6815
follow are some comments on the document itself
section 4.1 - Leaf-Spine used but not defined
section 4.4 - using old software versions seems to add complexity with
little
benefit section 4.7 - would be useful to say that the variance over the
repeated tests should be reported in the "test reporting" section -
e.g., in
section 5.1.1 I would add a row that reports the variance - same for all
tests
section 5.1.1 - is the topology discovery process slow enough that a 3
second
granularity of the measurement will show differences between systems?
section
5.1.2 - procedure step 2 - what inserts the timestamp in the response
message
(R1)? section 5.1.2 - the ID says "successful messages exchanged" -
might it be
useful to report the % of unsuccessful exchanges? section 5.1.6 - the
title &
objective description do not match - the title says "rate" but the
objective is
a count - the test seems to try to get the rate section 5.1.7 - same
issue as
section 5.1.6 section 5.2.3 - procedure step 1 - this reads as if the
addresses
are unique but unchanging - if that is not what is meant then it should
specifically say that the addresses are changing section 5.3.1 - it
might be
useful to establish specific "invalid messages" since I could imagine
that the
devices could handle different types in different ways that could have
an
impact on speed of handling section 5.3.2 - "a huge number" is somewhat
undefined
do you mean to say TCP SYN messages rather than TCP SYNC
messages?
section 6 - RFC 2026 is referenced in the introduction but not listed in
the
references section 9 - I have now retired so no affiliation should be
listed
_______________________________________________
bmwg mailing list
bmwg@xxxxxxxx
https://urldefense.proofpoint.com/v2/url?u=https-
3A__www.ietf.org_mailman_listinfo_bmwg&d=DwICAg&c=LFYZ-
o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Oip5-
2t1wtJ1ZpUIlQWQcvgNAVgydAS4jCoQi_LL50Q&s=WP7X4VUxjR59g9gvjUFj_8zTQrz-
iKc2jY9FOaPTlZY&e=