Hi Scott, Thanks for your review and insightful comments! Something modified your original formatting, so I've tried to restore it below. Authors, please reply to Scott's comments and add changes to your working version of the draft. regards, Al doc shepherd -=-=-=-=-=-=-=-=-=-=-=- Reviewer: Scott Bradner Review result: Has Issues This ID specifies the methodology to be used in testing the performance of SDN controllers. It is a standard BMWG methodology document and thus, cannot have any impact, operational or otherwise, on operational networks - see RFC 6815 The following are some comments on the document itself: section 4.1 - Leaf-Spine used but not defined section 4.4 - using old software versions seems to add complexity with little benefit section 4.7 - would be useful to say that the variance over the repeated tests should be reported in the "test reporting" section - e.g., in section 5.1.1 I would add a row that reports the variance - same for all tests section 5.1.1 - is the topology discovery process slow enough that a 3 second granularity of the measurement will show differences between systems? section 5.1.2 - procedure step 2 - what inserts the timestamp in the response message (R1)? section 5.1.2 - the ID says "successful messages exchanged" - might it be useful to report the % of unsuccessful exchanges? section 5.1.6 - the title & objective description do not match - the title says "rate" but the objective is a count - the test seems to try to get the rate section 5.1.7 - same issue as section 5.1.6 section 5.2.3 - procedure step 1 - this reads as if the addresses are unique but unchanging - if that is not what is meant then it should specifically say that the addresses are changing section 5.3.1 - it might be useful to establish specific "invalid messages" since I could imagine that the devices could handle different types in different ways that could have an impact on speed of handling section 5.3.2 - "a huge number" is somewhat undefined do you mean to say TCP SYN messages rather than TCP SYNC messages? section 6 - RFC 2026 is referenced in the introduction but not listed in the references section 9 - I have now retired so no affiliation should be listed > -----Original Message----- > From: bmwg [mailto:bmwg-bounces@xxxxxxxx] On Behalf Of Scott Bradner > Sent: Monday, January 29, 2018 9:17 PM > To: ops-dir@xxxxxxxx > Cc: ietf@xxxxxxxx; bmwg@xxxxxxxx; draft-ietf-bmwg-sdn-controller- > benchmark-meth.all@xxxxxxxx > Subject: [bmwg] Opsdir last call review of draft-ietf-bmwg-sdn- > controller-benchmark-meth-07 > > Reviewer: Scott Bradner > Review result: Has Issues > > This ID specifies the methodology to be used in testing the performance > of SDN > controllers. It is a standard BMWG methodology document and thus, cannot > have > any impact, operational or otherwise, on operational networks - see RFC > 6815 > > follow are some comments on the document itself > > section 4.1 - Leaf-Spine used but not defined > section 4.4 - using old software versions seems to add complexity with > little > benefit section 4.7 - would be useful to say that the variance over the > repeated tests should be reported in the "test reporting" section - > e.g., in > section 5.1.1 I would add a row that reports the variance - same for all > tests > section 5.1.1 - is the topology discovery process slow enough that a 3 > second > granularity of the measurement will show differences between systems? > section > 5.1.2 - procedure step 2 - what inserts the timestamp in the response > message > (R1)? section 5.1.2 - the ID says "successful messages exchanged" - > might it be > useful to report the % of unsuccessful exchanges? section 5.1.6 - the > title & > objective description do not match - the title says "rate" but the > objective is > a count - the test seems to try to get the rate section 5.1.7 - same > issue as > section 5.1.6 section 5.2.3 - procedure step 1 - this reads as if the > addresses > are unique but unchanging - if that is not what is meant then it should > specifically say that the addresses are changing section 5.3.1 - it > might be > useful to establish specific "invalid messages" since I could imagine > that the > devices could handle different types in different ways that could have > an > impact on speed of handling section 5.3.2 - "a huge number" is somewhat > undefined > do you mean to say TCP SYN messages rather than TCP SYNC > messages? > section 6 - RFC 2026 is referenced in the introduction but not listed in > the > references section 9 - I have now retired so no affiliation should be > listed > > > _______________________________________________ > bmwg mailing list > bmwg@xxxxxxxx > https://urldefense.proofpoint.com/v2/url?u=https- > 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwICAg&c=LFYZ- > o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Oip5- > 2t1wtJ1ZpUIlQWQcvgNAVgydAS4jCoQi_LL50Q&s=WP7X4VUxjR59g9gvjUFj_8zTQrz- > iKc2jY9FOaPTlZY&e=