Greg,
Thanks for considering my review comments. My responses to specific points are found below.
Kind regards,
-Peter
From: Greg Mirsky <gregimirsky@xxxxxxxxx>
Sent: Wednesday, October 11, 2023 1:54 AM
To: Peter Yee <peter@xxxxxxxxxx>
Cc: gen-art <gen-art@xxxxxxxx>; draft-ietf-ippm-pam.all@xxxxxxxx; IETF IPPM WG <ippm@xxxxxxxx>; Last Call <last-call@xxxxxxxx>
Subject: Re: Genart last call review of draft-ietf-ippm-pam-06
Hi Peter,
thank you for your thorough review and detailed comments. Please find my notes below under the GIM>> tag. I've attached the diff highlighting all the updates.
Regards,
Greg
On Tue, Oct 10, 2023 at 5:42 AM Peter Yee via Datatracker <noreply@xxxxxxxx> wrote:
Reviewer: Peter Yee
Review result: Ready with Issues
I am the assigned Gen-ART reviewer for this draft. The General Area
Review Team (Gen-ART) reviews all IETF documents being processed
by the IESG for the IETF Chair. Please treat these comments just
like any other last call comments.
For more information, please see the FAQ at
<https://wiki.ietf.org/en/group/gen/GenArtFAQ>.
Document: draft-ietf-ippm-pam-06
Reviewer: Peter Yee
Review Date: 2023-10-09
IETF LC End Date: 2023-10-20
IESG Telechat date: Not scheduled for a telechat
Summary: This specification offers a scheme for metrics that are used in
determining if what you’re getting on a network is what you contracted for. I
have some issues with the draft and the usual nits. [Ready with issues]
Major issues: None
Minor issues:
General: For something that's Standards Track, I find the heavy use of "can",
"could", and “might” leaving me wondering if this specification should be
Informational instead. It all seems a bit loose. You could do this or that, if
you wanted… I get that the specific metrics are defined here, but even so,
there's not a lot of meat on the bone.GIM>> As the result of the discussion with our AD Martin Duke, the -07 version of the document was set onto the Informational track. In your opinion, the wording you've pointed out could be appropriate for an informational document?
PEY> The wording used and the lack of specificity made me think the document was useful conceptually, giving guidance to the general idea of PAMs but not something from which I would directly drive an implementation.
Page 4, 1st full paragraph, 2nd sentence: I've not followed this work, so
pardon me if I'm repeating a previous argument. However, I would find terms
like "compliance", "compliant", and "non-compliant" (maybe "incompliant")
preferable over "availability", "available", and "unavailable". Sure, you get
PCM for an initialism instead of a nice acronym like PAM, but "available" also
has a common meaning that makes it less desirable, in my opinion. Compliance is
already used in the previous paragraph and seems like it could be substituted
without a lot of drama. Of course, this will also affect section 3.3, should
you choose the change the terms.GIM>> We, the authors, over the course of developing the document, discussed the terminology among ourselves,e.g., the use of "conformance" and "compliance", and with the WG. Current terminology, as I understand it, is consistent and reflects the consensus of the WG.
PEY> I’m completely fine with that and realize that I lack the knowledge of he discussions that went on before.
Page 8, 2nd paragraph, 3rd sentence: I don’t find availability based on
successive intervals all that good a way of making that determination. A
service with an unavailability threshold set would be deemed "available" if it
had a pattern of "unavailability threshold" - 1 interval violations followed by
one VFI, rinse and repeat. Might you not want something like a percentage of
VIs over a larger window of time instead? I realize this a couched in
permissive language, but then the following text runs with the idea that
successive intervals is the way to go. This also goes back to my general issue
above about the looseness of this specification.GIM>> Thank you for the question. Violated Interval Ratio is one of the PAM metrics that could be defined as noted in the last paragraph of Section 3.3. That and other metrics can be used to improve the understanding of the state of a service regarding the defined SLO thresholds.
PEY> Thanks for that answer. I think the language in that paragraph also reinforces my feeling that this draft is better as an Informational specification.
Nits/editorial comments:
Page 2, center header: I'm not sure how “PAM for Multi-SLO” applies as a
compression of the title. “Multi-SLO” never appears in any other context than
the header of the document.GIM>> The intention of the shortened title was to highlight that PAM can be particularly useful for services governed by more than one SLO, multiple SLOs, as in the full title of the document - "Precision Availability Metrics for Services Governed by Service Level Objectives (SLOs)". If you find the short title not reflecting the intent of the document, could you kindly suggest an alternative?
PEY> Touché. I’m not entirely sure what I would use. It seemed to me that the document wasn’t really emphasizing the multiplicity of SLOs, although it certainly does use the plural for several terms such as parameter or SLOs. On the other hand, it also works were there a single SLO being measured. When reading the document, I wasn’t struck by a need for there to be multiple SLOs or that there was any difference in whether one or multiple SLOs were being measured. The SLOs seem to be independent variables (at least at the high level of this document), so I was surprised at the emphasis on the multiple part in the short title. Perhaps, “Guidelines for PAMs”?
Page 4, 1st full paragraph, 3rd sentence: delete “, and which” along with
“therefore” and if that parses better. Otherwise, the sentence feels like a
phrase.GIM>> That text has been edited in the current version -07:
OLD TEXT in -06:
"Precision" refers to the fact that services whose end-to-end service
levels are governed by SLOs, and which must therefore be precisely
delivered according to the associated quality and performance
requirements.
NEW TEXT in -07:
"Precision" refers to services whose end-to-end service levels are
governed by SLOs and must be delivered precisely according to the
associated quality and performance requirements.
Would you find the current version clearer and acceptable?
PEY> Yes, that helps.
Page 4, 2nd full paragraph, 1st sentence: change “is” to “are”.
GIM>> Agreed. Also s/a separate topic/separate topics/ Right?
PEY> Indeed!
Page 5, section 3.1, 1st paragraph, 3rd sentence: “second” is given of an
example of a different granularity, but that’s the granularity given in the
second sentence, so it doesn’t make a good example of a different granularity.
Decamillisecond is fine.GIM>> Removed "second or". Is that acceptable?
PEY> Yes.
Page 6, definitions for “VPC” and “SVPC”: How is a severely VPC different from
a VPC?GIM>> Although, the applicability of these two metrics is the same, they reflect aspects of different severity.
PEY> Okay.
It’s also not clear to me from the text that the performance parameters
for VI/SVI apply to individual packets, so how is this count generated? Only
from parameters that can be applied on a per-packet basis (so not things like
jitter)?GIM>> Indeed, some listed metrics in this list provided as examples of what could be used. Our intention is to define new metrics in the future, in a separate standard document. Do you see that as a reasonable approach for the informational document or would suggest another path forward?
PEY> That approach is fine to me.
Page 7, 6th and 7th bullet items: I’d change “measurement interval” to
“measurement session” (session was used previously) or “period”. Otherwise, the
term “interval” becomes unnecessarily overloaded.GIM>> I think that using "interval" in these sentences is intentional to reference VI, SVI, and VFI. Could you kindly suggest how we can make it clearer?
PEY> My argument would be for “measurement session”, as used in the first paragraph of 3.1. Interval is used here as the period for a VI, SVI, or VFI. I see the session as the period covering all of the intervals.
Id-nits says:
== There is 1 instance of lines with non-ascii characters in the document.
I’m not sure where that character is and I admit to not searching for it.GIM>> It appears that the warning is triggered by 'ø' in the following in Acknowledgment:
The authors greatly appreciate review and comments by Bjørn Ivar
I hope that this is tolerable.
PEY> Completely. I just couldn’t easily find (on the particular OS I was using) what was causing idnits to complain.
<<< text/html; charset="US-ASCII"; name="draft-ietf-ippm-pam-08.diff.html": Unrecognized >>>
-- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
