Bob,
Thank you very much for reviewing
the draft and provided in-depth
comments. I am very sorry for the
delayed response due to traveling.
Replies to your comments are
inserted below marked by [Linda]:
Reviewer: Bob Briscoe
Review result: Not Ready
I have been selected as the
Transport Directorate reviewer for
this draft. The Transport Directorate
seeks to review all transport or
transport-related drafts as they pass
through IETF last call and IESG
review, and sometimes on special
request. The purpose of the review is
to provide assistance to the Transport
ADs. For more information about the
Transport Directorate Reviews and the
Transport Area Review Team, please see
https://trac..ietf.org/trac/tsv/wiki/TSV-Directorate-Reviews
In this case, very very few of the
review comments relate to transport
issues, although the greatest issue
concerns a desire that the network
could pause or stop connections during
L3 VM Mobility, which is certainly a
transport issue.
[Linda] There is “Hot Migration”
with transport service continuing, and
there is a “Cold Migration”, which is
a common practice in many data
centers, which stop the task running
on the old place and move to the new
place before restart as described in
the Task Migration.
Is it helpful to add this
description to the draft?
==Summary==
The technical aspects of the draft
concerning L2 VM mobility (within a
subnet) seem sound. However, this is
only part of the draft, which has the
following
issues:
#. The introduction does not say
what the purpose of publishing this
draft is.
It seems that, rather than
describing a specific protocol or
protocols, it intends to describe the
overall system procedure that would
typically be used in DCs for VM
mobility. It is tagged as a BCP, but
it does not say who needs this BCP,
why it is useful for the IETF to
publish this BCP, how wide the
authors' knowledge is of current
practice (given DCs are private), or
why this is a BCP rather than a
protocol spec.
[Linda] The first paragraph on Page
3 has the description why VM Mobility
is needed. Is it helpful to move this
paragraph to the beginning of the
Introduction Section?
“Virtualization
which is being used in almost all
of today’s data
centers enables
many virtual machines to run on a
single physical
computer or
compute server. Virtual machines
(VM) need hypervisor
running on the
physical compute server to provide
them shared
processor/memory/storage.
Network connectivity is provided
by the
network
virtualization edge (NVE)
[RFC8014]. Being able to move VMs
dynamically, or
live migration, from one server to
another allows for
dynamic load
balancing or work distribution and
thus it is a highly
desirable feature
[RFC7364].”
The draft starts out (S.3) as if it
intends to say what a good VM Mobility
protocol should or shouldn't do, but
the rest of the document doesn't give
any reasoning for these
recommendations, it just asserts what
appears to be one view of how a whole
VM Mobility system works, sometimes
referring to one example protocol RFC
for a component part, but more often
with no references or details.
[Linda] Is it helpful to move the
paragraph above to the beginning of
the Introduction Section? So that
audience is aware of why VM Mobility
is needed. And then follow up with
what a good VM Mobility protocol
should or shouldn't do?
#. It does not seem as if the NVO
WG has discussed the purpose of using
normative text in this draft. See
detailed comments.
[Linda] The “Intended status” of
the draft is “Best Current Practice”.
So all the text are not “normative”.
Is it Okay?
#. The draft silently slips back
and forth between VM mobility and VM
redundancy, without recognizing the
differences. See detailed comments.
[Linda] There is only one usage of
“redundancy” in the entire document,
used under the context of “Hot standby
option”, indicating the “redundancy”
of “the VMs in both primary and
secondary domains have identical
information and can provide services
simultaneously as in load-share mode
of operation” being expensive.
#. Please adopt different
terminology than "source NVE" and
"destination NVE", which are really
poor choices of terms for an
intermediate node. See detailed
comments. Why not use "old NVE" and
"new NVE", which is what you mean?
[Linda] Thanks for the suggestion.
We will change to “Old NVE”, and “new
NVE”.
#. Applicability is fairly clearly
outlined, but it is not clear whether
hosts corresponding with the mobile
VMs are part of the same controlled
environment or on the uncontrolled
public Internet. See detailed
comments.
[Linda] “Hosts” are the App running
on the VM. It is the under the same
controlled environment. Not on
uncontrolled public internet.
#. Section 4.2.1 on L3 VM mobility
reads like some potential
half-thought-through ideas on how to
solve L3 mobility, rather than current
practice, let alone best current
practice. Either current practice
should be described instead, or the
scope of the draft should be narrowed
solely to L2 VM mobility. See detailed
comments.
[Linda] This is refereeing to “Cold
Migration”, which is a common practice
in many data centers.
# The VM's file system is described
as state that moves with the VM (S.6),
but VM mobility solutions often move
the VM but stitch it back to its
(unmoved) storage. Conversely, the
storage can also move independent of
the VM.
[Linda] It depends. When a VM move
to a different zone, the storage/file
can becomes inaccessible.
#. The draft omits some of the
security, transport and management
aspects of VM mobility. See detailed
comments.
[Linda] Can you provide some text?
#. The draft reads as if different
sections have been written by
different authors and no-one has
edited the whole to give it a coherent
structure, or to ensure consistency
(both technical and editorial) between
the parts. See detailed comments.
[Linda] we can improve.
#. The quality of the English
grammar does not allow a reviewer to
concentrate on the technical aspects
rather than the English. It would have
been useful if one of the
English-speaking co-authors had
improved the English before submission
for review. See detailed comments.
[Linda] can you help? Becoming a
co-author to improve?
==Detailed Comments==
===#. Normative statements===
In the body of the document, there
is just one occurrence of normative
text (actually two "MUST"s, but both
state a common requirement - just
written separately for IPv4 and IPv6).
This merely serves to imply that
everything else the document says is
less important or optional, which was
probably not the intention.
[Linda] The goal is to indicate any
solution in moving the VM “MUST”
follow this rule. They make sense,
aren’t they?
At the start there is a
requirements section, which states
what a VM Mobility protocol "SHOULD"
or "SHOULD NOT" do. I think this is
intended as a set of goals for the
rest of the document. If so, these
"SHOULDs" are not intended to apply to
implementations, so they ought not to
be capitalized.
[Linda] okay, will change.
The first requirement, "Data center
network SHOULD support virtual machine
mobility in IPv6", is written as a
requirement on all DC networks, not on
implementations. I assume this was
intended to read as "Data center
network virtual machine mobility
protocols SHOULD support IPv6". Even
then, it doesn't really add anything
to say VM mobility should support v6
and it should support v4. A L2
solution won't. While undoubtedly, a
L3 solution will at least support one
of them.
[Linda]Agree. Will change it to
“Data center that support IPv6 address
should …”
I'm not sure that 'protocol' is the
right word anyway; I think 'VM
Mobility procedure' would be a better
phrase, because it includes steps such
as suspending the VM, which is more
than a protocol.
[Linda] yes. Will change to
“Procedure”.
The requirement "Virtual machine
mobility protocol MAY support host
routes to accomplish virtualization",
is not followed up at all in the rest
of the draft.
Even if this requirement stays, the
last 3 words should be deleted.
[Linda] will change to “Host Route
can be used to support the Virtual
Machine Mobility Procedure.”
By the end of the draft, the
solution falls far short of the most
relevant "Requirements" anyway, so one
assumes the title of the section ought
to have been "Goals". Specifically,
even in the simpler case of L2 VM
mobility, S.4.1 says that triangular
routing and tunnelling persist "until
a neighbour cache entry times out". A
cache timeout is about 10 orders of
magnitude longer than the requirement
to only persist "while handling
packets in flight", which would be a
few milliseconds at most (the time for
packets to clear the network that were
already launched into flight when the
old VM stopped).
Whatever, it would be preferable
for the draft to give rationale for
these requirements, rather than just
assert them. This would help to shed
light on the merits of the different
trade offs that solutions choose.
[Linda] Agree, will add.
===#. Mobility vs. Redundancy===
Redundancy and mobility have a lot
of similarities, but they have
different goals. With mobility, it is
necessary to know the exact instant
when one set of state is identical to
the other so it can hand over. With
redundancy, the aim is to keep two (or
more) sets of state evolving through
the same sequence of changes, but
there is no need to know the point at
which one is the same as the other was
at a certain point.
[Linda] Agree with what you said.
There is only one usage of
“redundancy” in the entire document,
used under the context of “Hot standby
option”, indicating the “redundancy”
of “the VMs in both primary and
secondary domains have identical
information and can provide services
simultaneously as in load-share mode
of operation” being expensive.
The draft slips from mobility to
resilience in the following places:
* S.2. Terminology: Warm VM
Mobility is defined without any
ending, as if it is permanent
replication. * S.7. "Handling of Hot,
Warm and Cold Virtual Machine
Mobility" is actually all about
redundancy, and doesn't address
mobility explicitly.
[Linda] Will add the definition
“Hot Migration”, “cold migration”, and
“warm migration”.
===#. Terminology===
Packets run from the source at A to
the destination at B via NVE1, then
via NVE2. Please don't call NVE1 and
NVE2 the source NVE and the
destination NVE.
In future, no-one will thank you
for the apparent contradictions when
they continually stumble over phrases
like this one in S.4.1: "...send their
packets to the source NVE"..
The term "packets in flight" is
used incorrectly to refer to all the
packets sent to the old NVE after the
VM has moved, even if they were
launched into flight long after the
old VM stopped receiving packets.
[Linda] thank for the comments.
Will change.
BTW, I think s/before/after/ in:
"that have old ARP or neighbor cache
entry before VM or task migration".
I think: s/IP-based VM mobility/L3
VM mobility/ throughout, because
"based"
sounds (to me) like the mobility
control protocol is over (i.e. based
on) IP.
===#. Applicability===
In section 4.2 it says that the
protocol mostly used as the IP based
task migration protocol is ILA. This
implies that all hosts corresponding
with the mobile VMs are either part of
the same controlled environment, or
they are proxied via nodes that are
part of the same controlled
environment (I only have passing
knowledge of ILA, but I understand
that it depends on ILA routers on the
path). If I am correct, this aspect of
scope needs to be made clear from the
start.
Also under the heading of
applicabiliy, the sentence "Since
migrations should be relatively rare
events" appears very late in the
document (S.4.2.1). The assumed level
of churn ought to be stated nearer the
start.
[Linda] yes, under the same
controlled environment.
===#. L3 Mobility===
L2 VM mobility is independent of
the application, because resolution of
L2 mappings is delegated to the stack.
In contrast, L3 VM mobility is only
feasible under certain conditions,
because an application needs an IP
address to open a socket (resolution
of DNS names is not delegated to the
stack, and apps can use IP addresses
directly anyway).
Examples of the 'certain
conditions':
a) /All/ applications used in the
whole DC load balancing scheme contain
IP address migration logic for /all/
their connections; b) VMs running
solely applications that support IP
address migration register this fact
with the NVA, and it only select such
VMs for mobility. c) An abstraction is
layered over /all/ the IP addresses
exposed to applications (at both ends)
so that the IP addresses that
applications use are solely
identifiers (e.g. ILA, LISP, HIP),
not also locators.
The introduction says the draft is
about VM mobility in a multi-tenant
DC, so the DC admin will not know the
range of applications being used. This
excludes condition (a) above. When the
draft says "...if all applications
running are known to handle this
gracefully...", it doesn't quantify
just how restrictive this condition
is, and it gives no explanation of how
this knowledge might be 'known' or
which function within the system
'knows' it.
S.4.2.1 contains what seems like
plenty of arm-waving.
* "TCP connections could be
automatically closed in the network
stack during a migration event."
o There is no TCP
connection state in the network stack.
o Even if the network
starts to drop every packet, the TCP
connection
state persists in the
end-points for a duration of the order
of 30-90
minutes (OS-dependent)
before TCP deems the connection is
broken. o
Other transport protocols
have similar designs (including the
app-layer
of protocols over UDP).
* "More involved approach to
connection migration":
o pausing the connection
[does this refer to an actual feature
of any
L4 protocol?] o packaging
connection state and sending to target
[does
this assume logic written
into the application, or is this
assuming the
stack handles this and the
app is restricted to using some form
of
separate identifier/locator
addresses?] o instantiating connection
state in the peer stack
[ditto?].
There's some arm-waving in S.7 too:
"Cold Virtual Machine mobility is
facilitated by the VM initially
sending an ARP or Neighbor
Discovery message at the destination
NVE
but the source NVE not receiving
any packets inflight."
[How is it arranged for the
source NVE not to receive any packets
in flight?]
And in S.7:
"In hot
standby option, regarding TCP
connections, one option is to start
with and maintain TCP
connections to two different VMs at
the same
time."
[This sounds like resilience
logic has been written into the
application,
which would be a special case
but not something VM mobility
infrastructure
could depend on.]
[Linda] will add.
===#. Gaps===
#. Security Considerations: repeats
issues in other drafts that are not
specific to mobility, but it does not
mention any security issues
specifically due to VM mobility. It
says that address spoofing may arise
in a DC (sort-of implying it is worse
than in non-DC environments, but not
saying why). The handshake at the
start of a connection (e.g. TCP, SCTP,
QUIC) checks for source address
spoofing. So L3 VM mobility would be
more vulnerable to source address
spoofing in cases where the mobile VM
was the connection initiator and there
was not a new handshake after the
move. However, this draft does not
contain any detailed mobility
protocols, so it is not possible to
identify any specific security flaws.
#. Transport Issues: Effect of
delay on the transport: Cold mobility
introduces significant delay, and
other forms less, but still some
delay. It should be pointed out that
some applications (e.g. real-time)
will therefore not be useful if
subjected to VM mobility. Similarly,
even a short period of delay will
drive most congestion controls to
severely reduce throughput. These
points might be self-evident, but
perhaps they should be stated
explicitly.
BTW, in the L3 VM mobility case,
the draft often refers to TCP
connections, but the address bindings
of any transport protocols would have
to be migrated due to VM mobility
(e.g. SCTP; sequences of datagrams
over UDP; streams over UDP such as
with RTP, QUIC).
#. Management Issues: perhaps the
draft ought to recommend statistics
gathering (e.g. time taken, amount of
duplicate data) to aid a DC's future
decisions on the cost-benefit of
moving a VM. The OPSDIR review says a
BCP does not /have/ to describe
management issues, but this document
seems to describe a whole system
procedure, not just a protocol, which
then surely includes the management
plane.
[Linda] can you become a co-author
and add those in?
===#. Incoherent Structure===
S.4.1. happens to talk about VMs
moving, while S.4.2. happens to talk
about tasks moving, but this is not
the distinguishing aspect of these two
sections (anyway, S.2. says "the draft
uses task and VM interchangeably"): *
"4.1 VM Migration" is about "L2 VM
Mobility" so this ought to be the
section heading, *
"4.2 Task Migration" is about "L3
VM Mobility" so this ought to be the
section heading. It would also help
not to switch from VM to task across
these sections
- it's just a distraction.
S.4.1 needs better signposting of
where each sub-case ends (Subsections
might be useful to solve this): * IPv4
* end-user client * 2 paras starting
"All NVEs communicating with this
virtual machine..." [Not clear that
the end-user case has ended and we
have returned to the general IPv4
case?] * IPv6 [Strictly, it still
hasn't said whether the end-user
client case has ended.] [Also, it
doesn't explain why there is no need
for an end-user client case under
IPv6?] Sections 5 & 6 seem to be
about either L2 or L3 mobility,
whereas Sections 7 &
8 seem to be restricted to L2.
The draft vacillates over what to
do with packets arriving at the old
NVE in the L3 case (see also L3
mobility above): * S4.2 first says
packets are dropped, possibly with an
ICMP error message;
o then later it says they are
silently dropped;
o then in the very next sentence
it says either silently drop them or
forward
them to the new location
* S.5 says they should not be lost,
but instead delivered to the
destination hypervisor
o then it describes how they are
tunnelled (which is not the same as
"forwarding").
The order in which all the stages
of mobilty are given is jumbled up
across sections that also appear in
arbitrary order: * S.5 prepares,
establishes uses then stops a tunnel,
but it doesn't say where the other
stages fit between these steps
o When tunneling packets,
it talks about the *migrating* VM not
the
*migrated* VM, which
implies tunnelling has started before
the new VM
is running. Does this imply
there is a huge buffer? o It says
"Stop
Tunneling Packets - When
source NVE stops receiving packets
destined
to..." but it is never
clear when a source has stopped
sending packets
to a destination, unless it
explicitly closes the connection (e.g.
with
a FIN in the case of TCP).
Often there are long gaps between
packets,
because many flows are
'thin' (meaning the application
frequently has
nothing to send). These
gaps can last for milliseconds, hours
or even
days without any
implication that the connection has
ended.
* Then S.6. describes moving state,
but doesn't say that this is not after
the previous tunnelling steps (or
where it fits within those steps). *
Then S.7 describes hot, warm and cold
mobility, but doesn't lay out the
tunnelling or steps to move state in
each case. * Then S.8 says it's about
VM life-cycle, but just gives the very
first 3 steps for allocation of
resources to a VM, then abruptly ends,
without even starting the VM, let
alone getting to move it.
S.5 exhibits another inconsistency
by talking about the hypervisor, not
the NVE.
==#. Nits==
Nits with the English are too
numerous to mention them all. Below
are pointers to general problems as
well as some individual instances.
S.4
"Layer 2 and Layer 3 protocols
are described next. In the following
sections, we examine more
advanced features."
s/following/subsequent/
S.4.1
Expand WSC, MSC and NVA on first
use.
s/the VM moves in the same link/the
VM moves in the same subnet/
"i.e. end-user clients ask for the
same MAC address upon migration. [...]
to ensure that the same IPv4 address
is assigned to the VM." I think
s/IPv4/MAC/ was intended?
" All NVEs communicating with this
virtual machine uses the old ARP
entry. If any VM in those NVEs
need to talk to the new VM in the
destination NVE, it uses the old
ARP entry."
Repetition: these 2 sentences say
the same. (The mistake is also
repeated when these 2 sentences are
repeated for IPv6).
S.4.2.1
s/Push the new mapping to
hosts./Push the new mapping to
communicating hosts./
S.5.
The IPv4/IPv6 pairs of paras for
"tunnel estabilshment" and "tunneling
packets"
only differ in the words
"IPv4"/"IPv6". So in each case a
single para could be given for IP
(irrespective of whether v4 or v6).
Thank you very much.
Linda Dunbar
