I'll send a separate email (in a new thread, so it doesn't get lost! ;-)
with a new draft of what the network XML should look like, but wanted to
respond to Dan's comments inline...
On 05/24/2011 10:21 AM, Daniel P. Berrange wrote:
On Fri, Apr 29, 2011 at 04:12:55PM -0400, Laine Stump wrote:
Okay, here's a brief description of what I *think* will work. I'll
build up the RNG based on this pseudo-xml:
For the<interface> definition in the guest XML, the main change
will be that<source .. mode='something'> will be valid (but
optional) when interface type='network' - in this case, it will just
be used to match against the source mode of the network on the host.
<virtualport> will also become valid for type='network', and will
serve two purposes:
1) if there is a mismatch with the virtualport on the host network,
the migrate/start will fail.
2) It will be ORed with<virtualport> on the host network to arrive
at the virtualport settings actually used.
For example:
<interface type='network'>
<source network='red-network' mode='vepa'/>
IMHO having a 'mode' here is throwing away the main reason for
using type=network in the first place - namely independance
from this host config element.
I agree, but was being accommodating :-) Since then, Dave has pointed
out that the same functionality can be achieved by having the management
application grab the XML for the network on the targetted host, and
check for matches of any important parameters before deciding to migrate
to that host. This has 2 advantages:
1) It is more flexible. The management application can check for more
than just mode='vepa', but also any number of other attributes of the
network on the target.
2) The result of a host's network not matching the desired mode will be
"management app looks elsewhere", rather than "migration fails".
The management application will need to do this anyway (even if just to
check that the given network is present at all) or, again, face the
prospect of the migration failing.
So I'll withdraw this piece from the next draft.
<virtualport type='802.1Qbg'>
<parameters instanceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
</virtualport>
<mac address='xx:xx:.....'/>
</interface>
(NB: if "mode" isn't specified, and the host network is actually a
bridge or virtual network, the contents of virtualport will be
ignored.)
<network> will be expanded by giving it an optional "type" attribute
(which will default to 'virtual'),<source> subelement, and
<virtualport> subelement. When type='bridge', you can specify source
exactly as you would in a domain<interface> definition:
<network type='bridge'>
<name>red-network</name>
<source bridge='br0'/>
</network>
When type='direct', again you can specify source and virtualport
pretty much as you would in an interface definition:
<network type='direct'>
<name>red-network</name>
<source dev='eth0' mode='vepa'/>
<virtualport type='802.1Qbg'>
<parameters managerid="11" typeid="1193047" typeidversion="2"
instanceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
</virtualport>
</network>
None of this really feels right to me. With this proposed
schema, there is basically nothing in common between the
existing functionality for<network> and this new functionality
except for the<name> and<uuid> elements.
Apps which know how to deal with existing<network> schema
will have no ability to interpret this new data at all.
Quite probably they will mis-interpet it as providing an
isolated virtual network, with no IP addr set, since this
design isn't actually changing any attribute value that
they currently look for.
Either we need to make this align with the existing schema,
or we need to put this under a completely separate set of
APIs. I think we can likely do better with the schema design
and achieve the former.
So the problem is that the new uses are so orthogonal to the current
usage that existing management apps encountering this new XML will
mistakenly believe that it's "old" XML with a bit of extra stuff that
can be ignored (thus leading to mayhem).
I think the most important thing is to make sure that a config for one
of these new types will have at least one change to an *existing*
element/attribute (mine just added a *new* attribute specifying type)
that causes existing apps to realize this isn't just an old school
network definition that happens to have a few kinks on the side. Your
suggestion of using new values for <forward mode="..."> seems like as
good an idea as any (actually I can't think of anything else that works
as well :-)
However, dev would be optional - if not specified, we would expect a
pool of interfaces to be defined within source, eg:
<network type='direct'>
<name>red-network</name>
<source mode='vepa'>
<pool>
<interface name='eth10' maxConnect='1'/>
<interface name='eth11' maxConnect='1'/>
<interface name='eth12' maxConnect='1'/>
<interface name='eth13' maxConnect='1'/>
<interface name='eth14' maxConnect='1'/>
<interface name='eth25' maxConnect='5'/>
</pool>
</source>
<virtualport ...... />
</network>
I don't really like the fact that this design has special
cased the num(intefaces) == 1 case to have a completely
different XML schema. eg we have this:
<source dev='eth0' mode='vepa'/>
And this
<source mode='vepa'>
<pool>
<interface name='eth10' maxConnect='1'/>
</pool>
both meaning the same thing. There should only be one
representation in the schema for this kind of thing.
BTW, for all the people asking about sectunnel, openvswitch, and vde
- can you see how those would fit in with this? In particular, do
you see any conflicts? (It's easy to add more stuff on later if
something is just missing, but much more problematic if I put
something in that is just plain wrong).
As mentioned above, I think this design is wrong, because it is not
taking any account of the current schema for<network> which defines
the various routed modes.
Currently<network> supports 3 connectivity modes
- Non-routed network, separate subnet (no<forward> element present)
- Routed network, separate subnet with NAT (<forward mode='nat'/>)
- Routed network, separate subnet (<forward mode='route'/>)
Following on from this, I can see another couple of routed modes
- Routed network, IP subnetting
- Routed network, separate subnet with VPN
And the core goal here is to replae type=bridge and type=direct from the
domain XML, which means we're adding several bridging modes
- Bridged network, eth + bridge + tap (akin to type=bridge)
- Bridged network, eth + macvtap (akin to type=direct)
- Bridged network, sriov eth + bridge + tap (akin to type=bridge)
- Bridged network, sriov eth + macvtap (akin to type=direct)
The macvtap can be in 4 modes, so perhaps it is probably better to
consider them separately
- Bridged network, eth + bridge + tap
- Bridged network, eth + macvtap + vepa
- Bridged network, eth + macvtap + private
- Bridged network, eth + macvtap + passthrough
- Bridged network, eth + macvtap + bridge
- Bridged network, sriov eth + bridge + tap
- Bridged network, sriov eth + macvtap + vepa
- Bridged network, sriov eth + macvtap + private
- Bridged network, sriov eth + macvtap + passthrough
- Bridged network, sriov eth + macvtap + bridge
I can also perhaps imagine another VPN mode:
- Bridged network, with VPN
The current routed modes can route to anywhere, or be restricted to
a particular network interface eg with<forward dev='eth0'/>. It
only allows for a single interface, though even for routed modes it
could be desirable to list multiple devs.
The other big distinction is that the<network> modes which do routing,
include interface configuration data (ie the IP addrs& bridge name)
which is configured on the fly. It looks like with the bridged modes,
you're assuming the app has statically configured the interfaces via
the virInterface APIs already, and this just points to an existing
configured interface. This isn't neccessarily a bad thing, just an
observation of a significant difference.
Right. Perhaps later it can be expanded (at least in some of the modes)
to setup these devices when the network is started, but right now the
network definition is just used to point to something that already
exists and is functioning.
So if we ignore the<ip> and<domain> elements from the current<network>
schema, then there are a handful of others which we need to have a plan
for
<forward mode='nat|route'/> (omitted completely for isolated networks)
<bridge name="virbr0" /> (auto-generated/filled if omitted)
<mac address='....'/> (auto-generated/filled if omitted)
The<forward> element can have an optional dev= attribute.
I think the key attribute is the<forward> mode= attribute. I think we
should be adding further values to that attribute for the new network
modes we want to support. We should also make use of the dev= attribute
on<forward> where practical, and/or extend it.
We could expand the list of<foward> mode values in a flat list
- route
- nat
- bridge (brctl)
- vepa
- private
- passthru
- bridge (macvtap)
NB: really need to avoid using 'bridge' in terminology, since all
5 of the last options are really 'bridge'.
Or we could introduce a extra attribute, and have a 2 level list
-<forward layer='link'/> (for all ethernet layer bridging)
Does that gain us anything, though? While it's correct information, it
seems redundant (the layer can always be implied from the mode).
-<forward layer='network'/> (for all IP layer bridging aka routing)
So the current modes would be
<forward layer='network' mode='route|nat'/>
And new bridging modes would be
<forward layer='link' mode='bridge-brctl|vepa|private|passthru|bridge-macvtap'/>
For the brctl/macvtap modes, the dev= attribute on<forward> could point to
the NIC being used, while with brctl modes,<bridge> would also be present.
Are you saying that in the case of a brctl mode, it would be required to
fill in both of these?
<forward mode="bridge-brctl" dev="br0" .../>
<bridge name="br0" .../>
I think I would prefer to only use the one in <forward>. Are you
suggesting putting it there to help older management apps cope with the
new modes? I don't really think it would help; it's really just an
accident of implementation that the device in "bridge-brctl" mode
happens to be a bridge device.
In the SRIOV case, we potentiallly need a list of interfaces. For this we
probably want to use
BTW, just to clarify, when you say "SRIOV", what you really mean is "any
situation where there are multiple network interface devices connected
to the same physical network, and identical connectivity to the guest
could be provided by any one of these devices". In other words, it
doesn't need to be an SRIOV ethernet card with multiple virtual
functions, it could also be an older style setup with multiple physical
cards, or multiple complete devices on a single card.
<forward dev='eth0'>
<interface dev='eth0'/>
<interface dev='eth1'/>
<interface dev='eth2'/>
...
</forward>
NB, the first interface is always to be listed both as a dev= attribute
(for compat with existing apps) *and* as a child<interface> element (for
apps knowing the new schema).
But since the pool of devices would only ever be used in one of the new
forward modes, which an existing app wouldn't understand anyway, would
that really buy us anything?
The maxConnect= attribute from your examples above is an interesting
thing. I'm not sure whether that is neccessarily a good idea. It feels
similar to VMWare's "port group" idea, but I don't think having a
simple 'maxConnect=' attribute is sufficient to let us represent the
vmware port group idea. I think we might need an more explicit
element eg
<portgroup count='5'>
<interface dev='eth2'/>
</portgroup>
eg, so this associates a port group which allows 5 clients (VM NICs)
with the uplink provided by eth2 (which is assumed to be listed
under<forward>).
I've thought about this a bit, and I think portgroup is a good idea, but
I don't think the name of the device being used fits there. portgroup is
a good place to put information about the characteristics of a set of
connections, but which device to use is a backend implementation detail,
and there isn't necessarily a 1:1 correspondence between the two.
portgroup would be used, for example, to configure bandwidth (that's
pretty much all VMWare uses it for, plus a blob of "vendor-specific"
data), and the guest interface XML would specify which portgroup a guest
was going to belong to - if you also set which physical device to use
based on portgroup, that would leave the guest XML specifying which
physical device to use, which is what we're trying to get away from.
(and also it would mean that each physical device would need its own
portgroup, which I don't think we want.
Thinking more about the maxCount thing, it seems like it might be
overkill for now. The case where there must be a limitation of 1 guest
per NIC is macvtap passthrough mode, but that's already implied by the
fact that it's passthrough. Other than that, libvirt can just attempt to
load-balance as best as possible by keeping track of how many
connections there are on each device, but not force any artificial
limit. We may need to provide some method of reporting the number of
connections to any particular network, to be used by a management
application for load balancing decisions (although the amount of traffic
is probably more important, and that can already be learned).
Conclusion on portgroup - a good idea, but not for this, probably for
configuration of bandwidth limiting.
So a complete SRIOV example might be
<network>
<name>Foo</name>
<forward dev='eth0' layer='link' mode='vepa'>
<interface dev='eth0'/>
<interface dev='eth1'/>
<interface dev='eth2'/>
...
</forward>
<portgroup count='10'>
<interface dev='eth0'/>
</portgroup>
<portgroup count='5'>
<interface dev='eth1'/>
</portgroup>
<portgroup count='5'>
<interface dev='eth2'/>
</portgroup>
</network>
The<virtualport> parameters for VEPA/VNLink could either be stored at
the top level under<network>, or inside<portgroup> or both.
Ah, now *there's* something that fits in portgroup (since that's likely
exactly what it's used for on the vepa/vnlink capable switch).
I think it's reasonable to put it in both places, at the top-level
(which would apply to all connections) and in portgroup (which would
override the global setting for connections using that portgroup). (I
think the bandwidth config could be done in the same way.
--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list