On 04/05/2010 08:35 AM, Dan Johansson wrote:
On Monday 05 April 2010 13.34:26 mathias dufresne wrote:
Hi,
For networks 1 and 2 (respectively 192.168.1.0/24 and 192.168.2.0/24, I'll
acll also network 3 for the last one) you'll have to use bridges.
For network 3 you can use vde which is quite simply to use and gives
possibility to add new virtual networks without the need of physical
interface.
For bridges configuration:
brctl addbr ${brname}
brctl setfd ${brname} 0
brctl sethello ${brname} 1
brctl stp ${brname} off
brct addif ${brname} ${physical_if}
Here you can add an IP to you're bridge interface if needed for the host
has access to this network. Host's IP must by set up on bridge interface
with configuration, if IP is set up on physical interface it is not
working. I'm wondering why, my network knowledge is not much :/
Then when you'll run qemu to set up a new VM you'll have to use a tap
device for the VM
One important thing when use several networks in the same host is then vlan
ID you'll use for networks. These vlans aren't (from my understanding : )
802.1Q vlans but pure qemu/kvm concepts. If you don't set up each VM's
interface on the right kvm's vlan they will all be set up on vlan number 0,
which means packets from all networks will appear everywhere (tcpdump on
192.168.1.0/24 card will see traffic for others networks).
Here is the command I use to launch a VM with two bridged interfaces and
one other on some vde network:
screen -d -m qemu-system-x86_64 -m 64 \
-net tap,vlan=0,ifname=tap0,script=/path/to/scripts/kvm-ifup.br0 \
-net nic,vlan=0,model=e1000,macaddr=52:54:00:01:ab:cd \
-net tap,vlan=3,ifname=tap1,script=/path/to/scripts/kvm-ifup.br1 \
-net nic,vlan=3,model=e1000,macaddr=52:54:00:03:ab:cd \
-net vde,vlan=2,sock=/var/vde/switches/vdelan0 \
-net nic,vlan=2,model=e1000,macaddr=52:54:00:02:ab:cd \
-hda /path/to/disks/vm.vmdk \
-nographic
Note the "vlan=X" option set on both -net arguments for each interface.
For VDE switchs:
/usr/bin/vde_switch -m 770 -g qemu -daemon -n 8 -t dummy10 -s
/var/vde/switches/vdelan0 -M /var/vde/management/vdelan0
I use dummy interface on host to set up vde networks because the host
mustn't have access on these networks.
-s directory where VDE's unix sockets will be store for this network
-M management socket (I haven't yet understood how to use them :p)
-n number of port on the switch
other options are quite easy to understand.
Finally my scripts to set up bridged interface are the following:
------------------
#!/bin/sh
#set -x
switch=`echo $0 | awk -F\. '{ print $NF}'`
if [ -n "$1" ];then
#/usr/bin/tunctl -u `whoami` -t $1
/sbin/ip link set $1 up
sleep 0.5s
/sbin/brctl addif $switch $1
exit 0
else
echo "Error: no interface specified"
exit 1
fi
------------------
I use one script per bridge named kvm-ifup.${brname} because the script's
name define the bridge on which the interface will be attached
(kvm-ifup.br0 -> br0 and kvm-ifup.br1 -> br1).
I use VDE switch for networks on which host isn't plugged because they are
simpler to use for me but it is not the only way to proceed.
A last thing, I have a physical card on which is plugged my modem. As I
don't want to expose the host on Internet I use a VM to act as a firewall.
As I just can't give this VM direct access on the physical card dedicated
to Internet I use a bridge:
modem ------- physical card ----- bridge ----- tap ---- VM's internal card
The modem knows only one MAC address: the one from physical interface but
this physical interface as no IP. The bridge has same MAC address as
physical interface but has no more IP. The card with an IP is the VM's
internal card so for packets pass through all these interface without IP I
use ebtables to "nat" MAC address between physical interface and VM's
internal card:
ebtables -t nat -A PREROUTING -i $PHYS_DEV_NAME -d $PHYS_DEV_MAC -j dnat
--to-destination $VIRT_DEV_MAC
ebtables -t nat -A POSTROUTING -o $VIRT_DEV_NAME -s $VIRT_DEV_MAC -j snat
--to-source $PHYS_DEV_MAC
Finally there are some interesting links:
forum's thread related to vlan issues
http://serverfault.com/questions/101477/issue-with-multiple-bridging-for-kv
m-hosts
something about bridges:
http://www.savelono.com/linux/using-multiple-interfaces-with-kvm-and-xen.ht
ml
different method to build networks using qemu/kvm:
http://people.gnome.org/~markmc/qemu-networking.html<http://people.gnome.or
g/%7Emarkmc/qemu-networking.html>
Hoping this helps.
Kindly regards,
mathias
2010/4/5 Dan Johansson<kvm@xxxxxx>
On Sunday 04 April 2010 22.23:28 Held Bernhard wrote:
Am 04.04.2010 20:02, schrieb Dan Johansson:
On Sunday 04 April 2010 15.00:26 sudhir kumar wrote:
On Sun, Apr 4, 2010 at 5:47 PM, Dan Johansson<kvm@xxxxxx> wrote:
Hi,
I am new to this list and to KVM (and qemu) so please be gentle
with me. Up until now I have been running my virtualizing using
VMWare-Server. Now I want to try KVM due to some issues with the
VMWare-Server and I am having some troubles with the networking
part
of
KVM.
This is a small example of what I want (best viewed in a fix-font):
+-----------------------------------+
| Host |
| +----------+ eth0 |---- 192.168.1.0/24
|
| | eth0|-- + |
| | VM1 eth1|---(---+------- eth1 |---- 192.168.2.0/24
| | eth2|---(---(---+ |
|
| +----------+ | | | |
|
| +----------+ +---(---(--- eth2 |---- 192.168.1.0/24
|
| | eth0|---+ | | |
| | VM2 eth1|-------+ +--- eth3 |---- 192.168.3.0/24
| | eth2|-----------+ |
|
| +----------+ |
+-----------------------------------+
Host-eth0 is only for the Host (no VM)
Host-eth1 is shared between the Host and the VM's (VM?-eth1)
Host-eth2 and Host-eth3 are only for the VMs (eth0 and eth2)
The Host and the VMs all have fixed IPs (no dhcp or likewise).
In this example th IPs could be:
Host-eth0: 192.168.1.1
Host-eth1: 192.168.2.1
Host-eth2: -
Host-eth3: -
VM1-eth0: 192.168.1.11
VM1-eth1: 192.168.2.11
VM1-eth2: 192.168.3.11
VM2-eth0: 192.168.1.22
VM2-eth1: 192.168.2.22
VM3-eth2: 192.168.3.22
And, yes, Host-eth0 and Host-eth2 are in the same subnet, with eth0
dedicated to the Host and eth2 dedicated to the VMs.
In VMWare this was quite easy to setup (three bridged networks).
Its easy with KVM too. You want 3 NICs per VM, so you need to pass
the corresponding parameters(including qemu-ifup script) for 3 NICs
to each VM.
In the host you need to create 2 bridges: say br-eth1 and br-eth2.
Make them as the interface on the host in place of the corresponding
eth interfaces.(brct addbr br-eth1; ifcfg eth1 0.0.0.0 up; brctl
addif br-eth eth1; assign eth1's ip and routes to breth1; same for
eth2). In the corresponding qemu-ifup scripts of each interface use
bridge=br-ethN (This basicaly translates to brctl addif br-ethN $1,
where $ is the tap device created)
This should work perfectly fine with your existing NW setup.
For a quick reference use: http://www.linux-kvm.org/page/Networking
Thanks for your help, but... I am still not able to get it to work
the way I want.
This is what I have don so far:
brctl addbr br-eth1
brctl addbr br-eth3
ip link set eth1 up
ip link set eth3 up
brctl addif br-eth1 eth1
brctl addif br-eth3 eth3
tunctl -b -t qtap1
tunctl -b -t qtap3
brctl addif br-eth1 qtap1
brctl addif br-eth3 qtap3
ifconfig qtap1 up 0.0.0.0 promisc
ifconfig qtap3 up 0.0.0.0 promisc
# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0d:88:52:51:24
inet addr:192.168.1.3 Bcast:192.168.1.255
Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:443638 errors:0 dropped:0 overruns:0 frame:0
TX packets:758540 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:47041686 (44.8 MiB) TX bytes:990115354 (944.2
MiB) Interrupt:19 Base address:0xec00
eth1 Link encap:Ethernet HWaddr 00:0d:88:52:51:25
inet addr:192.168.4.1 Bcast:192.168.4.255
Mask:255.255.255.0
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:6
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:360 (360.0 B)
Interrupt:18 Base address:0xe880
eth3 Link encap:Ethernet HWaddr 00:0d:88:52:51:27
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:240 (240.0 B)
Interrupt:16 Base address:0xe480
qtap1 Link encap:Ethernet HWaddr 26:c0:de:df:c5:e4
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:351 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:14742 (14.3 KiB) TX bytes:0 (0.0 B)
qtap3 Link encap:Ethernet HWaddr 26:3e:ba:2d:97:bc
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:252 (252.0 B) TX bytes:0 (0.0 B)
# brctl show
bridge name bridge id STP enabled interfaces
br-eth1 8000.000d88525125 no eth1
qtap1
br-eth3 8000.000d88525127 no eth3
qtap3
This is the way I start the guest:
kvm -net nic,vlan=1,model=rtl8139,macaddr=52:54:00:12:34:56 -net
tap,vlan=1,ifname=qtap1,script=no,downscript=no -net
nic,vlan=3,model=rtl8139,macaddr=52:54:00:12:34:58 -net
tap,vlan=3,ifname=qtap3,script=no,downscript=no Robbie.img -m 1024
The eth3/br-eth3/qtap3 looks OK (I can ping the "default-GW" on that
network from the guest) but the connection to the "shared" interface
(eth1/br- eth1/qtap1) does not work, I can not ping or ssh to/from
the guest from/to the host. Do not ask me if I can ping any other
host on that network - there are no other host on the network yet,
just the
Host
and the guest.
Any suggestions?
eth1 should not have an IP address:
# ifconfig eth1 0.0.0.0
br-eth1 is not activated (it's missing in `ifconfig`), and it needs an
IP address:
# ifconfig br-eth1 192.168.2.1/24
Even if it works I would explicitly activate br-eth3 too:
# ifconfig br-eth3 0.0.0.0 up
Looking at the output of `ifconfig` shows that the IP-address of eth0
(192.168.1.3) doesn't match 192.168.1.1 from your address list, and
eth1 (192.168.4.1) is in a different network than the specified
192.168.2.1/24.
OK, the 192.168.1.3 - 192.168.1.1 is my typo, 192.168.1.3 is the correct
IP for the Host on this interface.
I was first planning to use the 192.168.2.0/24 network here, but I had
forgotten that that was already used somewhere else in my network, so I
just
picked the next "free" network address. Sorry for the confusion.
I have tried you suggestions above but I still have no network
connection.
# ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:0d:88:52:51:25
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:3
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:180 (180.0 B)
Interrupt:18 Base address:0xe880
# ifconfig br-eth1
br-eth1 Link encap:Ethernet HWaddr 00:0d:88:52:51:25
inet addr:192.168.4.1 Bcast:192.168.4.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:65 errors:0 dropped:0 overruns:0 frame:0
TX packets:62 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3518 (3.4 KiB) TX bytes:4750 (4.6 KiB)
# ifconfig br-eth3
br-eth3 Link encap:Ethernet HWaddr 00:0d:88:52:51:27
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Regards,
OK, lets simplify it a bit. This is now the layout that I am playing with and
trying to get KVM to work .
+-----------------------------------+
| Host |
| +----------+ eth0 |---- 192.168.1.0/24
| | VM eth1|------qtap1 |
| | | | |
| | eth0 | br-eth1---eth1 |---- 192.168.4.0/24
| +----------+ |
| |
+-----------------------------------+
Host-eth0 is only for the Host (no VM I/O)
Host-eth1 is shared between the Host and the VM (Host<--> VM, Host<--
"external", VM<--> "external")
VM-eth0 is unused at the moment.
This is the commands I have been using to setup the Networking:
brctl addbr br-eth1
brctl setfd br-eth1 0
brctl sethello br-eth1 1
brctl stp br-eth1 off
brctl addif br-eth1 eth1
tunctl -b -t qtap1
brctl addif br-eth1 qtap1
ifconfig qtap1 up 0.0.0.0 promisc
ifconfig br-eth1 192.168.4.1/24 up
ifconfig eth1 0.0.0.0 up
And this is the way I start the VM:
kvm -net nic,vlan=1,model=rtl8139,macaddr=52:54:00:12:34:56 -net
tap,vlan=1,ifname=qtap1,script=no,downscript=no Robbie.img -m 1024 -k de-ch
On the Host I have the following:
# ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:0d:88:52:51:25
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:18 errors:0 dropped:0 overruns:0 carrier:18
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1080 (1.0 KiB)
Interrupt:18 Base address:0xe880
# ifconfig br-eth1
br-eth1 Link encap:Ethernet HWaddr 00:0d:88:52:51:25
inet addr:192.168.4.1 Bcast:192.168.4.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:501 errors:0 dropped:0 overruns:0 frame:0
TX packets:479 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:26378 (25.7 KiB) TX bytes:35080 (34.2 KiB)
# ifconfig qtap1
qtap1 Link encap:Ethernet HWaddr da:77:3a:a9:40:23
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:503 errors:0 dropped:0 overruns:0 frame:0
TX packets:481 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:33524 (32.7 KiB) TX bytes:35240 (34.4 KiB)
# brctl show
bridge name bridge id STP enabled interfaces
br-eth1 8000.000d88525125 no eth1
qtap1
# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.4.0 0.0.0.0 255.255.255.0 U 0 0 0 br-
eth1
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
239.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
And in the VM I have:
# ifconfig
eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:56
inet addr:192.168.4.4 Bcast:192.168.4.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1040 (1.0 KiB) TX bytes:1018 (1018.0 B)
Interrupt:11 Base address:0xc000
# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.4.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 192.168.4.1 0.0.0.0 UG 0 0 0 eth1
But Still I get the following:
From the Host to the VM:
$ ssh root@xxxxxxxxxxx
ssh: connect to host 192.168.4.4 port 22: Connection refused
And from the VM to the Host:
# ssh root@xxxxxxxxxxx
ssh: connect to host 192.168.4.1 port 22: Connection refused
Those messages imply a deliberate act by host or guest (e.g. RST in
response to SYN). IOW, your sshd or iptables on the source or
destination is seeing and willingly rejecting the connection attempt.
It's possible it could be caused by an ICMP No Route To Host response on
the route actually followed by the TCP SYN from ssh but your routing
looks OK to me and it wouldn't take many default routes of default
routes for an unreachable destination to become a black hole and that
would cause ssh to report "Connection timed out" (in my experience).
Any more suggestions?
Based on those messages I'd capture network traffic simultaneously on
the host and guest and repeat the connection attempt. Analyze the
traffic and diagnose the cause. I'd capture on the host via br-eth1 and
in the guest on it's eth1.
--
David.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
