Chapter 7: Oracle VM Networking from A to 802.1Q

Last update 12-03-2010

Copyright © 2009 - 2012 Roddy Rodstein. All rights reserved.

 

Chapter 7 covers the following topics:

Oracle VM ships with a flexible networking configuration that can be used as-is or modified to meet your business requirements. Oracle VM network configuration management is not supported by Oracle VM Manager or the Oracle VM Management Pack; it must therefore be performed by hand in dom0. Both 802.3AD NIC bonding and 802.1Q VLANning are supported, although they must also be configured by hand in dom0.

 

The default Oracle VM server networking configuration routes all dom0 and guest traffic through a Xen bridge. A Xen bridge operates at layer 2 of the OSI model, effectively acting as a layer 2 (L2) switch passing packets to the egress port; it relies on the TCP protocol for rate control and packet loss.

 

Table 1 shows the OSI model. Note that Xen bridges operate in layer 2.

Oracle VM's default network configuration pairs each network interface (NIC) with a Xen bridge. The Xen bridges are created by default by the Xen SysV startup script (/etc/init.d/xend). An Oracle VM server with two NICs will have two Xen bridges, eth0/xenbr0 and eth1/xenbr1. The first Xen bridge, eth0/xenbr0, is configured with an IP address on xenbr0 and is dedicated to Oracle VM management and HA traffic. The second Xen bridge will not have an IP address assigned; it effectively acts as a layer-two switch for guest traffic. Any Xen bridge, including the ent0/xenbr0 pair, can be used for guest traffic. Guest virtual NICs can be used with any Xen bridges by editing the guest's vm.cfg file or by editing the guest's network properties using Oracle VM Manager or the Oracle VM Management Pack.

 

Tip: In an HA-enabled pool, the loss of network connectivity for the Oracle VM management interface causes an HA event. When an HA event occurs, an Oracle VM server is fenced from the pool and reboots, then all HA-enabled guests are restarted on a live Oracle VM pool member.

 

It's your prerogative how to physically wire your Oracle VM server's NICs. For example, you could wire each NIC into a managed switch using VLANs to segregate the traffic from each Xen bridge. Another option is to wire your NICs into an unmanaged switch on a flat network, using the Xen bridges to distribute traffic across the switch.

 

Figure 1 shows the default Oracle VM Xen bridge configuration: an Oracle VM server with two NICs wired into a single switch, hosting three guests.

 

As shown in Figure 1, the default Oracle VM server network configuration will pair eth0 with xenbr0 and use the eth0/xenbr0 pair for Oracle VM agent, OCFS2 cluster heartbeat, and Live Migration traffic, and pair eth1 with xenbr1. If an Oracle VM server had four or more NICs, the default configuration would pair eth0 with xenbr0, used for Oracle VM management traffic, and then create additional Xen bridges, that is, eth1/xenbr1, eth2/xenbr2, eth3/xenbr3, and so on. Any Xen bridge can be used for guest traffic, even eth0/xenbr0.

 

In Figure 1, packets that arrive at the physical NIC (eth0) are handled by dom0’s Ethernet driver and appear on xenbr0. Xen bridges distribute packets like a layer-two switch for dom0 and guests. Xen bridges route guest packets based on the guest's MAC address.

 

The next example shows the output from “ifconfig” and “brctl show” from an Oracle VM server with two NICs without any guests.

The output from “ifconfig” in the above example shows, eth0, eth1, lo, xenbr0, and xenbr1. Next, the output from “brctl show” shows the Xen bridge configurations, that is, xenbr0 is connected to eth0 and xenbr1 is connected to eth1.

 

The Oracle VM server's dom0 interfaces are managed using native Linux network scripts, for example, /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-*. After an Oracle VM Server 2.2 boots, the Xen /etc/xen/xend-config.sxp script (network-script network-bridge) entry is referenced and creates an independent Xen bridge for each physical NIC. For example, typing "brctl addif xenbr0 eth0" creates the eth0/xenbr0 pair and typing "brcrtl addif xenbr0 eth1" creates the eth1/xenbr1 pair.

 

dom0's management interface is selected during the Oracle VM server installation and can be edited after the installation in the /etc/ovs-config file. From an HA perspective, the NIC used for Oracle VM management traffic is the only network interface that is monitored for HA events. If HA is a requirement for both the Oracle VM hosts and guests, it’s important to design your Oracle VM servers with a single bond used by all Xen bridges for dom0 and guests.

 

The next example shows Oracle VM's default /etc/sysconfig/network-scripts/ifcfg* scripts from an Oracle VM server with two NICs.

Note: Oracle VM 2.2 does not use the default Xen.org pseudo devices in dom0, for example, no vif or peth devices are used.

Beware of Baseboard Management Controllers and Broadcom NetXtreme II NICs

Beware of Baseboard Management Controllers that virtualize ethernet ports! Baseboard Management Controllers that virtualize ethernet ports may not work well with dom0's bonded interfaces and might result in broadcast storms on the connected switches. A broadcast storm could cause a switch to disable the ports that detect the broadcast storm. HA enabled Oracle VM servers connected to a switch port that gets disabled would promptly fence from the cluster, reboot and the HA enabled guests would restart on available pool members.

Bonded Oracle VM server's that use the Broadcom NetXtreme II bnx2x driver on hardware with Baseboard Management Controllers are subject to broadcast storms. The solution to eliminate the broadcast storms is to a) change the network interface driver from the bnx2x driver to the tg3 driver or b) disable the Baseboard Management Controller's management firmware and/or c) both a and b.

Typing “dmesg|grep Broad” from dom0 will validate if an Oracle VM server is using Broadcom NetXtreme II network interfaces. The next example shows the result from typing “dmesg|grep Broad” on an Oracle VM server using Broadcom NetXtreme II network interfaces.

Oracle VM networking can be broken down into two parts. The first part of the networking configuration consists of the Oracle VM servers and the second part of the networking configuration is for the guests. As explained above, Oracle VM server networking is configured in dom0 using native Linux networking scripts along with the default Xen SysV startup script (/etc/init.d/xend). Guest networking is configured in each guest's vm.cfg file and within the each guest’s operating system.

 

Guest networking is configured in each guest's vm.cfg file in the “vif” and “vif_other_config” directives. Each guest will have a unique vm.cfg file, located in the storage repository, for example, the /var/ovs/mount/<UUID>/running_pool/<VIRTUAL MACHINE NAME> directory. The next example shows a vm.cfg file from the Oracle VM 11g template. The “vif” and “vif_other_config” directives are highlighted.

In the above example, the settings in the “vif” directive show that:

The “vif_other_config” directive is empty, indicating that there is not an Oracle VM Manager network policy. A guest with the proper “vif” and OS networking setup will appear on the network from the assigned Xen bridge in the guest's vm.cfg file.

 

Tip: To generate a unique MAC address for a guest, type the following text while logged in as root:

PYTHONPATH=/opt/ovs-agent-2.2 python -c "from OVSCommons import randomMAC; print randomMAC()"

 

The next example shows the output from “ifconfig” and “brctl show” from an Oracle VM server with two NICs hosting three guests; each guest has a single NIC.

The output from “ifconfig” in the above example shows eth0, eth1, lo, vif2.0, vif3.0, vif4.0, xenbr0, and xenbr1. vif2.0, vif3.0, vif4.0 are the guest virtual NICs. Next, the output from “brctl show” shows the bridge configurations, that is, xenbr0 is connected to vif2.0 and eth0, and xenbr1 is connected to vif3.0, vif4.0, and eth1.

In this section we will examine the following three Oracle VM networking configurations:

Figure 1 shows the default Oracle VM Xen bridge configuration, with an Oracle VM server with two NICs wired into a single unmanaged switch. The Oracle VM server is hosting three guests that use xennbr1; xenbr0 is dedicated to management traffic.

 

In Figure 1, the Oracle VM management traffic and HA monitoring are isolated to the eth0/xenbr0 interface and the loss of network connectivity for eth0/xenbr0 will cause an HA event. If the eth1/xenbr1 interface looses connectivity and all the guests are dropped from the network, an HA event will not occur.

 

Figure 2 shows the default Oracle VM Xen bridge configuration with a two-server Oracle VM pool. Each Oracle VM server has two NICs wired in to a single unmanaged switch. The Oracle VM server is hosting three guests that use xenbr1; xenbr0 is dedicated to management traffic.

 

In Figure 2, the Oracle VM management traffic and HA monitoring are isolated to eth0/xenbr0 and the loss of network connectivity for eth0/xenbr0 will cause an HA event. If the eth1/xenbr1 interface loses connectivity and all the guests are dropped from the network an HA event will not occur.

Figure 3 shows an 802.3AD NIC bonding configuration with one Xen bridge on bond1.

 

The Oracle VM server in Figure 3 has four NICs configured with two bonds, bond0 and bond1. Each NIC from each bond is wired into a separate switch for high availability. The Oracle VM server is hosting three guests using xenbr1; bond0 and eth0:0 are dedicated to management traffic.

 

Tip: If a bond is dedicated for dom0 management traffic it is not necessary to configure a Xen bridge for dom0; a Xen bridge is only a requirement for guests to access the network.

 

In Figure 3, the Oracle VM management traffic and HA monitoring are isolated to bond0 and the eth0:0 interface and the loss of network connectivity for bond0 or the eth0:0 interface would cause an HA event. If the bond1/xenbr1 interface loses connectivity and all the guests are dropped from the network, an HA event will not occur.

 

Figure 4 shows an 802.3AD NIC bonding configurations with one Xen bridge with a two-server Oracle VM pool.

 

Each Oracle VM server in Figure 4 has four NICs configured with two bonds, bond0 and bond1. Each NIC from each bond is wired into a separate switch for high availability. Each Oracle VM server is hosting three guests that use xenbr1; bond0 and eth0:0 are dedicated to management traffic.

 

In Figure 4 the Oracle VM management traffic and HA monitoring are isolated to bond0 and the eth0:0 interface and the loss of network connectivity for bond0 or the eth0:0 interface would cause an HA event. If the bond1/xenbr1 interface loses connectivity and all the guests are dropped from the network, an HA event will not occur.

 

Figure 5 shows an 802.3AD NIC bonding configuration that uses Xen bridges with 802.1Q VLANning.

 

The Oracle VM server in Figure 5 has four NICs configured with one bond, bond0. Each NIC from each bond is wired into a separate switch for high availability. The Oracle VM server is hosting three guests that use vlan2, vlan3, and vlan4; vlan1 is dedicated to management traffic.

 

Note: In Figure 5 vlan1, vlan2, vlan3 and vlan4 are Xen bridges. Xen bridges can have discriptive names, i.e. vlanX, xenbrX, etc...

 

In Figure 5, the Oracle VM management traffic and HA monitoring are isolated to the bond0:1 interface over vlan1. If dom0 loses network connectivity due to bond0, bond0:1 or vlan1 an HA event would occur causing the Oracle VM server to fence from the pool and reboot, all HA-enabled guests would restart on a live Oracle VM pool member.

 

Figure 6 shows an 802.3AD NIC bonding configurations with a two-server Oracle VM pool that uses Xen bridges with 802.1Q VLAN and VLAN tagging.

 

Each Oracle VM server in Figure 5 has four NICs configured with one bond, bond0. Each NIC from each bond is wired into a separate switch for high availability. Each Oracle VM server is hosting three guests that use vlan2, vlan3, and vlan4, respectively; vlan1 is dedicated to management traffic. The loss of network connectivity with the bond0 interface would cause an HA event that fences the Oracle VM server and reboots it, and restart all HA-enabled guests on a live Oracle VM pool member.

This section will explain how to configure NIC bonding with a single Xen bridge on an Oracle VM server with two NICs. The example configuration has a single Xen bridge used for dom0 and the guests. In this example, the Oracle VM server and the guests are on the same network and use the same Xen bridge. Each Oracle VM server in a pool should have the identical networking configuration.

 

Figure 7 shows an example configuration with NIC bonding using a single Xen bridge.

 

The example configuration will create a bond with two NICs, one interface and one Xen bridge, used by dom0 and guests. After configuring the bonding interface, dom0 will have the following interfaces:

Note: This example configuration provides HA for both dom0 and the guests if the Oracle VM server loses network connectivity.

 

Step 1: Disable the default xend network configuration. From dom0 type the following commands while logged in as root to disable the xend network configuration:

Step 2: Create a file in the /etc/xen/scripts directory named “network-bridge-ovs” that contains the following lines:

 

#!/bin/sh

/bin/true

 

For example, from dom0 while logged in as root type “vi /etc/xen/scripts/network-bridge-ovs” to create a file named network-bridge-ovs, then enter the text in the next example:

Step 3: Make the etc/xen/scripts/network-bridge-ovs file executable by typing the text in the next example.

Step 4: Edit the /etc/xen/xend-config.sxp file and change the following line from “network-script network-bridges” to “network-script network-bridge-ovs”.

 

Step 5: Edit /etc/modprobe.conf and load and configure the bonding modules. For example:

 

alias eth0 <DriverName>

alias eth1 <DriverName>

alias bond0 bonding

options bonding mode=1 miimon=100 downdelay=200 updelay=200

 

Change the <DriverName> to the correct driver for your NICs, for example, tg3, e1000, or the like.

 

We have done extensive testing with the various bonding modes and suggest mode 1 for all Oracle VM configurations. For example, the existing 2.6.18 kernel has known issues with mode 4 and 6. 

 

Note: Please review Bonding with LACP does not work if your testing mode 4. Consider testing the  "xmit_hash_policy=layer2+3" setting in the above modprobe.conf file. Adding the  "xmit_hash_policy=layer2+3" setting in the modprobe.conf file may help spread the load between the NICs in the bond and provide better throughput. 

 

To list the NIC drivers, try one of the following commands from dom0 while logged in as root:

 

# dmesg | grep 'Ethernet driver'

# grep 'Ethernet driver' /var/log/dmesg

# grep ethX /etc/modules.conf

 

Step 6: Configure the network devices and the Xen bridge.

 

To configure the eth0 and eth1 interfaces, edit the /etc/sysconfig/network-scripts/ifcfg-eth0 file as follows:

 

DEVICE=eth0

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes

USERCTL=no

HWADDR=<YOUR MAC ADDRESS>

 

For /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes

USERCTL=no

HWADDR=<YOUR MAC ADDRESS>

 

The HWADDR parameter is the pointer to the NIC's MAC address. The SLAVE parameter defines the network card as a slave of a bond-device, in this case bond0. The MASTER parameter points to the actual bonding device that this network interface will be part of.

 

Next, create the bonding device bond0 by creating a file named /etc/sysconfig/network-scripts/ifcfg-bond0, containing the following lines:

 

DEVICE=bond0

ONBOOT=yes

BRIDGE=xenbr0

USERCTL=no

 

The next example shows how to create and save a file named /etc/sysconfig/network-scripts/ifcfg-bond0 with the bond device parameters.

The BRIDGE parameter specifies that the bond interface does not have an IP address, but will be connected to bridge xenbr0.

 

Next, create the Xen bridge by creating a file named /etc/sysconfig/network-scripts/ifcfg-xenbr0 containing the following lines:

 

DEVICE=xenbr0

ONBOOT=yes

STP=off

USERCTL=no

IPADDR=<YOUR IP ADDRESS>

NETMASK=<YOUR NETMASK>

 

The next example shows how to create and save a file named ifcfg-xenbr0 that contains the Xen bridge parameters.

An IP address is assigned to xenbr0 using the IPADDR and NETMASK lines. The NETWORK and BROADCAST parameters are deprecated and are automatically calculated with ipcalc.

 

Step 5: Restart the network and xend services from dom0 by typing “service network restart” followed by “service xend restart,” as shown in the next example:

Once xend and the network service have been restarted, the bond with two NICs with one Xen bridge will be ready for use. In Oracle VM Manager each guest should be configured to use xenbr0.

 

To validate the configurations, from dom0 type “ifconfig” to list the interfaces and the Xen bridges, as shown in the next example.

As shown in the next example, type “brctl show” to confirm that the desired interface is using the appropriate Xen bridge.

The output for “brctl show” lists the bridge name, bridge ID, spanning-tree protocol (STP) configuration, and the configured interfaces.

 

The next example shows a vm.cfg file from a guest that is configured to use xenbr0.

Tip: All Oracle VM servers using bonded interfaces should modify two default O2CB heartbeat configurations to allow a bond fail-over event to occur without the Oracle VM server fencing from the pool. Edit the /etc/sysconfig/o2cb file using your favorite text editor and change "O2CB_HEARTBEAT_THRESHOLD=" to "O2CB_HEARTBEAT_THRESHOLD= 61" and "O2CB_IDLE_TIMEOUT_MS=" to "O2CB_IDLE_TIMEOUT_MS= 60000". The next example shows the suggested modifications made to a /etc/sysconfig/o2cb file.

 

# vi /etc/sysconfig/o2cb

#
# This is a configuration file for automatic startup of the O2CB
# driver. It is generated by running /etc/init.d/o2cb configure.
# Please use that method to modify this file
#
# O2CB_ENABELED: 'true' means to load the driver on boot.
O2CB_ENABLED=true
# O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.
O2CB_BOOTCLUSTER=rmvsx
# O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead.
O2CB_HEARTBEAT_THRESHOLD= 61
# O2CB_IDLE_TIMEOUT_MS: Time in ms before a network connection is considered dead.
O2CB_IDLE_TIMEOUT_MS= 60000
# O2CB_KEEPALIVE_DELAY_MS: Max time in ms before a keepalive packet is sent
O2CB_KEEPALIVE_DELAY_MS=

This section will examine how to configure NIC bonding with VLANning using multiple Xen bridge on an Oracle VM server with two NICs. To be able to use 802.1Q VLANning, your switches will need to support VLAN trunking and be correctly configured. The example configuration uses a single bond with multiple Xen bridges for dom0 and guests. Each Oracle VM server in a pool should have the identical networking configuration.

 

Figure 8 shows an example 802.3AD NIC bonding and Xen bridges with 802.1Q VLANning.

 

The example configuration will have an 802.3AD bond with two NICs with one interface using four Xen bridges. After the bonding interface is configured, dom0 will have the following interfaces:

 

The example configuration offers HA for both dom0 and the guest Xen bridges if the Oracle VM server loses network connectivity.

 

Note: Please note that the :XX part of the interface name, that is, bond0:1, bond0:2, bond0:3, bond0:4, is referencing VLAN 1, 2, 3, and 4, respectively.

 

Step 1: Disable the default xend network configuration. From dom0 type the following commands while logged in as root to disable the xend network configuration:

Step 2: Create a file in the /etc/xen/scripts directory named “network-bridge-ovs” that contains the lines:

 

#!/bin/sh

/bin/true

 

For example, from dom0 while logged in as root type “vi /etc/xen/scripts/network-bridge-ovs” to create a file named network-bridge-ovs, then enter the text in the next example.

Step 3: Make the etc/xen/scripts/network-bridge-ovs file executable by typing the text in the next example.

Step 4: Edit the /etc/xen/xend-config.sxp file and change the line“network-script network-bridges” to “network-script network-bridge-ovs”.

 

Step 5: Edit /etc/modprobe.conf and load and configure the bonding modules. For example:

 

alias eth0 <DriverName>

alias eth1 <DriverName>

alias bond0 bonding

options bonding mode=1 miimon=100 downdelay=200 updelay=200

 

Change the <DriverName> to the correct driver for your NICs, for example, tg3, e1000, or the like.

 

 

To list the NIC drivers, use one of the following commands from dom0 while logged in as root.

 

# dmesg | grep 'Ethernet driver'

# grep 'Ethernet driver' /var/log/dmesg

# grep ethX /etc/modules.conf

 

Step 6: Configure the network devices and the Xen bridge.

 

To configure the eth0 and eth1 interfaces, edit the /etc/sysconfig/network-scripts/ifcfg-eth0 file as follows:

 

DEVICE=eth0

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes

USERCTL=no

HWADDR=<YOUR MAC ADDRESS>

 

For /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes

USERCTL=no

HWADDR=<YOUR MAC ADDRESS>

 

The HWADDR parameter is the pointer to the NIC's MAC address. The SLAVE parameter defines the network card as a slave of a bond-device, in this case, bond0. The MASTER parameter points to the actual bonding device that this network interface will be part of.

 

Next, create the bonding device bond0 by creating a file named /etc/sysconfig/network-scripts/ifcfg-bond0 containing the following lines:

 

DEVICE=bond0

ONBOOT=yes

USERCTL=no

 

The next example shows how to create and save a file named /etc/sysconfig/network-scripts/ifcfg-bond0 with the bond-device parameters.

Next, create the interfaces by making a file for each interface named /etc/sysconfig/network-scripts/ifcfg-<INTERFACE NAME>. The following examples show four interface files.

 

Note: Any descriptive name can be used for the interface and Xen bridge. You can create as many interfaces and Xen bridges as needed for your environment.

 

The next example is the file for the bond0:1 interface that uses vlan1.

 

DEVICE=bond0:1

ONBOOT=yes

BOOTPROTO=none

VLAN=yes

TYPE=Ethernet

BRIDGE=vlan1

 

The next example is the file for the bond0:2 interface that uses vlan2.

 

DEVICE=bond0:2

ONBOOT=yes

BOOTPROTO=none

VLAN=yes

TYPE=Ethernet

BRIDGE=vlan2

 

The next example is the file for the bond0:3 interface that uses vlan3.

 

DEVICE=bond0:3

ONBOOT=yes

BOOTPROTO=none

VLAN=yes

TYPE=Ethernet

BRIDGE=vlan3

 

The next example is the file for the bond0:4 interface that uses vlan4.

 

DEVICE=bond0:4

ONBOOT=yes

BOOTPROTO=none

VLAN=yes

TYPE=Ethernet

BRIDGE=vlan4

 

Next, create the Xen bridges by making a file for each Xen bridge, named /etc/sysconfig/network-scripts/ifcfg-<XEN BRIDGE NAME>. The following examples show four Xen bridge files.

 

The next example is the file for the vlan1 that is used by bond0:1.

 

DEVICE=vlan1

BOOTPROTO=static

ONBOOT=yes

IPADDR=<YOUR IP ADDRESS FOR DOM0>

NETMASK=<YOUR NETMASK FOR DOM0>

 

An IP address is assigned to xenbr11 using the IPADDR and NETMASK sections. The NETWORK and BROADCAST parameters are deprecated and are automatically calculated with ipcalc.

 

The next example is the file for the vlan2 that is used by bond0:2.

 

DEVICE=vlan2

BOOTPROTO=none

ONBOOT=yes

Type=Bridge

 

The next example is the file for the vlan3 that is used by bond0:3.

 

DEVICE=vlan3

BOOTPROTO=none

ONBOOT=yes

Type=Bridge

 

The next example is the file for the vlan4 that is used by bond0:4.

 

DEVICE=vlan4

BOOTPROTO=none

ONBOOT=yes

Type=Bridge

 

The next example shows how to create and save a file named ifcfg-vlan2 that contains the Xen bridge parameters.

Step 5: Restart the network and xend services from dom0 by typing “service network restart” followed by “service xend restart” as shown in the next example.

Once xend and the network service have been restarted, the new configuration will be ready for use. In Oracle VM Manager, each guest can be configured to use any of the Xen bridges.

 

To validate the configurations, from dom0 type “ifconfig” to list the interfaces and the Xen bridges, as shown in the next example.

As shown in the next example, type “brctl show” to confirm that the desired interfaces are using the correct Xen bridges.

The output for “brctl show” lists the bridge name, bridge ID, spanning-tree protocol (STP) configuration and the configured interfaces. 

Tip: All Oracle VM servers using bonded interfaces should modify two default O2CB heartbeat configurations to allow a bond fail-over event to occur without the Oracle VM server fencing from the pool. Edit the /etc/sysconfig/o2cb file using your favorite text editor and change "O2CB_HEARTBEAT_THRESHOLD=" to "O2CB_HEARTBEAT_THRESHOLD= 61" and "O2CB_IDLE_TIMEOUT_MS=" to "O2CB_IDLE_TIMEOUT_MS= 60000". The next example shows the suggested modifications made to a /etc/sysconfig/o2cb file.