Chapter 2: Oracle VM Architectural Review

Last update 7-28-2010

This chapter provides an architectural review of Oracle VM Manager followed by Oracle VM server. The goal of this chapter is to provide the architectural background to help plan, design and support an Oracle VM environment.

Table of Contents

Oracle VM Manager Introduction

...The Oracle VM Manager Command Line Interface
Oracle VM Manager Intra-component Communication & Firewall Requirements
The Oracle VM Management Pack Intra-component Communication & Firewall Requirements
Oracle VM Manager and Oracle VM Server Pool Management
...Oracle VM Server Agent Roles
Oracle VM Manager System Design Considerations & Server Pool Examples

Oracle VM Storage

Oracle VM High Availability

...High Availability Introduction
...Oracle VM High Availability
......Oracle VM HA Network Considerations
...Oracle VM Live Migration
......Live Migration Requirements
......How Oracle VM Live Migration Works
......Oracle VM Live Migration and HA Network Considerations
Oracle VM Server Introduction
...The Xen Hypervisor
......Dom0
......DomU - Paravirtualized (PV) Guests
......DomU - Full Virtualization – Hardware Virtualization Machines (HVM)
......Domain Management Services
......Xend
......XM
......Xenstored
......Libxenctrl
......Quemu-dm

......Virtual Frame Buffer
......Oracle VM Agent

Oracle VM Networking

Oracle VM Manager Introduction

Oracle VM Manager is a private source cluster ready Oracle Application Development Framework (ADF) application that offers browser based management for Oracle VM server pools and guests. Oracle VM Manager is distributed in three formats; a) as a pre-packaged production ready Oracle VM template b) as an ISO file that installs on Enterprise Linux and c) as the Oracle VM Management Pack plug-in.

Figure 1 shows the Oracle VM Manager 2.2 user interface.

The Oracle VM Manager template is a self-contained, pre-installed, and pre-configured Oracle VM Manager image that contains Oracle Enterprise Linux 5 with Oracle VM Manager. The Oracle VM Manager template can be downloaded from the Oracle eDelivery portal uncompressed and setup directly on an Oracle VM Server. The Oracle VM Manager template is configured directly from the Oracle VM Server console.

Figure 2 shows the steps to deploy the Oracle VM Manager template on a single Oracle VM Server.

The Oracle VM Manager ISO file is distributed from the Oracle eDelivery portal. The Oracle VM Manager ISO file contains a collection of applications needed to install Oracle VM Manager. Oracle VM Manager ships with the Oracle 10g Express Edition Database, although Oracle 10g Standard Edition Database, Oracle 10g Enterprise Edition Database, or RAC is supported as the Oracle VM database repository. Table 3 lists the packaged applications in the Oracle VM Manager ISO file.

Table 1

Oracle VM Manager Applications
Application	Capability
Oracle Database 10g Express Edition	The Oracle Database 10g Express Edition is packaged in the Oracle VM ISO.
Oracle VM Manager package	The Oracle VM Manager package contains the Oracle VM Manager web application. The Oracle VM Manager web application runs on Oracle Application Server 10g Release 3 (10.1.3) and is deployed into an OC4J container.
Oracle Containers for J2EE (OC4J) Standalone 10.1.3 packaged with Application Development Framework (ADF) 10.1.3.3	Oracle Containers for J2EE (OC4J) is the J2EE runtime environment for the Oracle Application Server. Oracle Application Development Framework (ADF) is a Java EE framework.
XML-RPC 3.0	The XML-RPC 3.0 package is installed during an Oracle VM Manager installation. XML-RPC is used for Oracle VM Manager to Oracle VM agent and Oracle VM agent to Oracle VM agent communication.

The Oracle VM Manager Command Line Interface

The Oracle VM Manager Command Line Interface is an Oracle VM Manager add on component that is available to Oracle Unbreakable Linux support customers via the Oracle Unbreakable Linux Network. The Oracle VM Manager Command Line Interface offers a command-line interface for Oracle VM Manager. The Oracle VM Manager Command Line Interface allows Oracle VM Manager functions to be executed from the command-line. Oracle VM Manager Command Line Interface commands can be scripted allowing automation of administrative tasks such as checking status, performing lifecycle management, and executing bulk actions.

List 1 highlights the most common use cases for the Oracle VM Manager Command Line Interface:

Guest, server and server pool lifecycle management:

Power On, Power Off, Clone, Save as Template, Import, Migrate, Pause, Unpause, Suspend, Resume and Delete virtual machines
Manage virtual machine resources, including ISO files, virtual machine templates, and virtual machine images
Manage Oracle VM Manager users and Oracle VM Manager groups
Create and configure server pools
Manage the Oracle VM Agent

Checking the status of guests, server, and server pools:

Get a list of running guests in server pool
Get a list of active pool servers
Diagnose and troubleshoot issues with guests, servers and server pools
Get configuration and status information for guests, servers and server pools

Performing bulk operations:

With multiple guests
With multiple servers
With multiple server pools

The Oracle VM Manager Command Line Interface (ovmcli) is a stand-alone application that is written in Python that leverages the Oracle VM Manager Web Services API to communicate with Oracle VM Manager. The Oracle VM Manager Command Line Interface can be installed on any Oracle Enterprise Linux or RHEL host with connectivity to your Oracle VM Manager system. The Oracle VM Manager Command Line Interface can be accessed from a local or remote console.

The Oracle VM Manager Command Line Interface has one prerequisite, namely the python-ZSI RPM. The ovmcli and python-ZSI RPMs are hosted at ULN. The ovmcli RPM is located in the el5_i386_oracle_addons and the el5_x86_64_oracle_addons channel. The python-ZSI RPM is located in the el5_i386_addons and el5_x86_64_addons channel.

Oracle VM Manager Intra-component Communication & Firewall Requirements

Now that we have reviewed the applications that comprise Oracle VM Manager we will continue with an overview of Oracle VM Manager’s intra-component communication and firewall requirements. The goal of this section is to provide an understanding of the communication ports and system passwords required to help plan, build and support an Oracle VM environment using Oracle VM Manager.

List 2 highlights Oracle VM communication ports and system passwords used with Oracle VM Manager.

Oracle VM Manager:

The default HTTPS port for the Oracle VM Manager console is 4443.
The default HTTP port for Oracle VM Manager console is 8888.
The default HTTP port for the Oracle Database 10g Express Edition console is 8080.
The default HTTP port for the Oracle Application Server 10g console is 8888.
The default Database listening port for the Oracle Database 10g Express Edition repository is 1521.
Oracle Database 10g Express Edition accounts are SYS and SYSTEM.
Guest console access between the Oracle VM Manager console and guests use ports 5900 through 5999.
Guest console access from the Oracle VM Manager console requires a VNC password without a user name. The VNC password is assigned from the Oracle VM Manager console.

Oracle VM Server:

The Oracle VM Agent's default listening port is 8899 and requires a password. The agent password is selected during the Oracle VM server installation and can be configured from dom0 by typing “service ovs-agent configure”.

Figure 3 illustrates Oracle VM Manager including intra-component communication and firewall requirements.

Next, we will review the Oracle VM Manager Command Line Interface intra-component communication and firewall requirements.

List 3 highlights the Oracle VM Manager Command Line Interface communication ports and system passwords:

The Oracle VM Manager Command Line Interface:

The default HTTPS port for the Oracle VM Manager console is 4443.
The default HTTP port for Oracle VM Manager console is 8888.
Access to the Command Line Interface host can use ssh. The default ssh port is 22.
Guest console access from the Command Line Interface host uses ports 5900 through 5999.
Guest console access from the Command Line Interface host requires a VNC password without a user name. The VNC password is assigned in Oracle VM Manager UI.

Figure 4 illustrates the Oracle VM Manager Command Line Interface intra-component communication and firewall requirements.

The Oracle VM Management Pack Intra-component Communication & Firewall Requirements

Next, we will review the intra-component communication and firewall requirements for the Oracle VM Management Pack. The goal of this section is to provide an understanding of the communication ports and system passwords required to help plan, build, and support an Oracle VM environment using the Oracle VM Management Pack.

List 4 highlights communication ports used with the Oracle VM Management Pack.

Oracle VM Management Pack:

The default HTTP port for the Enterprise Manager UI is 4889.
The default Database listening port for the Oracle Database repository is 1521.
Guest console access from the Enterprise Manager UI uses ports 5900 through 5999.
Guest console access from the Enterprise Manager UI requires a VNC password without a user name. The VNC password is assigned from the Enterprise Manager UI .

Oracle VM Server:

The Oracle VM Agent's default listening port is 8899 and requires a password. The agent password is selected during the Oracle VM server installation and can be configured from dom0 by typing “service ovs-agent configure”.

Oracle VM Guests:

The Oracle Management Agent to Oracle Management Service communication is HTTPS on 4889.

Figure 5 illustrates Oracle VM Management Pack architecture, including intra component communication and firewall requirements.

Oracle VM Manager and Oracle VM Server Pool Management

Along with virtual machine life cycle management, Oracle VM Manager and the Oracle VM Management Pack provide management of Oracle VM servers and server pools. An Oracle VM server pool defines the management boundaries and storage requirements of one or more Oracle VM Servers.

The first step after installing Oracle VM Manager or the Oracle VM Management Pack is to create a server pool. The server pool wizard walks through the server pool creation process, which entails selecting a name for the server pool, selecting the pool owner and finally adding an Oracle VM Server to the pool. Once the server pool is created, resources such as ISO files, templates, administrative users and groups can be configured. Additional server pool members can be added to the server pool as long as the server pool members are accessing a shared repository.

A server pool that is configured with local storage is limited to a single Oracle VM server, without HA or Live Migration functionality. To add additional Oracle VM Servers to a pool, and to enable HA, a shared repository is a requirement. Shared storage allows virtual machines to start, run and migrate to any Oracle VM Server within a pool.

Oracle VM Server Agent Roles

Oracle VM Manager and the Oracle VM Management Pack facilitate centralized guest and server pool management using an agent-based architecture. When an Oracle VM server is added to a server pool, Oracle VM agent roles are assigned to each server pool member. There are a total of three Oracle VM agent roles; 1) the Server Pool Master, 2) the Utility Server and 3) the Virtual Machine Server. When an Oracle VM server is added to a server pool it can be assigned one, two, or all three of the agent roles.

List 5 explains each of the three Oracle VM agent roles.

Server Pool Master
- The server pool master is the principal server pool role within a server pool. The server pool master is the only agent role that communicates with Oracle VM Manager. The server pool master dispatches commands from Oracle VM Manager to other servers within a server pool. There can be only one server pool master in a server pool at any one point in time. The server pool Virtual IP feature in 2.2 and above, will detect and automatically failover a failed server pool master server to the first pool member that can lock the OVS repository.
Utility Server
- The utility server role is responsible for I/O intensive operations such as virtual machine creation and removal, server pool creation and removal as well as copying and moving files. The server pool master dispatches operations to utility server agents. There can be one or more utility server agents in a server pool. When there are multiple utility server agents in a pool, the server pool master will select the least loaded utility server to conduct a task. For production Oracle VM systems, consider using dedicated utility servers to isolate the impact of I/O intensive operations. For example, co-locating the utility server agent with the virtual machine server agent could impact guest performance during utility server operations.
Virtual Machine Server
- Servers with the virtual machine server agent role are responsible for allocating CPU, memory, and disk resources to the guests in a server pool. There can be one or more virtual machine servers in a server pool.

Oracle VM Manager System Design Considerations & Server Pool Examples

Next, we will review Oracle VM Manager system design considerations followed by four different Oracle VM server pool examples.

Oracle VM employs a scalable architecture with Oracle VM Manager dispatching commands to each server pool master, which in turn dispatches commands to pool members. Oracle VM architecture is exceptionally bandwidth efficient since the intra-component traffic is limited between; a) Oracle VM Manager and each server pool master agent and b) each server pool master agent and it's pool members. For example, an Oracle VM server farm with 20 pools, each pool with 20 servers, would have a total of 20 communication channels between Oracle VM Manager and each pool master. If Oracle employed a direct Oracle VM Manager to Oracle VM server relationships, the same Oracle VM server farm with 20 pools, each pool with 20 servers, would have a total of 400 communication channels.

The total number of Oracle VM servers that Oracle VM Manager can support is limited by the type of repository used by Oracle VM Manager. The default Oracle XE database has a 4GB limit for on-disk storage, which could become a bottleneck supporting Oracle VM server farms with more than 50 Oracle VM servers. Using Oracle Standard Edition, Oracle Standard Edition One or Oracle Enterprise Edition, eliminates the 4GB limit for on-disk storage limitation. For example, if your Oracle VM repository is not Oracle XE but an Oracle Standard Edition database on a reasonable server platform with a gigE networking, Oracle VM Manager could support 1000+ Oracle VM servers with thousands of guests.

Note: Oracle VM has a soft limit of 32 servers per HA enabled server pool.

Next, we will evaluate four separate Oracle VM server pool examples. The first example shows a server pool with one Oracle VM server, hosting four virtual machines, managed by a dedicated Oracle VM Manager server. Next, we show a server pool with two Oracle VM pool members with shared storage, hosting six virtual machines, managed by a dedicated Oracle VM Manager server. This is followed by an example showing an Oracle VM farm with two pools. One of the server pools has two Oracle VM pool members with shared storage and the second pool has a single Oracle VM pool member with local storage. We will conclude with an example showing a multi-site Oracle VM farm with two server pools, managed by a dedicated Oracle VM Manager server.

Figure 6 shows a server pool with one Oracle VM server, hosting four virtual machines, managed by a dedicated Oracle VM Manager server.

Note: Oracle VM Manager can run on a physical server or as a virtual machine.

Figure 6 shows an Oracle VM server farm with two physical servers. One of the servers is an Oracle VM server pool member with all three agent roles enabled, hosting four virtual machines, with local storage. The second server is hosting Oracle VM Manager. Oracle VM Manager is responsible for the management of the server pool, the server pool resources and the virtual machines.

Oracle VM Manager communicates with Oracle VM agents using XML-RPC. A single Oracle VM Manager or Oracle VM Management Pack instance can manage an unlimited number of Oracle VM servers and server pools as long as; a) there is sufficient bandwidth between Oracle VM Manager or Enterprise Manager and the server pools and b) that Oracle VM Manager repository can store the Oracle VM Manager data, i.e. the default 10G XE database has a 4GB limit for on-disk storage. If you use Oracle 10G or 11G Standard Edition or Oracle 10G or 11G Enterprise Edition, then there is no limit besides database resources.

After the creation of an Oracle VM server pool, additional members can be added to a server pool as long as all server pool members are using the same-shared storage. For example, the server pool shown in Figure 6 can contain only one pool member because the Oracle VM server uses local, not shared storage. If the server pool in Figure 6 used shared storage, additional member servers could be added to the server pool.

Figure 7 shows a server pool with two Oracle VM pool members with shared storage, hosting six virtual machines, managed by a dedicated Oracle VM Manager server.

The next example, Figure 8 shows an Oracle VM farm with two pools. One of the server pools has two Oracle VM member servers with shared storage and the second server pool has a single Oracle VM member server with local storage. A dedicated Oracle VM Manager server manages both server pools.

The next example, Figure 9 shows a multi-site Oracle VM farm with two server pools managed by a dedicated Oracle VM Manager server.

Oracle VM Storage

In the server pool section, we reviewed that Oracle VM supports both local and shared storage. A server pool with local storage is limited to a single Oracle VM server without the ability to use HA or Live Migration. To add additional Oracle VM servers to a pool and to enable HA and Live Migration, a shared repository is required. A shared repository allows virtual machines to start, run and migrate to any Oracle VM server within the pool.

A default Oracle VM server installation creates a “local” OCFS2 virtual machine file system (VMFS) that is mounted under /OVS. When a server pool has multiple Oracle VM servers a shared repository is required. Oracle VM supports OCFS2 on SAN or iSCSI as well as NFS shared repositories. A shared repository is referred to as an OVS repository. With Oracle VM 2.2 the /OVS directory is a symbolic link mounted under the /var/ovs/mount/uuid directory. With Oracle VM 2.1 an OVS repository is mounted under /OVS. Local and shared storage is also referred to as back-end storage.

When an Oracle VM 2.2 server boots the o2cb and ocfs2 services are started which bring up the OCFS2 clusterstack. Once the OCFS2 clusterstack is running, the Oracle VM agent queries the OCFS2 clusterstack to assign the pool master role or validate that the pool master role is active. Next, the Oracle VM agent queries the pool master agent and re-configures its /etc/ocfs2/cluster.conf file and adds all active pool member servers to the cluster. As each Oracle VM server in the pool starts, you'll see the cluster.conf being updated across the pool. Next, the Oracle VM agent mounts the root repository and checks that /OVS is symlinked correctly. Finally, the Oracle VM agent mounts any remaining extended repositories.

Note: The o2cb and ocfs2 services are located in the /etc/rc.d/init.d directory.

With Oracle VM 2.1.x, the ovsrepositories service will automatically try to mount the OVS repository during the boot process. After ovsrepositories mounts the repository the Oracle VM agent will pull the cluster configuration from Oracle VM master agent and configure the cluster and start the cluster heartbeat.

Note: The ovsrepositories service is located in the /etc/rc.d/init.d directory.

As shown in the next example, typing mount on a 2.2 Oracle VM server would list the OVS repository.

#mount

/dev/sdb1 on /var/ovs/mount/AEFAB9DB88114C08AABA6340195AB564 type ocfs2 (rw,_netdev,heartbeat=local)

As shown in the next example, typing ls –l on an Oracle VM 2.2 server would list the /OVS symbolic link pointing to the /var/ovs/mount/uuid directory.

# ls -l /OVS

lrwxrwxrwx 1 root root 47 Oct 18 14:54 /OVS -> /var/ovs/mount/AEFAB9DB88114C08AABA6340195AB564

Oracle’s OCFS2 file system is a general-purpose cluster file system with distributed lock management (DLM) and other cluster-aware code that is integrated in to Oracle VM. To provide OCFS2 functionality with an NFS repository, Oracle creates a hidden OCFS2 file-backed block device that uses OCFS2 DLM management. The ability to use OCFS2 DLM management for OCFS2 and NFS allows Oracle VM to monitor both OCFS2 and NFS shared disk subsystem, as well as the network, to ensure cluster awareness and to better respond to HA events.

The following example shows the default OVS repository directory structure, with a brief explanation of each sub directory:

/OVS (Root directory)

| iso_pool (ISO files storage, requires VT chip extensions)

| lost+found (The lost and found directory)

| publish_pool (Public virtual machine storage)

| running_pool (Published virtual machine storage)

| seed_pool (Virtual Machine template storage)

| sharedDisk (Shared virtual disk storage)

Figure 10 shows an Oracle VM server pool with an OVS root repository.

Oracle VM provides a tool set to manage OVS repositories. For example, Oracle VM 2.2 uses the /opt/ovs-agent-2.3/utils/repos.py python script to list, add, delete, initialize and to flag repositories. You can type /opt/ovs-agent-2.3/utils/repos.py –h to list the help menu. Oracle VM 2.1 uses the deprecated ovs-makerepo script to manage repositories.

Increasing the capacity of an OVS repository involves creating additional LUNs and adding the LUNs to extend the root OVS repository. Adding a LUN to an existing root OVS repository is called extending the OVS repository. The OCFS2 file system used with Oracle VM 2.x does not have volume management, so adding a new LUN to an existing OVS root repository will actually mount the new LUN under the /var/ovs/mount/uuid directory, and add a symbolic link in the root OVS repository.

For example, typing the mount command from dom0 on a 2.2 Oracle VM server with an OVS root and extended repository would display the following.

#mount

/dev/sdb1 on /var/ovs/mount/AEFAB9DB88114C08AABA6340195AB564 type ocfs2 (rw,_netdev,heartbeat=local)

/dev/sdc1 on /var/ovs/mount/0E67A3C1B113443C91C4FB65D6AD150E type ocfs2 (rw,_netdev,heartbeat=local)

As shown below typing ls –l on the same Oracle VM 2.2 server would still only display the root OVS repository.

# ls -l /OVS

lrwxrwxrwx 1 root root 47 Oct 18 14:54 /OVS -> /var/ovs/mount/AEFAB9DB88114C08AABA6340195AB564

As shown in the next example to view the root and extended OVS repositories from dom0 type “/opt/ovs-agent-2.3/utils/repos.py –l”.

# /opt/ovs-agent-2.3/utils/repos.py -l

[ * ] aefab9db-8811-4c08-aaba-6340195ab564 => /dev/sdb1

[ ] 0e67a3c1-b113-443c-91c4-fb65d6ad150e => /dev/sdc1

In the above example the “*” indicates the root OVS repository.

The following example shows a root OVS repository with an extended OVS repository.

/OVS (Root directory)

| 0E67A3C1B113443C91C4FB65D6AD150E (Extended file system)

| iso_pool (ISO files storage, requires VT chip extensions)

| lost+found (The lost and found directory)

| publish_pool (Public virtual machine storage)

| running_pool (Published virtual machine storage)

| seed_pool (Virtual Machine template storage)

| sharedDisk (Shared virtual disk storage)

Figure 11 shows an Oracle VM server pool with a root and extended OVS repository.

Extending an OVS repository provides several advantages; a) the ability to increase storage capacity and b) the ability to profile and group together virtual machines on LUNs with similar I/O needs. For example, you could extend an OVS repository with multiple LUNs, each LUN with a different QOS setting. Virtual machines with low I/O requirements can be placed on a lower performance LUN, while virtual machines with high I/O needs can be placed on their own high performance LUN.

When an OVS repository is extended with Oracle VM 2.2 the new LUN will be mounted under the /var/ovs/mount/uuid directory and a symbolic link to the LUN is added in the root OVS repository. Oracle VM Manager and the Oracle VM Virtualization Pack will automatically place resources such as virtual machines, templates or ISO files on the OVS repository with available space. If you wish to place virtual machines into a specific repository to take advantage of QOS settings, the virtual machine must be powered off and then moved to the desired LUN. Once the virtual machine is placed on the desired LUN, the virtual machine must be imported from Oracle VM Manager or the Oracle VM Virtualization Pack via the Resources tab.

List 6 highlights Oracle VM storage best practices:

The storage back-end should be configured with some form or striping (with parity), e.g. RAID 5.
Use integrated multi-pathing support in Oracle VM Server for SAN and iSCSI storage - DM-MPIO (device-mapper-multipath)
Use integrated drivers with Oracle VM dom0 – i.e. not third-party drivers.
Present all LUNs directly from SAN (FC and/or iSCSI/etc) to dom0 then export to the LUN to domUs via vm.cfg's i.e. physical backed block devices.
OCFS2 doesn't currently support online file system resize, so rather than extending current LUN at SAN storage array, extend the VMFS using /usr/lib/ovs/ovs-makerepo or with 2.2 the /opt/ovs-agent-2.3/utils/repos.py script.

Oracle VM High Availability

Oracle VM offers two high availability (HA) options; a) guest HA and b) Live Migration. Oracle VM HA is used to minimize unplanned downtime and Live Migration is used to eliminate planned downtime. In this section, we will briefly review high availability and service level agreements (SLAs), Oracle’s high availability solutions and Oracle VM HA and Live Migration functionality and architecture.

High Availability Introduction

High availability is a strategy that maintains operational continuity in the event of system failure. For example, if you can not access your email due to system failure, the email system is unavailable. We typically use the term "downtime" to refer to systems when they are unavailable. We employ high availability strategies to minimize or eliminate planned and unplanned downtime.

Planned downtime represents scheduled outages that are planned in advance to reduce the loss of availability to a system. Planned downtime is typically the result of maintenance events that revolve around repairs, backups, or upgrades. Unplanned downtime can be the result of any uncontrollable random failures.

High availability is an integral data center strategy that allows us to meet service level agreements (SLAs) by minimizing or eliminating planned and unplanned downtime. A service level agreement is a part of a service contract that defines the level of service. An SLA will typically specify the levels of availability, which determine which high availability solution will allow you to meet your availability SLA. For example, a mission critical application would have an SLA that requires operational continuity in the event of system failure, which would necessitate the use of high availability technologies. Conversely, a non mission critical application might have an SLA that allows several hours of downtime, which may or may not require the use of high availability technologies.

Oracle has a wide variety of high availability solutions for databases, applications and virtualization that offer different levels of availability. For example, RAC is one of Oracle’s database high availability solutions that offers operational continuity in the event of node failure. If one RAC node in a two node RAC cluster fails, the Oracle database would continue to run without disruption of service. Oracle DataGuard and Oracle ApplicationGuard are two other high availability solutions that offer operational continuity in the event of failure.

Oracle VM offers two high availability options; a) guest HA and b) Live Migration. Oracle VM HA automatically restart guests when; a) a guest is detected in an unexpected state, i.e. the guest's status is "Running" although the guest is indeed powered off or b) when an Oracle VM pool member fails or restarts, which minimize unplanned downtime by restarting guests. Live Migration is used to eliminate planned downtime by migrating running guests off one Oracle VM pool member to another during a maintenance event, i.e. for repairs or an upgrade.

In contrast to RAC, DataGuard, ApplicationGuard and Oracle VM Live Migration, Oracle VM HA does not offer operational continuity. For example, when an Oracle VM HA event occurs a guest or group of guests will be restarted and the services running on the guests will be unavailable for the duration of the HA event. An Oracle VM HA event "typically" takes no longer than 5 minutes. Live Migration offers operational continuity, by allowing an administrator to migrate running guests from one pool member to another.

Note: All x86 virtualization solution i.e. VMware, Hyper-V and XenServer do not offer operational continuity. When a HA event occurs with any x86 virtualization solution, a guest or group of guests will be restarted and the services running on the guests will be unavailable for the duration of the HA event.

The key takeaway is that SLAs will dictate the level of high availability. An SLA for a mission critical application will define the need for operational continuity in the event of failure. An SLA for a non mission critical application might have an SLA that allows several hours of downtime, which may or may not require the use of high availability technologies. If your SLA allows for a brief outage, Oracle VM HA could replace or augment your existing high availability clustering solutions.

Oracle VM HA is complementary with RAC, DataGuard and ApplicationGuard. For example, Oracle VM HA offers value with RAC, DataGuard and ApplicationGuard by restarting a failed RAC, DataGuard and ApplicationGuard node.

Oracle VM High Availability

Oracle VM HA will automatically restart guests when; a) a guest is detected in an unexpected state, i.e. the guest's status is "Running" although the guest is indeed powered off or b) when an Oracle VM pool member fails or restarts. Oracle VM leverages a lightly modified OCFS2 cluster stack, o2dlm to monitor the status of each Oracle VM server using a network and storage heartbeat, which is also referred to as a “quorum disk”. Each Oracle VM server in an HA enabled pool will read and write small amounts of status data to the same segment of the root shared repository on a pre-scheduled basis. Having each Oracle VM server in a pool regularly read and write status data to the root storage repository allows the Oracle VM pool to quickly detect and respond to Oracle VM server failures.

Oracle VM HA is enabled at the pool and guest level using Oracle VM Manager or the Oracle VM Management Pack. To enabled HA for guests, HA must first be enabled at the pool level, which requires a shared root storage repository. Enabling HA will allow you to configure if a guest will or will not be restarted if an HA event occurs.

List 7 reviews the HA settings that are maintained on each HA enabled server pool member:

Node management information. The node management information is stored in the cluster.conf file in /ect/ocfs2/cluster.conf. The cluster.conf file lists all of the hosts in the cluster, their node #, etc.
Heartbeat status information. The heartbeat status information polls if any nodes failed to update/respond to their heartbeat as expected. The heartbeat traffic is TCP on port 7777. Each Oracle VM server in a pool must be able to communicate to all of the pool members over TCP on port 7777.
Network, Storage (quorum disk info) & Distributed lock manager. The Network, Storage and Distributed lock manager is used to prevent any server and guest from accessing shared storage unless it has exclusive lock on that storage to prevent corruption.

Oracle VM HA Network Considerations

Oracle VM HA puts negligible traffic over the network, 1 packed every 2 seconds for heartbeat traffic plus Distributed Lock Manager (DLM) traffic when locks are taken during guest start and stops.

The HA heartbeat traffic is latency sensitive, and the allowable latency for heartbeat traffic is 60 seconds. For example, network heartbeat latency is 30 seconds, however, after 30 seconds the cluster will attempt to establish a disk heartbeat. If the cluster establishes a disk heartbeat, then it will try and reconnect via the network. The second disk heartbeat timeout latency is set to 30 seconds, so it takes a full 60 seconds AND inability to see shared disk to cause a cluster HA event.

Oracle VM Live Migration

Live Migration moves running guests between Oracle VM pool members across a LAN without loss of availability. Live Migration has two primary use cases. The first use case is to eliminate planned downtime by migrating running guests from one Oracle VM pool member to another, during planned maintenance events. The second use case is to migrate running guests from an over utilized pool member to a pool member with available resources.

Live Migration Requirements

There are three requirements for Live Migration. The first requirement is that the source and target servers CPUs must be identical. The second requirement is a shared OVS repository. The final requirement is that the server pool must have excess CPU and memory capacity to accommodate the CPU and memory requirements of the largest guest running on any pool member at all times.

As of this writing, Oracle VM uses Xen 3.4.0. Live Migration with Xen 3.4.0 requires that the source and target servers CPUs are identical. To be more specific, the CPU family and model of the source and target servers must be the same. You can validate the CPU family and model by look at /proc/cpuinfo.

Live Migration copies a guest’s memory pages from one server to the other. Once the guest moves from the source to the target server, the guest expects the same network, the same bridge, the same connections and the same storage. An Oracle VM pool configuration will ensure that a migrated guest’s environment is identical on all pool members.

How Oracle VM Live Migration Works

Oracle VM uses an iterative pre-copy method to migrate running guests over a TCP SSL connection between two pool members. A Live Migration event starts when the source server sends a migration request to the target server, which contains the guest resource requirements. If the target accepts the migration request, the source starts the iterative pre-copy phase. The iterative pre-copy phase starts by iteratively copying the guest’s memory pages from the source to the target server. During the pre-copy phase if a memory page changes it is marked dirty and resent. Once the majority of the pages are copied, the stop and copy phase begins. The stop and copy phase starts by pausing the guest while the remaining dirty pages are copied to the target, which usually takes 60 to 300 milliseconds. Once the pages are copied to the target, the guest is started on target server.

The following example will walk through each step of a Live Migration between server 1 and server 2.

Server 1 sends a migration request to the server 2.
Next, server 2 accepts the migration request and the in memory content of the guest are copied from server 1 to server 2, while the guest is still running.
Next, the stop and copy phase starts by pausing the guest while the remaining dirty memory pages are copied to server 2.
Once the dirty memory pages are synchronized between server 1 and server 2, the guest is unpaused and is running on server 2.

Oracle VM Live Migration and HA Network Considerations

Collocating the HA heartbeat and Live Migration traffic on the same network is suitable for most environments. You may want to separate the HA heartbeat and Live Migration traffic if you are making extensive use of Live Migration. In extreme cases, Live Migration traffic may cause unacceptable latencies with the cluster heartbeats.

For example, if the guest has 1 GB of memory allocated there should be no issues collocating the HA heartbeat and Live Migration traffic on the same network / VLAN. If a guest has 8 or 16 GB of allocated memory, the first pass of the Live Migration will be very intensive and could saturate the network. With a 16GB VM about 20GB actually gets copied over the wire during a Live Migration, since some contents change during the process and thus get copied several times. Gig Ethernet means roughly 100MB/sec wirespeed so about 200sec to complete the migration. 60sec is the heartbeat timeout based on retries on both network and storage heartbeats. However that doesn't mean you will have a problem that NOTHING is getting through during the 200 second live migration, particularly since the migration is performed at the page-level rather than one huge file. With the heartbeats being so small, it is very likely that they will squeeze through OK.

The key takeaway is if Live Migration is not heavily utilized, it would be ideal to combine the heartbeat and management functions (HA, Live Migration and Agent traffic) onto one private network and multi-home the Oracle VM Manager or Oracle VM Management Pack Server (on the heartbeat VLAN and on a public VLAN) for management access.

Oracle VM Server Introduction

This section will start with a brief introduction to Oracle VM Server followed by a high level architectural overview of Oracle VM server and each Oracle VM server component.

Oracle VM server is Oracle’s Open Source Xen distribution, which allows multiple guest operating systems to run concurrently on a single piece of x86 hardware. Oracle VM server is a type 1 hypervisor that installs directly on x86 or x86-64 hardware unlike a type 2 hypervisor which is a software application that runs on top of an operating system. Oracle VM servers are managed as server pools from either Oracle VM Manager or the Oracle VM Management Pack, but not both. The Oracle VM server source code and ISO files can be freely downloaded from the Oracle eDelivery portal.

Oracle VM server is comprised of several components that allow both paravirtualized and hardware virtualized guests to share the CPU and memory resources of a single piece of x86 hardware. List 8 highlights Oracle VM server’s components.

The Xen hypervisor
Domain 0
Domain Management Services (DMS)
Domain U Paravirtualized (PV) Guests
Domain U Hardware Virtualized Machine (HVM) Guests

Figure 12 shows an architectural overview of the Oracle VM server components.

The Xen Hypervisor

Oracle VM server uses the Xen hypervisor as its virtual machine monitor (VMM). Xen is a type 1 hypervisor since Xen installs directly on the host hardware, unlike type 2 hypervisors which are software applications that run on top of an operating system. Xen is responsible to monitor guest operating systems and to schedule guest operating system requests to the host CPU and memory. Xen is not responsible and has no knowledge of networking, external storage, video, or common I/O functions.

Oracle makes subtle changes to the Xen.org code that creates a unique Xen distribution which is re-distributed as Oracle VM server. Oracle has developed a Xen configuration for Oracle VM that enhances database and middleware performance that other Xen distributions do not use. As an example, we could ask, "Are Red Hat Linux and SUSE Linux the same?" Though Red Hat Linux and SUSE Linux are both GPL Linux, they are still different. Red Hat Linux and SUSE Linux often learn from each other and may eventually add functionality that was developed by the other, but they are not the same. The same logic applies to Oracle VM and all other Xen distributions i.e. XenServer, Red Hat, Novell, Debian, etc...

Xen supports two virtualization modes, paravirtualization mode and hardware virtualization mode. Xen can support paravirtualization mode on any x86 or x86-64 hardware platform. Hardware virtualization mode requires CPU virtualization extensions on the host i.e. Intel VT or AMD-V processors. Xen supports both paravirtualization mode and hardware virtualization mode on a single host with CPU virtualization extensions.

Paravirtualization mode requires the guest operating system to run a special Xen kernel and Xen network and IO drivers. Guests that run under paravirtualization mode are called paravirtualized guests. Paravirtualized guests are aware of the Xen hypervisor and run without emulated hardware. Paravirtualization allows guest hypervisor interaction to be grouped together and reused, which has a very low overhead. Xen paravirtualized guest kernels are available for Linux, NetBSD, FreeBSD, and Novell Netware operating systems.

In hardware virtualization mode, guests require exclusive hypervisor interaction as well as a dedicated QEMU process for each guest. Guests that run under hardware virtualization mode are called hardware virtualized machines (HVM). Paravirtualization mode offers near native performance in contrast to hardware virtualization mode. Hardware virtualization mode has a higher overhead, due in part, to the use of device emulation and the use of the CPU virtualization extensions.

Oracle VM supports both Red Hat Linux and Oracle Enterprise Linux in paravirtualization mode and Red Hat Linux, Oracle Enterprise Linux, Solaris 10 and Windows in hardware virtualization mode.

Note: From Red Hat Linux and Oracle Enterprise Linux 4.7 onwards the stock kernels provide paravirtualized network and IO drivers for hardware virtualized guests. From Solaris 10 Solaris 10 10/09 onwards the stock kernels also provide paravirtualized network and IO drivers for hardware virtualized guests. Windows guests do not have native paravirtualization support. Oracle has released paravirtualized network and IO driver available at http://edelivery.oracle.com/linux that should be installed in each Windows guests to provide a performance boost.

Table 2 lists the pros and cons between Paravirtualization and Hardware virtualization mode.

Techniques	Purpose	Pros	Cons
Paravirtualization (PV) Oracle VM & Xen	Modify the guest operating systems to integrate with the hypervisor	Excellent performance vs. bare-metal	PV Xen kernel is required‏ for the guest operating systems
Hardware virtualization (HVM) Oracle VM & Xen	Design the hardware to create elements of the virtualization and emulation environment	Can use unmodified guest operating systems	Poor performance verses PV and CPU virtualization extensions are required

Dom0

The Xen hypervisor works in concert with a control domain, which, in Xen terminology, is called dom0 or domain 0. Xen, as well as dom0, are automatically started when Oracle VM server boots. Dom0’s role includes providing an interface to Xen, hardware detection, device support, domU management, and presenting domUs with networking, storage, video, and I/O interfaces. In Xen terminology unprivileged domains are called domU or domain U. The industry term for domU is virtual machine. Dom0 and domUs are both virtual machines.

Each Xen system has a unique dom0. Oracle VM’s dom0 was built from the ground up to provide the best performance for high IO Oracle workloads. Oracle’s dom0 is a purpose built paravirtualized 32bit Oracle Enterprise Linux guest with broad device, file systems, software RAID and volume management support. Oracle’s dom0 can be considered an appliance, which is completely maintained by Oracle.

Although Oracle’s dom0 is GPL Linux, each Xen distribution has unique dom0. For example, when Xen is enabled on a Linux operating system, the entire Linux operating system, along with user applications becomes dom0. For example, a default Oracle VM Server installation contains a tuned version of Xen and dom0 with a little over 300 RPM packages. When you enable Xen on Linux you get a vanilla Xen configuration with a dom0 that could contain thousands of RPM packages, which is not an optimized virtualization platform. Oracle VM’s Xen and dom0 configurations are tuned to provide the best performance for high IO Oracle workloads.

DomU

Paravirtualized (PV) Guests

DomU paravirtualized guests use a special Xen kernel and a Xen paravirtualized block and a Xen paravirtualized network driver. Paravirtualization allows domU hypervisor interaction to be grouped together and reused, which has significantly lower overhead when compared to hardware virtualization mode. In hardware virtualization mode domU interaction requires exclusive hypervisor interaction.

Dom0 includes two drivers; a) the network backend driver and b) the block backend driver to support network and IO requests for paravirtualized domUs. The network backend driver communicates through dom0 to the local networking hardware to process all domU networking requests. The block backend driver communicates through dom0 to the local storage to process all domU storage related requests. The key takeaway is that dom0 runs the paravirtualized backend drivers, and domU runs the paravirtualized frontend drivers.

Figure 13 shows the placement of the network and block backend drivers in dom0 along with the network and block frontend drivers in a paravirtualized domU.

DomU

Full Virtualization – Hardware Virtualization Machines (HVM)

In hardware virtualization mode, either Intel VT or AMD-V processors are required. Intel VT or AMD-V processors are not a requirement for paravirtualized mode. An advantage of hardware virtualization mode is the ability to run unmodified operating systems like Windows and legacy Linux kernels that do not have Xen support. Hardware virtualization mode runs QEMU device emulation in dom0, which exposes a set of virtual hardware to domUs.

Dom0 also runs a dedicated QEMU process for each hardware virtualized machine. Hardware virtualized machines do not use the same dom0 backend and domU frontend paravirtualized drivers as paravirtualized domUs. Hardware virtualized machine use emulated drivers that are created and managed by the quemu-dm processes.

Figure 14 shows the quemu-dm process in dom0 and the emulated QEMU interface provided to a hardware virtualized domU.

Domain Management Services

Dom0 runs various standardized Linux daemons that are used for domain control and management. The services work together with Xen to provide the management infrastructure for Oracle VM server. Next, we will review the xend, xm, xenstored, libxenctrl and the Oracle VM agent services.

Xend

The Xen daemon, (xend), is a python application that runs as root in dom0 as a background process. Xend is the system manager for Xen that executes privileged system management functions from dom0. The xm (Xen Master) user command is the primary source of the privileged system management commands that are passed to the Xen daemon via XML RPC. Xend then processes the privileged system management commands and uses the libxenctrl library communicate the privileged system management commands to the Xen hypervisor.

The Xen Master (xm) command is the primary user interface used to pass privileged system management commands via the Xen daemon to the Xen hypervisor.

Xenstored

The XenStore daemon, xenstored creates a centralized configuration database called the XenStore that is shared between domains. User space tools and kernel code can both write to the XenStore. The XenStore database stores key-value pairs as a trivial database (tdb) file located at /var/lib/xenstored. The structure of the XenStore is similar to a filesystem that’s composed of directories which can contain other directories.

We can query the XenStore using the xenstore-ls and xenstore-list commands. The next example lists the XenStore top-level categories from the root by typing “xenstore-list /” from dom0 as root.

# xenstore-list /

tool

local

To list the contents of the vm directory type “xenstore-list /vm” as shown in the next example.

# xenstore-list /vm

00000000-0000-0000-0000-000000000000

3dfe8997-84c4-ca2c-9f4a-a3b2a8fb207e

96b37296-2836-8c84-268e-ef9d7be8e74a

7c88b1ca-583c-926c-186a-8f9585710f50

cbbfa165-99e9-ab76-e2de-22600020de8e

362f239d-dece-79df-a1c6-d8d3aa0d5627

d428ba07-31b9-5667-2085-8753a0342425

f8725f79-c6c8-26d4-a51e-1f32cf010c84

68d1f842-0e45-7d14-51d4-80319d839d41

The above example lists the contents of the vm directory that shows the Universal Unique Identifier (UUID) of each domain on the Oracle VM server.

To view the entries for a specific domain type “xenstore-ls /vm/UUID” as shown in the next example.

# xenstore-ls /vm/68d1f842-0e45-7d14-51d4-80319d839d41

image = "(linux (kernel ) (videoram 4) (device_model /usr/lib/xen/bin/qemu-dm) (notes (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pa\..."

ostype = "linux"

kernel = "/var/run/xend/boot/boot_kernel.7385hs"

cmdline = "ro root=/dev/VolGroup00/LogVol00 rhgb quiet "

ramdisk = "/var/run/xend/boot/boot_ramdisk.hoNXgE"

vncpasswd = "\000"

device = ""

vkbd = ""

0 = ""

frontend = "/local/domain/9/device/vkbd/0"

frontend-id = "9"

backend-id = "0"

backend = "/local/domain/0/backend/vkbd/9/0"

vfb = ""

0 = ""

frontend = "/local/domain/9/device/vfb/0"

frontend-id = "9"

backend-id = "0"

backend = "/local/domain/0/backend/vfb/9/0"

console = ""

0 = ""

frontend = "/local/domain/9/device/console/0"

frontend-id = "9"

backend-id = "0"

backend = "/local/domain/0/backend/console/9/0"

vbd = ""

768 = ""

frontend = "/local/domain/9/device/vbd/768"

frontend-id = "9"

backend-id = "0"

backend = "/local/domain/0/backend/vbd/9/768"

vif = ""

0 = ""

frontend = "/local/domain/9/device/vif/0"

frontend-id = "9"

backend-id = "0"

backend = "/local/domain/0/backend/vif/9/0"

on_xend_stop = "ignore"

shadow_memory = "0"

uuid = "68d1f842-0e45-7d14-51d4-80319d839d41"

on_reboot = "restart"

start_time = "1256145759.78"

on_poweroff = "destroy"

bootloader_args = "-q"

on_xend_start = "ignore"

on_crash = "restart"

xend = ""

restart_count = "1"

last_shutdown_reason = "reboot"

vcpus = "1"

vcpu_avail = "1"

bootloader = "/usr/bin/pygrub"

name = "v5164node1"

Libxenctrl

Libxenctrl is a C library interface that allows Xend to communicate with the Xen hypervisor from dom0. Libxenctrl leverages a special dom0 driver, privcmd to send requests to the Xen hypervisor.

Quemu-dm

Hardware virtualization mode runs QEMU device emulation in dom0, which exposes a set of virtual hardware to domUs. Dom0 also runs a dedicated QEMU process, quemu-dm for each hardware virtualized machine. Hardware virtualized machines do not use the paravirtualized drivers, hardware virtualized machines use emulated drivers that are created and managed by quemu-dm processes.

Virtual Frame Buffer

Each running guest has a virtual frame buffer with a unique VNC virtual frame buffer port, starting from 5900. The virtual frame buffer runs on the Oracle VM Server that’s hosting the guest and acts like a remote KVM console for guests. The virtual frame buffer supports guest console access using a VNC client or Oracle VM Manager's integrated Java client.

Guest console access has many advantages such as not requiring the guest’s networking to be setup or started in order to access a guest console, which is very handy if networking is unavailable and when the administrator is remote. Of course, you can also set up a separate VNC or RDC service within the guest itself to access a guest console just like on a physical system.

If you select the guest radio button in Oracle VM Manager you will see a 'Console' button, which will start the Java TightVNC client. If the 'Console' button is grayed out or does not work, you are missing the prerequisite software. Please visit Connecting to a Virtual Machine's Console for details on how to set up and install the guest console prerequisite software.

Oracle VM Agent

The Oracle VM agent is a python application that is installed by default in dom0. As discussed in the Oracle VM Manager section, Oracle VM Manager and the Oracle VM Management Pack communicate with a server pool’s master agent that dispatches requests to other non-master pool members.

Oracle VM Networking

In this section we will explain the default Oracle VM 2.1.x network configuration.

Oracle VM ships with a flexible network configuration that can be used as-is or can be modified to meet your business requirements. Oracle VM 2.1.x uses Xen bridging to set up the networking for guest traffic. The bridging configuration allows all domUs to appear on the network as individual hosts. All virtual network device communication routes through a Xen bridge.

Oracle VM networking can be quite confusing because dom0 shows a lot of different network interfaces and it is not clear which interface is doing what. Let’s start our Oracle VM network configuration review by examining the output from “ifconfig” followed by “brctl show” from an Oracle VM Server with two NICs.

# ifconfig

eth0 Link encap:Ethernet HWaddr 00:30:48:7F:30:84

inet addr:192.168.4.8 Bcast:192.168.4.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:25096112 errors:0 dropped:0 overruns:0 frame:0

TX packets:24382466 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:1325145485 (1.2 GiB) TX bytes:3139987566 (2.9 GiB)

eth1 Link encap:Ethernet HWaddr 00:30:48:7F:30:85

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:8829 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:1541737 (1.4 MiB) TX bytes:0 (0.0 b)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:780880 errors:0 dropped:0 overruns:0 frame:0

TX packets:780880 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:212372625 (202.5 MiB) TX bytes:212372625 (202.5 MiB)

peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF