Oracle VM Centralized Logging

Oracle VM for x86 Troubleshooting Made Easier

Distribution or derivative of the work in any form for commercial purposes is prohibited unless prior permission is obtained from the copyright holder.

Author Martin Foster

Editor Roddy Rodstein

By default Oracle VM 2.x logs all events locally. Logging events locally makes troubleshooting Oracle VM server pool issues a challenge, because different log information is being echoed to different local log files. In this chapter, we will walk through a centralized logging configuration for Oracle VM that makes troubleshooting Oracle VM much easier when compared to the default local Oracle VM logging configuration.

As of Oracle VM 2.2, the Oracle VM agent's logging functionality is customizable by using the Python Logger class configuration file. The Oracle VM Manager application runs on OC4J, a JSP container that has log4j style configuration capabilities. Both log4j and python's logger do support logging to syslog.

Change Log

Revision	Change Description	Updated By	Date
1	Draft Release	Roddy Rodstein	03/28/11

Table of Contents

Change Log

Upgrade the Oracle VM Server and Manager Local Syslog Daemon

Oracle VM Server Syslog Configuration

...Example /etc/ovs-agent/logger_server.ini file

How to Configure Rsyslog on Oracle VM Server

...Example /etc/rsyslog.conf file

...Example /etc/sysconfig/rsyslog file

How to Configure the Central Log Host

...Example /etc/sysconfig/rsyslog file

The Oracle VM Manager Syslog Configuration

...OC4J Logging

...Oracle VM Manager Logging Configuration Files

…...Example /etc/rsyslog.conf file

...How to Make Rsyslog Read the oc4j.log file

...Oracle VM Manager /etc/rsyslog.conf and Adding imfile Support

…...Example /etc/rsyslog.conf file

...Empty the Existing Oracle VM Manager oc4j.log

...Configure /etc/rsyslog.conf to Forward local2 Events

Configuring the Central Log Host

...Example central log host /etc/rsyslog.conf file

How to Configure the Oracle VM Server's Logrotation

Upgrade the Oracle VM Server and Manager Local Syslog Daemon

The first step to configure centralized logging for Oracle VM is to upgrade the local syslog daemon to the Oracle Linux 5.5 rsyslog package on the Oracle VM servers and on the Oracle VM Manager host.

Note: The default Oracle VM 2.x server configuration does not have rsyslog.

The next three steps show how to install and configure rsyslog on an Oracle VM 2.x server and on an Oracle VM Manager x86 or x86-x64 host:

Download and install the Oracle Linux 5.5 rsyslog rpm (3.22 at time of writing) using wget and the rpm programs.

Oracle VM Server: The Oracle VM server will always use the i386 RPM regardless of the hardware platform, i.e. both x86 or x86-x64 servers both use a x86 dom0 and will use the i386 RPM package.

Oracle VM Manager: Depending on the hardware/OS platform for your Oracle VM Manager host, use the i386 RPM package for x86 or the x86_64 package x64.

The next two examples show how to download and install the rsyslog rpm package for the i386 and x86-64 platforms using wget and the rpm programs.

I386

# wget http://public-yum.oracle.com/repo/EnterpriseLinux/EL5/5/base/i386/rsyslog-3.22.1-3.el5.i386.rpm

# rpm -Uvh rsyslog-3.22.1-3.el5.i386.rpm

x86-64

# wget http://public-yum.oracle.com/repo/EnterpriseLinux/EL5/5/base/x86_64/rsyslog-3.22.1-3.el5.x86_64.rpm

# rpm -Uvh rsyslog-3.22.1-3.el5.x86_64.rpm

In the next example, we use the syslog configuration file for rsyslog. We also disable syslog and enable rsyslog:

# cp /etc/syslog.conf /etc/rsyslog.conf

# chkconfig syslog off

# chkconfig rsyslog on

# service syslog stop

# service rsyslog start

Next, check /var/log/messages to validate that rsyslog has started. For example, type “tail /var/log/messages”

Oracle VM Server Syslog Configuration

The /etc/ovs-agent/logger_server.ini file stores the logger class configuration and is read when the Oracle VM agent starts.

Note: For more information please refer to: http://docs.python.org/library/logging.html#sysloghandler

The next list shows the changes that will be made to the ovs-agent Python logger:

Maintain the various handlers that Oracle uses, for consistency with Oracle support.
Propagate all handers to the parent (root) handler; all information logged by the ovs-agent will be available at this handler.
Set the log level to NOTSET, which is everything (more than DEBUG).
Forward logs to local syslog over udp/514 (default syslog port).
Set all loggers to "propagate=1", so they forward up logs to their parent handlers.
By default performance and macip logging doesn't propagate up. We need "propagate=1" for centralization.
Write to unix socket /dev/log, which must be created by rsyslog!

Note: We don't use the localhost 514/udp destination because this will create a message from a hostname of localhost or 127.0.0.1, which is of no use for centralized logging. By writing to the socket, the syslog daemon appends its hostname, which is necessary for centralized logging.

Send all to syslog by altering the root handler
Create a separate formatter that matches the syslog "name: message" style.

Example /etc/ovs-agent/logger_server.ini file

The following example shows a logger_server.ini file that meets the above prerequisites:

# cat /etc/ovs-agent/logger_server.ini

[loggers]

keys=root,performance,operation,query

[logger_root]

handlers=root

level=NOTSET

[logger_operation]

qualname=ovs.operation

handlers=operation

level=DEBUG

propagate=1

[logger_performance]

qualname=ovs.performance

handlers=performance

level=DEBUG

# default propagate is 0

propagate=1

[logger_query]

qualname=ovs.query

handlers=query

level=DEBUG

propagate=1

[logger_macip]

qualname=ovs.macip

handlers=macip

level=DEBUG

# default propagate is 0

propagate=1

;----------------------------------------------------------------------

[handlers]

keys=root,performance,operation,query,macip

[handler_root]

class=handlers.RotatingFileHandler

;append to log file, and file size is 1M with 3 archive files

args=("%(log.dir)s/ovs_root.log", "a", 1024*1024, 3)

formatter=ovs

level=DEBUG

# Syslog handler

# - log to local syslog daemon, which can forward to central loghost

# - using unix socket, which must match the socket created by the syslog daemon

# - could use UDP to localhost, which loses the originating host information

# (get messages tagged with an IP that makes no sense centrally, like 127.0.0.1

# for the localhost config)

# - SysLogHandler unix socket: args=('/dev/log', handlers.SysLogHandler.LOG_LOCAL3)

# - SysLogHandler to 514/udp: (('localhost', handlers.SYSLOG_UDP_PORT),

handlers.SysLogHandler.LOG_LOCAL3)

# - if the level is set on the syslog handler, all messages sent through that handler

# inherit this level. Preferable to set level on a per-logger basis

# - if level inherited by the loggers, then a simple formatter can be used that best

matches

# the expectation of syslog parsers. see formatters section for details. eg:

# format=%(name)s: %(message)s

# address: /dev/log unix socket

# facility: LOG_LOCAL3

# level: do not set at the handler level

# ref: http://docs.python.org/library/logging.html#sysloghandler

class=handlers.SysLogHandler

#args=(('localhost', handlers.SYSLOG_UDP_PORT), handlers.SysLogHandler.LOG_LOCAL3)

args=('/dev/log', handlers.SysLogHandler.LOG_LOCAL3)

formatter=syslog

[handler_operation]

class=handlers.RotatingFileHandler

;append to log file, and file size is 1M with 3 archive files

args=("%(log.dir)s/ovs_operation.log", "a", 1024*1024, 3)

formatter=ovs

level=DEBUG

[handler_performance]

class=handlers.RotatingFileHandler

;append to log file, and file size is 1M with 3 archive files

args=("%(log.dir)s/ovs_performance.log", "a", 1024*1024, 3)

formatter=ovs

level=DEBUG

[handler_query]

class=handlers.RotatingFileHandler

;append to log file, and file size is 1M with 3 archive files

args=("%(log.dir)s/ovs_query.log", "a", 1024*1024, 3)

formatter=ovs

level=DEBUG

[handler_macip]

class=handlers.RotatingFileHandler

;append to log file, and file size is 1M with 2 archive files

args=("%(log.dir)s/ovs_macip.log", "a", 1024*1024, 2)

formatter=ovs

level=DEBUG

;----------------------------------------------------------------------

[formatters]

keys=ovs,syslog

[formatter_ovs]

class=logging.Formatter

format=%(asctime)s %(levelname)s=> %(message)s

datefmt="%Y-%m-%d %H:%M:%S"

# formatter for syslog

# - no datefmt required, syslog tags with event time

# - if level is not specified by the handler, then we do not need to add

# it to the formatter

# - formatter options

# with level in syslog message field: format=%(name)s: %(levelname)s %(message)s

# closest to traditional syslog: format=%(name)s: %(message)s

[formatter_syslog]

class=logging.Formatter

format=%(name)s: %(message)s

How to Configure Rsyslog on Oracle VM Server

The next list shows the steps to configure rsyslog on an Oracle VM server as root:

Create a UNIX syslog socket at /dev/log.
Eliminate the local3 channel from /var/log/messages to keep the ovs-agent logs in one place and not to spread the ovs-agent logs to multiple logfiles.
Route all local3 facility messages to a file, /var/log/ovs-agent/all.log.
Route all local3 facility messages to a central log host, over port 601/tcp.

The following /etc/rsyslog.conf file example contains rsyslog specific configuration elements. The "@@" remote forwarding terminology means "use TCP", whereas "@" means "use UDP".

Enter your system details in the following sections:

Example /etc/rsyslog.conf file

# cat /etc/rsyslog.conf

# ----------------------------- Queues (required for forwarding)

-----------------------------

$WorkDirectory /var/spool/rsyslog

$ActionQueueType LinkedList # use asynchronous processing

$ActionQueueFileName srvrfwd # set file name, also enables disk mode

$ActionResumeRetryCount -1 # infinite retries on insert failure

$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

# ----------------------------- Modules & Functions -----------------------------

# module: kernel logs, klogd replacement

$ModLoad imklog

# module: MARK messages (usually disable)

$ModLoad immark

# module: input udp

# NOTE: documetation says InputUDPServerRun, that's wrong

$ModLoad imudp

$UDPServerAddress *

$UDPServerRun 514

# module: input tcp (works fine)

$ModLoad imtcp

$InputTCPServerRun 601

$InputTCPServerRun 4096

# module: input socket

$ModLoad imuxsock

$InputUnixListenSocketHostName /dev/log

$InputUnixListenSocketIgnoreMsgTimestamp on

# ----------------------------- Templates -----------------------------

# Use traditional timestamp format

# eg: "Jan 1 15:00:01 "

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# RFC5424 format

# also known as RSYSLOG_SyslogProtocol23Format, and

draft-internet-ietf-syslog-protocol-23 (now RFC5424)

# the "1" in "<%PRI%>1 " denotes syslog protocol version 1, as per the RFC

# eg: <21>1 2011-01-01T16:09:05+00:00 <MAIL PROXY HOST NAME GOES HERE> perdition 14185 - -

$template RFC5424FMT,"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"

# ----------------------------- Destinations -----------------------------

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

kern.* /var/log/kernel

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none;local3.none /var/log/messages

# The authpriv file has restricted access.

authpriv.* /var/log/secure

# MAIL: Log locally & forward a copy to syslog1-syd.internal for further analysis

mail.*

-/var/log/mail/mail.log;RFC5424FMT

mail.*

@@syslog1-<FQDN OF CENTRAL LOG HOST>:601

# Log cron stuff

cron.* /var/log/cron

# Everybody gets emergency messages

#*.emerg *

# Save news errors of level crit and higher in a special file.

#uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log

local7.* /var/log/boot.log

# All, save mail

*.debug;mail.none;authpriv.none;cron.none -/var/log/debug

# Oracle VM: Log locally & forward a copy to syslog1-syd.internal for further analysis

local3.*

-/var/log/ovs-agent/all.log;RFC5424FMT

local3.*

@@syslog1-<FQDN OF CENTRAL LOG HOST>:601

Example /etc/sysconfig/rsyslog file

The next example shows a rsyslog file.

# cat /etc/sysconfig/rsyslog

# Options to syslogd

# -m 0 disables 'MARK' messages.

# -rPortNumber Enables logging from remote machines. The listener will listen to the specified port.

# -x disables DNS lookups on messages received with -r

# See syslogd(8) for more details

SYSLOGD_OPTIONS="-c3 "

# Options to klogd

# -2 prints all kernel oops messages twice; once for klogd to decode, and

# once for processing with 'ksymoops'

# -x disables all klogd processing of oops messages entirely

# See klogd(8) for more details

KLOGD_OPTIONS="-x"

Next, enable the changes by reloading rsyslog by typing, “service rsyslog reload”. After rsyslog has been reloaded, there should be an empty file named all.log in the /var/log/ovs-agent/ directory.

Next, reload the Oracle VM agent by typing “service ovs-agent stop --disable-nowayout; service ovs-agent start” as shown in the next example.

# service ovs-agent stop --disable-nowayout; service ovs-agent start

OVSAgentServer shutdown...

OVSAgentServer forced stop.

OVSAgentServer is now starting...

OVSAgentServer started.

After the Oracle VM agent has been reloaded, messages will appear in the local /var/log/ovs-agent/all.log file, as shown in the next example.

<158>1 2011-01-25T01:03:03.245088+00:00 virtual5-syd root - - - OVSAgentServer start serving!

<158>1 2011-01-25T01:03:08.633781+00:00 virtual5-syd ovs.operation - - - clusterm_init_sr:

success.

<158>1 2011-01-25T01:03:08.658913+00:00 virtual5-syd ovs.operation - - - cluster_init_sr:

success.

<158>1 2011-01-25T01:03:10.863599+00:00 virtual5-syd ovs.operation - - - ha_set_shutdown_mode:

remove shutdown mode flag.

<158>1 2011-01-25T01:03:14.944380+00:00 virtual5-syd ovs.performance - - -

sys_vmstat_info:cpu_idle=0.99;mem_free=427

…

Note: If you see "localhost.localdomain" or "127.0.0.1" instead of the hostname in the all.log file, use the unix socket instead of udp in configuring syslog for the Oracle VM agent.

How to Configure the Central Log Host

The next list shows the configuration steps to configure the central log host.

Receive remote syslog over port 601/tcp
Write local3 messages to /var/log/ovs-agent/all.log
Keep local3 message from other major logfiles

Note: For the /etc/rsyslog.conf file, please see the Centralized Log Host full rsyslog.conf section.

Example /etc/sysconfig/rsyslog file

The next example shows the /etc/sysconfig/rsyslog file with the above prerequisites.

# cat /etc/sysconfig/rsyslog

# Options to syslogd

# -m 0 disables 'MARK' messages.

# -rPortNumber Enables logging from remote machines. The listener will listen to the specified port.

# -x disables DNS lookups on messages received with -r

# See syslogd(8) for more details

#SYSLOGD_OPTIONS="-c3 -x -m 0 -r514 -t601,4096" # syslog compat

SYSLOGD_OPTIONS="-c3 " # rsyslog

# Options to klogd

# -2 prints all kernel oops messages twice; once for klogd to decode, and

# once for processing with 'ksymoops'

# -x disables all klogd processing of oops messages entirely

# See klogd(8) for more details

KLOGD_OPTIONS="-x"

Next, reload the rsyslog service by typing, “service rsyslog reload”. After the rsyslog service has been reloaded, the central log host will receive messages to /var/log/ovs-agent/all.log.

Notes: The rsyslog configuration must allow inbound 601/tcp. Check the central log host firewall to ensure that inbound 601/tcp is enabled. The central log host's /var/log/ovs-agent directory must exist. If SElinux is in use, it must have the "user_u:object_r:var_log_t" context

The Oracle VM Manager Syslog Configuration

OC4J uses java.util.logging, which cannot directly log to syslog. To log Oracle VM Manager events centrally, we will use rsyslog's "imfile" module to read in the existing oc4j logfile and send it to the central log host line by line.

OC4J Logging

The Oracle VM Manager application is a J2EE Web application, running in an OC4J container. Assuming OC4J is installed to /opt/oc4j, then the default logging is configured via the

“/opt/oc4j/j2ee/home/config/j2ee-logging.xml” file.

Oracle VM Manager Logging Configuration Files

The deployment descriptors for Oracle VM Manager application are installed by default to:

/opt/ovs-manager-2.2/bin/default-server.xml (the non-ssl server)
/opt/ovs-manager-2.2/bin/secure-server.xml (the ssl server)

By default the deployment descriptors both point to the j2ee-logging.xml configuration file. So we only need to edit the j2ee-logging.xml configuration file.

How to Make Rsyslog Read the oc4j.log file

The default logfile for OC4J is in the /var/log/ovm-manager/oc4j.log file. Our configuration will read the /var/log/ovm-manager/oc4j.log file to the local2/debug priority with the rsyslog imfile module.

Note: The configuration will result in the lose of the event severity.

Oracle VM Manager /etc/rsyslog.conf and Adding imfile Support

Next, we will add the “input file” module to /etc/rsyslog.conf. This configuration will write the state file to “$WorkDirectory”.

Note: The full imfile module documentation can be found here: http://www.rsyslog.com/doc/imfile.html

Example /etc/rsyslog.conf file

# cat /etc/rsyslog.conf

# module: input file

# - emits each line of the given file to syslog, for apps that don't do syslog

# - keeps track of position, file rotation

$ModLoad imfile

$InputFileName /var/log/ovm-manager/oc4j.log

$InputFileTag OVM:

$InputFileStateFile state-oc4j

$InputFileSeverity debug

$InputFileFacility local2

$InputRunFileMonitor

Empty the Existing Oracle VM Manager oc4j.log

When the imfile module starts, it will read every line in the existing oc4j.log file, which is too much information for the imfile module to start. Empty the oc4j.log file by typing “cp /dev/null /var/log/ovm-manager/oc4j.log”.

Configure /etc/rsyslog.conf to Forward local2 Events

Next, add the following lines to the /etc/rsyslog.conf file to forward local2 events to the central log host. Enter your central log host FQDN in the <CENTRAL LOG HOST> section.

# Oracle VM Manager: Log locally & forward a copy to syslog1-syd.internal for further analysis

# - Oracle VM Manager to local2, not logging to file here as the input is a file

local2.* @@<CENTRAL LOG HOST>:601

Configuring the Central Log Host

On the central log host add the following line to /etc/rsyslog.conf to receive local2 events.

# Oracle VM: Log locally

# - Oracle VM Manager on local2

# - Oracle VM agent on local3

local2.* -/var/log/ovm/ovm-managers.log;RFC5424FMT

local3.* -/var/log/ovm/ovs-agents.log;RFC5424FMT

Next, restart the rsyslog service on the central log host and on Oracle VM Manager. After the rsyslog service is restarted, log host should see inbound messages:

<151>1 2011-01-25T13:12:01+00:00 vmmanager1-syd OVM - - - INFO: ServerPoolSync.run: sync virtual

machines status succeeded.

<151>1 2011-01-25T13:12:01+00:00 vmmanager1-syd OVM - - - INFO: ServerPoolSync.run: sync cluster

master status succeeded.

<151>1 2011-01-25T13:12:01+00:00 vmmanager1-syd OVM - - - INFO: ServerPoolSync.run: sync agent

version succeeded.

<151>1 2011-01-25T13:12:01+00:00 vmmanager1-syd OVM - - - INFO: master server is:10.200.28.202

Example central log host /etc/rsyslog.conf file

The next example shows the complete central log host rsyslog.conf file.

# cat /etc/rsyslog.conf

# ----------------------------- Queues (required for forwarding) -----------------------------

$WorkDirectory /var/spool/rsyslog

$ActionQueueType LinkedList # use asynchronous processing

$ActionQueueFileName srvrfwd # set file name, also enables disk mode

$ActionResumeRetryCount -1 # infinite retries on insert failure

$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

# ----------------------------- Modules & Functions -----------------------------

# module: kernel logs, klogd replacement

$ModLoad imklog

# module: MARK messages (usually disable)

$ModLoad immark

# module: input udp

# NOTE: documentation says InputUDPServerRun, that's wrong

$ModLoad imudp

$UDPServerAddress *

$UDPServerRun 514

# module: input tcp (works fine)

$ModLoad imtcp

$InputTCPServerRun 601

$InputTCPServerRun 4096

# ----------------------------- Templates -----------------------------

# Use traditional timestamp format

# eg: "Jan 1 15:00:01 "

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# RFC5424 format

# also known as RSYSLOG_SyslogProtocol23Format, and draft-internet-ietf-syslog-protocol-23 (now

RFC5424)

# the "1" in "<%PRI%>1 " denotes syslog protocol version 1, as per the RFC

# eg: <21>1 2011-01-01T16:09:05+00:00 <MAIL PROXY HOST NAME> perdition 14185 - -

$template RFC5424FMT,"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID%

%STRUCTURED-DATA% %msg%\n"

# ----------------------------- Destinations -----------------------------

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.* /dev/console

kern.* /var/log/kernel

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none;local3.none /var/log/messages

# The authpriv file has restricted access.

authpriv.* /var/log/secure

# MAIL: Log locally & forward a copy to sysdev1-syd3 for further analysis

# - RFC5424 parseable output

mail.* -/var/log/mail/mail.log;RFC5424FMT

mail.* @@<SMTP HOST>:601

# Log cron stuff

cron.* /var/log/cron

# Everybody gets emergency messages

#*.emerg *

# Save news errors of level crit and higher in a special file.

#uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log

local7.* /var/log/boot.log

# All, save mail

*.debug;mail.none;authpriv.none;cron.none -/var/log/debug

# Oracle VM: Log locally

# - OVM manager on local2

# - OVS agent on local3

local2.* -/var/log/ovm/ovm-managers.log;RFC5424FMT

local3.* -/var/log/ovm/ovs-agents.log;RFC5424FMT

How to Configure the Oracle VM Server's Logrotation

The next example show the recommended Oracle VM server and the central log host configurations by creating an executable file named ovs-agent in the /etc/logrotate.d/ directory.

# cat /etc/logrotate.d/ovs-agent

/var/log/ovs-agent/all.log {

create 0644 root root

missing ok

# keep for 14 days

rotate 14

# dateext tags with date, max 1 rotation/day

dateext

# compression parameters

compress

compresscmd /usr/bin/bzip2

compressext .bz2

compressoptions -9

# notify syslog or rsyslog

postrotate

/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true

/bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true

endscript

}

The above configuration can be tested verbosely by typing “logrotate -v -f /etc/logrotate.d/ovs-agent”.

Menu Search

Oracle VM Centralized Logging

Oracle VM for x86 Troubleshooting Made Easier