Oracle Cloud Reference Design 01-14-2014

Mokum is the only full-time Oracle virtualization
integrator with the expertise to help you virtualize your

Production, Test and DR Oracle workloads.
sales@mokumsolutions.com
Copyright 2014 Mokum Solutions, Inc. All rights reserved.
Distribution of the Oracle Cloud Cookbook or derivative of the work in any form
is prohibited unless prior permission is obtained from the Copyright holder.
About Mokum Solutions, Inc.
Founded in March 2011, Mokum Solutions, Inc. specializes in the implementation,
delivery and support of Oracle technologies in private and public clouds. Mokum
corporate headquarters are located in San Francisco, CA http://mokumsolutions.com
or call 1 415 252 9164
About the Author
The author of the Oracle Cloud Cookbook is none other than the owner of
Mokum Solutions, Inc., Roddy Rodstein. Roddy is one of the most respected
Oracle Cloud Computing experts, having designed and managed many of the
worlds largest and most complex Oracle private clouds. Before establishing
Mokum in March 2011, Roddy spent three years at Oracle on the Oracle VM
and Oracle Linux team designing and supporting Oracle's largest and most
complex customer environments. Before Oracle, Roddy spent six years at Citrix,
designing and supporting Citrix's largest and most complex customer environments,
Including Oracle's. With Mr. Rodsteins rich background and knowledge, there
can be no better resource for revealing the Oracle Cloud recipe.
Audience
The Oracle Cloud Cookbook is a comprehensive, field tested reference design that
guides you through each step to move to your Oracle software portfolio to an elastic
Oracle cloud using the Oracle VM product line, Oracle Linux, Oracle Engineered
Systems managed by Oracle Enterprise Manager 12c, with total control over Oracle
processor licensing.
http://mokumsolutions.com
Last update: 01/14/2014

Table of Contents
The Oracle Cloud Reference Design Introduction
The Oracle Cloud Reference Design Implementation Overview
The Oracle Cloud Reference Design Support Infrastructure
Oracle Cloud Architectural Design
Oracle VM Hardware Architecture
Oracle VM Server Pool Design
Oracle VM Security Standards
Oracle VM Manager Security Controls
Oracle VM Server Security Controls
Virtual Machine Operating System Standards
Oracle VM Disaster Recovery
Oracle VM Application Integration
Change Log
Revision
Change Description
Updated By
Date
2.0
Document Creation
Roddy Rodstein
09/30/2013
2.1
Content Refresh
Roddy Rodstein
01/14/2014
The Oracle Cloud Reference Design Introduction

This chapter of the Oracle Cloud Cookbook presents the Oracle Cloud reference design.
The Oracle Cloud reference designs encompass the software, hardware, storage, and
network components required to deploy a scalable, secure, and supportable internal or
external Oracle cloud.
The Oracle Cloud reference design is a eld-tested best-practice standard, designed with
simplicity, reproducibility, usability, scalability, supportability and security. The Oracle
Cloud reference designs represent a complete Oracle Cloud standard that can be
leveraged as a vanilla solution or modied to more accurately reect organization-specic
needs. The Oracle Cloud reference design includes the following categories:
Software as a Service SaaS
Platform as a Service PaaS
Infrastructure as a Service
Oracle Applications
Oracle Fusion Middleware
Oracle Database (DBaaS)
Oracle Enterprise Manager

Oracle VM Manager
Open Source
Virtual Machines
Mokum Solutions, Inc. +1 415 252-9164
4 of 41
Oracle VM for x86

IaaS
x86 64 Servers
Storage & Network
Services
Note: A detailed explanation of each category and solution in the Oracle Cloud reference
design is presented in the architectural overview section.
The Oracle Cloud Reference Design Implementation

Overview
The Oracle Cloud reference design provides a well dened starting point for each Oracle
Cloud implementation. It also serves as a baseline upon which all solution additions,
revisions, and tools will be based. As such, there is an increasing value to Oracle Cloud
reference design in keeping implementations as close to the reference design as possible.
Prior to implementing an Oracle Cloud, its important that an infrastructure assessment
(IA) and gap analysis (GA) be performed. During the IA/GA, the architecture of the
solution will match the customers business needs while maintaining the integrity of the
Oracle Cloud reference design. Implementation and support will follow the analysis phase
after careful consideration has been given to any specic design modications that
deviate from the Oracle Cloud reference design.
This document outlines the decision points necessary for implementing the Oracle Cloud
reference design. For decisions that rely on preexisting factors or specic organizational
needs, the appropriate best practice will be discovered in the infrastructure assessment
(IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions
should be made based on organizational needs, existing architecture, and budget
resource availability.
The Oracle Cloud reference design is designed to be scalable and resilient for ease of
implementation, high availability, and ease of maintenance for internal and external
Oracle clouds. The complete solution is made up of four architectural components that
work together to provide exibility and options with respect to on-demand self-service,
broad network access, resource pooling, elasticity, measured service, high availability,
security and ease of maintenance. The design breaks down into the following four
components:
Software as a Service (SaaS). Software as a Service is the capability to host and
deliver applications over the Internet, accessible from various client devices. The
provider manages the cloud infrastructure and application portfolio that is accessed
by the consumer. The Oracle Cloud reference design outlines the decision points
necessary for implementing an Oracle Software as a Service delivery model using
Oracle SGD.
5 of 41
Platform as a Service (PaaS). Platform as a Service is the capability to host and

allow access to a computing platform and software stack for application
development. The provider hosts the computing platform and software stack on the
cloud infrastructure that is accessed by the consumer. The consumer manages the
computing platform and software stack used for application development. The
Oracle Cloud reference design outlines the decision points necessary for
implementing the cloud infrastructure and the Oracle Platform as a Service delivery
model using the self service portal in Oracle Enterprise Manager Cloud Control 12c.
Infrastructure as a Service (IaaS). Infrastructure as a Service is the capability to
provision and deliver fundamental computing resources as a service to the
consumer. The Oracle Cloud reference design outlines the decision points necessary
for implementing an Oracle VM cloud infrastructure to deliver Infrastructure as a
Service using pre-congured virtual machine templates from the self service portal
in Oracle Enterprise Manager Cloud Control 12c.
The next Figure shows a high-level overview of the Oracle Cloud reference design
components.
The Oracle Cloud reference design isolates Oracle VM server pools into the following four
security domains:
Controlled: A controlled security domain is used to restrict access between security
domains. A controlled security domain could contain groups of users with their
network equipment or a demilitarized zone (DMZ).
Uncontrolled: An uncontrolled security domain refers to any network not in control
of an organization, such as the Internet.
Restricted: A restricted security domain can represent an organizations
production, test and development networks. Access is restricted to authorized
personnel, and there is no direct access from the Internet.
Secured: A secured security domain is a network that is only accessible to a small
group of highly trusted users, such as administrators and auditors.
Note: The classication of security domains is very similar to data classications. FIPS
PUB 199 is the Standards for Security Categorization of Federal Information and
Information Systems. FIPS PUB 199 can be used to determine the security category of
6 of 41
systems and within which security domain systems should reside.
The Oracle Cloud Reference Design Support

Infrastructure
Support is an integral part of the Oracle Cloud reference design and includes a
combination of Oracle support agreements and on-site and o-site support from the
implementing party. Administrators will have several options for support, including live
assistance, phone support, and forums.
This table outlines the decision points for the support infrastructure for the Oracle Cloud
reference design. For decisions that rely on preexisting factors or specic organizational
needs, the appropriate best practice will be discovered in the infrastructure assessment
(IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions
should be made based on organizational needs, existing architecture, and budget
resource availability.
Decision Point
Decision
Justication
Oracle Support
Agreements
Oracle Support Agreements for

the Oracle technologies will be
active and up to date.
Support is an integral part of

every successful IT project.
Oracle support agreements are
necessary to be able to create and
manage service requests as well
as to be able to receive software
patches and updates from Oracle
Enterprise Manager and My
Oracle Support.
On-site and
O-site support
On-site and o-site support from

the implementing party will be
used for maintenance, site
reviews, upgrades, and security
audits.
On-site and o-site support from

the implementing party for
problem resolution, system
maintenance, site reviews,
upgrades, and security audits
augments the Oracle support
agreement and internal IT
operations sta.
Oracle Cloud Architectural Design

The following sections provides the decision matrices for the Oracle Cloud reference
design. Implementers of the Oracle Cloud reference design can use the
decision matrices as quick reference guide to identify settings and conguration
decisions to be implemented in the environment. These decisions should be carefully
analyzed during a gap analysis phase.
7 of 41
Oracle VM Hardware Architecture

The server hardware for your Oracle VM environment is a critical component in the
success of your Oracle cloud project. The rst step in selecting an Oracle VM hardware
platform is to size the server hardware, followed by calculating the total number of
servers required to be in each Oracle VM server pool. The formula to calculate Oracle VM
server sizing is: The total aggregate virtual machine CPU, RAM and Storage requirements
plus your N+x availability requirements provides the total server count along with the
hardware requirements.
Oracle VM server sizing is calculated by adding the aggregate CPU, RAM and storage
requirements for all of the virtual machines that could run on an Oracle VM server, and
then selecting server hardware with ample CPU, RAM and storage resources. Once the
server hardware has been selected, the number of servers in a server pool is calculated
by selecting enough servers to support the aggregate CPU, RAM and storage
requirements of all of the virtual machines within a server pool, including the number of
additional servers for availability, i.e. HA, Live Migration and Distributed Resource
Scheduling (DRS). Oracle VM server pools that use HA, Live Migration and DRS must
have excess CPU and RAM capacity for hardware failures and virtual machine migrations.
The number of network interfaces for an Oracle VM server is determined by the network
switch VLAN setup and the total number of Oracle VM management network ports, and
the virtual machine network ports for your environment.
Oracle VM server can be installed on an x86 64 bit server with up to 160 CPU cores or
threads, up to 4 TB of RAM, with up to 40 network ports. The default behavior of the
Oracle VM server installation program is to allocate only 3 GB of disk space for the entire
Oracle VM installation, regardless of the amount of available disk space. Since Oracle VM
server only requires 3 GB of storage, you might consider procuring disk-less hardware
with a ash storage module or boot from SAN to reduce operating costs.
Tip: I have had the opportunity to support and benchmark Oracle VM server installations
on a slow single 4 GB SSD Drive (18 MB/second Read Transfer Rate,17 MB/second Write
Transfer Rate) as well as Oracle VM server installations using local 7k, 10k and 15k disks.
The read and write performance from either type of Oracle VM server installation disk on
the remote virtual machine storage (SAN, NFS or iSCSI) from the Oracle VM server and
the virtual machines was identical. The disk speed from the Oracle VM server installation
does not aect the remote storage read and write performance.
The next table shows the maximum number of CPUs, RAM and NICs for Oracle VM server
release 3.2.x and above.
Item
Maximum
CPU Cores or Threads
160
RAM
4 TB
NICs
40
Oracle VM Server CPU, RAM and storage hardware sizing is calculated by determining
the total number of virtual machines CPU, RAM, and storage (I/O and disk) requirements
8 of 41
per Oracle VM server. For example, if a single virtual machine with 16 CPUs, 128 GB
RAM, 1 TB of disk space with 1500 IOPS will run on one Oracle VM server, the Oracle VM
server hardware should have at least 16 CPU cores or threads, 130 GB RAM, 1 TB of disk
space and the ability to support 1500 IOPS with local or remote storage. If two virtual
machines each with 16 CPUs, 128 GB RAM, 1 TB of disk space with 1500 IOPS will run on
one Oracle VM server, the Oracle VM server hardware must have at least 32 CPU cores
or threads, 300G RAM, 2 TB of disk space and the ability to support 3000 IOPS with local
or remote storage.
A single Oracle VM 3.2.x server can support up to 160 CPU cores or threads, 4 TB of
memory with local or remote storage. An Oracle VM server with 4 TB of RAM and 160
CPU cores or threads could allocate the majority of the 4 TB of RAM and more than 160
CPU cores or threads to running virtual machines. Oracle VM server supports CPU
oversubscription. CPU oversubscription means that an Oracle VM server with 160 CPU
cores could overallocate the total number of CPU cores to virtual machines. Oracle VM
server does not support memory oversubscription, which means that an Oracle VM server
with 4 TB of RAM cannot overallocate RAM to virtual machines. By default, each Oracle
VM server reserves 512 MB of RAM for Oracle VM server (dom0). The average memory
overhead for each running virtual machine on an Oracle VM server is approximately 20
MB plus 1% of each virtual machine' memory allocation. The remaining RAM can be
allocated to virtual machines.
A best practice is to avoid oversubscribing CPU-bound workloads such as the Oracle
Database. CPU oversubscription with CPU-bound workloads negatively aects the
performance and availability of an Oracle VM server along with all of the virtual machines
running on the server. CPU oversubscription for non-CPU-bound workloads, such as
Oracle Fusion Middleware products, is highly recommended. It is common to
oversubscribe CPU cores 3-to-1 with non-CPU-bound workloads. For example, one CPU
core could be allocated to 3 virtual CPUs for non-CPU-bound workloads without a
performance penalty.
Note: Virtual machines cannot aggregate CPU and memory resources from more than
one Oracle VM server. That is, a virtual machine consumes resources only from the
Oracle VM server where the virtual machine is running.
Oracle VM has two high-availability features, HA and Live Migration. Oracle VM HA and
Live Migration along with Distributed Resource Scheduling (DRS) must be considered to
calculate the total number of servers required to respond to hardware failures and virtual
machine migrations.
The next Figure shows Oracle VM server pool designed with excess CPU and RAM
capacity to be able to use HA, DRS and Live Migration. Excess CPU and RAM capacity is
a requirement for HA, DRS and Live Migration.
This image shows an
Oracle VM server pool
with excess capacity able
to use HA, Live Migration
and DRS.
This image shows an

Oracle VM server
pool responding to a HA
event, with DRS and/or
Live Migration moving
This image shows an

Oracle VM server
pool migrating running
virtual machines using DRS
and/or Live Migration.
9 of 41
running virtual machines.
Oracle VM HA automatically restarts virtual machines when an Oracle VM pool member

fails or restarts. Oracle VM HA minimizes unplanned downtime by restarting virtual
machines when an Oracle VM server fails or restarts. Live Migration is used to eliminate
planned downtime by migrating running virtual machines from one Oracle VM pool
member to another during a maintenance event, for example, for repairs or an upgrade.
DRS is an Oracle VM feature which provides policy based real-time utilization monitoring
of Oracle VM servers with the goal to distribute virtual machine loads across a server
pool. DRS migrates virtual machines from heavily utilized Oracle VM servers to less
utilized Oracle VM servers. Both HA, Live Migration and DRS require a server pool with
at least three servers with excess CPU and RAM capacity to be able to run and migrate
virtual machines across the server the pool even if one Oracle VM servers fails.
Tip: There is a known limitation with OCFS2 two node cluster and network failures that
cause the node with the higher node number to self-fence. For example, with a two node
Oracle VM server pool, if one node has a network failure that triggers a HA event, both
Oracle VM server will reboot. A best practice is to use a minimum of three Oracle VM
servers for a server pool to eliminate the two node OCFS2 limitation.
Oracle VM HA monitors the status of each server pool member using a network and
storage heartbeat. If a server pool member fails to update or respond to network and/or
storage heartbeats due to hardware failure, the server pool member is fenced from the
pool, promptly reboots, then all HA-enabled virtual machines are restarted on a live node
in the pool. Oracle VM does not support memory oversubscription, which means that an
Oracle VM server pool must have sucient RAM capacity to be able to respond to a
hardware failure using HA, or to support virtual machine migrations.
The Oracle VM Live Migration and DRS move running virtual machines between server
pool members across a LAN without loss of availability. Live Migration and DRS have two
primary use cases. The rst use case is to eliminate planned downtime by Live Migrating
10 of 41
running virtual machines from one server pool member to another during planned
maintenance events. The second use case is to use DRS policies to load balance running
virtual machines from heavily utilized Oracle VM servers to less utilized Oracle VM
servers. Since Oracle VM does not support memory oversubscription, an Oracle VM
server pool must have available RAM capacity to be able to migrate virtual machines
between servers.
DRS is an Oracle VM feature which provides policy based real-time utilization monitoring
of Oracle VM servers with the goal to distribute virtual machine loads across a server
pool. DRS migrates virtual machines from heavily utilized Oracle VM servers to less
utilized Oracle VM servers.
The exact number of network interfaces for an Oracle VM server is determined by the
network switch VLAN setup and the number of Oracle VM management and virtual
machine network ports. Oracle VM supports both 802.1Q trunk port VLANs as well as
port based VLANs, with Linux bonding Modes 1 (Active-Backup), 4 (802.3ad) and 6
(Adaptive load balancing). 802.1q trunk ports can have two or more VLANs per port, in
contrast to port based VLANS that are limited to one VLAN per port or port channel.
802.1Q uses fewer network switch ports and fewer Oracle VM server NICs compared to
port based VLANs that require a dedicated switch port and NIC per network. A network
switch VLAN conguration must rst be selected to be able to calculate the exact number
of network switch ports and NICs for your Oracle VM servers.
Oracle VM uses a total of ve discrete networks for the Oracle VM server management
functions; server management, cluster heartbeat, live migration, storage (only for NFS
and iSCSI) and virtual machines. Each Oracle VM server pool should have a discrete
network for each of the ve aforementioned server management networks, as well as a
discrete network for each virtual machine network. For example, an Oracle VM Server on
a 1-gigabit copper network with NFS or iSCSI storage could easily use 12 or more
bonded NICs with access ports just for the server management networks and one virtual
machine network. In contrast to the latter 1-gigabit copper network example, an Oracle
VM Server on a 10-gigabit ber network using 802.1q trunk ports with NFS or iSCSI
storage could easily use up to 4 bonded ports just for the server management and 2
bonded ports for the virtual machine networks.
Tip: In an clustered Oracle VM server pool, the loss of network connectivity for the
Oracle VM cluster heartbeat network will causes a HA event. When a HA event occurs,
the Oracle VM server that loses cluster heartbeat connectivity is fenced from the server
pool and reboots, then all HA-enabled guests are restarted on a live Oracle VM pool
member.
Prior to implementing an Oracle Cloud, its important that an infrastructure assessment
(IA) and gap analysis (GA) be performed. During the IA/GA, the hardware specications
will be matched to the customers business needs.
This table outlines the decision points for the for Oracle VM for x86 server hardware. For
decisions that rely on preexisting factors or specic organizational needs, the appropriate
best practice will be discovered in the infrastructure assessment (IA) and gap analysis
(GA). The best practices should be analyzed carefully and decisions should be made based
on organizational needs, existing architecture, and budget resource availability.
11 of 41
Decision
Point
Decision
Certication
The server hardware must be jointly

supported by the hardware vendor and
Oracle.
CPU
Server hardware will be ordered with

two socket Intel or AMD multiple-core
CPUs for small and medium workloads
and four socket multiple-core CPUs for
large CPU-bound workloads.
Justication
Only jointly supported

hardware product receive
vendor support when
problems occur and service
tickets are created. The
Note: The following link is the Oracle'
server hardware must be
hardware certication page.
http://linux.oracle.com/pls/apex
jointly supported by the
/f?p=117:1:5773793518142288::NO:RP:: hardware vendor and Oracle.
The Maximum Number of
CPU cores or threads an
Oracle VM server can support
is 160. Oracle VM server
maps a virtual CPU to a
hardware thread on a CPU
core in a CPU socket.
Oracle VM Server supports
CPU oversubscription. CPU
oversubscription allows an
Oracle VM Server with 160
CPU cores to overallocate the
total number of CPU cores to
virtual machines. For
example, a server with an
Intel Xeon processor
5600-series CPU with
hyperthreading can have up
to six cores and twelve
threads per socket. A two
socket server with an Intel
Xeon processor 5600-series
CPU could allocate twenty
four virtual CPUs without
oversubscribing the physical
CPUs.
CPU-bound workloads, such
as Oracle Databases, should
not be on Oracle VM Servers
with oversubscribed CPUs.
RAM
Server hardware will be ordered with

the maximum amount of physical
memory.
Note: Oracle VM Server supports up to
Oracle VM Server does not

support memory
oversubscription. For
example, an Oracle VM
Server with 1TB of RAM
12 of 41
4TB of RAM.
cannot overallocate RAM to

virtual machines. By default,
each Oracle VM Server
reserves 512MB of RAM for
dom0. The average memory
overhead for each running
guest on a dom0 is
approximately 20MB plus 1%
of the guests memory size.
The remaining physical RAM
can be allocated to guests.
An Oracle VM Server in a
server pool with Live
Migration, DRS, DPM and/or
HA must
have excess RAM capacity to
accept virtual machines from
a Live Migration, DRS, DPM
and/or HA operation. Oracle
VM pool members without
available RAM can not
support Live Migration, DRS,
DPM and/or HA. Having
available RAM on each server
provides exibility in terms of
adding new virtual machines
to the server pool, and to
allow Live Migration, DRS,
DPM and/or HA within a
server pool.
Storage
Unless the Oracle VM server is booting

from SAN, redundant SSD internal hard
drives are recomended.
Oracle VM Server requires

only 3 GB of local storage
for the entire Oracle VM
Server installation. The
Virtual machine image and conguration design goal for Oracle VM is
les are hosted on shared SAN, iSCSI, or to support multiple node
NFS repositories.
Oracle VM Server pools with
shared bre channel SAN,
iSCSI and/or NFS storage.
Oracle VM supports local
storage without HA or Live
Migration. With local storage,
the OCFS2 virtual machine
le system must be on a
dedicated non SAS hard dirve.
For example, a partition on
13 of 41
same disk as Oracle VM

server installation is not
supported. Local SAS storage
for virtual machines is not
supported.
Network
Interface
Cards
A minimum of one Ethernet network

interface (NIC) card is required just to
install Oracle VM, although four or more
10G NICs is strongly recommended. NIC
bonding with port-based VLANs
and/or 802.1Q tag-based VLANs are
supported and congured post Oracle
VM Server installation with Oracle VM
Manager or Enterprise Manager. Oracle
VM 3.0.1 through 3.1.1 supports two
NIC ports per network bond, and a total
of ve network bonds per Oracle VM
Server. Oracle VM 3.2.x and above
supports four NIC ports per network
bond, and a total of ten network bonds
per Oracle VM Server.
The exact number of network interfaces
for an Oracle VM Server entirely
depends on your organizations business
requirements and network and storage
infrastructure capabilities. For example,
an Oracle VM Server with four 10G
NICs, congured with two 802.1Q bonds
could support the most demanding
network and storage requirements, with
only four 10G NICs. By contrast, an
Oracle VM Server using access
ports/port-based VLANs or 802.1Q
tag-based VLANS on a 1G copper
network, could easily use the maximum
number of supported NIC ports
(<= 3.1.1 = 10 ports, >= 3.2 = 40
ports) to meet the minimum network
requirements.
NAME
Rate(bit/s) Rate(byte/s)
Gigabit Ethernet
1 Gbit/s 125 MB/s
10 Gigabit Ethernet 10 Gbit/s 1.25 GB/s
Inniband DDR
16 Gbit/s
2 GB/s
Tip: One thing to consider is NIC
rmware levels between bonded internal
Both 802.3AD NIC bonds,

port-based VLANs and/or
802.1Q tag-based VLANs are
supported and congured
post Oracle VM Server
installation with Oracle VM
Manager. Network
redundancy, i.e. 802.3AD NIC
bonding doubles the number
of required NICs.
Oracle VM uses a total of ve
discrete networks; Server
Management, Cluster
Heartbeat, live Migration,
Storage and Virtual Machines.
All ve networks can be
supported using one or more
802.1Q tag-based VLANs (2
NICs) or using up to ve
802.3AD bond (10 NICs).
Each Oracle VM server pool
should have a discrete
network for the Server
Management, Cluster
Heartbeat, Live Migration,
Storage and Virtual Machines.
Isolating the Server
Management, Cluster
Heartbeat, Live Migration and
Storage networks protect the
server pool from unexpected
server reboots by eliminating
OCSF2 heartbeat
interruptions that could cause
a pool member to loose
network connectivity, fence
from the pool and reboot.
Each Oracle VM Server will
be assigned a unique IP
address on the Server
14 of 41
Host Bus
Adapter
Cards
NIC ports and PCI NIC ports. Consider

only bonding internal NICs with internal
NICs and PCI NICs with PCI NICs.
Management, Cluster
Heartbeat, live Migration and
Storage network.
SAN Storage: 2 Host Bus Adapter Cards

(HBAs).
2 HBAs are used to eliminate

a single point of failure.
NAME
4GFC
8GFC
10GFC
16GFC
20GFC
Line-Rate Throughput MBps

4.25
8.5
10.52
14.025
21.04
800
1600
2550
3200
5100
Oracle VM Server Pool Design

Oracle VM uses the concept of a "server pool" to group together and centrally manage
one or more server pools with up to 32 Oracle VM servers. If more than one location
exists, Oracle VM server pools may be dispersed to dierent locations. Oracle VM
Manager with Oracle Enterprise Manager 12c provide a single point of administration for
one or more dispersed Oracle VM server pools.
Oracle VM server pools can accommodates organization-specic needs, i.e., Oracle
technology license management (hard and soft partitioning) , defense in depth, the
principle of least privilege, compartmentalization of information, security domains and
dierent applications and their performance, authentication, and security requirements.
The next Figure shows a high-level overview of how server pools can be used to
implement security domains, defense in depth, the principle of least privilege and
compartmentalization of information.
This table outlines the decision points for an Oracle VM server pool. For decisions that
rely on preexisting factors or specic organizational needs, the appropriate best practice
will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best
practices should be analyzed carefully and decisions should be made based on
organizational needs, existing architecture, and budget resource availability.
15 of 41
Decision
Point
Decision
Justication
Oracle VM
Server Pool
Design
Prior to implementing an Oracle Cloud, its

important that an infrastructure
assessment (IA) and gap analysis (GA) be
performed. During the IA/GA, the
architecture of the solution will be
matched to the customers business needs.
Server pool design is a

strategic, architectural
security decision. Server
pools can be used to
control Oracle licensing
costs (hard and soft
partitioning) and as a way
to implement security
domains, defense in depth,
the principle of least
privilege and
compartmentalization of
information.
Oracle VM
Manager
The Oracle VM Manager installer provides

two installation options. Oracle VM 3.0.1
up to 3.1.1 oers
a Demo or Production installation. Oracle
VM 3.2.1 and above oers
a Simple or Custom installation.
For large environments

(>33 hosts), the Oracle VM
Manager Database
repository should be on
dedicated virtual or
physical servers. If your
Oracle VM environment
starts out small and scales
out, make sure to have a
plan to scale up Oracle VM
Manager with more RAM
and CPUs and scale out the
Oracle VM Manager
Database repository on
dedicated virtual or
physical servers with RAC.
Oracle VM Manager will be installed in

Production, Simple or Custom mode on a
dedicated physical or virtual server.
Production and Custom mode uses a local
or external Oracle 11g Enterprise or RAC
database on a dedicated physical or virtual
server. Simple mode uses a local MySQL
database.
The Oracle VM Manager Database
repository will not be shared with other
production or test databases on the same
server.
The Oracle Enterprise Manager Agent and
the Virtualization plug-in will be installed
to enable Oracle Enterprise Manager
integration.
For the Oracle VM

Manager Database
repository, scaling out
means moving from a
single server Database to a
multi node RAC cluster. An
important consideration
when scaling out an Oracle
VM Manager environment
is to determine if the
underlying hardware where
the Oracle VM Manager
Database repository runs is
capable to transition to
RAC. If the hardware is not
capable to transition to
16 of 41
RAC, it is possible to move

and/or export the Oracle
VM Manager Database
repository to a dierent
system with more
resources.
Monitoring
and Alerting
The Oracle VM product family; Oracle VM

Server, Oracle VM Manager, virtual
machines, Oracle VM Templates and
Assemblies can be managed and monitored
with Oracle VM Manager and Oracle
Enterprise Manager 12c. Unlike Oracle
VM 2.x, which could only be managed by
Oracle VM Manager or Oracle Enterprise
Manager, not both, Oracle VM 3 and above
can be managed simultaneously by Oracle
VM Manager along with Oracle Enterprise
Manager 12c Cloud Control.
When things go wrong

within an Oracle VM server
pool, being able to quickly
determine the root cause
of an issue can eliminate or
reduce down time. The
most eective way to
identify problems with an
Oracle VM server pool is to
analyze the Oracle VM
Manager, the Oracle VM
Servers, and the virtual
machines performance
Oracle VM Manager is a stand-alone
statistics, and log les
management solution for Oracle VM, with
using Oracle Enterprise
limited monitoring and alerting
Manager, SNMP based
functionality. Oracle VM is a default Oracle monitoring solutions, and a
Enterprise Manager 12c feature that
central log host.
provides Infrastructure as a Service (IaaS),
Database as a Service (DaaS), Platform as
a Service (PaaS) and Testing as a Service
(TaaS) provisioning with a self-service
portal. Oracle VM should be enabled in
Cloud Control by installing an Oracle
Management Agent with the Virtualization
plug-in on a managed Linux target with
Oracle VM Manager. Once Oracle VM is
enabled in Cloud Control, Oracle VM
Manager, Oracle VM Servers, and all the
virtual machines can be managed, and
setup with performance monitoring
proles and alerts that can be used for
root cause and statistical analysis.
A central log host should be congured to
capture the Oracle VM Server, the Oracle
VM Manager, and the virtual machine
operating systems log les.
Network
Time
Protocol
(NTP)
With Oracle VM, accurate time is essential

to maintain system stability due to
time-sensitive cluster transactions between
Oracle VM Servers. Without accurate time,
With Oracle VM, accurate

time is essential to
maintain system stability
due to time-sensitive
17 of 41
Oracle VM clusters can be brought to a

complete standstill.
By default, Oracle VM Servers (up to
Release 3.1.1) that are discovered by
Oracle VM Manager are congured to use
the Oracle VM Manager host as the
upstream NTP time host. A best practice is
to set-up the Oracle VM Manager hosts as
the upstream NTP time host to synchronize
with upstream Coordinated Universal Time
(UTC) sources as well as provide time
services to Oracle VM Servers.
cluster transactions
between Oracle VM
Servers. Without accurate
time, Oracle VM clusters
can be brought to a
complete standstill.
A best practice is to set-up

the Oracle VM Manager
hosts as the upstream NTP
time host to synchronize
with upstream Coordinated
Universal Time (UTC)
sources as well as provide
Oracle Linux and Red Hat Enterprise Linux time services to Oracle VM
ship with a default /etc/ntp.conf le that
Servers.
points to three of Red Hat's upstream
public UTC time sources. A best practice is
to have two internal NTP servers on your
local network to provide time services for
internal systems and devices. Using
internal time servers normalizes system
event time-stamps across the Enterprise as
well as reduces NTP Internet bandwidth
usage.
Oracle VM
Server Agent
Roles
Oracle VM Manager facilitates centralized

management of server pools and their
resources using an agent-based
architecture. When an Oracle VM server is
added to a server pool, up to three Oracle
VM agent roles can be enabled. There are
a total of three Oracle VM agent roles; 1)
Master Server, 2) Utility Server and 3) VM
Server. When an Oracle VM server is
added to a server pool, it can be assigned
one, two, or all three of the agent roles.
Master Server
By default each clustered
server pool has one Master
Server with the Virtual IP
feature enabled.
Utility Server
The Utility Server role is
responsible for
I/O-intensive operations
such as virtual machine
creation and removal, as
well for as creating,
Master Server
deleting, modifying,
copying and moving virtual
The Master Server is the principal
machine les. Enabling the
server pool role within a server pool. Utility Server agent role
The Master Server is the server that with the VM Server role on
communicates with Oracle VM
the same Oracle VM server
Manager. The Master
may negatively aect
Server dispatches commands
running virtual machines
received from Oracle VM Manager to during Utility Server
other servers within a server pool.
operations. Server pools
There can be only one Master
18 of 41
Server in a server pool at any

instant. The Virtual IP feature is a
mandatory server pool property that
detect the loss of the server pool
master agent and responds with
automatic failover to the rst pool
member that can lock the pool le
systsm. The server pool Virtual IP
feature removes the single point of
failure (SPOF) for the server pool
master agent role.
Utility Server
The Utility Server role is responsible
for I/O-intensive operations such as
virtual machine creation and
removal, as well for as creating,
deleting, modifying, copying and
moving virtual machine les.
The Master Server dispatches
operations to Utility Servers. There
can be one or more Utility Servers in
a server pool. When there are
multiple Utility Servers in a pool, the
server Master Server will select the
least loaded utility server to conduct
a task.
that are not static and

support the self service
portal in Oracle Enterprise
Manager Cloud Control 12c
should have one or
more dedicated utility
servers to isolate the
impact of I/O intensive
operations to Utility
Servers.
VM Server
Unless a server pool is
static, VM Servers should
only have the VM Server
role enabled to be able to
dedicate CPU, RAM and I/O
resources to running virtual
machines, eliminating the
eect of Utility Server
operations.
Tip: Oracle VM environments that

dynamically grow should have one or
more dedicated Utility Servers to
isolate I/O jobs to Utility Servers. For
example, an I/O intensive that
runs on an Oracle VM Server with
the VM Server and Utility Server role
will impact the performance of all of
the virtual machines running on the
Oracle VM Server.
VM Server
Servers with the VM Server role are
responsible for allocating CPU,
memory, and disk resources to the
virtual machines in a server pool.
There can be one up to 32 VM
Servers in a server pool.
19 of 41
Storage
Back-end storage
Each Oracle VM server pool uses one
dedicated OCFS2 12G mount point (OCFS2
or NFS) for the server pool's cluster
congurations and one or more shared
OCFS2 or NFS repositories to host virtual
machine conguration les and images.
An Oracle VM storage
solution consists of three
distinct layers. Each layer
has its own unique
requirements,
congurations,
dependencies and features.
Front-end storage
The virtual machine layer is where the
storage is presented to virtual machines as
either a at le (UUID.img), as RAW disks
(LUN), or as a combination of at les and
RAW disks.
The rst layer is the

storage array, which is
referred to as back-end
storage. Oracle VM
supports Fibre Channel and
iSCSI SAN and NFS
back-end storage.
The second layer is the
server layer consisting of
the Oracle VM Server's
Device-Mapper Multipath
congurations and the
shared Oracle Cluster File
System 2 (OCFS2) or NFS
virtual machine le system.
Note: OCFS does not
factor disk space
exhaustion including space
for virtual machine les as
well as volume
metadata. OCFS2 metadata
can consume over 6% of an
OCFS2 volumes free disk
space.
The third layer is the
guest front-end storage
consisting of multiple guest
storage (le and RAW) and
driver options. RAW disks
have the best performance
of the two front-end storage
storage options. In most
cases, RAW disks are the
best option for high I/O
workloads like Oracle
Databases.
Networks
Each Oracle VM server pool will have

isolated Oracle VM management networks
Each Oracle VM server

pool should have a discrete
20 of 41
and isolated virtual machine networks.
network for the Server

Management, Cluster
Oracle VM uses a total of ve discrete
Heartbeat, live Migration,
networks; Server Management, Cluster
Storage and Virtual
Heartbeat, live Migration, Storage and
Machines. Isolating the
Virtual Machines.
Server Management,
Cluster Heartbeat, live
The exact number of network interfaces
Migration and Storage
for an Oracle VM Server entirely depends
networks protect the server
on your organizations business
pool from unexpected
requirements and network and storage
server reboots by
infrastructure capabilities. For example, an eliminating OCSF2
Oracle VM Server with four 10G NICs,
heartbeat interruptions
congured with two 802.1Q bonds could
that cause pool members to
support the most demanding network and
fence from the pool and
storage requirements, with only four 10G
reboot.
NICs. By contrast, an Oracle VM Server
using access ports/port-based VLANs
Each Oracle VM Server will
or 802.1Q tag-based VLANS on a 1G
be assigned a unique IP
copper network, could easily use the
address on the Server
maximum number of supported NIC ports
Management, Cluster
(<= 3.1.1 = 10 ports, >= 3.2 = 40 ports)
Heartbeat, live Migration
to meet the minimum network
and Storage network.
requirements.
Note: The heartbeat trac
is TCP on port 7777. Each
Oracle VM server in a pool
must be able to
communicate to all of the
pool members over TCP on
port 7777.
RAM
The server pool must be designed with

excess RAM capacity to accommodate the
memory requirements of virtual machines
that could migrate or start on any pool
member.
Oracle VM server does not

support memory
oversubscription, which
means that an Oracle VM
server cannot accept a
DRS, Live Migration or HA
requests unless the server
has available RAM for the
virtual machines. Having
excess RAM on each Oracle
VM server provides
exibility in terms of
adding new virtual
machines to the server
pool, and to allow DRS,
Live Migration and HA to
operate within a server
21 of 41
pool.
Oracle VM Security Standards

The security controls used to secure Oracle VM are similar to the security controls used
to protect your existing physical and virtual IT resources. As with physical and virtual IT
resources, securing Oracle VM is dependent on the security posture of each of its
components, from the design, hardware, hypervisor, network, and storage to the virtual
machine operating systems and installed applications. In short, if the organization has a
security policy for virtualization, networking, storage, operating systems and
applications, the security policies could be applied to Oracle VM.
Security controls should be employed using industry standard frameworks and standards
in the context of the organization's Enterprise Architecture (EA). Organizations turn to
their Enterprise Architecture to understand how Oracle VM ts within their information
system. An Enterprise Architecture is articulated in diagrams and written policies that
dene organizational standards and best practices to plan, build, run, and monitor
technologies, including Oracle VM.
Enterprise Architecture has well dened principles and processes and an approach that
generates a comprehensive, layered policy infrastructure used to communicate
managements goals, instructions, procedures, and response to laws and regulatory
mandates. A policy infrastructure consists of written tier 1, tier 2, and tier 3 policies that
encompass people, systems, data, and information. Policies are broken down into high
level policies and lower level standards, procedures, baselines, and guidelines.
Oracle VM policies typically fall within the layered policy infrastructure of the platform
architecture domain. Platform architecture policies are the foundation used to manage
the entire lifecycle of an Oracle VM environment.
This table outlines the decision points for Oracle VM Manager security controls. For
Oracle VM Manager Security Controls

Decision Point
Decision
Justication
Oracle VM
Manager and
DMZs
The Oracle VM Manager

The Oracle VM Manager
application was not designed to be application was not designed to be
an Internet facing application. If
an Internet facing application.
Internet access is a requirement
for Oracle VM Manager, VPN
access should be used to access
the Oracle VM Manager GUI.
22 of 41
Network Time
Protocol (NTP)
With Oracle VM, accurate time is

essential to maintain system
stability due to time-sensitive
cluster transactions between
Oracle VM Servers. Without
accurate time, Oracle VM clusters
can be brought to a complete
standstill.
With Oracle VM, accurate time is

essential to maintain system
stability due to time-sensitive
cluster transactions between
Oracle VM Servers. Without
accurate time, Oracle VM clusters
can be brought to a complete
standstill.
By default, Oracle VM Servers (up

to Release 3.1.1) that are
discovered by Oracle VM Manager
are congured to use the Oracle
VM Manager host as the upstream
NTP time host. A best practice is
to set-up the Oracle VM Manager
hosts as the upstream NTP time
host to synchronize with upstream
Coordinated Universal Time
(UTC) sources as well as provide
time services to Oracle VM
Servers.
A best practice is to set-up the

Oracle VM Manager hosts as the
upstream NTP time host to
synchronize with upstream
Coordinated Universal Time
(UTC) sources as well as provide
time services to Oracle VM
Servers.
Oracle Linux and Red Hat

Enterprise Linux ship with a
default /etc/ntp.conf le that
points to three of Red Hat's
upstream public UTC time
sources. A best practice is to have
two internal NTP servers on your
local network to provide time
services for internal systems and
devices. Using internal time
servers normalizes system event
time-stamps across the Enterprise
as well as reduces NTP Internet
bandwidth usage.
Virtual Machine
Console Access
Oracle VM uses the RAS proxy

(Remote Access Service) java
applet to proxy virtual machine
console trac from Oracle VM
Manager to the administrator's
Client PC. An Oracle VM Manager
administrative account is a
requirement to access a virtual
machine's console. Any rewall
between Oracle VM Manager
and the administrator's Client
PC conecting to a virtual machine
All Oracle VM administrative

users have root access to all of the
objects managed by Oracle VM
Manager. Virtual machine end
users such as DBAs and
application administrators should
only have access to thier virtual
machines, not root access to all of
the objects managed by Oracle
VM Manager. End user access to
virtual machines can be controled
using Enterprise Manager Cloud
23 of 41
console must have TCP port

15901 open for the RAS proxy.
Control. With Cloud Control, each

object managed by Oracle VM
Manager can be congured with
Oracle VM Manager does not
role based access control,
support role based access control. including each virtual machine
All administrative users with
console.
access to the Oracle VM Manager
GUI have root administrative
access to all of the objects
managed by Oracle VM Manager,
including all of the virtual
machine consoles. If an Oracle VM
Manager account is not an option
for a user, for example for DBAs,
Opertaions. or application
administators, Oracle VM role
based access control can be
congured using Enterprise
Manager Cloud Control. With
Cloud Control, each object
managed by Oracle VM Manager
can be congured with role based
access control, including each
virtual machine console.
Host rewall
Host
rewall failed
connection
logging
The iptables service should be

enabled on each Oracle VM
Manager host using a ruleset
managed in /etc/syscong
/iptables. In order to use Oracle
VM Manager, the Core API and
the Oracle Management Agent
with iptables, it is necessary to
open tcp ports 7001, 7002,
tcp-54321 or tcps-54322, 15901
and 3872 as well as UDP 123.
Iptables failed connection logging
should be enabled on each Oracle
VM Manager host.
Host rewalls, for example

iptables, are a fundamental part
of information security that
protect hosts from attacks and
intrusions.
Failed connect logging is a

fundamental part of information
security that allows detection of
attacks and intrusions.
The following two lines will be

added prior to the last REJECT
line in the /etc/syscong/iptables
le:
-A RH-Firewall-1-INPUT -m limit
--limit 15/minute -j LOG
--log-prex "FW Drop:"
-A RH-Firewall-1-INPUT -j REJECT
--reject-with icmp-hostprohibited
24 of 41
Systems administrators should

access the Oracle VM Manager
host with non-root individual user
accounts and use sudo to perform
selected administrative
tasks. Sudo stands for either
"substitute user do" or "super
user do".
Root ssh access

and sudo
Root ssh access should be

disabled on the Oracle VM
Manager host. Sudo should should
be used to congure ne-grained
permissions to allow
administrative users to perform
selected administrative tasks with
logging.
By default, Oracle Linux permit

ssh access using the root super
user account.
One of the most important
security measure that can be
taken with Oracle VM is to
prevent unauthorized access to
the root user account by disabling
root ssh access. A best practice is
to only provision non-root
individual user accounts that can
be audited, disabled, expired and
managed using sudo.
Disables Root Access:

To disable root ssh access, edit
the default /etc/ssh/sshd_cong
le and uncomment the
the #PermitRootLogin yes
Note: All sudo user access will be
line and change the yes to no; that
tracked and logged in the /var/log
is, PermitRootLogin no. Next,
/secure le.
restart the sshd service by typing
service sshd restart to enable
the change.
The visudo command is used to
edit the /etc/sudoers le. Consult
the sudoers man page for sudo
conguration details.
Pre and post SSH login banners
should be congured on each
Oracle VM Manager host.
SSH login
banners
Pre-login banner:
Edit the /etc/ssh/sshd_cong and
add the following directive:
Banner /etc/banner.net
Next, create the /etc/banner.net
le and add your login banner
text, i.e.
This system is restricted to
authorized access only. All
activities on this system are
To be able to successfully
prosecute individuals who
improperly use a computer, the
computer must have a warning
banner displayed at all access
points.
SSH login banners presents a
denitive warning or disclaimer to
all users that wish to access your
systems using SSH. SSH login
banners should clarify which
types of activities are illegal as
well as advise legitimate users of
their obligations relating to the
acceptable use of the system.
25 of 41
recorded and logged.

Unauthorized access will be fully
investigated and reported to the
appropriate law enforcement
agencies.
Once the le has been created and
the banner text is added and
saved, restart the sshd by typing:
# service sshd restart
Post login banner:
Edit /etc/motd and add your login
banner text, i.e.
agencies.
Once the le has been edited and
Central log host
A central log host should be used

to log all user logins and iptables
connection failures.
Centralized logging for user logins

and iptables connection failures
simplies security management
for the detection of attacks and
intrusions.
This table outlines the decision points for Oracle VM Server security controls. For
Oracle VM Server Security Controls

Decision Point
Decision
Justication
Oracle VM
Server and
DMZs
Oracle VM Servers hosting

Internet facing virtual machines
can be placed in a DMZ without
connectivity to the Internet or
internal network segments to
Oracle VM Servers in a DMZ

should be restricted from inbound
and outbound Internet
connectivity to reduce the attack
surface.
26 of 41
reduce the attack

surface. TCP/8899 is necessary to
and from the Oracle VM Servers
to Oracle VM Manager to enable
centralized management using
Oracle VM Manager.
Build Process
Before any Oracle VM Servers are

placed on the production network,
a standard build processes should
be executed to ensure that all
Oracle VM Servers are installed,
congured and maintained in a
manner that prevents
unauthorized access,
unauthorized use and disruptions
in service.
An Oracle VM Server build

document provides employees
with an approved procedure to
install and congure Oracle VM
Server. An Oracle VM Server build
document is used with other IT
infrastructure policies to address
interoperability and security of
Oracle VM in the context of the
entire information system.
Patch
Management
A key component of a successful

Oracle VM deployment is
acquiring and vetting new
releases, patches and updates for
production systems. New Oracle
VM releases, patches and updates
must be researched to identify
which release, patches and
updates are applicable to your
environment. Newly released
versions, patches and updates
should be vetted before being
deployed into production. A best
practice is to run the latest stable
release of Oracle VM.
A patch management program is

an integral component of an
organization's information
security program used to mitigate
the risk from security
vulnerabilities (bugs) that are
inherent in all operating systems
and applications.
A key component of patch
management is acquiring and
vetting patches for production
systems. Patches must be
researched to identify which
patches, security xes, and
updates are applicable to your
environment. Newly released
patches, security updates, and
application updates will be tested
before being deployed in to
production using time stamped
local custom repositories.
Oracle VM Servers should be

congured to use local custom
yum repositories. Local yum
repositories with point-in-time
static channel for each supported
Oracle VM release ensures all
like Oracle VM server are patched
in a consistent manner across the Pre- and post-production audits
organization.
will be conducted in accordance
with <Company Name>s Change
All patches should be regression
Management Policy to validate
tested in the lab environment
conguration and patch
before they are deployed on
compliance.
production systems. High-priority
upgrades will be applied as
27 of 41
needed in accordance with

<Company Name>s Change
Management Policy.
All production systems should
undergo security audits in
accordance with <Company
Name>s Change Management
Policy to validate conguration
and patch compliance.
Host rewall
Host
rewall failed
connection
logging
Host rewalls, for example

The iptables service will be
iptables, are a fundamental part
enabled on each Oracle VM server
of information security that
using the default policy and
protect hosts from attacks and
ruleset in /etc/syscong/iptables.
intrusions.
Iptables failed connection logging
will be enabled on the Oracle VM
Manager host and each Oracle
VM server.
Failed connect logging is a

security that allows detection of
attacks and intrusions.
The following two lines will be

added prior to the last REJECT
line in the /etc/syscong/iptables
le:
-A RH-Firewall-1-INPUT -m limit
--limit 15/minute -j LOG
--log-prex "FW Drop:"
-A RH-Firewall-1-INPUT -j REJECT
--reject-with icmp-hostprohibited
Systems administrators should
access Oracle VM Servers
with non-root individual user
accounts and use sudo to perform
selected administrative
tasks. Sudo stands for either
"substitute user do" or "super
user do".
Root ssh access
and sudo
Root ssh access should be

disabled on the each Oracle VM
servers. Sudo should should be
used to congure ne-grained
permissions to allow
administrative users to perform
selected administrative tasks with
logging.
Disables Root Access:
By default, Oracle VM Server

permit ssh access using the root
super user account.
One of the most important
security measure that can be
taken with Oracle VM is to
prevent unauthorized access to
the root user account by disabling
root ssh access. A best practice is
to only provision non-root
individual user accounts, that can
be audited, disabled, expired and
managed using sudo.
Note: All sudo user access will be
tracked and logged in the /var/log
/secure le.
28 of 41
To disable root ssh access, edit the

default /etc/ssh/sshd_cong le
and uncomment the
the #PermitRootLogin yes line
and change the yes to no; that is,
PermitRootLogin no. Next,
restart the sshd service by typing
service sshd restart to enable
the change.
Note: To enable sudo on Oracle
VM Servers, it is neccessary to
install the ovs-support-tools
meta-package that includes sudo.
The visudo command is used to
edit the /etc/sudoers le. Consult
the sudoers man page for
conguration details.
Pre and post SSH login banners
should be congured on each
Oracle VM Manager host and
Oracle VM Server.
Pre-login banner:
Edit the /etc/ssh/sshd_cong and
add the following directive:
Banner /etc/banner.net
Next, create the /etc/banner.net
le and add your login banner
text, i.e.
SSH login
banners

agencies.
Once the le has been created
and the banner text is added and
To be able to successfully
prosecute individuals who
improperly use a computer, the
computer must have a warning
banner displayed at all access
points.
SSH login banners presents a
denitive warning or disclaimer to
all users that wish to access your
systems using SSH. SSH login
banners should clarify which
types of activities are illegal as
well as advise legitimate users of
their obligations relating to the
acceptable use of the system.
Post login banner:

Edit /etc/motd and add your login
29 of 41
banner text, i.e.

agencies.
Once the le has been edited and
Note: By default Oracle VM
Server's /etc/motd le displays the
following warning
message: Warning: making
manual modications in the
management domain
might cause inconsistencies
between Oracle VM Manager and
the server.
Rootkit
prevention and
monitoring
Wikipedia describes a rootkit as

A rootkit is software that enables
continued privileged access to a
computer while actively hiding its
presence from administrators by
subverting standard operating
system functionality or other
applications.
Monitoring the hypervisor (Oracle

VM Server) for rootkits is
security used to detect rootkits to
prevent attacks and intrusions.
Each Oracle VM Server should
have a rootkit prevention solution
in place, such as chkrootkit, that
monitors the host for rootkits.
A Hypervisor (Oracle VM Server)

may be one of the most sensitive
operating systems in the data
center because it controls the
hardware as well as all of the
virtual machines on it. If the
hypervisor is compromised direct
access to the hardware and all of
the virtual machines is possible,
and other code could be
monitored and controlled by the
attacker.
Central log host
A central log host will be used to

log all user logins and iptables
connection failures.
Centralized logging for user logins

and iptables connection failures
simplies security management
30 of 41
for the detection of attacks and

intrusions.
Virtual Machine Operating System Standards

This table outlines the decision points for the for virtual machine operating systems
hosted on Oracle VM. For decisions that rely on preexisting factors or specic
organizational needs, the appropriate best practice will be discovered in the
infrastructure assessment (IA) and gap analysis (GA). The best practices should be
analyzed carefully and decisions should be made based on organizational needs, existing
architecture, and budget resource availability.
Decision Point
Decision
Justication
Virtual Machine
Operating
Systems
A small number of virtual

machine operating systems
should be be used. For example,
stadardizing on Oracle Linux 6
latest, in contract to supporting
Oracle Linux 5U2, 5U3, 5U4, 5U5,
5U6, 5U6, 5U8, 5U9, 5U10, 6,
6U1, 6U2, 6U3, 6U4, etc...
Standardizing on a small number

of virtual machine operating
systems streamlines operations
and increases the level of le
duplication on production and
archival virtual machine data.
This design reduces complexity
and increases operational
eciency by limiting the number
of supported operating systems.
Virtual Machine
Operating
System
Versioning
Virtual Machine
Operating
System
Deployments
In accordance with <Company

Name>s Application Software
Policy and Application Software
Standards, applications will
determine the operating system
type and version.
All new virtual machine operating

systems will be deployed using a
virtual machine template in
Name>s Server Policy, and
Server Security Policy.
Each application has an operating

system support matrix that lists
the supported operating systems,
patch levels, and software
prerequisites.
In accordance with <Company
Name>s Application Software
Policy and Application Software
Standards, applications will
determine the operating system
type and version.
A virtual machine template is a
self-contained, precongured
virtual machine with an operating
system and optionally an
application installed in
Name>s Server Policy, Server
Security Policy, and Operating
System Installation Guidelines.
Each time a new virtual machine
31 of 41
is deployed using a virtual

machine template, <Company
Name>s standards are applied to
each new virtual machine.
Linux virtual machines will be

congured to use local custom
yum repositories.
Patch
Management
All patches will be regression

production systems.
High-priority patches, security
xes, and application upgrades
updates will be applied as needed
in accordance with <Company
Policy.
Noncritical xes will be applied
on a Quarterly basis in
Policy.
All production systems will

application updates are applicable
to your environment. Newly
released patches, security
updates, and application updates
will be tested before being
deployed in to production using
time stamped local custom
repositories.
Local yum repositories will be
maintained for patch testing and
production using a point-in-time
static channel for each supported
operating system to ensure all
like operating systems are
patched in a consistent manner
across the organization.
Pre- and post-production audits
compliance.
Oracle VM Disaster Recovery

An Oracle VM disaster recovery architecture includes the design and process to maintain
business continuity following a disastrous event aecting the availability of an
organization's primary site. Failover to a disaster recovery site is prompted by the results
of a disaster assessment. The failover process is the restoration of the primary site's
services at the disaster recovery site.
Note: Disaster recovery requirements are calculated using Service-level Agreements
(SLA), Recovery Time Objectives (RTOs) and Recovery Point Objectives (RPOs) objectives.
SLA, RPO and RTO objectives and budget inuence the disaster recovery architecture
and design.
32 of 41
Oracle VM uses the concept of a server pool to group together and manage one or more
clustered Oracle VM servers. Once an Oracle VM server pool is created, the physical and
virtual resources are managed within the boundary of the server pool. Physical resources
include server hardware, networks, storage, infrastructure services (DNS, NTP, LDAP,
HTTP, etc..), operating system installation media and administrative accounts. The virtual
resources include virtual disks, virtual network interfaces, and virtual machine
conguration les. For example, an Oracle VM environment with multiple server pools
located in one or more sites could be managed from a single Oracle VM Manager instance
with each server pool's resources isolated to their respected server pool. An Oracle VM
server pool's resources from one site can be replicated and restored to another site for
disaster recovery.
Restoration of the primary site's services at a disaster recovery site requires a replica of
the primary site's physical and virtual resources at the disaster recovery site. A disaster
recovery site hosts a replica of the primary site's Oracle VM physical and virtual
resources, i.e. server hardware, networks, storage, infrastructure services, virtual disks,
and virtual machine conguration les. The failover process involves restoring the
primary sites virtual machines at the disaster recovery site, then systematically starting
the virtual machines and services.
Note: Oracle VM Servers are not backed up and restored at the DR site. The time
required to backup and restore an Oracle VM Server is signicantly greater then a PXE
boot kickstart installation.
A disaster recovery site can be a warm failover site waiting idle to respond to a disastrous
occurrence, or part of a multi-site high availability design. A multi-site design uses excess
capacity with application high availability to mirror services across sites to handle the
lose of one or more sites.
The next Figure shows a warm Oracle VM failover site waiting idle to respond to a
disastrous occurrence.
33 of 41
The next Figure shows a warm Oracle VM failover site responding to a disastrous
occurrence and running the primary sites services.
34 of 41
The next Figure shows a multi-site Oracle VM design with application high availability
solutions to mirror services across sites as well as excess capacity to handle the lose of
one or more sites.
35 of 41
Virtual machines that are restored at a disaster recovery site expect the same networks,
storage, and infrastructure services as in the primary site. In the event that the disaster
recovery site has dierent networks, storage, and infrastructure services, the properties
of each virtual machines would need to be edited to use the new networks, storage and
infrastructure services before services can be restored.
The virtual machine operating systems are typically installed in virtual disks that are
actually at les hosted on shared OCFS2 or NFS repositories. RAW disks such as ASM
Disks, Log and Archive Files, etc.. are presented to the virtual machines from the Oracle
VM Servers as local devices. Each virtual machine's virtual network interface card(s)
(vNIC) are connected to one or more discrete networks using Xen bridges that are
managed and presented to the virtual machines by the Oracle VM pool members. Virtual
disks and virtual network interface card(s) allocations are managed using Oracle VM
Manager and/or Oracle Enterprise Manager with the congurations saved in each virtual
machines vm.cfg le.
The virtual machine vm.cfg les, virtual disk images and RAW disks (ASM disks) can be
replicated between sites using storage array replication and/or mirroring solutions. Rsync
is an option if an array does not have replication and/or mirroring functionality.
As soon as the replicated storage repositories are available, the failover process for a
warm recovery site starts with the installation of Oracle VM Manager with the
runInstall.sh --uuid option using the primary sites Oracle VM Manager UUID. An Oracle
VM Manager --uuid installation allows Oracle VM Manager to use the primary site'
replicated repositories with the virtual machines.
36 of 41
Tip: The Oracle VM Manager UUID is listed in the .cong le on the Oracle VM
Manager host in the /u01/app/oracle/ovm-manager-3/ directory as well as in each server
pool' .ovsrepo le in the pool le system.
The next example shows the content of the .cong le with the UUID in bold.
# cat /u01/app/oracle/ovm-manager-3/.cong
DBHOST=localhost
SID=orcl
LSNR=1521
APEX=None
OVSSCHEMA=ovs
WLSADMIN=weblogic
OVSADMIN=admin
COREPORT=54321
UUID=0004fb00000100009edfaa0f93184f44
BUILDID=3.0.3.126
The next example shows the content of the .ovsrepo le with the UUID in bold.
# cat .ovsrepo
OVS_REPO_UUID=0004fb0000030000554308a6997a6b2f
OVS_REPO_MGR_UUID=0004fb00000100009edfaa0f93184f44
OVS_REPO_VERSION=3.0
This table outlines the decision points for an Oracle VM disaster recovery solution. For
Decision
Point
Decision
Justication
Disaster
Recovery
Design
Prior to implementing an Oracle

VM Disaster Recovery solution,
its important that an
infrastructure assessment (IA)
and gap analysis (GA) be
performed. During the IA/GA, the
architecture of the solution will
be matched to the customers
SLA, Recovery Time Objectives
(RTOs) and Recovery Point
Objectives (RPOs) objectives.
Implementing a Disaster
Recovery is a strategic decision.
Disaster recovery requirements
are calculated using SLA,
Recovery Time Objectives (RTOs)
and Recovery Point Objectives
(RPOs) objectives. SLA, RPO and
RTO objectives and budget
inuence the disaster recovery
architecture and design.
Oracle VM
Manager
Oracle VM Manager will be

installed in Production mode
using the runInstall.sh --uuid
option with the primary site's
As soon as the replicated storage

repositories are available, the
failover process for a warm
recovery site starts with the
37 of 41
Oracle VM Manager UUID.

Oracle VM Manager will be
hosted on a dedicated physical
server using an external or local
Oracle 11g Standard, Enterprise
or RAC database.
Once Oracle Enterprise Manager
is restored, the Oracle Enterprise
Manager Agent and Virtualization
plug-in will be installed to enable
Oracle Enterprise Manager
integration.
installation of Oracle VM
Manager with the runInstall.sh
--uuid option using the primary
sites Oracle VM Manager UUID.
An Oracle VM Manager --uuid
installation allows Oracle VM
Manager to use the primary site'
replicated repositories and virtual
machines.
The Oracle VM Manager UUID is
listed in the .cong le on the
Oracle VM Manager host in the
/u01/app/oracle/ovm-manager-3/
directory as well as in each server
pool' .ovsrepo le in the pool le
system.
Oracle VM
Server Builds
Oracle VM Servers will be

installed using an automated
build process.
Oracle VM servers are installed

using an automated PXE boot
conguration to ensure that each
server has a consistent
installation conguration.
Oracle VM
Server Backups
Oracle VM Servers will not

backed up at the primary site and
restored at the DR site.
The time required to backup and

restore an Oracle VM Server is
signicantly greater then an
automated PXE boot kickstart
installation.
Oracle VM servers are installed
using an automated PXE boot
conguration to ensure that each
server has a consistent
installation conguration.
Storage
A replica of the primary site's

repositories with the virtual
machine resources and RAW disks
will be hosted at the disaster
recovery site.
As soon as the replicated storage

repositories and RAW disks are
available, the failover process for
a warm recovery site starts with
the installation of Oracle VM
Manager with the runInstall.sh
--uuid option using the primary
sites Oracle VM Manager UUID.
An Oracle VM Manager --uuid
installation allows Oracle VM
Manager to use the primary site'
replicated repositories and virtual
machines.
38 of 41
Virtual machines that are

restored at a disaster recovery
site expect the same storage as in
the primary site. In the event that
the disaster recovery site has
dierent storage each virtual
machine would need to be
recreated or edited to use the
new storage before services can
be restored.
Networks

Oracle VM networks will be
maintained at the disaster
recovery site.

site expect the same networks as
in the primary site. In the event
that the disaster recovery site has
dierent networks each virtual
machine would need to be edited
to use the new networks before
services can be restored.
Infrastructure
Services

infrastructure services will be
maintained at the disaster
recovery site.

site expect the same
infrastructure services as in the
primary site. In the event that the
disaster recovery site has
dierent infrastructure services,
each virtual machine operating
system would need to be edited to
use the new infrastructure
services before services can be
restored.
Oracle VM Application Integration

This table outlines the decision points for an Oracle VM hosted application integration.
For decisions that rely on preexisting factors or specic organizational needs, the
appropriate best practice will be discovered in the infrastructure assessment (IA) and gap
analysis (GA). The best practices should be analyzed carefully and decisions should be
made based on organizational needs, existing architecture, and budget resource
availability.
Decision Point
Application
Support
Decision
Justication
Applications must be supported

by the independent software
vendor (ISV) on the latest version
of Oracle VM to be included in the
Oracle VM environment.
Only applications that are

supported by Oracle on Oracle
VM should be hosted on Oracle
VM. Only applications with ISV
support for the latest version of
39 of 41
Oracle VM can be deployed and

supported by <Company Name>,
the ISV, and Oracle on Oracle VM.
Application
Requirements
and
Dependencies
Applications will be analyzed for

requirements and dependencies
and tested in accordance with
<Company Name>'s Software
Installation Standards.
Applications will be analyzed for

requirements and dependencies
and tested to ensure compliance
with ISV specications.
Applications should be installed

and packaged using Oracle VM
Template Builder or Oracle
Enterprise Manager.
Application
Installation,
Packaging, and
Distribution
Applications that are installed and

packaged using Oracle VM
Template Builder will be deployed
as Oracle VM templates.
Application installations that are

packaged in Oracle VM templates
or deployed using Oracle
Enterprise Manager have a
consistent installation
conguration.
Applications that are installed

using Oracle Enterprise Manager
will be installed on Oracle VM
templates.
Application
sunsetting
Applications will be sunsetted in

Name>'s Hardware and Software
Sunset Policy
Applications that have reached

the end of their life cycle and are
no longer supported by a vendor
will be given a sunset date. The
sunset date is when the product is
scheduled to be removed from
production.
Sunsetting applications that have
reached the end of their life cycle
results in better customer service
and reduced costs.
Patch
Management
All patches will be regression

production systems in accordance
Management Policy.
Noncritical xes will be applied
on a Quarterly basis in

application updates are applicable
to your environment. Newly
released patches, security
updates, and application updates
will be tested before being
40 of 41
deployed in to production using

time stamped local custom
repositories.
Policy.
All production systems will
Pre- and post-production audits

compliance.
41 of 41

Oracle Cloud Reference Design 01-14-2014

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Oracle Cloud Reference Design 01-14-2014

Uploaded by

Copyright:

Available Formats

Mokum is the only full-time Oracle virtualization

integrator with the expertise to help you virtualize your

Last update: 01/14/2014

The Oracle Cloud Reference Design Introduction

Oracle Enterprise Manager

Mokum Solutions, Inc. +1 415 252-9164

Oracle VM for x86

The Oracle Cloud Reference Design Implementation

Mokum Solutions, Inc. +1 415 252-9164

Platform as a Service (PaaS). Platform as a Service is the capability to host and

Mokum Solutions, Inc. +1 415 252-9164

systems and within which security domain systems should reside.

The Oracle Cloud Reference Design Support

Oracle Support Agreements for

Support is an integral part of

On-site and o-site support from

On-site and o-site support from

Oracle Cloud Architectural Design

Mokum Solutions, Inc. +1 415 252-9164

Oracle VM Hardware Architecture

CPU Cores or Threads

Mokum Solutions, Inc. +1 415 252-9164

This image shows an

This image shows an

Mokum Solutions, Inc. +1 415 252-9164

running virtual machines.

Oracle VM HA automatically restarts virtual machines when an Oracle VM pool member

Mokum Solutions, Inc. +1 415 252-9164

Mokum Solutions, Inc. +1 415 252-9164

The server hardware must be jointly

Server hardware will be ordered with

Only jointly supported

Server hardware will be ordered with

Oracle VM Server does not

Mokum Solutions, Inc. +1 415 252-9164

cannot overallocate RAM to

Unless the Oracle VM server is booting

Oracle VM Server requires

Mokum Solutions, Inc. +1 415 252-9164

same disk as Oracle VM

A minimum of one Ethernet network

Both 802.3AD NIC bonds,

Mokum Solutions, Inc. +1 415 252-9164

NIC ports and PCI NIC ports. Consider

SAN Storage: 2 Host Bus Adapter Cards

2 HBAs are used to eliminate

Line-Rate Throughput MBps

Oracle VM Server Pool Design

Mokum Solutions, Inc. +1 415 252-9164

Prior to implementing an Oracle Cloud, its

Server pool design is a

The Oracle VM Manager installer provides

For large environments

Oracle VM Manager will be installed in

For the Oracle VM

Mokum Solutions, Inc. +1 415 252-9164

RAC, it is possible to move

The Oracle VM product family; Oracle VM

When things go wrong

With Oracle VM, accurate time is essential

With Oracle VM, accurate

Mokum Solutions, Inc. +1 415 252-9164

Oracle VM clusters can be brought to a

A best practice is to set-up