Professional Documents
Culture Documents
sales@mokumsolutions.com
Copyright 2014 Mokum Solutions, Inc. All rights reserved.
Distribution of the Oracle Cloud Cookbook or derivative of the work in any form
is prohibited unless prior permission is obtained from the Copyright holder.
About Mokum Solutions, Inc.
Founded in March 2011, Mokum Solutions, Inc. specializes in the implementation,
delivery and support of Oracle technologies in private and public clouds. Mokum
corporate headquarters are located in San Francisco, CA http://mokumsolutions.com
or call 1 415 252 9164
About the Author
The author of the Oracle Cloud Cookbook is none other than the owner of
Mokum Solutions, Inc., Roddy Rodstein. Roddy is one of the most respected
Oracle Cloud Computing experts, having designed and managed many of the
worlds largest and most complex Oracle private clouds. Before establishing
Mokum in March 2011, Roddy spent three years at Oracle on the Oracle VM
and Oracle Linux team designing and supporting Oracle's largest and most
complex customer environments. Before Oracle, Roddy spent six years at Citrix,
designing and supporting Citrix's largest and most complex customer environments,
Including Oracle's. With Mr. Rodsteins rich background and knowledge, there
can be no better resource for revealing the Oracle Cloud recipe.
Audience
The Oracle Cloud Cookbook is a comprehensive, field tested reference design that
guides you through each step to move to your Oracle software portfolio to an elastic
Oracle cloud using the Oracle VM product line, Oracle Linux, Oracle Engineered
Systems managed by Oracle Enterprise Manager 12c, with total control over Oracle
processor licensing.
http://mokumsolutions.com
Change Description
Updated By
Date
2.0
Document Creation
Roddy Rodstein
09/30/2013
2.1
Content Refresh
Roddy Rodstein
01/14/2014
Oracle Applications
Oracle Fusion Middleware
Oracle Database (DBaaS)
Virtual Machines
4 of 41
http://mokumsolutions.com
x86 64 Servers
Storage & Network
Services
Note: A detailed explanation of each category and solution in the Oracle Cloud reference
design is presented in the architectural overview section.
5 of 41
http://mokumsolutions.com
The Oracle Cloud reference design isolates Oracle VM server pools into the following four
security domains:
Controlled: A controlled security domain is used to restrict access between security
domains. A controlled security domain could contain groups of users with their
network equipment or a demilitarized zone (DMZ).
Uncontrolled: An uncontrolled security domain refers to any network not in control
of an organization, such as the Internet.
Restricted: A restricted security domain can represent an organizations
production, test and development networks. Access is restricted to authorized
personnel, and there is no direct access from the Internet.
Secured: A secured security domain is a network that is only accessible to a small
group of highly trusted users, such as administrators and auditors.
Note: The classication of security domains is very similar to data classications. FIPS
PUB 199 is the Standards for Security Categorization of Federal Information and
Information Systems. FIPS PUB 199 can be used to determine the security category of
6 of 41
http://mokumsolutions.com
Decision
Justication
Oracle Support
Agreements
On-site and
O-site support
7 of 41
http://mokumsolutions.com
Maximum
160
RAM
4 TB
NICs
40
Oracle VM Server CPU, RAM and storage hardware sizing is calculated by determining
the total number of virtual machines CPU, RAM, and storage (I/O and disk) requirements
8 of 41
http://mokumsolutions.com
per Oracle VM server. For example, if a single virtual machine with 16 CPUs, 128 GB
RAM, 1 TB of disk space with 1500 IOPS will run on one Oracle VM server, the Oracle VM
server hardware should have at least 16 CPU cores or threads, 130 GB RAM, 1 TB of disk
space and the ability to support 1500 IOPS with local or remote storage. If two virtual
machines each with 16 CPUs, 128 GB RAM, 1 TB of disk space with 1500 IOPS will run on
one Oracle VM server, the Oracle VM server hardware must have at least 32 CPU cores
or threads, 300G RAM, 2 TB of disk space and the ability to support 3000 IOPS with local
or remote storage.
A single Oracle VM 3.2.x server can support up to 160 CPU cores or threads, 4 TB of
memory with local or remote storage. An Oracle VM server with 4 TB of RAM and 160
CPU cores or threads could allocate the majority of the 4 TB of RAM and more than 160
CPU cores or threads to running virtual machines. Oracle VM server supports CPU
oversubscription. CPU oversubscription means that an Oracle VM server with 160 CPU
cores could overallocate the total number of CPU cores to virtual machines. Oracle VM
server does not support memory oversubscription, which means that an Oracle VM server
with 4 TB of RAM cannot overallocate RAM to virtual machines. By default, each Oracle
VM server reserves 512 MB of RAM for Oracle VM server (dom0). The average memory
overhead for each running virtual machine on an Oracle VM server is approximately 20
MB plus 1% of each virtual machine' memory allocation. The remaining RAM can be
allocated to virtual machines.
A best practice is to avoid oversubscribing CPU-bound workloads such as the Oracle
Database. CPU oversubscription with CPU-bound workloads negatively aects the
performance and availability of an Oracle VM server along with all of the virtual machines
running on the server. CPU oversubscription for non-CPU-bound workloads, such as
Oracle Fusion Middleware products, is highly recommended. It is common to
oversubscribe CPU cores 3-to-1 with non-CPU-bound workloads. For example, one CPU
core could be allocated to 3 virtual CPUs for non-CPU-bound workloads without a
performance penalty.
Note: Virtual machines cannot aggregate CPU and memory resources from more than
one Oracle VM server. That is, a virtual machine consumes resources only from the
Oracle VM server where the virtual machine is running.
Oracle VM has two high-availability features, HA and Live Migration. Oracle VM HA and
Live Migration along with Distributed Resource Scheduling (DRS) must be considered to
calculate the total number of servers required to respond to hardware failures and virtual
machine migrations.
The next Figure shows Oracle VM server pool designed with excess CPU and RAM
capacity to be able to use HA, DRS and Live Migration. Excess CPU and RAM capacity is
a requirement for HA, DRS and Live Migration.
This image shows an
Oracle VM server pool
with excess capacity able
to use HA, Live Migration
and DRS.
9 of 41
http://mokumsolutions.com
10 of 41
http://mokumsolutions.com
running virtual machines from one server pool member to another during planned
maintenance events. The second use case is to use DRS policies to load balance running
virtual machines from heavily utilized Oracle VM servers to less utilized Oracle VM
servers. Since Oracle VM does not support memory oversubscription, an Oracle VM
server pool must have available RAM capacity to be able to migrate virtual machines
between servers.
DRS is an Oracle VM feature which provides policy based real-time utilization monitoring
of Oracle VM servers with the goal to distribute virtual machine loads across a server
pool. DRS migrates virtual machines from heavily utilized Oracle VM servers to less
utilized Oracle VM servers.
The exact number of network interfaces for an Oracle VM server is determined by the
network switch VLAN setup and the number of Oracle VM management and virtual
machine network ports. Oracle VM supports both 802.1Q trunk port VLANs as well as
port based VLANs, with Linux bonding Modes 1 (Active-Backup), 4 (802.3ad) and 6
(Adaptive load balancing). 802.1q trunk ports can have two or more VLANs per port, in
contrast to port based VLANS that are limited to one VLAN per port or port channel.
802.1Q uses fewer network switch ports and fewer Oracle VM server NICs compared to
port based VLANs that require a dedicated switch port and NIC per network. A network
switch VLAN conguration must rst be selected to be able to calculate the exact number
of network switch ports and NICs for your Oracle VM servers.
Oracle VM uses a total of ve discrete networks for the Oracle VM server management
functions; server management, cluster heartbeat, live migration, storage (only for NFS
and iSCSI) and virtual machines. Each Oracle VM server pool should have a discrete
network for each of the ve aforementioned server management networks, as well as a
discrete network for each virtual machine network. For example, an Oracle VM Server on
a 1-gigabit copper network with NFS or iSCSI storage could easily use 12 or more
bonded NICs with access ports just for the server management networks and one virtual
machine network. In contrast to the latter 1-gigabit copper network example, an Oracle
VM Server on a 10-gigabit ber network using 802.1q trunk ports with NFS or iSCSI
storage could easily use up to 4 bonded ports just for the server management and 2
bonded ports for the virtual machine networks.
Tip: In an clustered Oracle VM server pool, the loss of network connectivity for the
Oracle VM cluster heartbeat network will causes a HA event. When a HA event occurs,
the Oracle VM server that loses cluster heartbeat connectivity is fenced from the server
pool and reboots, then all HA-enabled guests are restarted on a live Oracle VM pool
member.
Prior to implementing an Oracle Cloud, its important that an infrastructure assessment
(IA) and gap analysis (GA) be performed. During the IA/GA, the hardware specications
will be matched to the customers business needs.
This table outlines the decision points for the for Oracle VM for x86 server hardware. For
decisions that rely on preexisting factors or specic organizational needs, the appropriate
best practice will be discovered in the infrastructure assessment (IA) and gap analysis
(GA). The best practices should be analyzed carefully and decisions should be made based
on organizational needs, existing architecture, and budget resource availability.
11 of 41
http://mokumsolutions.com
Decision
Point
Decision
Certication
CPU
Justication
RAM
12 of 41
http://mokumsolutions.com
4TB of RAM.
Storage
13 of 41
http://mokumsolutions.com
Rate(bit/s) Rate(byte/s)
Gigabit Ethernet
1 Gbit/s 125 MB/s
10 Gigabit Ethernet 10 Gbit/s 1.25 GB/s
Inniband DDR
16 Gbit/s
2 GB/s
Tip: One thing to consider is NIC
rmware levels between bonded internal
14 of 41
http://mokumsolutions.com
Host Bus
Adapter
Cards
Management, Cluster
Heartbeat, live Migration and
Storage network.
NAME
4GFC
8GFC
10GFC
16GFC
20GFC
800
1600
2550
3200
5100
This table outlines the decision points for an Oracle VM server pool. For decisions that
rely on preexisting factors or specic organizational needs, the appropriate best practice
will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best
practices should be analyzed carefully and decisions should be made based on
organizational needs, existing architecture, and budget resource availability.
15 of 41
http://mokumsolutions.com
Decision
Point
Decision
Justication
Oracle VM
Server Pool
Design
Oracle VM
Manager
16 of 41
http://mokumsolutions.com
Network
Time
Protocol
(NTP)
17 of 41
http://mokumsolutions.com
cluster transactions
between Oracle VM
Servers. Without accurate
time, Oracle VM clusters
can be brought to a
complete standstill.
Master Server
By default each clustered
server pool has one Master
Server with the Virtual IP
feature enabled.
Utility Server
The Utility Server role is
responsible for
I/O-intensive operations
such as virtual machine
creation and removal, as
well for as creating,
Master Server
deleting, modifying,
copying and moving virtual
The Master Server is the principal
machine les. Enabling the
server pool role within a server pool. Utility Server agent role
The Master Server is the server that with the VM Server role on
communicates with Oracle VM
the same Oracle VM server
Manager. The Master
may negatively aect
Server dispatches commands
running virtual machines
received from Oracle VM Manager to during Utility Server
other servers within a server pool.
operations. Server pools
There can be only one Master
18 of 41
http://mokumsolutions.com
19 of 41
http://mokumsolutions.com
Storage
Back-end storage
Each Oracle VM server pool uses one
dedicated OCFS2 12G mount point (OCFS2
or NFS) for the server pool's cluster
congurations and one or more shared
OCFS2 or NFS repositories to host virtual
machine conguration les and images.
An Oracle VM storage
solution consists of three
distinct layers. Each layer
has its own unique
requirements,
congurations,
dependencies and features.
Front-end storage
The virtual machine layer is where the
storage is presented to virtual machines as
either a at le (UUID.img), as RAW disks
(LUN), or as a combination of at les and
RAW disks.
Networks
20 of 41
http://mokumsolutions.com
21 of 41
http://mokumsolutions.com
pool.
Decision
Justication
Oracle VM
Manager and
DMZs
22 of 41
http://mokumsolutions.com
Network Time
Protocol (NTP)
23 of 41
http://mokumsolutions.com
Host rewall
Host
rewall failed
connection
logging
24 of 41
http://mokumsolutions.com
SSH login
banners
Pre-login banner:
Edit the /etc/ssh/sshd_cong and
add the following directive:
Banner /etc/banner.net
Next, create the /etc/banner.net
le and add your login banner
text, i.e.
This system is restricted to
authorized access only. All
activities on this system are
To be able to successfully
prosecute individuals who
improperly use a computer, the
computer must have a warning
banner displayed at all access
points.
SSH login banners presents a
denitive warning or disclaimer to
all users that wish to access your
systems using SSH. SSH login
banners should clarify which
types of activities are illegal as
well as advise legitimate users of
their obligations relating to the
acceptable use of the system.
25 of 41
http://mokumsolutions.com
This table outlines the decision points for Oracle VM Server security controls. For
decisions that rely on preexisting factors or specic organizational needs, the appropriate
best practice will be discovered in the infrastructure assessment (IA) and gap analysis
(GA). The best practices should be analyzed carefully and decisions should be made based
on organizational needs, existing architecture, and budget resource availability.
Decision
Justication
Oracle VM
Server and
DMZs
26 of 41
http://mokumsolutions.com
Patch
Management
27 of 41
http://mokumsolutions.com
Host rewall
Host
rewall failed
connection
logging
28 of 41
http://mokumsolutions.com
To be able to successfully
prosecute individuals who
improperly use a computer, the
computer must have a warning
banner displayed at all access
points.
SSH login banners presents a
denitive warning or disclaimer to
all users that wish to access your
systems using SSH. SSH login
banners should clarify which
types of activities are illegal as
well as advise legitimate users of
their obligations relating to the
acceptable use of the system.
29 of 41
http://mokumsolutions.com
30 of 41
http://mokumsolutions.com
Decision
Justication
Virtual Machine
Operating
Systems
Virtual Machine
Operating
System
Versioning
Virtual Machine
Operating
System
Deployments
31 of 41
http://mokumsolutions.com
Patch
Management
32 of 41
http://mokumsolutions.com
Oracle VM uses the concept of a server pool to group together and manage one or more
clustered Oracle VM servers. Once an Oracle VM server pool is created, the physical and
virtual resources are managed within the boundary of the server pool. Physical resources
include server hardware, networks, storage, infrastructure services (DNS, NTP, LDAP,
HTTP, etc..), operating system installation media and administrative accounts. The virtual
resources include virtual disks, virtual network interfaces, and virtual machine
conguration les. For example, an Oracle VM environment with multiple server pools
located in one or more sites could be managed from a single Oracle VM Manager instance
with each server pool's resources isolated to their respected server pool. An Oracle VM
server pool's resources from one site can be replicated and restored to another site for
disaster recovery.
Restoration of the primary site's services at a disaster recovery site requires a replica of
the primary site's physical and virtual resources at the disaster recovery site. A disaster
recovery site hosts a replica of the primary site's Oracle VM physical and virtual
resources, i.e. server hardware, networks, storage, infrastructure services, virtual disks,
and virtual machine conguration les. The failover process involves restoring the
primary sites virtual machines at the disaster recovery site, then systematically starting
the virtual machines and services.
Note: Oracle VM Servers are not backed up and restored at the DR site. The time
required to backup and restore an Oracle VM Server is signicantly greater then a PXE
boot kickstart installation.
A disaster recovery site can be a warm failover site waiting idle to respond to a disastrous
occurrence, or part of a multi-site high availability design. A multi-site design uses excess
capacity with application high availability to mirror services across sites to handle the
lose of one or more sites.
The next Figure shows a warm Oracle VM failover site waiting idle to respond to a
disastrous occurrence.
33 of 41
http://mokumsolutions.com
The next Figure shows a warm Oracle VM failover site responding to a disastrous
occurrence and running the primary sites services.
34 of 41
http://mokumsolutions.com
The next Figure shows a multi-site Oracle VM design with application high availability
solutions to mirror services across sites as well as excess capacity to handle the lose of
one or more sites.
35 of 41
http://mokumsolutions.com
Virtual machines that are restored at a disaster recovery site expect the same networks,
storage, and infrastructure services as in the primary site. In the event that the disaster
recovery site has dierent networks, storage, and infrastructure services, the properties
of each virtual machines would need to be edited to use the new networks, storage and
infrastructure services before services can be restored.
The virtual machine operating systems are typically installed in virtual disks that are
actually at les hosted on shared OCFS2 or NFS repositories. RAW disks such as ASM
Disks, Log and Archive Files, etc.. are presented to the virtual machines from the Oracle
VM Servers as local devices. Each virtual machine's virtual network interface card(s)
(vNIC) are connected to one or more discrete networks using Xen bridges that are
managed and presented to the virtual machines by the Oracle VM pool members. Virtual
disks and virtual network interface card(s) allocations are managed using Oracle VM
Manager and/or Oracle Enterprise Manager with the congurations saved in each virtual
machines vm.cfg le.
The virtual machine vm.cfg les, virtual disk images and RAW disks (ASM disks) can be
replicated between sites using storage array replication and/or mirroring solutions. Rsync
is an option if an array does not have replication and/or mirroring functionality.
As soon as the replicated storage repositories are available, the failover process for a
warm recovery site starts with the installation of Oracle VM Manager with the
runInstall.sh --uuid option using the primary sites Oracle VM Manager UUID. An Oracle
VM Manager --uuid installation allows Oracle VM Manager to use the primary site'
replicated repositories with the virtual machines.
36 of 41
http://mokumsolutions.com
Tip: The Oracle VM Manager UUID is listed in the .cong le on the Oracle VM
Manager host in the /u01/app/oracle/ovm-manager-3/ directory as well as in each server
pool' .ovsrepo le in the pool le system.
The next example shows the content of the .cong le with the UUID in bold.
# cat /u01/app/oracle/ovm-manager-3/.cong
DBHOST=localhost
SID=orcl
LSNR=1521
APEX=None
OVSSCHEMA=ovs
WLSADMIN=weblogic
OVSADMIN=admin
COREPORT=54321
UUID=0004fb00000100009edfaa0f93184f44
BUILDID=3.0.3.126
The next example shows the content of the .ovsrepo le with the UUID in bold.
# cat .ovsrepo
OVS_REPO_UUID=0004fb0000030000554308a6997a6b2f
OVS_REPO_MGR_UUID=0004fb00000100009edfaa0f93184f44
OVS_REPO_VERSION=3.0
This table outlines the decision points for an Oracle VM disaster recovery solution. For
decisions that rely on preexisting factors or specic organizational needs, the appropriate
best practice will be discovered in the infrastructure assessment (IA) and gap analysis
(GA). The best practices should be analyzed carefully and decisions should be made based
on organizational needs, existing architecture, and budget resource availability.
Decision
Point
Decision
Justication
Disaster
Recovery
Design
Implementing a Disaster
Recovery is a strategic decision.
Disaster recovery requirements
are calculated using SLA,
Recovery Time Objectives (RTOs)
and Recovery Point Objectives
(RPOs) objectives. SLA, RPO and
RTO objectives and budget
inuence the disaster recovery
architecture and design.
Oracle VM
Manager
37 of 41
http://mokumsolutions.com
installation of Oracle VM
Manager with the runInstall.sh
--uuid option using the primary
sites Oracle VM Manager UUID.
An Oracle VM Manager --uuid
installation allows Oracle VM
Manager to use the primary site'
replicated repositories and virtual
machines.
The Oracle VM Manager UUID is
listed in the .cong le on the
Oracle VM Manager host in the
/u01/app/oracle/ovm-manager-3/
directory as well as in each server
pool' .ovsrepo le in the pool le
system.
Oracle VM
Server Builds
Oracle VM
Server Backups
Storage
38 of 41
http://mokumsolutions.com
Infrastructure
Services
Application
Support
Decision
Justication
39 of 41
http://mokumsolutions.com
Application
Requirements
and
Dependencies
Application
sunsetting
Patch
Management
40 of 41
http://mokumsolutions.com
Policy.
All production systems will
undergo security audits in
accordance with <Company
Name>s Change Management
Policy to validate conguration
and patch compliance.
41 of 41