Student Guide

Information Storage and
Management Version 3
Student Guide
Education Services
August 2015
Welcome to Information Storage and Management Version 3.

Copyright 2015 EMC Corporation. All Rights Reserved. Published in the USA. EMC believes the information in this publication is accurate as of its publication date.
The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH
RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. The trademarks, logos, and service marks
(collectively "Trademarks") appearing in this publication are the property of EMC Corporation and other parties. Nothing contained in this publication should be
construed as granting any license or right to use any Trademark without the prior written permission of the party that owns the Trademark.
EMC, EMC AccessAnywhere Access Logix, AdvantEdge, AlphaStor, AppSync ApplicationXtender, ArchiveXtender, Atmos, Authentica, Authentic Problems,
Automated Resource Manager, AutoStart, AutoSwap, AVALONidm, Avamar, Bus-Tech, Captiva, Catalog Solution, C-Clip, Celerra, Celerra Replicator, Centera,
CenterStage, CentraStar, EMC CertTracker. CIO Connect, ClaimPack, ClaimsEditor, Claralert ,cLARiiON, ClientPak, CloudArray, Codebook Correlation Technology,
Common Information Model, Compuset, Compute Anywhere, Configuration Intelligence, Configuresoft, Connectrix, Constellation Computing, EMC ControlCenter,
CopyCross, CopyPoint, CX, DataBridge , Data Protection Suite. Data Protection Advisor, DBClassify, DD Boost, Dantz, DatabaseXtender, Data Domain, Direct Matrix
Architecture, DiskXtender, DiskXtender 2000, DLS ECO, Document Sciences, Documentum, DR Anywhere, ECS, elnput, E-Lab, Elastic Cloud Storage,
EmailXaminer, EmailXtender , EMC Centera, EMC ControlCenter, EMC LifeLine, EMCTV, Enginuity, EPFM. eRoom, Event Explorer, FAST, FarPoint, FirstPass, FLARE,
FormWare, Geosynchrony, Global File Virtualization, Graphic Visualization, Greenplum, HighRoad, HomeBase, Illuminator , InfoArchive, InfoMover, Infoscape, Infra,
InputAccel, InputAccel Express, Invista, Ionix, ISIS,Kazeon, EMC LifeLine, Mainframe Appliance for Storage, Mainframe Data Library, Max Retriever, MCx, MediaStor
, Metro, MetroPoint, MirrorView, Multi-Band Deduplication,Navisphere, Netstorage, NetWorker, nLayers, EMC OnCourse, OnAlert, OpenScale, Petrocloud, PixTools,
Powerlink, PowerPath, PowerSnap, ProSphere, ProtectEverywhere, ProtectPoint, EMC Proven, EMC Proven Professional, QuickScan, RAPIDPath, EMC RecoverPoint,
Rainfinity, RepliCare, RepliStor, ResourcePak, Retrospect, RSA, the RSA logo, SafeLine, SAN Advisor, SAN Copy, SAN Manager, ScaleIO Smarts, EMC Snap,
SnapImage, SnapSure, SnapView, SourceOne, SRDF, EMC Storage Administrator, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix, Symmetrix
DMX, Symmetrix VMAX, TimeFinder, TwinStrata, UltraFlex, UltraPoint, UltraScale, Unisphere, Universal Data Consistency, Vblock, Velocity, Viewlets, ViPR, Virtual
Matrix, Virtual Matrix Architecture, Virtual Provisioning, Virtualize Everything, Compromise Nothing, Virtuent, VMAX, VMAXe, VNX, VNXe, Voyence, VPLEX, VSAMAssist, VSAM I/O PLUS, VSET, VSPEX, Watch4net, WebXtender, xPression, xPresso, Xtrem, XtremCache, XtremSF, XtremSW, XtremIO, YottaYotta, Zero-Friction
Enterprise Storage.
Revision Date: August 2015

Revision Number: MR-1VP-ISMv3-1504
Copyright 2015 EMC Corporation. All rights reserved.
Module: Course Introduction
Information Storage and Management (ISM) is a unique course that provides a comprehensive
understanding of the various storage infrastructure components in data center environments. It
enables participants to make informed decisions on storage-related technologies in an
increasingly complex IT environment, which is fast changing with the adoption of software-defined
infrastructure management and third platform technologies (cloud, Big Data, social, and mobile
technologies). It provides a strong understanding of storage technologies and prepares
participants for advanced concepts, technologies, and processes. Participants will learn the
architectures, features, and benefits of intelligent storage systems including block-based, filebased, object-based, and unified storage; software-defined storage; storage networking
technologies such as FC SAN, IP SAN, and FCoE SAN; business continuity solutions such as
backup and replication; the highly-critical areas of information security; and storage infrastructure
management. This course takes an open-approach to describe all the concepts and technologies,
which are further illustrated and reinforced with EMC-related product examples.
This module focuses on digital data, the types of digital data, and information. This module also
focuses on data center and its key characteristics. Further this module focuses on the key data
center management processes. Finally, this module focuses on the evolution of computing
platforms.
Module 1: Introduction to Information Storage
We live in a digital universe a world that is created and defined by software. A massive amount
of digital data is continuously generated, collected, stored, and analyzed through software in the
digital universe. According to the 2014 Digital Universe Study conducted by International Data
Corporation (IDC), it is estimated that the digital universe produces approximately 4.4 trillion
gigabytes (GB) of data annually, which is doubling every two years. By these estimates, it is
projected that by the year 2020, the digital universe will expand to 44 trillion GB of data. The
data in the digital universe comes from diverse sources, including individuals living and working
online, organizations employing information technology (IT) to run their businesses, and from a
variety of smart electronic devices connected to the Internet.
In organizations, the volume and importance of information for business operations continue to
grow at astounding rates. Individuals constantly generate and consume information through
numerous activities, such as web searches, e-mails, uploading and downloading content and
sharing media files. The rapid proliferation of online social networking and Internet-enabled
smartphones and tablets has also contributed significantly to the growth of the digital universe.
The advent of the Internet of Things (IoT) is also gradually adding to the growth of the digital
universe. The IoT is a technology trend wherein smart devices with embedded electronics,
software, and sensors exchange data with other devices over the Internet. Examples of such
devices are wearable gadgets smartwatches and fitness activity trackers; electronic sensors
temperature sensors and heart monitoring implants; and household appliances televisions,
thermostats, and lighting. The IoT has vast applications and is driving the development of several
innovative technology solutions. Some application areas include weather monitoring remote
monitoring and analysis of temperature and atmospheric conditions; healthcare health
monitoring devices can enable doctors to remotely monitor patients and be notified in case of
emergencies; and infrastructure management technicians can remotely monitor equipment and
proactively schedule repair activities for maintenance crews.
Organizations have become increasingly information-dependent in the twenty-first century, and

information must be available whenever and wherever it is required. It is critical for users and
applications to have continuous, fast, reliable, and secure access to information for business
operations to run as required. Some examples of such organizations and processes include
banking and financial institutions, government departments, online retailers, airline reservations,
billing and transaction processing, social networks, stock trading, scientific research, and
healthcare.
It is essential for organizations to store, protect, process, and manage information in an efficient
and cost-effective manner. Legal, regulatory, and contractual obligations regarding the
availability, retention, and protection of data further add to the challenges of storing and
managing information.
Organizations also face newer challenges in the form of requirement to extract value from the
information generated in the digital universe. Information can be leveraged to identify
opportunities to transform and enhance businesses and gain a competitive edge. For example, an
online retailer may need to identify the preferred product types and brands of customers by
analyzing their search, browsing, and purchase patterns. The retailer can then maintain a
sufficient inventory of popular products, and also advertise relevant products to the existing and
potential customers. Furthermore, the IoT is expected to lead to new consumer and business
behavior in the coming years creating new business opportunities.
To meet all these requirements and more, organizations are increasingly undertaking digital
transformation initiatives to implement intelligent storage solutions. These solutions not only
enable efficient and optimized storage and management of information, but also enable extraction
of value from information to derive new business opportunities, gain a competitive advantage,
and create new sources of revenue.
A generic definition of data is that it is a collection of facts, typically collected for the purpose of
analysis or reference. Data can exist in a variety of forms such as facts stored in a person's mind,
photographs and drawings, alphanumeric text and images in a book, a bank ledger, and tabled
results of a scientific survey. Originally, data is the plural form of datum. However, data is now
generally treated as a singular or mass noun representing a collection of facts and figures. This is
especially true when referring to digital data.
In computing, digital data is a collection of facts that is transmitted and stored in electronic form,
and processed through software. Digital data is generated by various devices, such as desktops,
laptops, tablets, mobile phones, and electronic sensors. It is stored as strings of binary values (0s
and 1s) on a storage medium that is either internal or external to the devices generating or
accessing the data. The storage devices may be of different types, such as magnetic, optical, or
solid state storage devices. Examples of digital data are electronic documents, text files, e-mails,
e-books, digital images, digital audio, and digital video.
Based on how it is stored and managed, digital data can be broadly classified as either structured
data or unstructured data. Structured data is organized in fixed fields within a record or file. For
data to be structured, a data model is required. A data model specifies the format for organizing
data, and also specifies how different data elements are related to each other. For example, in a
relational database, data is organized in rows and columns within named tables. Semi-structured
data does not have a formal data model but has an apparent, self-describing pattern and
structure that enable its analysis. Examples of semi-structured data include spreadsheets that
have a row and column structure, and XML files that are defined by an XML schema. Quasistructured data consists of textual data with erratic data formats, and can be formatted with
effort, software tools, and time. An example of quasi-structured data is a clickstream or
clickpath that includes data about which webpages a user visited and in what order which is
the result of the successive mouse clicks the user made. A clickstream shows when a user
entered a website, the pages viewed, the time spent on each page, and when the user exited.
Unstructured data does not have a data model and is not organized in any particular format.
Some examples of unstructured data include text documents, PDF files, e-mails, presentations,
images, and videos.
As indicated by the figure on the slide, the majority, which is more than 90 percent, of the data
generated in the digital universe today is non-structured data (semi-, quasi-, and unstructured).
Although the figure shows four different and separate types of data, in reality a mixture of these
is typically generated. For instance, in a call center for customer support of a software product, a
classic relational database management system (RDBMS) may store call logs with structured data
such as date/time stamps, machine types, and problem type entered by the support desk person.
In addition, there may be unstructured or semi-structured data, such as an e-mail ticket of the
problem, call log information, or the actual call recording.
The terms data and information are closely related and it is common for the two to be used
interchangeably. However, it is important to understand the difference between the two. Data, by
itself, is simply a collection of facts that needs to be processed for it to be useful. For example a
set of annual sales figures of an organization is data. When data is processed and presented in a
specific context it can be interpreted in a useful manner. This processed and organized data is
called information. For example, when the annual sales data is processed into a sales report, it
provides useful information, such as the average sales for a product (indicating product demand
and popularity), and a comparison of the actual sales to the projected sales. Information thus
creates knowledge and enables decision-making.
As discussed previously, processing and analyzing data is vital to any organization. It enables
organizations to derive value from data, and create intelligence to enable decision-making and
organizational effectiveness. It is easier to process structured data due to its organized form. On
the other hand, processing non-structured data and extracting information from it using
traditional applications is difficult, time-consuming, and requires considerable resources. New
architectures, technologies, and techniques (described in Module 2, Third Platform Technologies)
have emerged that enable storing, managing, analyzing, and deriving value from unstructured
data coming from numerous sources.
In a computing environment, storage devices (or simply storage) are devices consisting of nonvolatile recording media on which information can be persistently stored. Storage may be internal
(for example, internal hard drive), removable (for example, memory cards), or external (for
example, magnetic tape drive) to a compute system. Based on the nature of the storage media
used, storage devices can be broadly classified as given below:
Magnetic storage devices: For example, hard disk drive and magnetic tape drive.
Optical storage devices: For example, Blu-ray, DVD, and CD.
Flash-based storage devices: For example, solid state drive (SSD), memory card, and USB
thumb drive (or pen drive).
Storage is a core component in an organizations IT infrastructure. Various factors such as the

media, architecture, capacity, addressing, reliability, and performance influence the choice and
use of storage devices in an enterprise environment. For example, disk drives and SSDs are used
for storing business-critical information that needs to be continuously accessible to applications;
whereas, magnetic tapes and optical storage are typically used for backing up and archiving data.
The different types of storage devices are covered in Module 3, Data Center Environment.
In enterprise environments, information is typically stored on storage systems (or storage
arrays). A storage system is a hardware component that contains a group of
homogeneous/heterogeneous storage devices assembled within a cabinet. These enterprise-class
storage systems are designed for high capacity, scalability, performance, reliability, and security
to meet business requirements. The compute systems that run business applications are provided
storage capacity from storage systems. Storage systems are covered in Module 4, Intelligent
Storage Systems (ISS). Organizations typically house their IT infrastructure, including compute
systems, storage systems, and network equipment within a data center.
A data center is a dedicated facility where an organization houses, operates, and maintains backend IT infrastructure including compute systems, storage systems, and network equipment along
with other supporting infrastructure. A data center centralizes an organizations IT equipment and
data-processing operations, and is vital for carrying out business operations.
A data center typically comprises the following:
Facility: It is the building and floor space where the data center is constructed. It typically has
a raised floor with ducts underneath holding power and network cables.
IT equipment: It includes equipment such as compute systems, storage systems, network

equipment and cables, and cabinets for housing the IT equipment.
Support infrastructure: It includes all the equipment necessary to securely sustain the
functioning of the data center. Some key support equipment are power equipment including
uninterruptible power sources, and power generators; environmental control equipment
including fire and water detection systems, heating, ventilation, and air conditioning (HVAC)
systems; and security systems including biometrics, keycard, and video surveillance systems.
An organization may build a data center to provide open access to applications over the Internet,
or for privately executing business applications within its operational environment. A data center
may be constructed in-house and located in an organizations own facility, or it may be
outsourced, with equipment being located at a third-party site. Large organizations often maintain
multiple data centers to distribute data-processing workloads and for disaster recovery.
Organizations are increasingly focusing on energy-efficient technologies and efficient management
practices to reduce the energy consumption of data centers and lessen the impact on the
environment. Such data centers are called as green data centers.
Data centers are designed and built to fulfill the key characteristics shown in the figure on the
slide. Although the characteristics are applicable to almost all data center components, the
discussion here primarily focuses on storage systems.
Availability: Availability of information as and when required should be ensured. Unavailability

of information can severely affect business operations, lead to substantial financial losses, and
damage the reputation of an organization.
Security: Policies and procedures should be established, and control measures should be
implemented to prevent unauthorized access to and alteration of information.
Capacity: Data center operations require adequate resources to efficiently store and process
large and increasing amounts of data. When capacity requirements increase, additional
capacity should be provided either without interrupting the availability or with minimal
disruption. Capacity may be managed by adding new resources or by reallocating existing
resources.
Scalability: Organizations may need to deploy additional resources such as compute systems,
new applications, and databases to meet the growing requirements. Data center resources
should scale to meet the changing requirements, without interrupting business operations.
Performance: Data center components should provide optimal performance based on the
required service levels.
Data integrity: Data integrity refers to mechanisms, such as error correction codes or parity
bits, which ensure that data is stored and retrieved exactly as it was received.
Manageability: A data center should provide easy, flexible, and integrated management of all
its components. Efficient manageability can be achieved through automation for reducing
manual intervention in common, repeatable tasks.
The activities carried out to ensure the efficient functioning of a data center can be broadly
categorized under the following key management processes:
Monitoring: It is a continuous process of gathering information on various resources in the

data center. The process involves monitoring parameters such as configuration, availability,
capacity, performance, and security of resources.
Reporting: It is a process of collating and presenting the monitored parameters such as

resource performance, capacity, and utilization of resources. Reporting enables data center
managers to analyze and improve the utilization of data center resources and identify
problems. It also helps in establishing business justifications and chargeback of costs
associated with data center operations.
Provisioning: It is the process of configuring and allocating the resources that are required to
carry out business operations. For example, compute systems are provisioned to run
applications and storage capacity is provisioned to a compute system. Provisioning primarily
includes resource management activities to meet capacity, availability, performance, and
security requirements.
Planning: It is a process of estimating the amount of IT resources required to support

business operations and meet the changing resource requirements. Planning leverages the
data collected during monitoring and enables improving the overall utilization and performance
of resources. It also enables estimation of future resource requirements. Data center managers
also determine the impact of incidents and devise contingency plans to resolve them.
Maintenance: It is a set of standard repeatable activities for operating the data center. It
involves ensuring the proper functioning of resources and resolving incidents such as
malfunctions, outages, and equipment loss. It also involves handling identified problems or
issues within the data center and incorporating changes to prevent future problem occurrence.
10
In general, the term platform refers to hardware and software that are associated with a
particular computing architecture deployed in a data center. Computing platforms evolve and
grow with advances and changes in technology. The figure on the slide displays the three
computing platforms of IT growth as specified by IDC. The first platform (or Platform 1) dates
back to the dawn of computing and was primarily based on mainframes and terminals. The
second platform (or Platform 2) emerged with the birth of the personal computer (PC) in the
1980s and was defined by the client-server model, Ethernet, RDBMSs, and web applications. The
third platform (or Platform 3) of today comprises cloud, Big Data, mobile, and social technologies.
Each computing platform is defined not so much by the comprising technologies but by the scale
of users and the scope of applications the technologies enable. The first platform supported
millions of users, with applications and solutions in the low thousands. The second platform
supported hundreds of millions of users and tens of thousands of applications. The third platform
is already supporting a user base of billions and has millions of applications and solutions. This is
evident from the fact that over 2.4 billion people (~36 percent of the world's population) are
currently connected to the Internet (more than half of them through mobile devices), and that
there are over one million applications available for iOS and Android devices alone.
11
Mainframes are compute systems with very large processing power, memory, and storage
capacity and are primarily used for centrally hosting mission-critical applications and databases in
an organizations data center. Multiple users simultaneously connect to mainframes through lesspowerful devices, such as workstations or terminals. All processing is performed on the
mainframe, while the terminals only provide an interface to use the applications and view results.
Although mainframes offer high reliability and security, there are several cost concerns associated
with them. Mainframes have high acquisition costs, and considerable floor space and energy
requirements. Deploying mainframes in a data center may involve substantial capital expense
(CAPEX) and operating expense (OPEX). Historically, large organizations such as banks, insurance
agencies, and government departments have used mainframes to run their business operations.
12
The client-server model uses a distributed application architecture, in which a compute system
called server runs a program that provides services over a network to other programs running
on various end-point devices called clients. Server programs receive requests for resources from
client programs and in response to the requests, the clients receive access to resources, such as
e-mail applications, business applications, web applications, databases, files, and printers. Client
devices can be desktops, laptops, and mobile devices. Clients typically communicate with servers
over a LAN or WAN, with users making use of either a client application or a web interface on a
browser.
In the client-server model, both the clients and the servers may have distinct processing tasks
that they routinely perform. For example, a client may run the business application while the
server may run the database management system (DBMS) to manage storage and retrieval of
information to and from a database. This is called a two-tier architecture. Alternatively, a client
may use an application or web interface to accept information while the server runs another
application that processes the information and sends the data to a second server that runs the
DBMS. This is called the three-tier architecture. This distributed application architecture can be
extended to any number of tiers (n-tier architecture). Because both client and server systems are
intelligent devices, the client-server model is completely different from the mainframe model.
The figure on the slide shows an example of the client-server model. In the example, clients
interact with the web server using a web browser. The web server processes client requests
through HTTP and delivers HTML pages. The application server hosts a business application and
the database server hosts a DBMS. The clients interact with the application server through client
software. The application server communicates with the database server to retrieve information
and provide results to the clients. In some implementations, applications and databases may even
be hosted on the same server.
(Contd)
13
Some challenges with the client-server model are associated with creation of IT silos,
maintenance overhead, and scalability issues. In organizations, it is common for business
units/departments to have their own servers running business applications. This leads to the
creation of application and information silos (individual, disparate systems). Silos make it difficult
to efficiently utilize or share IT resources, and are challenging to manage and integrate. Though
the cost of server hardware is considerably less than mainframes, there is still a significant OPEX
involved in maintenance of multiple servers and clients, and the software running on them.
Furthermore, in this model, it is challenging to meet todays rapid growth in users, information,
and applications workloads. Adding more servers does not necessarily lead to better workload
management. It is also necessary to optimally distribute processing and application logic across
servers and application instances.
Note: In general, a compute system is a device with an operating system (OS) that runs
applications. Physical servers, hosts, desktops, laptops, and mobile devices are examples of
compute systems. In this course, the term compute system or compute is used to refer to
physical servers and hosts on which business applications of an organization are deployed.
14
The term third platform was coined by IDC, and Gartner refers to the same as a nexus of
forces. The third platform is built on a foundation of cloud, Big Data, mobile, and social
technologies. These are the four major disruptive technologies that are significantly
transforming businesses, economies, and lives globally.
At its core, the third platform has the cloud that enables a consumer to provision IT resources as
a service from a cloud provider. Big Data enables analytics that create deeper insights from data
for improved decision-making. Mobile devices enable pervasive access to applications and
information. Social technologies connect individuals, and enable collaboration and information
exchange.
Over the past three decades, it was essential for organizations to intelligently leverage the second
platform for their businesses. According to IDC, over the next three decades, the third platform
will represent the basis for solution development and business innovation. The third platform is
being used for the digital transformation, evolution, and expansion of all industries and for
developing major new sources of competitive advantage. Business strategists, IT leaders, and
solution developers are already building disruptive new business models and consumer services
around third platform technologies.
Third platform technologies are an enhancement of second platform technologies rather than a
substitution. A key aspect of third platform is that it is a convergence of cloud, Big Data, mobile,
and social technologies and not just each technology taken in isolation. The real key is combining
two or more of the technologies to create high-value industry solutions known as mashups. For
example, some of the top drivers of cloud include social and mobile solutions. This means that
organizations already see the greatest value in solutions that are mashups across all four
technologies. The combinations of third platform technologies are already transforming
organizations such as retail, financial services, government departments, telecommunications,
and healthcare.
(Contd)
15
According to IDC, it is estimated that currently over 80 percent of the infrastructure and
applications in most data centers belong to the second platform. Second platform technologies
also currently account for 74 percent of worldwide IT spending. This means that for organizations
that have a significant investment in second platform technologies, an immediate and complete
shift to the third platform may not be cost-effective and practical. This has led to an intermediate
computing platform called Platform 2.5, between the second and third platforms. Platform 2.5
includes the solutions and technologies that enable organizations to bridge the gap between the
second and third platforms. Platform 2.5 technologies enable organizations to use a combination
of second and third platform technologies. Organizations would be able to deliver second platform
applications and build third platform outcomes without duplicating and moving data. For example,
platform 2.5 technologies would allow an organization to run second platform applications using
traditional data structures and protocols, while enabling the same data to be leveraged for
analytics using Big Data technologies.
IDC predicts that future global IT spending will primarily focus on segments such as wireless data,
smartphones and tablets, cloud services, Big Data analytics, and IoT. This spending is estimated
to be in the hundreds of billions of dollars in each of the segments. This indicates the growing
industry trend towards the large-scale adoption of third platform technologies. It is estimated that
by 2020 third platform technologies would account for over 40 percent of IT spending. Module 2
covers third platform technologies.
16
This module covered digital data, the types of digital data, and information. This module also
covered data center and its key characteristics. Further, this module covered the key data center
management processes. Finally, this module covered the evolution of computing platforms.
17
18
This module focuses on the four technologies that make up the third platform, namely cloud, Big
Data, social, and mobile technologies. This module also focuses on cloud computing and its
essential characteristics. Then, this module focuses on cloud service models and cloud
deployment models. Additionally, this module focuses on Big Data analytics. Further, this module
focuses on social networking and mobile computing. Lastly, this module focuses on the key
characteristics of third platform infrastructure and the key imperatives for transforming to the
third platform.
Module 2: Third Platform Technologies
This lesson covers the definition of cloud computing and the essential characteristics of cloud
computing. This lesson also covers cloud service models and cloud deployment models.
The National Institute of Standards and Technology (NIST)a part of the U.S. Department of
Commercein its Special Publication 800-145 defines cloud computing as a model for enabling
convenient, on-demand network access to a shared pool of configurable computing resources
(e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and
released with minimal management effort or service provider interaction.
The term cloud originates from the cloud-like bubble that is commonly used in technical
architecture diagrams to represent a system, such as the Internet, a network, or a compute
cluster. In cloud computing, a cloud is a collection of IT resources, including hardware and
software resources that is deployed either in a single data center, or across multiple
geographically-dispersed data centers that are connected over a network. A cloud infrastructure is
built, operated, and managed by a cloud service provider. The cloud computing model enables
consumers to hire IT resources as a service from a provider. A cloud service is a combination of
hardware and software resources that are offered for consumption by a provider. The cloud
infrastructure contains IT resource pools, from which resources are provisioned to consumers as
services over a network, such as the Internet or an intranet. Resources return to the pool when
released by consumers.
The cloud model is similar to utility services such as electricity, water, and telephone. When
consumers use these utilities, they are typically unaware of how the utilities are generated or
distributed. The consumers periodically pay for the utilities based on usage. Similarly, in cloud
computing, the cloud is an abstraction of an IT infrastructure. Consumers simply hire IT resources
as services from the cloud without the risks and costs associated with owning the resources.
Cloud services are accessed from different types of client devices over wired and wireless network
connections. Consumers pay only for the services that they use, either based on a subscription or
based on resource consumption.
When organizations use cloud services, their IT infrastructure management tasks are reduced to
managing only those resources that are required to access the cloud services. The cloud
infrastructure is managed by the provider, and tasks such as software updates and renewals are
also handled by the provider. The figure on the slide illustrates a generic cloud computing
environment.
The figure on the slide illustrates a generic cloud computing environment. The cloud provides
various types of hardware and software services that are accessed by consumers from different
types of client devices over wired and wireless network connections. The figure includes some
virtual components for relevance and accuracy. Virtualization will be introduced in Module 3, Data
Center Environment and covered in detail in relevant sections of the later modules.
In SP 800-145, NIST specifies that a cloud infrastructure should have the five essential
characteristics described below:
On-demand self-service: A consumer can unilaterally provision computing capabilities, such

as server time or networked storage, as needed automatically without requiring human
interaction with each service provider. NIST
Broad network access: Capabilities are available over the network and accessed through
standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g.,
mobile phones, tablets, laptops, and workstations). NIST
Resource pooling: The providers computing resources are pooled to serve multiple
consumers using a multi-tenant model, with different physical and virtual resources
dynamically assigned and reassigned according to consumer demand. There is a sense of
location independence in that the customer generally has no control or knowledge over the
exact location of the provided resources but may be able to specify location at a higher level of
abstraction (e.g., country, state, or datacenter). Examples of resources include storage,
processing, memory, and network bandwidth. NIST
Rapid elasticity: Capabilities can be rapidly and elastically provisioned, in some cases
automatically, to scale rapidly outward and inward commensurate with demand. To the
consumer, the capabilities available for provisioning often appear to be unlimited and can be
appropriated in any quantity at any time. NIST
Measured service: Cloud systems automatically control and optimize resource use by
leveraging a metering capability at some level of abstraction appropriate to the type of service
(e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be
monitored, controlled, and reported, providing transparency for both the provider and
consumer of the utilized service. NIST
A cloud service model specifies the services and the capabilities that are provided to consumers.
In SP 800-145, NIST classifies cloud service offerings into the three primary models listed below:
Infrastructure as a Service (IaaS)
Platform as a Service (PaaS)
Software as a Service (SaaS)
Cloud administrators or architects assess and identify potential cloud service offerings. The
assessment includes evaluating the services to be created and upgraded, the necessary feature
set for each service, and the service level objectives (SLOs) of each service aligned to consumer
needs and market conditions. SLOs are specific measurable characteristics such as availability,
throughput, frequency, and response time. They provide a measurement of performance of the
service provider. SLOs are key elements of a service level agreement (SLA), which is a legal
document that describes items such as what service level will be provided, how it will be
supported, service location, and the responsibilities of the consumer and the provider.
Note: Many alternate cloud service models based on IaaS, PaaS, and SaaS are defined in various
publications and by different industry groups. These service models are specific to the cloud
services and capabilities that are provided. Examples of such service models are Backup as a
Service (BaaS), Desktop as a Service (DaaS), Test Environment as a service (TEaaS), and
Disaster Recovery as a Service (DRaaS). However, these models eventually belong to one of the
three primary cloud service models.
Infrastructure as a Service: The capability provided to the consumer is to provision

processing, storage, networks, and other fundamental computing resources where the
consumer is able to deploy and run arbitrary software, which can include operating
systems and applications. The consumer does not manage or control the underlying
cloud infrastructure but has control over operating systems, storage, and deployed
applications; and possibly limited control of select networking components (for example,
host firewalls). NIST
IaaS pricing may be subscription-based or based on resource usage. The provider pools the
underlying IT resources and they are typically shared by multiple consumers through a multitenant model. IaaS can even be implemented internally by an organization, with internal IT
managing the resources and services.
Platform as a Service: The capability provided to the consumer is to deploy onto the
cloud infrastructure consumer-created or acquired applications created using
programming languages, libraries, services, and tools supported by the provider. The
consumer does not manage or control the underlying cloud infrastructure including
network, servers, operating systems, or storage, but has control over the deployed
applications and possibly configuration settings for the application-hosting environment.
NIST
In the PaaS model, a cloud service includes compute, storage, and network resources along with
platform software. Platform software includes software such as OS, database, programming
frameworks, middleware, and tools to develop, test, deploy, and manage applications.
Most PaaS offerings support multiple operating systems and programming frameworks for
application development and deployment. PaaS usage fees are typically calculated based on
factors, such as the number of consumers, the types of consumers (developer, tester, and so on),
the time for which the platform is in use, and the compute, storage, or network resources
consumed by the platform.
Software as a Service: The capability provided to the consumer is to use the providers
applications running on a cloud infrastructure. The applications are accessible from various client
devices through either a thin client interface, such as a web browser (for example, web-based
email), or a program interface. The consumer does not manage or control the underlying cloud
infrastructure including network, servers, operating systems, storage, or even individual
application capabilities, with the possible exception of limited user-specific application
configuration settings. NIST
In the SaaS model, a provider offers a cloud-hosted application to multiple consumers as a
service. The consumers do not own or manage any aspect of the cloud infrastructure. In SaaS, a
given version of an application, with a specific configuration (hardware and software) typically
provides service to multiple consumers by partitioning their individual sessions and data. SaaS
applications execute in the cloud and usually do not need installation on end-point devices. This
enables a consumer to access the application on demand from any location and use it through a
web browser on a variety of end-point devices. Some SaaS applications may require a client
interface to be locally installed on an end-point device. Customer Relationship Management
(CRM), email, Enterprise Resource Planning (ERP), and office suites are examples of applications
delivered through SaaS.
A cloud deployment model provides a basis for how cloud infrastructure is built,
managed, and accessed. In SP 800-145, NIST specifies the four primary cloud
deployment models listed below:
Public cloud
Private cloud
Hybrid cloud
Community cloud
Each cloud deployment model may be used for any of the cloud service models: IaaS, PaaS, and
SaaS. The different deployment models present a number of tradeoffs in terms of control,
scale, cost, and availability of resources.
10
Public cloud: The cloud infrastructure is provisioned for open use by the general public. It may
be owned, managed, and operated by a business, academic, or government organization, or some
combination of them. It exists on the premises of the cloud provider. NIST
Public cloud services may be free, subscription-based, or provided on a pay-per-use model. A
public cloud provides the benefits of low up-front expenditure on IT resources and enormous
scalability. However, some concerns for the consumers include network availability, risks
associated with multi-tenancy, visibility and control over the cloud resources and data, and
restrictive default service levels.
11
Private cloud: The cloud infrastructure is provisioned for exclusive use by a single
organization comprising multiple consumers (for example, business units). It may be
owned, managed, and operated by the organization, a third party, or some combination
of them, and it may exist on or off premises. NIST
Many organizations may not wish to adopt public clouds due to concerns related to privacy,
external threats, and lack of control over the IT resources and data. When compared to a public
cloud, a private cloud offers organizations a greater degree of privacy and control over the cloud
infrastructure, applications, and data.
There are two variants of private cloud: on-premise and externally-hosted, as shown in figure 1
and figure 2 respectively on the slide. The on-premise private cloud is deployed by an
organization in its data center within its own premises. In the externally-hosted private
cloud (or off-premise private cloud) model, an organization outsources the implementation of the
private cloud to an external cloud service provider. The cloud infrastructure is hosted on the
premises of the provider and may be shared by multiple tenants. However, the organizations
private cloud resources are securely separated from other cloud tenants by access policies
implemented by the provider.
12
Community cloud: The cloud infrastructure is provisioned for exclusive use by a specific
community of consumers from organizations that have shared concerns (for example, mission,
security requirements, policy, and compliance considerations). It may be owned, managed, and
operated by one or more of the organizations in the community, a third party, or some
combination of them, and it may exist on or off premises. NIST
The organizations participating in the community cloud typically share the cost of
deploying the cloud and offering cloud services. This enables them to lower their
individual investments. Since the costs are shared by a fewer consumers than in a public
cloud, this option may be more expensive. However, a community cloud may offer a
higher level of control and protection than a public cloud. As with the private cloud, there
are two variants of a community cloud: on-premise and externally-hosted.
In an on-premise community cloud, one or more organizations provide cloud services that are
consumed by the community. The cloud infrastructure is deployed on the premises of the
organizations providing the cloud services. The organizations consuming the cloud services
connect to the community cloud over a secure network. The figure on the slide illustrates an
example of an on-premise community cloud.
13
In the externally-hosted community cloud model, the organizations of the community outsource
the implementation of the community cloud to an external cloud service provider. The cloud
infrastructure is hosted on the premises of the provider and not within the premises of any of the
participant organizations. The provider manages the cloud infrastructure and facilitates an
exclusive community cloud environment for the organizations.
The IT infrastructure of each of the organizations connects to the externally-hosted community
cloud over a secure network. The cloud infrastructure may be shared with multiple tenants.
However, the community cloud resources are securely separated from other cloud tenants by
access policies implemented by the provider.
14
Hybrid cloud: The cloud infrastructure is a composition of two or more distinct cloud
infrastructures (private, community, or public) that remain unique entities, but are
bound by standardized or proprietary technology that enables data and application
portability (for example, cloud bursting for load balancing between clouds.) NIST
The figure on the slide illustrates a hybrid cloud that is composed of an on-premise private cloud
deployed by enterprise P, and a public cloud serving enterprise and individual consumers in
addition to enterprise P.
15
The hybrid cloud has become the model of choice for many organizations. Some use cases of the
hybrid cloud model are described below.
Cloud bursting: Cloud bursting is a common usage scenario of a hybrid cloud. In cloud bursting,
an organization uses a private cloud for normal workloads, but optionally accesses a public cloud
to meet transient higher workload requirements. For example, an application can get additional
resources from a public cloud for a limited time period to handle a transient surge in workload.
Web application hosting: An organization may use the hybrid cloud model for web application
hosting. The organization may host mission-critical applications on a private cloud, while less
critical applications are hosted on a public cloud. By deploying less critical applications in the
public cloud, an organization can leverage the scalability and cost benefits of the public cloud. For
example, e-commerce applications use public-facing web assets outside the firewall and can be
hosted in the public cloud.
Packaged applications: An organization may also migrate standard packaged applications, such
as email and collaboration software out of the private cloud to a public cloud. This frees up
internal IT resources for higher value projects and applications.
Application development and testing: An organization may also use the hybrid cloud model for
application development and testing. An application can be tested for scalability and under heavy
workload using public cloud resources, before incurring the capital expense associated with
deploying it in a production environment. Once the organization establishes a steady-state
workload pattern and the longevity of the application, it may choose to bring the application into
the private cloud environment.
16
This lesson covered the definition of cloud computing and the essential characteristics of cloud
computing. This lesson also covered cloud service models and cloud deployment models.
17
This lesson covers the definition of Big Data and its key characteristics. This lesson also covers
the components of a Big Data analytics solution. Further, this lesson covers some use cases of Big
Data analytics.
18
Big Data represents the information assets whose high volume, high velocity, and high variety
require the use of new technical architectures and analytical methods to gain insights and for
deriving business value.
The definition of Big Data has three principal aspects: characteristics of data, data processing
needs, and business value.
Characteristics of data: Big Data includes data sets of considerable sizes containing both
structured and non-structured digital data. Apart from its size, the data gets generated and
changes rapidly, and also comes from diverse sources. These and other characteristics are
covered next.
Data processing needs: Big Data also exceeds the storage and processing capability of
conventional IT infrastructure and software systems. It not only needs a highly-scalable
architecture for efficient storage, but also requires new and innovative technologies and methods
for processing. These technologies typically make use of platforms such as distributed processing,
massively-parallel processing, and machine learning. The emerging discipline of Data Science
represents the synthesis of several existing disciplines, such as statistics, mathematics, data
visualization, and computer science for Big Data analytics.
Business value: Big Data analytics has tremendous business importance to organizations.
Searching, aggregating, and cross-referencing large data sets in real-time or near-real time
enables gaining valuable insights from the data. This enables better data-driven decision making.
Many organizations such as government departments, retail, telecommunications, healthcare,
social networks, banks, and insurance companies employ data science techniques to benefit from
Big Data analytics.
19
Apart form the characteristics of volume, velocity, and varietypopularly known as the 3Vs,
the three other characteristics of Big Data include variability, veracity, and value.
Volume: The word Big in Big Data refers to the massive volumes of data. Organizations are
witnessing an ever-increasing growth in data of all types, such as transaction-based data
stored over the years, sensor data, and unstructured data streaming in from social
media. This growth in data is reaching Petabyteand even Exabytescales. The excessive
volume not only requires substantial cost-effective storage, but also gives rise to challenges in
data analysis.
Velocity: Velocity refers to the rate at which data is produced and changes, and also how fast
the data must be processed to meet business requirements. Today, data is generated at an
exceptional speed, and real-time or near-real time analysis of the data is a challenge for
many organizations. It is essential for the data to be processed and analyzed, and the
results to be delivered in a timely manner. An example of such a requirement is real-time
face recognition for screening passengers at airports.
Variety: Variety refers to the diversity in the formats and types of data. Data is generated by
numerous sources in various structured and non-structured forms. Organizations face the
challenge of managing, merging, and analyzing the different varieties of data in a cost-effective
manner. The combination of data from a variety of data sources and in a variety of formats is a
key requirement in Big Data analytics. An example of such a requirement is combining a large
number of changing records of a particular patient with various published medical research to find
the best treatment.
(Contd)
20
Variability: Variability refers to the constantly changing meaning of data. For example, analysis
of natural language search and social media posts requires interpretation of complex and highlyvariable grammar. The inconsistency in the meaning of data gives rise to challenges related to
gathering the data and in interpreting its context.
Veracity: Veracity refers to the varying quality and reliability data. The quality of the data being
gathered can differ greatly, and the accuracy of analysis depends on the veracity of the source
data. Establishing trust in Big Data presents a major challenge because as the variety and
number of sources grows, the likelihood of noise and errors in the data increases. Therefore, a
significant effort may go into cleaning data to remove noise and errors, and to produce accurate
data sets before analysis can begin. For example, a retail organization may have gathered
customer behavior data from across systems to analyze product purchase patterns and
to predict purchase intent. The organization would have to clean and transform the data
to make it consistent and reliable.
Value: Value refers to the cost-effectiveness of the Big Data analytics technology used and the
business value derived from it. Many large enterprise scale organizations have maintained large
data repositories, such as data warehouses, managed non-structured data, and carried out realtime data analytics for many years. With hardware and software becoming more affordable and
the emergence of more providers, Big Data analytics technologies are now available to a much
broader market. Organizations are also gaining the benefits of business process enhancements,
increased revenues, and better decision making.
21
Data for analytics typically comes from repositories such as enterprise data warehouses and data
lakes.
A data warehouse is a central repository of integrated data gathered from multiple different
sources. It stores current and historical data in a structured format. It is designed for query and
analysis to support an organizations decision making process. For example, a data warehouse
may contain current and historical sales data that is used for generating trend reports for sales
comparisons.
A data lake is a collection of structured and non-structured data assets that are stored as exact or
near-exact copies of the source formats. The data lake architecture is a store-everything
approach to Big Data. Unlike conventional data warehouses, data is not classified when it is stored
in the repository, as the value of the data may not be clear at the outset. The data is also not
arranged as per a specific schema and is stored using an object-based storage architecture. As a
result, data preparation is eliminated and a data lake is less structured compared to a data
warehouse. Data is classified, organized, or analyzed only when it is accessed. When a business
need arises, the data lake is queried, and the resultant subset of data is then analyzed to provide
a solution. The purpose of a data lake is to present an unrefined view of data to highly-skilled
analysts, and to enable them to implement their own data refinement and analysis techniques.
22
The technology layers in a Big Data analytics solution include storage, MapReduce technologies,
and query technologies. These components are collectively called the SMAQ stack.
Storage is the foundational layer of the stack, and is characterized by a distributed architecture
with primarily non-structured content in non-relational form.
The intermediate layer consists of MapReduce technologies that enable the distribution of
computation across multiple compute systems for parallel processing. It also supports a batchoriented processing model of data retrieval and computation as opposed to the record-set
orientation of most SQL-based databases.
The query layer typically implements a NoSQL database for storing, retrieving, and processing
data. It also provides a user-friendly platform for analytics and reporting.
SMAQ solutions may be implemented as a combination of multi-component systems or offered as
a product with a self-contained system comprising storage, MapReduce, and query all in one.
23
MapReduce is the driving force behind most Big Data processing solutions. It is a parallel
programming framework for processing large data sets on a compute cluster. The key innovation
of MapReduce is the ability to take a query over a data set, divide it, and run it in parallel over
multiple compute systems or nodes. This distribution solves the issue of processing data that is
too large to be processed by a single machine.
MapReduce works in two phasesMap and Reduceas suggested by its name. An input data
set is split into independent chunks which are distributed to multiple compute systems. The Map
function processes the chunks in a completely parallel manner, and transforms them into multiple
smaller intermediate data sets. The Reduce function condenses the intermediate results and
reduces them to a summarized data set, which is the desired end result. Typically both the input
and the output data sets are stored on a file-system. The MapReduce framework is highly scalable
and supports the addition of processing nodes to process chunks. Apaches Hadoop MapReduce is
the predominant open source Java-based implementation of MapReduce.
The figure on the slide depicts a generic representation of how MapReduce works and can be used
to illustrate various examples. A classic example of MapReduce is the task of counting the number
of unique words in a very large body of data including millions of documents. In the Map phase,
each word is identified and given the count of 1. In the Reduce phase, the counts are added
together for each word. Another example is the task of grouping customer records within a data
set into multiple age groups, such as 20-30, 30-40, 40-50, and so on. In the Map phase, the
records are split and processed in parallel to generate intermediate groups of records. In the
Reduce phase, the intermediate data sets are summarized to obtain the distinct groups of
customer records (depicted by the colored groups).
24
MapReduce fetches data sets and stores the results of the computation in storage. The data
must be available in a distributed fashion, to serve each processing node. The design and
features of the storage layer are important not just because of the interface with MapReduce, but
also because they affect the ease with which data can be loaded and the results of computation
extracted and searched.
A storage system in the SMAQ stack is based on either a proprietary or an open-source
distributed file system, such as Hadoop Distributed File System (HDFS). The storage system may
also support multiple file systems for client access. The storage system consists of multiple
nodescollectively called a clusterand the file system is distributed across all the nodes in the
cluster. Each node in the cluster has processing capability as well as storage capacity. The system
has a highly-scalable architecture, and additional nodes can be added dynamically to meet the
workload and the capacity needs.
The distributed file systems like HDFS typically provide only an interface similar to that of
regular file systems. Unlike a database, they can only store and retrieve data and not
index it, which is essential for fast data retrieval. To mitigate this and gain the
advantages of a database system, SMAQ solutions may implement a NoSQL database on
top of the distributed file system. NoSQL databases may have built-in MapReduce
features that allow processing to be parallelized over their data stores. In many
applications, the primary source of data is in a relational database. Therefore, SMAQ
solutions may also support the interfacing of MapReduce with relational database
systems.
25
It is unintuitive and inconvenient to specify MapReduce jobs in terms of distinct Map and
Reduce functions in a programming language. To mitigate this, SMAQ systems

incorporate a higher-level query layer to simplify both the specification of the MapReduce
operations, and the analysis of the results. The query layer implements high-level
languages that enable users to describe, run, and monitor MapReduce jobs. The
languages are designed to handle not only the processing, but also the loading and
saving of data from and to the MapReduce cluster. The languages typically support
integration with NoSQL databases implemented on the MapReduce cluster.
26
Big Data analytics solutions have created a world of new opportunities for organizations, such as
healthcare, finance, retail, and governments.
Healthcare: In healthcare, Big Data analytics solutions provide consolidated diagnostic
information and enable healthcare providers to analyze patient data; improve patient care and
outcomes; minimize errors; increase patient engagement; and improve operations and services.
These solutions also enable healthcare providers to monitor patients and analyze their
experiences in real time.
Finance: In finance, organizations use Big Data analytics for activities such as correlating
purchase history, profiling customers, and analyzing behavior on social networks. This also
enables in controlling customer acquisition costs and target sales promotions more effectively. Big
Data analytics is also being used extensively in detecting credit card frauds.
Retail: In retail, organizations use Big Data analytics to gain valuable insights for competitive
pricing, anticipating future demand, effective marketing campaigns, optimized inventory
assortment, and improved distribution. This enables them to provide optimal prices and services
to customers, and also improve operations and revenue.
Government: In government organizations, Big data analytics enables improved efficiency and
effectiveness across a variety of domains such as social services, education, defense, national
security, crime prevention, transportation, tax compliance, and revenue management.
27
This lesson covered the definition of Big Data and its key characteristics. This lesson also covered
the components of a Big Data analytics solution, namely storage, MapReduce, and query. Further,
this lesson covered some of the use cases of Big Data analytics.
28
This lesson covers social networking, social network analysis, and social network use cases. This
lesson also covers mobile computing and its use cases.
29
Social networking is the practice of individuals establishing connections with other individuals for
expanding social and/or business contacts. It results in the formation of a structure of many-tomany human connections called a social network, which represents the relationships and flows
between individuals and groups. A social network enables the sharing of information with the
entire network or subsets of it.
A variety of online services provide a global web-based platform to build social networks among
individuals (and organizations) who share interests, activities, and real-life connections. Online
social networking has grown immensely over the past decade with the proliferation of the Internet
and mobile devices. These social networking services enable the creation, discovery, sharing,
promotion, distribution, and consumption of a variety of digital content for community and social
activities across geographic locations. Most social networking services enable individuals and
organizations to create their personal profiles and connect to each other. They also typically
enable the sharing of opinions, activities, blogs, events, messages, pictures, videos, and other
media. Some provide a specialized set of features, such as enabling connections with co-workers
within an organization, professionals of different fields, or with potential future employers. Some
of the most popular online social networking services are Facebook, Twitter, LinkedIn, Pinterest,
Instagram, and Google+.
30
The increasing use of online social networking services has led to a massive growth of data in the
digital universe. The immense volumes of data hold a tremendous value for organizations.
Through Big Data analytics, organizations can gain valuable insights from the data generated
through social networking. Social network analysis (SNA) is the process of analyzing patterns of
relationships in social networks. SNA involves collecting data from multiple sources (such as social
media posts, surveys, e-mails, blogs, and other electronic artifacts), using analytics on the data
to identify relationships, and mining it for new information. It is useful for examining the social
structure, information flow, and interdependencies (or work patterns) of individuals or
organizations. SNA tools scan social media to determine the quality or effectiveness of a
relationship, identify influential people, associations, and trends.
SNA enables the identification and discovery of complex dynamics, growth, and evolution patterns
in social networks using machine learning and data mining approaches. SNA uses a
multidisciplinary approach involving the use of a wide range of techniques from social sciences,
mathematics, statistics, physics, network science, and computer science. SNA enables the
discovery and analysis of communities, personalization for solitary activities (for example, search)
and social activities (for example, discovery of potential friends), the analysis of user behavior in
open forums (for example, conventional sites, blogs, and communities) and in commercial
platforms (for example, e-commerce). SNA has a wide range of application including
engineering, science, economics, national security, criminology, fraud detection, and ecommerce.
31
Apart from providing a platform for connecting people and organizations, online social networking
also has many applications in many areas such as education, science, problem-solving, sales, and
marketing. Some use cases of social networking are described below:
Brand networking: It is the use of social networking to provide consumers with a platform of
relevant content associated with a particular brand. Organizations use brand networking by
creating social network pages and communities that showcase products, provide information on
promotional offers and events, and enable customer interaction. Brand networking provides a
higher-level of customer interaction and participation, gives global visibility to brands, and
enables reaching a broader customer base. By actively engaging in social networking,
organizations also seek to improve their visibility on search engines. Through analytics tools,
organizations can also gain insights on their customer base that helps in creating more effective
sales campaigns.
Enterprise collaboration: Collaboration is enabled in organizations through the use of enterprise
social networking to enable employees to communicate and share information. These networks
enable employees to collaborate better, share ideas, and exchange information. In essence,
enterprise social networks provide all the features and benefits of online social networking
services with additional specific features and greater control.
Marketing: The use of social networks is becoming a standard approach for marketing. Social
media marketing has the potential to help increase sales and revenue. Organizations advertise
their products and services on the pages of individuals with the advertisements either linking back
to the organizations social media page or to their sales website. Organizations may also mine
social content for identifying potential customers. This helps them in finding new target audiences
more effectively for marketing.
Customer support: Organizations are also increasingly using social networks to engage
customers for enhanced and faster support. By monitoring customer comments on social media,
organizations proactively identify and resolve customer issues.
32
A mobile device is a handheld compute system that has a display with either touch input and/or a
hardware keyboard. Mobile devices typically have features such as voice calling, Bluetooth and/or
NFC for file sharing, Wi-Fi and/or data services (for example, HSPA+ and LTE) for Internet
connectivity, GPS, and audio-video capabilities. Examples of mobile devices are laptops, tablets,
smartphones, and personal digital assistants (PDAs). Mobile computing is the use of mobile
devices to access applications and/or information on the go over a wireless network.
The convergence of wireless technologies, advanced electronics, and the Internet has led to the
emergence of pervasive computing (also called ubiquitous computing) and the Internet of Things
(IoT). Pervasive computing is the growing trend of embedding processors in devices such as
sensors and wearable gadgets and enabling them to communicate over the Internet. Pervasive
computing devices are continuously connected and available, and are contributing to the growth
of the mobile computing ecosystem.
The figure on the slide depicts an application server in an enterprise data center/cloud being
accessed by various mobile clients through wireless connections over the Internet or a private
network, such as a wireless LAN (WLAN).
33
Mobile computing has vast applications in numerous industries and domains. Some use cases of
mobile computing are described below.
Enterprise mobility: The rapid adoption of smart mobile devices is changing the way individuals
and organizations interact and collaborate. Organizations are increasingly providing their
workforce with ubiquitous access to information and business applications over mobile devices.
This enables the employees to stay informed and to carry out business operations irrespective of
their location. This increases the collaboration and enhances the workforce productivity.
Organizations are also increasingly exploring the option of Bring Your Own Device (BYOD),
whereby employees are allowed to use non-company devices, such as laptops and tablets as
business machines. BYOD enables employees to have access to applications and information from
their personal devices while on the move. It also creates an opportunity to reduce acquisition and
operational costs.
Mobility-based products and services: Organizations and service providers offer customers a
wide range of mobility-based applications. It facilitates ubiquitous availability of software products
and services to customers, improves customer service, increases market penetration, and leads
to a potential increase in profitability. A wide variety of mobility-based solutions, such as social
networking services, mobile banking, mobile e-commerce, location-based services, cloud storage,
mobile ticketing, and mobile marketing are extensively available globally.
Mobile cloud computing: Mobile cloud computing is the convergence of cloud computing,
Internet, and wireless technologies. With the rapid growth in the use of mobile devices, cloud
service providers are increasingly enabling mobile access to cloud services. For example, todays
SaaS cloud providers offer a variety of mobile applications for cloud storage, travel and expense
management, and customer relationship management. Mobile cloud computing is also prevalent
within organizations, with corporate IT making enterprise cloud services available to a mobile
workforce.
34
This lesson covered social networking, social network analysis, and social network use cases. This
lesson also covered mobile computing and its use cases.
35
This lesson covers the key drivers for transforming to the third platform, characteristics of third
platform infrastructure, and business and IT imperatives for third platform transformation.
36
For organizations worldwide, there are several drivers for transforming to the third platform.
Some key drivers are described below.
New/improved business models: Organizations constantly strive to create new business
models, new offerings, and new cost structures to drive revenue growth and expand their value.
Third platform technologies enable organizations to improve existing business models and develop
new models to create more value for users/customers, measure market performance, and
dynamically respond to changing user/customer needs. For example, analytics reveals valuable
insights from data that can significantly improve decision-making and enable better responses to
opportunities and threats.
Agility and innovation: In todays competitive world, organizations seek to have agile
operations and reduce the time-to-market for products and services. Third platform technologies
enable organizations to operate in a more agile manner and facilitate innovation. For example,
instead of following the traditional process of resource acquisition, an application development
team in an organization can provision computing resources from a clouds self-service portal, as
and when required. This agility enables rapid development, reduces the time-to-market, and
facilitates innovation and experimentation, which is essential for the development of new products
and services.
Intelligent operations: Organizations globally depend on the smart combination of people and
technology for efficient operations. Inefficient processes, poor quality data, and ineffective
communication and collaboration among asset teams can severely hinder operational efficiency.
The new possibilities to increase operational effectiveness and efficiency are constantly emerging
through the use of third platform technologies. For example, analytics enables organizations to
develop efficient and cost-effective equipment maintenance and replacement strategies.
Equipment can be monitored and analytics tools can process the data in real time to spot or
predict device failures. This also reduces downtime due to equipment failures.
(Contd)
37
New products and services: Creating new products and services is an essential process in
organizations for sustaining business growth and profitability. The third platform enables
organizations to create new or additional products and services on top of traditional products. This
allows them to monetize the new offerings and thereby create new revenue streams. For
instance, manufacturers are using data obtained from sensors embedded in products to offer
innovative after-sales service to customers, such as proactive maintenance to avoid failures in the
products. Analytics also allows organizations to have a more precise segmentation of their
customers and offer tailored products or services.
Mobility: Todays workforce and customers have ubiquitous access to information and business
applications over mobile devices. This increases workforce collaboration and productivity, and
potentially increases market penetration and profitability.
Social networking: Consumer-facing organizations are rapidly embracing online social
networking. For instance, many retail organizations leverage online social networking services to
advertise their products and services, and to increase their global visibility. Social networking
makes organizations more accessible for new customers, and makes them more familiar for
existing customers. Using social networks is becoming a standard approach for customer
engagement, marketing, and for providing a better level of customer service. Organizations also
mine social media to gain insights on existing and potential customers. This helps them in
providing better services to existing customer and in finding new target audiences for expanding
their customer base. Organizations that engage on social networking channels also tend to enjoy
greater loyalty from their customers.
38
Module 1, Introduction to Information Storage described the key characteristics of a data center.
Although a third platform infrastructure has similar key characteristics, there are additional
requirements from the infrastructure to support capabilities such as mobility, social interaction,
analytics, and delivering IT resources as services in a cost-effective manner. Some key
characteristics of a third platform infrastructure are described below.
Availability: Resilient design of infrastructure components and applications enable to achieve
high availability of applications and information. Resilience is characterized by the infrastructures
ability to continue to provide critical services efficiently following a disruptive event. The
techniques such as redundant components, clustering, service availability zones, and looselycoupled application design are implemented for protection and recoverability from planned
outages, system failures, and natural disasters. These and other techniques are covered later in
the course.
Security: With third platform technologies, there are several security challenges such as
unauthorized data access, data loss, hacking, malware, data ownership, and loss of governance
and compliance. Several security mechanisms such as authentication, access control, firewall, and
encryption are implemented to ensure security across multiple third platform technologies. These
and other techniques are covered later in the course. Security tools may also support threat
detection, security incident response, compliance reporting and incident investigation through the
real-time collection and historical analysis of security events from a wide variety of event and
contextual data sources.
Scalability: A third platform infrastructure is typically characterized by massive scalability. The
infrastructure is capable of non-disruptively accommodating changes in application workloads and
data volume. For example, in Big Data analytics, a scale-out architecture (covered later in the
course) is typically used, which involves adding nodes to a cluster to meet growth in processing
and storage requirements. The workload and data is then automatically balanced across the
available resources.
(Contd)
39
Performance: A third platform infrastructure encounters mixed workloads, that may have
varying combinations of sequential/random reads and/or writes for different operations such as
transaction processing, analytics, and backup and recovery. Some applications such as Big Data
analytics solutions use batch processing and require real-time or near-real time processing
capabilities. The infrastructure should maintain optimal performance of applications, while
ensuring high throughput and low latency. Apart from installing high-performance components, a
number of techniques such as load balancing, caching, and storage tiering are used to ensure the
performance required to meet service levels. These and other techniques are covered later in the
course.
Ease of access: One of the key drivers for third platform adoption is the ability to access
applications and information from any location over mobile devices. Organizations require
infrastructure, software, and application development platforms to enable mobile access
to information and current and new applications.
Interoperability: In a third platform environment, multiple technologies and
heterogeneous hardware and software components integrate to deliver services to end
users. Interoperability is the ability of two or more systems or components to share and
use information and services. Integration services for interoperability include detailed
design and implementation services that link the IT infrastructure, data, and application
functionality (custom and/or off-the-shelf software) with each other. The specific
activities may include project planning and management, detailed design or
implementation of application programming interfaces (APIs), web services, or
middleware.
Manageability: In a third platform environment, simplicity and flexibility of infrastructure
management is vital. Management tools provide a single pane to carry out all management
processes on the entire infrastructure. Tools that enable multi-party orchestration and automation
of tasks are deployed for improved efficiency and faster provisioning of resources. Automation
and service orchestration also support self-service (which enables agility) and measured service.
Pre-integrated/commodity components are also widely used for ease of installation and
management. Management processes are covered later in the course.
40
Today, CIOs and line-of-business executives in organizations find themselves in the midst of
unprecedented opportunity that comes with the emergence of the third platform. The third
platform is fueling enterprise innovation and growth, and major new sources of competitive
advantage are being built by creatively leveraging third platform technologies. For third platform
adoption, organizations need to transform the way in which they engage with their workforce and
customers, the speed at which they deliver their products and services, and the efficiency and
resiliency of their operations. Organizations need to provide support for the proliferation of new
devices coming into the workplace, meet the demands of a highly mobile workforce, manage
rapidly expanding data volumes, and ensure the value and security of information in multiple data
sources both within and outside the enterprise. All of these are transforming the traditional IT
environment and changing the way IT roles are performed.
Some key organizational imperatives for third platform transformation are described below.
Operating model transformation: Traditionally, the requests to IT are channeled through a
lengthy approval and delivery process, which is detrimental to agility and innovation. The
adoption of the IT as a Service (ITaaS) operating model transforms traditional IT delivery to a
service paradigm. IT becomes a utility to the business, and services are packaged and
orchestrated for convenient consumption. IT services are maintained in a service catalog which
enables Lines of Business to provision resources or deploy new applications in a self-service
manner. This enables organizations to achieve business agility. ITaaS provides a flexible cost
model, reduces fixed costs, and permits multiple options for sourcing resources. The availability of
vast tools and technologies is enabling the delivery of any IT resource as a service
Anything/Everything as a Service (XaaS). ITaaS represents an important means to provide the
responsiveness and cost-effectiveness that organizations seek. This involves significant
automation of routine IT tasks, presentation of services through a self-service portal, and
streamlined operational processes.
(Contd)
41
Organizational transformation: Organizational transformation focuses on how the IT

organization and the roles within it change when transforming to the third platform. New
roles and responsibilities emerge to establish and manage services instead of technology.
The new roles involve performing tasks related to service definition and creation, service
administration and management, service governance and policy formulation, and service
consumer management. For example, IT may move from a cost center to a strategic
business partner within the organization and IT managers could be called on to act as
cloud advisors or financial managers of the IT services business. Some other examples of
new roles in a third platform environment include service manager, account manager,
cloud architect, capacity planner, and service operation manager.
Technology transformation: Technology transformation has two dimensions:
application transformation and infrastructure transformation. Application transformation
involves aligning application delivery through the cloud, and empowering applications with
analytics and mobile delivery capabilities. Cloud software sourcing options allow consolidation of
similar applications. Organizations can also make a single sourcing decision for each application
type from private or public clouds, while leaving some applications on legacy infrastructure. This
right-sourcing of applications for the optimal cloud model reduces the number of integration
points for IT, and produces greater standardization and operational efficiency. Infrastructure
transformation involves building a standardized, cost-efficient, and agile third platform
infrastructure, capable of supporting existing and transformed applications, and providing a
foundation for automation. This is enabled through the use of virtualization and software-defined
technologies (covered later in the course). These technologies enable IT to consolidate
resources, improve resource utilization, eliminate IT silos, and reduce long cycle times to
respond to new business requests.
Skills transformation: The changing roles for IT staff entails the need for skills in new
technologies, and also in more business-facing skills focused on communications, marketing, and
service management. Many organizations were built around second platform technologies, and
implementing innovative third platform solutions require newer skills and expertise. The skills in
the areas of cloud, Big Data, social, and mobile technologies are predicted by IDC to become the
new core IT competencies over the next two decades. Apart from technical skills, having strong
soft skills like communication, collaboration, networking, creativity, relationship building, and
problem solving is considered equally important. Organizations may also take the hybrid approach
of adding specific skills to their in-house teams to focus on their core competencies, while
sourcing non-core activities from partners, suppliers, and third party service providers.
42
This lesson covered the key drivers for transforming to the third platform, characteristics of third
platform infrastructure, and the imperatives for third platform transformation.
43
The Concepts in Practice section covers VMware vCloud Air, Pivotal Cloud Foundry, EMC
Syncplicity, Pivotal GemFire, and Pivotal Greenplum Database.
Note:
For the latest information on VMware products, visit www.vmware.com.
For the latest information on Pivotal products, visit www.pivotal.io.
For the latest information on EMC Syncplicity, visit www.syncplicity.com.
44
VMware vCloud Air is a secure public cloud owned and operated by VMware that offers
Infrastructure as a Service for enterprise use cases, such as extending existing data center
workloads into the public cloud, migrating applications from on-premise clouds to the public
cloud, new application development, and disaster recovery. It is built on the foundation of
vSphere and is compatible with existing VMware on-premise clouds. It enables organizations to
adopt the hybrid cloud model by seamlessly extending their on-premise clouds into the public
cloud. vCloud Air allows existing applications to run in the public cloud without the need to rewrite
or re-architect them. Organizations can use the same networking, security, and management
tools, skills, and policies that are used in their on-site environments. A consolidated view of
allocated resources is provided to enable administrators to manage resource utilization. vCloud Air
has three primary service offerings (with more expected in the future): Dedicated Cloud (singletenant, physically isolated cloud service), Virtual Private Cloud (logically isolated, multi-tenant
cloud service), and Disaster Recovery (cloud-based disaster recovery service). vCloud Air offers
both term-based subscription and pay-as-you-go options.
Pivotal Cloud Foundry (CF) is an enterprise Platform as a Service, built on the foundation of the
Cloud Foundry open-source PaaS project. The Cloud Foundry open-source project is sustained by
the Cloud Foundry Foundation, which has many leading global enterprises as members. Pivotal
CF, powered by Cloud Foundry, enables streamlined application development, deployment, and
operations in both private and public clouds. It supports multiple programming languages and
frameworks including Java, Ruby, Node.js, PHP, and Python. It supports agile application
development and enables developers to continuously deliver updates to and horizontally scale
web and third platform applications with no downtime. Developers can rapidly develop and deploy
applications without being concerned about configuring and managing the underlying cloud
infrastructure. Pivotal CF also supports multiple leading data services such as Jenkins, MongoDB,
MySQL, Redis, and Hadoop. The use of open standards enables migration of applications between
compatible public and private clouds. Pivotal CF provides a unified management console for the
entire platform that enables in-depth application and infrastructure monitoring.
(Contd)
45
EMC Syncplicity is an enterprise-grade online file sharing, collaboration, and data protection
SaaS solution. It enables a business user to securely share files and folders, and collaborate with
other users. It supports both mobile and web access to files from any device, and the files are
also available offline. It synchronizes file changes across all devices in real time, so
documents are always protected and available on any device. If a device fails, access to
files would still be available from other devices. It enables a bring-your-own-device (BYOD)
workforce, while providing access controls, single sign-on (SSO), data encryption, and other
enterprise-grade features. Syncplicity currently has four offerings: Personal Edition (for
individuals), Business Edition (for small and medium businesses), Department Edition
(for enterprise departments), and Enterprise Edition. The Enterprise Edition has support
for public, on-premise, and hybrid deployment options.
46
Pivotal GemFire is an in-memory distributed database for high-scale custom NoSQL applications.
GemFire stores all operational data in the RAM across distributed nodes to provide fast access to
data while minimizing the performance penalty of reading from the storage drives. This provides
low latency data access to applications at massive scale with many concurrent transactions
involving Terabytes of operational data. Designed for maintaining consistency of concurrent
operations across its distributed data nodes, GemFire supports ACID (Atomicity, Consistency,
Isolation, Durability) transactions for massively-scaled applications, such as stock trading,
financial payments, and ticket sales having millions of transactions a day. GemFire provides linear
scalability that allows to predictably increase the capacity and the data storage by adding
additional nodes to a cluster. Data distribution and system resource usage is automatically
adjusted as nodes are added or removed, making it easy to scale up or down to quickly meet the
expected or unexpected spikes of demand. GemFire offers built in fail-over and resilient selfhealing clusters to allow developers to meet the most stringent service level requirements for
data accessibility. It provides native support for Java, C++, and C# programming languages,
while applications written in other programming languages are supported via a REST API.
Pivotal Greenplum Database is a complete SMAQ solution, designed for business intelligence
and Big Data analytics. It has a linearly scalable, massively parallel processing (MPP) architecture
that stores and analyzes Terabytes to Petabytes of data. In this architecture, each server node
acts as a self-contained database management system that owns and manages a distinct portion
of the overall data. It provides automatic parallelization with no need for manual partitioning or
tuning. The system automatically distributes data and parallelizes query workloads across all
available hardware. In-database analytics is enabled via the support of high-performance and
flexible data exchange between Hadoop and Greenplum Database. It has embedded support for
SQL, MapReduce, and programmable analytics. It also provides tools for database management,
backup, and disaster recovery.
47
This module covered cloud computing and its essential characteristics. This module also covered
cloud service models, and cloud deployment models. Additionally, this module covered Big Data
analytics. Further, this module covered social networking and mobile computing. Lastly, this
module covered the key characteristics of third platform infrastructure and the key imperatives
for transforming to the third platform.
48
This module focuses on the building blocks of a data center environment. This module also
focuses on compute system, its components, and its types. Additionally, this module
focuses on compute virtualization, application virtualization, and desktop virtualization.
Further, this module focuses on an overview of storage and connectivity in a data center.
Finally, this module focuses on an overview of software-defined data center.
Module 3: Data Center Environment
This lesson covers the building blocks of a data center infrastructure. It covers the components
and functions of the five layers of a data center. It also covers the three cross-layer functions in a
data center. Further, this lesson covers best-of-breed versus converged infrastructure.
Module 1, Introduction to Information Storage, defined data center and specified the IT
infrastructure and support infrastructure that comprise a data center. The figure on the slide is a
block diagram depicting the core IT infrastructure building blocks that make up a data center. The
IT infrastructure is arranged in five logical layers and three cross-layer functions. The five layers
are physical infrastructure, virtual infrastructure, software-defined infrastructure, orchestration,
and services. Each of these layers has various types of hardware and/or software components as
shown in the figure. The three cross-layer functions are business continuity, security, and
management. Business continuity and security functions include mechanisms and processes that
are required to provide reliable and secure access to applications, information, and services. The
management function includes various processes that enable the efficient administration of the
data center and the services for meeting business requirements.
The building blocks depicted in the figure may be implemented in part or in whole to create either
a second platform, platform 2.5, or a third platform data center. Applications that are deployed in
the data center may be a combination of internal applications, business applications, and third
platform applications that are either custom-built or off-the-shelf. By ensuring the fulfillment of
the five essential cloud characteristics, the infrastructure can be transformed into a cloud
infrastructure that could be either private or public. Further, by integrating cloud extensibility, the
infrastructure can be connected to an external cloud to leverage the hybrid cloud model.
The physical infrastructure forms the foundation layer of a data center. It includes equipment
such as compute systems, storage systems, and networking devices along with the operating
systems, system software, protocols, and tools that enable the physical equipment to perform
their functions. A key function of physical infrastructure is to execute the requests generated by
the virtual and software-defined infrastructure, such as storing data on the storage devices,
performing compute-to-compute communication, executing programs on compute systems, and
creating backup copies of data. Compute systems are covered later in this module. Different
storage systems are covered in Modules 4, Intelligent Storage Systems (ISS), 5, Block-based
Storage System, 6, File-based Storage System (NAS), and 7, Object-based and Unified
Storage. Networking is covered in Modules 9, Fibre Channel (FC) SAN, 10, Internet Protocol (IP)
SAN, and 11, FC over Ethernet (FCoE) SAN.
Virtualization is the process of abstracting physical resources, such as compute, storage, and
network, and creating virtual resources from them. Virtualization is achieved through the use of
virtualization software that is deployed on compute systems, storage systems, and network
devices. Virtualization software aggregates physical resources into resource pools from which it
creates virtual resources. A resource pool is an aggregation of computing resources, such as
processing power, memory, storage, and network bandwidth. For example, storage virtualization
software pools the capacity of multiple storage devices to create a single large storage capacity.
Similarly, compute virtualization software pools the processing power and memory capacity of a
physical compute system to create an aggregation of the power of all processors (in megahertz)
and all memory (in megabytes). Examples of virtual resources include virtual compute (virtual
machines), virtual storage (LUNs), and virtual networks.
Virtualization enables a single hardware resource to support multiple concurrent instances of
systems, or multiple hardware resources to support a single instance of system. For example, a
single disk drive can be partitioned and presented as multiple disk drives to a compute system.
Similarly, multiple disk drives can be concatenated and presented as a single disk drive to a
compute system. With virtualization, it is also possible to make a resource appear larger or
smaller than it actually is.
Virtualization offers several benefits in a data center. It enables the consolidation of physical IT
resources, and supports a multitenant environment. This optimizes the utilization of physical
resources that, in turn, results in an increased return-on-investment (ROI) and enables reducing
the costs of purchasing of new hardware. Virtualization also reduces space and energy
requirements and simplifies infrastructure management. It also increases the flexibility of
resource provisioning through the dynamic creation and reclamation of virtual resources.
Virtualization is a key enabling technology to meet the resource pooling and rapid elasticity
characteristics of cloud computing.
(Contd)
Compute virtualization is covered later in this module, while different storage virtualization and
network virtualization techniques are covered later in the course in the storage modules and
network modules respectively.
Note: While deploying a data center, an organization may choose not to deploy virtualization. In
such an environment, the software-defined layer is deployed directly over the physical
infrastructure. Further, it is also possible that part of the infrastructure is virtualized and rest is
not virtualized.
The software-defined infrastructure layer is deployed either on the virtual layer or on the
physical layer. In the software-defined approach, all infrastructure components are virtualized and
aggregated into pools. This abstracts all underlying resources from applications. The softwaredefined approach enables ITaaS, in which consumers provision all infrastructure components as
services. It centralizes and automates the management and delivery of heterogeneous resources
based on policies. The key architectural components in the software-defined approach include
software-defined compute (equivalent to compute virtualization), software-defined storage (SDS),
and software-defined network (SDN). Software-defined data center is covered later in this
module. Software-defined storage is covered in Module 8, whereas software-defined network is
covered in the network modules.
The orchestration layer includes the orchestration software. The key function of this layer
is to provide workflows for executing automated tasks to accomplish a desired outcome.
Workflow refers to a series of inter-related tasks that perform a business operation. The
orchestration software enables this automated arrangement, coordination, and
management of the tasks. This helps to group and sequence tasks with dependencies
among them into a single, automated workflow.
Associated with each service listed in the service catalog, there is an orchestration
workflow defined. When a service is selected from the service catalog, an associated
workflow in the orchestration layer is triggered. Based on this workflow, the orchestration
software interacts with the components across the software-defined layer and the BC, security,
and management functions to invoke the provisioning tasks to be executed by the entities.
Similar to a cloud service, an IT service is a means of delivering IT resources to the end users to
enable them to achieve the desired business results and outcomes without having any liabilities
such as risks and costs associated with owning the resources. Examples of services are
application hosting, storage capacity, file services, and email. The service layer is accessible to
applications and end users. This layer includes a service catalog that presents the information
about all the IT resources being offered as services. The service catalog is a database of
information about the services and includes a variety of information about the services, including
the description of the services, the types of services, cost, supported SLAs, and security
mechanisms. The provisioning and management requests are passed on to the orchestration
layer, where the orchestration workflowsto fulfill the requestsare defined.
The business continuity (BC) cross-layer function specifies the adoption of proactive and reactive
measures that enable an organization to mitigate the impact of downtime due to planned and
unplanned outages. The proactive measures include activities and processes such as business
impact analysis, risk assessment, and technology solutions such as backup, archiving, and
replication. The reactive measures include activities and processes such as disaster recovery and
disaster restart to be invoked in the event of a service failure. This function supports all the
layersphysical, virtual, software-defined, orchestration, and servicesto provide uninterrupted
services to the consumers. The BC cross-layer function of a cloud infrastructure enables a
business to ensure the availability of services in line with the service level agreement (SLA). BC
and BC solutions are covered in Modules 12, Introduction to Business Continuity, 13, Backup
and Archive, and 14, Replication.
10
The security cross-layer function supports all the infrastructure layersphysical, virtual, softwaredefined, orchestration, and serviceto provide secure services to the consumers. Security
specifies the adoption of administrative and technical mechanisms that mitigate or minimize the
security threats and provide a secure data center environment. Administrative mechanisms
include security and personnel policies or standard procedures to direct the safe execution of
various operations. Technical mechanisms are usually implemented through tools or devices
deployed on the IT infrastructure. Examples of technical mechanisms include firewall, intrusion
detection and prevention systems, and antivirus software.
Governance, risk, and compliance (GRC) specifies processes that help an organization in ensuring
that their acts are ethically correct and in accordance with their risk appetite (the risk level an
organization chooses to accept), internal policies, and external regulations. Security mechanisms
should be deployed to meet the GRC requirements. Security and GRC are covered in Module 15,
Securing the Storage Infrastructure.
11
The management cross-layer function specifies the adoption of activities related to data center
operations management. Adoption of these activities enables an organization to align the
creation and delivery of IT services to meet their business objectives. This course focuses on
the aspect of storage infrastructure management.
Storage operation management enables IT administrators to manage the data center
infrastructure and services. Storage operation management tasks include handling of
infrastructure configuration, resource provisioning, problem resolution, capacity, availability, and
compliance conformance. This function supports all the layers to perform monitoring,
management, and reporting for the entities of the infrastructure. Storage infrastructure
management is covered in Module 16, Managing the Storage Infrastructure.
12
There are two options for building the data center infrastructure by integrating best-of-breed
infrastructure components, or by acquiring and deploying a converged infrastructure.
Best-of-breed infrastructure: In this approach, organizations integrate the best-of-breed
infrastructure components (hardware and software) purchased from multiple different vendors.
This enables the organizations to leverage the advantages of high quality products and services
from the respective leading vendors in the segment. It provides the flexibility to change the
individual vendors in case the committed support is not provided and the SLAs are not met.
Additionally, this approach allows organizations to repurpose the existing infrastructure
components, providing a cost benefit. However, this approach requires significant CAPEX, OPEX,
and time as it involves evaluation, purchase, testing, deployment, configuration, and integration
of multiple disparate hardware and software components. Further, scaling of such an
infrastructure takes longer because each new component goes through the process from
evaluation to integration.
Converged infrastructure: A converged infrastructure integrates hardware and software
components that make up a data center into a single packaged solution. This package is a selfcontained unit that can be deployed independently, or aggregated with other packages to meet
the additional capacity and performance requirements. The package is pre-configured and
optimized, which reduces the time to acquire and deploy the infrastructure. It also lowers power
and space requirements. Vendors also provide cloud-ready converged infrastructure with built-in
capabilities for secure multi-tenancy. Converged infrastructure has a single management
software capable of managing all hardware and software within the package. A potential area of
concern regarding the converged infrastructure solutions is the lack of flexibility to use
infrastructure components from different vendors. Some vendors may provide the flexibility to
choose multi-vendor infrastructure components such as network devices, compute systems, and
hypervisors for the solution.
13
This lesson covered the building blocks of a data center infrastructure. It covered the components
and functions of the five layers of a data center. It also covered the three cross-layer functions of
a data center. Further, this lesson covered best-of-breed versus converged infrastructure.
14
This lesson covers compute system, and its key physical and logical components. This lesson also
covers the types of compute systems.
15
A compute system is a computing device (combination of hardware, firmware, and system

software) that runs business applications. Examples of compute systems include physical
servers, desktops, laptops, and mobile devices. As mentioned previously in Module 1,

Introduction to Information Storage in this course, the term compute system refers to
physical servers and hosts on which platform software, management software, and
business applications of an organization are deployed.
A compute systems hardware consists of processor(s), memory, internal storage, and
I/O devices. The logical components of a compute system include the operating system
(OS), file system, logical volume manager, and device drivers. The OS may include the
other software or they can be installed individually.
In an enterprise data center, applications are typically deployed on compute clusters for
high availability and for balancing computing workloads. A compute cluster is a group of two or
more compute systems that function together, sharing certain network and storage resources,
and logically viewed as a single system. Compute clustering is covered in detail in Module 12,
Introduction to Business Continuity.
16
A compute system comprises multiple physical hardware components assembled inside a metal
enclosure. Some key components are described below.
Processor: A processor, also known as a Central Processing Unit (CPU), is an integrated circuit
(IC) that executes the instructions of a software program by performing fundamental arithmetical,
logical, and input/output operations. A common processor/instruction set architecture is the x86
architecture with 32-bit and 64-bit processing capabilities. Modern processors have multiple cores
(independent processing units), each capable of functioning as an individual processor.
Random-Access Memory (RAM): The RAM or main memory is an IC that serves as a volatile
data storage internal to a compute system. The RAM is directly accessible by the processor, and
holds the software programs for the execution and the data used by the processor.
Read-Only Memory (ROM): A ROM is a type of non-volatile semiconductor memory from which
data can only be read but not written to. It contains the boot firmware (that enables a compute
system to start), power management firmware, and other device-specific firmware.
Motherboard: A motherboard is a printed circuit board (PCB) to which all compute system
components connect. It has sockets to hold components such as the microprocessor chip, RAM,
and ROM. It also has network ports, I/O ports to connect devices such as keyboard, mouse, and
printers, and essential circuitry to carry out computing operations. A motherboard may
additionally have integrated components, such as a graphics processing unit (GPU), a network
interface card (NIC), and adapters to connect to external storage devices. Motherboards (and
other internal components) receive power from a power supply unit.
(Cont'd)
17
Chipset: A chipset is a collection of microchips on a motherboard and it is designed to perform

specific functions. The two key chipset types are Northbridge and Southbridge. Northbridge
manages processor access to the RAM and the GPU, while Southbridge connects the processor to
different peripheral ports, such as USB ports.
Secondary storage: Secondary storage is a persistent storage device, such as a hard disk drive
or a solid state drive, on which the OS and the application software are installed. The processor
cannot directly access secondary storage. The desired applications and data are loaded from the
secondary storage on to the RAM to enable the processor to access them.
Based on business and performance requirements, cost, and expected rate of growth, an
organization has to make multiple important decisions about the choice of compute system
hardware to be deployed in a data center. These decisions include the number of compute
systems to deploy, the number, the type, and the speed of processors, the amount of RAM
required, the motherboards RAM capacity, the number and type of expansion slots on a
motherboard, the number and type of I/O cards, and installation and configuration effort.
18
The key logical components of a compute system are:
Operating system
Virtual memory
Logical volume manager
File system
A detailed description of the logical components is beyond the scope of this course. However, the
components are covered in brief next.
19
The operating system (OS) is a software that acts as an intermediary between a user of a
compute system and the compute system hardware. It controls and manages the hardware and
software on a compute system. The OS manages hardware functions, applications execution, and
provides a user interface (UI) for users to operate and use the compute system. The figure on the
slide depicts a generic architecture of an OS. Some functions (or services) of an OS include
program execution, memory management, resources management and allocation, and
input/output management. An OS also provides networking and basic security for the access and
usage of all managed resources. It also performs basic storage management tasks while
managing other underlying components, such as the device drivers, logical volume manager, and
file system. An OS also contains high-level Application Programming Interfaces (APIs) to enable
programs to request services.
To interact with a particular hardware resource, an OS requires a device driver, which is a special
system software that permits the OS to interact with the specific device. For example, hardware
such as printer, mouse, disk drive, network adapters, and graphics cards require device drivers. A
device driver enables the OS to recognize the device, and to access and control it. Device drivers
are hardware-dependent and OS-specific.
20
The amount of physical memory (RAM) in a compute system determines both the size and the
number of applications that can run on the compute system. Memory virtualization presents
physical memory to applications as a single logical collection of contiguous memory locations
called virtual memory. While executing applications, the processor generates logical addresses
(virtual addresses) that map into the virtual memory. The memory management unit of the
processor then maps the virtual address to the physical address. The OS utility, known as the
virtual memory manager (VMM), manages the virtual memory and also the allocation of physical
memory to virtual memory.
An additional memory virtualization feature of an OS enables the capacity of secondary storage
devices to be allocated to the virtual memory. This creates a virtual memory with an address
space that is much larger than the actual physical memory space present in the compute system.
This enables multiple applications and processes, whose aggregate memory requirement is
greater than the available physical memory, to run on a compute system without impacting each
other. The VMM manages the virtual-to-physical memory mapping and fetches data from the
secondary storage when a process references a virtual address that points to data at the
secondary storage. The space used by the VMM on the secondary storage is known as a swap
space. A swap space (also known as page file or swap file) is a portion of the storage drive that is
used as physical memory.
In a virtual memory implementation, the memory of a system is divided into contiguous blocks of
fixed-size pages. A process known as paging moves inactive physical memory pages onto the
swap file and brings them back to the physical memory when required. This enables efficient use
of the available physical memory among different applications. The OS typically moves the leastused pages into the swap file so that enough RAM is available for processes that are more active.
The access to swap file pages is slower than physical memory pages because swap file pages are
allocated on the storage drive, which is slower than the physical memory.
21
Logical Volume Manager (LVM) is software that runs on a compute system and manages logical
and physical storage. LVM is an intermediate layer between the file system and the physical
drives. It can partition a larger-capacity disk into virtual, smaller-capacity volumes (partitioning)
or aggregate several smaller disks to form a larger virtual volume (concatenation). LVMs are
mostly offered as part of the OS. Earlier, an entire storage drive would be allocated to the file
system or the other data entity used by the OS or application. The disadvantage of this was a lack
of flexibility. When a storage drive ran out of space, there was no easy way to extend the file
systems size. As the storage capacity of the disk drive increased, allocating the entire disk drive
for the file system often resulted in underutilization of the storage capacity. The evolution of LVMs
enabled dynamic extension of file system capacity and efficient storage management. The LVM
provides optimized storage access and simplifies storage resource management. It hides details
about the physical disk and the location of data on the disk. It enables administrators to change
the storage allocation even when the application is running.
The basic LVM components are physical volumes, logical volume groups, and logical volumes. In
LVM terminology, each physical disk connected to the compute system is a physical volume (PV).
A volume group is created by grouping together one or more PVs. A unique physical volume
identifier (PVID) is assigned to each PV when it is initialized for use by the LVM. Physical volumes
can be added or removed from a volume group dynamically. They cannot be shared between
different volume groups; which means, the entire PV becomes part of a volume group. Each PV is
divided into equal-sized data blocks called physical extents when the volume group is created.
Logical volumes (LV) are created within a given volume group. A LV can be thought of as a disk
partition, whereas the volume group itself can be thought of as a disk. The size of a LV is based
on a multiple of the number of physical extents. The LV appears as a physical device to the OS. A
LV is made up of noncontiguous physical extents and may span over multiple physical volumes. A
file system is created on a logical volume. These LVs are then assigned to the application. A
logical volume can also be mirrored to provide enhanced data availability.
22
Disk partitioning was introduced to improve the flexibility and utilization of disk drives. In
partitioning, a disk drive is divided into logical containers called logical volumes. For example, a
large physical drive can be partitioned into multiple LVs to maintain data according to the file
system and application requirements. The partitions are created from groups of contiguous
cylinders when the hard disk is initially set up on the host. The hosts file system accesses the
logical volumes without any knowledge of partitioning and physical structure of the disk.
Concatenation is the process of grouping several physical drives and presenting them to the host
as one big logical volume.
23
A file is a collection of related records or data stored as a single named unit in contiguous logical
address space. Files are of different types, such as text, executable, image, audio/video, binary,
library, and archive. Files have a number of attributes, such as name, unique identifier, type, size,
location, owner, and protection.
A file system is an OS component that controls and manages the storage and retrieval of files in a
compute system. A file system enables easy access to the files residing on a storage drive, a
partition, or a logical volume. It consists of logical structures and software routines that control
access to files. It enables users to perform various operations on files, such as create, access
(sequential/random), write, search, edit, and delete.
A file system typically groups and organizes files in a tree-like hierarchical structure. It enables
users to group files within a logical collection called a directory, which are containers for storing
pointers to multiple files. A file system maintains a pointer map to the directories, subdirectories
(if any), and files that are part of the file system. It also stores all the metadata (file attributes)
associated with the files.
A file system block is the smallest unit allocated for storing data. Each file system block is a
contiguous area on the physical disk. The block size of a file system is fixed at the time of its
creation. The file system size depends on the block size and the total number of file system
blocks. A file can span multiple file system blocks because most files are larger than the
predefined block size of the file system. File system blocks cease to be contiguous and become
fragmented when new blocks are added or deleted. Over the course of time, as files grow larger,
the file system may become fragmented.
(Contd)
24
File system may be broadly classified as follows disk-based, network-based, and virtual file
systems. These are described below.
Disk-based file system: A disk-based file system manages the files stored on storage devices
such as solid-state drives, disk drives, and optical drives. Examples of disk-based file systems are
Microsoft NT File System (NTFS), Apple Hierarchical File System (HFS) Plus, Extended File System
family for Linux, Oracle ZFS, and Universal Disk Format (UDF).
Network-based file system: A network-based file system uses networking to allow file system
access between compute systems. Network-based file systems may use either the client-server
model, or may be distributed/clustered. In the client-server model, the file system resides on a
server, and is accessed by clients over the network. The client-server model allows clients to
mount the remote file systems from the server. NFS for UNIX environment and CIFS for Windows
environment (both covered in Module 6, File-based Storage System (NAS)) are two standard
client-server file sharing protocols. A clustered file system is a file system that is simultaneously
mounted on multiple compute systems (or nodes) in a cluster. It allows the nodes in the cluster
to share and concurrently access the same storage device. Clustered file systems provide features
like location-independent addressing and redundancy. A clustered file system may also spread
data across multiple storage nodes, for redundancy and/or performance. Examples of networkbased file systems are Microsoft Distributed File System (DFS), Hadoop Distributed File System
(HDFS), VMware Virtual Machine File System (VMFS), Red Hat GlusterFS, and Red Hat CephFS.
Virtual file system: A virtual file system is a memory-based file system that enables compute
systems to transparently access different types of file systems on local and network storage
devices. It provides an abstraction layer that allows applications to access different types of file
systems in a uniform way. It bridges the differences between the file systems for different
operating systems, without the applications knowledge of the type of file system they are
accessing. The examples of virtual file systems are Linux Virtual File System (VFS) and Oracle
CacheFS.
25
The following is the process of mapping user files to the storage that uses an LVM:
1. Files are created and managed by users and applications.
2. These files reside in the file systems.
3. The file systems are mapped to file system blocks.
4. The file system blocks are mapped to logical extents of a logical volume.
5. These logical extents in turn are mapped to the physical extents either by the OS or by the
LVM.
6. These physical extents are mapped to the sectors in a storage subsystem.
If there is no LVM, then there are no logical extents. Without LVM, file system blocks are directly
mapped to sectors.
Apart from the files and directories, the file system also includes a number of other related
records, which are collectively called the metadata. The metadata of a file system must be
consistent for the file system to be considered healthy.
26
The compute systems used in building data centers are typically classified into three categories:
tower compute system, rack-mounted compute system, and blade compute system
A tower compute system, also known as a tower server, is a compute system built in an upright
standalone enclosure called a tower, which looks similar to a desktop cabinet. Tower servers
have a robust build, and have integrated power supply and cooling. They typically have individual
monitors, keyboards, and mice. Tower servers occupy significant floor space and require complex
cabling when deployed in a data center. They are also bulky and a group of tower servers
generate considerable noise from their cooling units. Tower servers are typically used in smaller
environments. Deploying a large number of tower servers in large environments may involve
substantial expenditure.
A rack-mounted compute system, also known as a rack server, is a compute system designed to
be fixed inside a frame called a rack. A rack is a standardized enclosure containing multiple
mounting slots called bays, each of which holds a server in place with the help of screws. A
single rack contains multiple servers stacked vertically in bays, thereby simplifying network
cabling, consolidating network equipment, and reducing the floor space use. Each rack server has
its own power supply and cooling unit. Typically, a console is mounted on a rack to enable
administrators to manage all the servers in the rack. Some concerns with rack servers are that
they are cumbersome to work with, and they generate a lot of heat because of which more
cooling is required, which in turn increases power costs. A rack unit (denoted by U or RU) is a
unit of measure of the height of a server designed to be mounted on a rack. One rack unit is 1.75
inches (44.45 mm). A 1 U rack server is typically 19 inches (482.6 mm) wide. The standard rack
cabinets are 19 inches wide and the common rack cabinet sizes are 42U, 37U, and 27U. The rack
cabinets are also used to house network, storage, telecommunication, and other equipment
modules. A rack cabinet may also contain a combination of different types of equipment modules.
(Contd)
27
A blade compute system, also known as a blade server, is an electronic circuit board containing
only core processing components, such as processor(s), memory, integrated network controllers,
storage drive, and essential I/O cards and ports. Each blade server is a self-contained compute
system and is typically dedicated to a single application. A blade server is housed in a slot inside a
blade enclosure (or chassis), which holds multiple blades and provides integrated power supply,
cooling, networking, and management functions. The blade enclosure enables interconnection of
the blades through a high-speed bus and also provides connectivity to external storage systems.
The modular design of the blade servers makes them smaller, which minimizes the floor space
requirements, increases the compute system density and scalability, and provides better energy
efficiency as compared to the tower and the rack servers. It also reduces the complexity of the
compute infrastructure and simplifies compute infrastructure management. It provides these
benefits without compromising on any capability that a non-blade compute system provides.
Some concerns with blade servers include the high cost of a blade system (blade servers and
chassis), and the proprietary architecture of most blade systems due to which a blade server can
typically be plugged only into a chassis from the same vendor.
28
This lesson covered compute system, and its key physical and logical components. This lesson
also covered the types of compute systems.
29
This lesson covers compute virtualization, hypervisor, and virtual machine. This lesson also covers
application virtualization and its techniques. Further, this lesson covers desktop virtualization and
its techniques.
30
Compute virtualization is a technique of abstracting the physical hardware of a compute system

from the operating system (OS) and applications. The decoupling of the physical hardware
from the OS and applications enables multiple operating systems to run concurrently on a single
or clustered physical compute system(s). Compute virtualization enables the creation of virtual
compute systems called virtual machines (VMs). Each VM runs an OS and applications, and is
isolated from the other VMs on the same compute system. Compute virtualization is achieved by
a hypervisor, which is virtualization software that is installed on a physical compute system. The
hypervisor provides virtual hardware resources, such as CPU, memory, storage, and network
resources to all the VMs. Depending on the hardware capabilities, a large number of VMs can be
created on a single physical compute system.
A VM is a logical entity; but to the OS running on the VM, it appears as a physical compute
system, with its own processor, memory, network controller, and disks. However, all VMs share
the same underlying physical hardware of the compute system and the hypervisor allocates the
compute systems hardware resources dynamically to each VM. From a hypervisors perspective,
each VM is a discrete set of files. This is covered later in this lesson.
31
In an x86-based physical compute system, the software and hardware are tightly coupled and it
can run only one OS at a time. A physical compute system often faces resource conflicts when
multiple applications running on the compute have conflicting requirements. For example,
conflicts may arise when applications need different values in the same registry entry, and
different versions of the same DLL. These issues are further compounded when an application has
high-availability requirements. As a result, a compute system is typically configured to serve only
one application at a time. Therefore organizations purchase and configure new compute systems
for every application they deploy, which is expensive, inflexible, and results in server sprawl and
creation of IT silos. Moreover, many applications do not take full advantage of the hardware
capabilities available to them. Consequently, resources such as processors, memory, and storage
frequently remain underutilized. A large number of compute systems also requires complex
network cabling and considerable floor space and power requirements. Hardware configuration,
provisioning, and management become complex and require more time. A physical compute is a
single point of failure because its failure leads to application unavailability.
Compute virtualization enables to overcome these challenges by allowing multiple operating
systems and applications to run on a single compute system. It converts physical machines to
virtual machines and consolidates the converted machines onto a single compute system. Server
consolidation significantly improves resource utilization and enables organizations to run their
data center with a fewer machines. This, in turn, reduces the hardware acquisition costs and
operational costs, and saves the data center space and energy requirements. Compute
virtualization increases the management efficiency and reduces the maintenance time. The
creation of VMs takes less time compared to a physical compute setup and organizations can
provision compute resources faster, and with greater ease to meet the growing resource
requirements. Individual VMs can be restarted, upgraded, or even crashed, without affecting the
other VMs on the same physical compute. Moreover, VMs are portable and can be copied or
moved from one physical compute to another without causing application unavailability.
32
Hypervisor is compute virtualization software that is installed on a compute system. It provides a

virtualization layer that abstracts the processor, memory, network, and storage of the compute
system and enables the creation of multiple virtual machines. Each VM runs its own OS, which
essentially enables multiple operating systems to run concurrently on the same physical compute
system. The hypervisor provides standardized hardware resources to all the VMs.
A hypervisor has two key components: kernel and virtual machine manager (VMM). A hypervisor
kernel provides the same functionality like the kernel of any OS, including process management,
file system management, and memory management. It is designed and optimized to run multiple
VMs concurrently. It receives requests for resources through the VMM, and presents the requests
to the physical hardware. Each virtual machine is assigned a VMM that gets a share of the
processor, memory, I/O devices, and storage from the physical compute system to successfully
run the VM. The VMM abstracts the physical hardware, and appears as a physical compute system
with processor, memory, I/O devices, and other components that are essential for an OS and
applications to run. The VMM receives resource requests from the VM, which it passes to the
kernel, and presents the virtual hardware to the VM.
Hypervisors are categorized into two types: bare-metal and hosted. A bare-metal hypervisor is
directly installed on the physical compute hardware in the same way as an OS. It has direct
access to the hardware resources of the compute system and is therefore more efficient than a
hosted hypervisor. A bare-metal hypervisor is designed for enterprise data centers and third
platform infrastructure. It also supports the advanced capabilities such as resource management,
high availability, and security. The figure on the slide represents a bare-metal hypervisor. A
hosted hypervisor is installed as an application on an operating system. The hosted hypervisor
does not have direct access to the hardware, and all requests pass through the OS running on the
physical compute system. A hosted hypervisor adds an overhead compared to a bare-metal
hypervisor. This is because there are other processes being executed by the OS that consume
compute resources. Therefore, a hosted hypervisor is more suitable for development, testing, and
training purposes.
33
A virtual machine (VM) is a logical compute system with virtual hardware on which a supported
guest OS and its applications run. A VM is created by a hosted or a bare-metal hypervisor
installed on a physical compute system. An OS, called a guest OS, is installed on the VM in the
same way it is installed on a physical compute system. From the perspective of the guest OS, the
VM appears as a physical compute system. A VM has a self-contained operating environment,
comprising OS, applications, and virtual hardware, such as a virtual processor, virtual memory,
virtual storage, and virtual network resources. As discussed previously, a dedicated virtual
machine manager (VMM) is responsible for the execution of a VM. Each VM has its own
configuration for hardware, software, network, and security. The VM behaves like a physical
compute system, but does not have direct access either to the underlying host OS (when a hosted
hypervisor is used) or to the hardware of the physical compute system on which it is created. The
hypervisor translates the VMs resource requests and maps the virtual hardware of the VM to the
hardware of the physical compute system. For example, a VMs I/O requests to a virtual disk drive
are translated by the hypervisor and mapped to a file on the physical compute systems disk
drive.
Compute virtualization software enables creating and managing several VMseach with a
different OS of its ownon a physical compute system or on a compute cluster. VMs are created
on a compute system, and provisioned to different users to deploy their applications. The VM
hardware and software are configured to meet the applications requirements. The different VMs
are isolated from each other, so that the applications and the services running on one VM do not
interfere with those running on other VMs. The isolation also provides fault tolerance so that if
one VM crashes, the other VMs remain unaffected.
34
When a VM is created, it is presented with virtual hardware components that appear as physical
hardware components to the guest OS. Within a given vendors environment, each VM has
standardized hardware components that make them portable across physical compute systems.
Based on the requirements, the virtual components can be added or removed from a VM.
However, not all components are available for addition and configuration. Some hardware devices
are part of the virtual motherboard and cannot be modified or removed. For example, the video
card and the PCI controllers are available by default and cannot be removed. The figure on the
slide shows the typical hardware components of a VM. This includes virtual processor(s), virtual
motherboard, virtual RAM, virtual disk, virtual network adapter, optical drives, serial and parallel
ports, and peripheral devices.
A VM can be configured with one or more virtual processors. The number of virtual processors in a
VM can be increased or reduced based on the requirements. When a VM starts, its virtual
processors are scheduled by the hypervisor kernel to run on the physical processors. Each VM is
assigned a virtual motherboard with the standardized devices essential for a compute system to
function. Virtual RAM is the amount of physical memory allocated to a VM and it can be configured
based on the requirements. The virtual disk is a large physical file, or a set of files that stores the
VMs OS, program files, application data, and other data associated with the VM. A virtual network
adapter functions like a physical network adapter. It provides connectivity between VMs running
on the same or different compute systems, and between a VM and physical compute systems.
Virtual optical drives and floppy drives can be configured to connect to either physical devices or
to image files, such as ISO and floppy images (.flp), on the storage. SCSI/IDE virtual controllers
provide a way for the VMs to connect to the storage devices. The virtual USB controller is used to
connect to a physical USB controller and to access the connected USB devices. Serial and parallel
ports provide an interface for connecting peripherals to the VM.
35
From a hypervisors perspective, a VM is a discrete set of files on a storage device. Some of the
key files that make up a VM are the configuration file, the virtual disk file, the memory file, and
the log file. The configuration file stores the VMs configuration information, including VM name,
location, BIOS information, guest OS type, virtual disk parameters, number of processors,
memory size, number of adapters and associated MAC addresses, SCSI controller type, and disk
drive type. The virtual disk file stores the contents of a VMs disk drive. A VM can have multiple
virtual disk files, each of which appears as a separate disk drive to the VM. The memory state file
stores the memory contents of a VM and is used to resume a VM that is in a suspended state. The
snapshot file stores the running state of the VM including its settings and the virtual disk, and
may optionally include the memory state of the VM. It is typically used to revert the VM to a
previous state. Log files are used to keep a record about the VMs activity and are often used for
troubleshooting purposes.
For managing VM files, a hypervisor may use a native clustered file system, or the Network File
System (NFS). A hypervisors native clustered file system is optimized to store VM files. It may be
deployed on Fibre Channel and iSCSI storage (covered later in the course), apart from the local
storage. The virtual disks are stored as files on the native clustered file system. Network File
System enables storing VM files on remote file servers (NAS device) accessed over an IP network.
The NFS client built into the hypervisor uses the NFS protocol to communicate with the NAS
device. NAS devices and NFS are covered in Module 6, File-based Storage System (NAS).
36
Application virtualization is the technique of decoupling an application from the underlying

computing platform (OS and hardware) in order to enable the application to be used on a
compute system without installation. In application virtualization, the application is either
deployed on a remote compute system and delivered to a client system over a network, or
encapsulated in a virtualized container along with the assets it requires for execution. In both the
cases, the application can be used without the need to install it on the application users compute
system. Some key benefits of application virtualization are described below.
Simplified application management: Application virtualization provides a solution to meet an
organizations need for simplified and improved application deployment, delivery and
manageability. It reduces system integration and maintenance costs by providing a common
software baseline across multiple diverse end-point devices.
Eliminate OS modifications: Since application virtualization decouples an application from the
OS, it leaves the underlying OS unaltered. This provides additional security, and protects the OS
from potential corruptions and problems that may arise due to changes to the file system and
registry.
Resolve application conflicts and compatibility issues: Application virtualization allows the
use of conflicting applications on the same end-point device. It also enables the use of
applications that otherwise do not execute on an end-point device due to incompatibility with the
underlying computing platform.
Simplified OS image management: Application virtualization simplifies OS image management.
Since application delivery is separated from the OS, there is no need to include "standard"
applications in end-point images. As a result, managing images is simpler, especially in the
context of OS patches and upgrades.
Flexibility of access: Application virtualization enables an organizations workforce and
customers to access applications hosted on a remote compute system from any location, and
through diverse end-point devices types.
37
There are three techniques for application virtualization: application encapsulation, application
presentation, and application streaming.
In application encapsulation, an application is aggregated within a virtualized container, along
with the assets, such as files, virtual registry, and class libraries that it requires for execution.
This process, known as packaging or sequencing, converts an application into a standalone, selfcontained executable package that can directly run on a compute system. The assets required for
execution are included within the virtual container. Therefore, the application does not have any
dependency on the underlying OS, and does not require a traditional installation on the compute
system. The applications virtual container isolates it from the underlying OS and other
applications, thereby minimizing application conflicts. During application execution, all function
calls made by the application to the OS for assets get redirected to the assets within the virtual
container. The application is thus restricted from writing to the OS file system or registry, or
modifying the OS in any other way.
In application presentation, an applications user interface (UI) is separated from its execution.
The application executes on a remote compute system, while its UI is presented to an end-point
client device over a network. When a user accesses the application, the screen pixel information
and the optional sound for the application are transmitted to the client. A software agent installed
on the client receives this information and updates the clients display. The agent also transmits
the keystrokes and graphical input information back from the client, allowing the user to control
the application. This process makes it appear as if the application is running on the client when, in
fact, it is running on the remote compute system. Application presentation enables the delivery of
an application on devices that have less computing power than what is normally required to
execute the application. In application presentation, application sessions are created in the
remote compute system and a user connects to an individual session from a client by means of
the software agent. Individual sessions are isolated from each other, which secures the data of
each user and also protects the application crashes.
(Contd)
38
In application streaming, an application is deployed on a remote compute system, and is

downloaded in portions to an end-point client device for local execution. A user typically
launches the application from a shortcut, which causes the client to connect to the
remote compute system to start the streaming process. Initially, only a limited portion of
the application is downloaded into memory. This portion is sufficient to start the
execution of the application on the client. Since a limited portion of the application is
delivered to the client before the application starts, the user experiences rapid
application launch. The streaming approach also reduces network traffic. As the user
accesses different application functions, more of the application is downloaded to the
client. The additional portions of the application may also be downloaded in the
background without user intervention. Application streaming requires an agent or client
software on clients. Alternatively, the application may be streamed to a web browser by
using a plug-in installed on the client. In some cases, application streaming enables
offline access to the application by caching them locally on the client.
39
With the traditional desktop machine, the OS, applications, and user profiles are all tied to a
specific piece of hardware. With legacy desktops, business productivity is impacted greatly when a
client device is broken or lost. Managing a vast desktop environment is also a challenging task.
Desktop virtualization decouples the OS, applications, and user state (profiles, data, and
settings) from a physical compute system. These components, collectively called a virtual
desktop, are hosted on a remote compute system, and can be accessed by a user from
any client device, such as laptops, desktops, thin clients, or mobile devices. A user accesses
the virtual desktop environment over a network on a client through a web browser or a
client application.
The OS and applications of the virtual desktop execute on the remote compute system,
while a view of the virtual desktops user interface (UI) is presented to the end-point
device. The view of the virtual desktop enables the user to interact with it by using
peripheral devices, such as keyboard and mouse, on the end-point device. Desktop
virtualization uses a remote display protocol to transmit the virtual desktops UI to the
end-point devices. The remote display protocol also sends back key strokes and graphical
input information from the end-point device, enabling the user to interact with the virtual
desktop. Although the virtual desktop runs remotely, the user experience is similar to
using an OS, and applications installed locally on an end-point device.
(Contd)
40
Some key benefits of desktop virtualization are describe below.

Simplified desktop infrastructure management: Desktop virtualization simplifies desktop
infrastructure management, and creates an opportunity to reduce the maintenance costs. New
virtual desktops can be configured and deployed faster than physical machines. The patches,
updates, and upgrades can be centrally applied to the OS and applications. This simplifies or
eliminates many redundant, manual, and time-consuming tasks. Virtual desktops are also based
on standardized images, which make the environment simpler to manage. It is also easier to
diagnose and troubleshoot problems.
Improved data protection and compliance: Applications and data are located
centrally, which ensures that business-critical data is not at risk in case of loss or theft of
the device. Virtual desktops are also easier to back up compared to deploying backup
solutions on end-point devices.
Flexibility of access: Desktop virtualization enables users to access their desktops and
applications without being bound to a specific end-point device. The virtual desktops can
be accessed remotely from different end-point devices. This creates a flexible work
scenario and enables user productivity from remote locations. Desktop virtualization also
enables Bring Your Own Device (BYOD), which creates an opportunity to reduce
acquisition and operational costs.
41
There are two techniques for desktop virtualization: remote desktop services (RDS) and virtual
desktop infrastructure (VDI).
In remote desktop services, the OS and applications are hosted on a remote compute
system and are shared by multiple users. RDS is similar to application presentation, but
provides the capability to deliver virtual desktops rather than individual applications.
Each user has an individual virtual desktop session within which applications execute. A
user typically connects to a remote desktop session by means of client software. To connect to a
remote desktop session, a user specifies the URL of the remote desktop service. When the client
connects to the remote desktop service, a unique session, in which the user can execute
applications, is created. This session provides a complete desktop experience to the user on the
end-point device. The desktops UI is transmitted to the end-point device through a remote
display protocol enabling the user to interact with the desktop and applications. Each
user session is isolated from the sessions of other users, which protects the application
and data. In RDS, users are typically restricted from installing or modifying applications on the
remote OS.
RDS supports a very high user density, as several desktop sessions can be served per processor
core. It is typically used in a scenario where a core set of applications are accessed on-demand by
a group of users. It provides a streamlined and standardized environment for those applications
to multiple users. However, unpredictable application loads may lead to inconsistent performance.
The multi-user environment used in RDS may not also be suitable for all applications. Some
applications may have to be re-written for use in RDS environments.
(Contd)
42
In virtual desktop infrastructure, the OS and applications are hosted on virtual machines running
on a remote compute system. Virtual desktop VMs are typically hosted on a bare-metal
hypervisor. Every user is provided a separate desktop VM with its own OS and applications. The
UI of the OS and applications on the desktop VM is transmitted to an end-point device via a
remote display protocol, which enables the user to interact with the virtual desktop. In VDI, a
desktop VM may be persistent or non-persistent. A persistent desktop VM retains a users data
and settings between logins and is typically dedicated to a specific user. A non-persistent desktop
VM does not retain user data and customizations. It is allocated to a user from a pool of desktop
VMs. When the user logs out, the desktop VM is returned to the desktop VM pool and is available
for reallocation to other users. Desktop VMs can be provisioned from templates. The changes
made to the templates are then automatically propagated to the corresponding desktop VMs
without affecting the user data and the applications.
In VDI, desktop VMs are typically accessed from client devices by means of connection broker
software. A connection broker establishes and manages the connection between an end-point
device and the desktop VM. If desktop VMs are provisioned from a pool, the connection broker
connects the user to an available desktop VM in the pool. The connection broker may authenticate
a desktop request before establishing a connection to the desktop VM. The connection broker may
have capabilities to suspend and resume desktop VMs, based on policies, to enable efficient usage
of resources, such as processor and memory. The connection broker may also support encryption
mechanisms to secure the connection between the end-point device and the desktop VM.
In VDI, desktop VMs are fully isolated from one another. This provides reliability and security. For
persistent desktop VMs, users may be given full administrative and local application installation
privileges in their personal desktop VM. Desktop VMs can be migrated within a cluster without
disruption, thereby enabling efficient manageability and maintenance operations. At the OS level,
a desktop VM is indistinguishable from a physical compute system, which provides greater
software compatibility with VDI as compared to remote desktop technology.
43
Application virtualization and desktop virtualization provide several benefits to organizations and
facilitate the transformation to the third platform. The two use cases of application virtualization
and desktop virtualization are described below.
Cloud application streaming: Cloud application streaming employs application virtualization to
stream applications from the cloud to client devices. Streaming applications from the cloud enable
organizations to reach more users on multiple devices, without modifying the application code
significantly. The application is deployed on a cloud infrastructure, and the output is streamed to
client devices, such as desktops, tablets, and mobile phones. Because the application runs in the
cloud, it can flexibly scale to meet the massive growth in processing and storage needs,
regardless of the client devices the end users are using. The cloud service can stream either all or
portions of the application from the cloud. Cloud application streaming enables an application to
be delivered to client devices on which it may not be possible to run the application natively.
Desktop as a Service: Desktop as a Service (DaaS) is a cloud service in which a virtual desktop
infrastructure (VDI) is hosted by a cloud service provider. The provider offers a complete,
business-ready VDI solution, delivered as a cloud service with either subscription-based or payas-you-go billing. The service provider (internal IT or public) manages the deployment of the
virtual desktops, data storage, backup, security, and OS updates/upgrades. The virtual desktops
are securely hosted in the cloud and managed by the provider. DaaS has a multi-tenant
architecture, wherein virtual desktops of multiple users share the same underlying infrastructure.
However, individual virtual desktops are isolated from each other and protected against
unauthorized access and crashes on other virtual desktops. The virtual desktops can be easily
provisioned by consumers and they are delivered over the Internet to any client device. DaaS
provides organizations with a simple, flexible, and efficient approach to IT. It enables to lower
CAPEX and OPEX for acquiring and managing end-user computing infrastructure.
44
This lesson covered compute virtualization, hypervisor, and virtual machine. This lesson also
covered application virtualization and its techniques. Further, this lesson covered desktop
virtualization and its techniques.
45
This lesson covers evolution of storage architecture and the types of storage devices. This lesson
also covers compute-to-compute and compute-to-storage connectivity. Further, this lesson covers
different storage connectivity protocols.
46
In a traditional environment, business units/departments in an organization have their own

servers running the business applications of the respective business unit/department. Storage
devices are connected directly to the servers and are typically internal to the server. These
storage devices cannot be shared with any other server. This is called server-centric storage
architecture. In this architecture, each server has a limited number of storage devices, and each
storage device exists only in relation to the server to which it is connected. The figure on the slide
depicts an example of server-centric architecture. In the figure, the servers of different
departments in an organization have directly-connected storage and clients connect to the servers
over a local area network (LAN) or a wide area network (WAN).
Traditional server-centric architecture has several limitations, and is therefore inadequate to
satisfy the growing demand for storage capacity in third platform environments. The number of
storage devices that can be connected to one server is limited, and it is not possible to scale the
storage capacity. Moreover, a server cannot directly access the unused storage space available on
other servers. A server failure or any administrative tasks, such as maintenance of the server or
increasing its storage capacity, also results in unavailability of information. Furthermore, the
proliferation of departmental servers in an organization results in silos of information, that are
difficult to manage and lead to an increase in capital expenditure (CAPEX) and operating
expenditure (OPEX).
47
To overcome the challenges of the server-centric architecture, storage evolved to the informationcentric architecture. In information-centric architecture, storage devices exist completely
independently of servers, and are managed centrally and shared between multiple compute
systems. Storage devices assembled within storage systems form a storage pool, and several
compute systems access the same storage pool over a specialized, high-speed storage area
network (SAN). A SAN is used for information exchange between compute systems and storage
systems, and for connecting storage systems. It enables compute systems to share storage
resources, improve the utilization of storage systems, and facilitate centralized storage
management. SANs are classified based on protocols they support. Common SAN deployment
types are Fibre Channel SAN (FC SAN), Internet Protocol SAN (IP SAN), and Fibre Channel over
Ethernet SAN (FCoE SAN). These are covered later in the course.
The figure on the slide depicts an example of information-centric architecture. In the figure, the
servers of different departments in an organization are connected to the shared storage over a
SAN, while clients connect to the servers over a LAN or a WAN. When a new server is deployed in
the environment, storage is assigned to the server from the same shared pool of storage devices.
The storage capacity can be increased dynamically and without impacting information availability
by adding storage devices to the pool. This architecture improves the overall storage capacity
utilization, while making management of information and storage more flexible and cost-effective.
48
A magnetic disk is a circular storage medium made of non-magnetic material (typically an alloy)
and coated with a ferromagnetic material. Data is stored on both surfaces (top and bottom) of a
magnetic disk by polarizing a portion of the disk surface. A disk drive is a device that comprises
multiple rotating magnetic disks, called platters, stacked vertically inside a metal or plastic casing.
Each platter has a rapidly moving arm to read from and write data to the disk. Disk drives are
currently the most popular storage medium for storing and accessing data for performanceintensive applications. Disks support rapid access to random data locations and data can be
written or retrieved quickly for a number of simultaneous users or applications. Disk drives use
pre-defined protocols, such as Advanced Technology Attachment (ATA), Serial ATA (SATA), Small
Computer System Interface (SCSI), Serial Attached SCSI (SAS), and Fibre Channel (FC). These
protocols reside on the disk interface controllers that are typically integrated with the disk drives.
Each protocol has its unique performance, cost, and capacity characteristics.
A solid-state drive (SSD) uses semiconductor-based memory, such as NAND and NOR chips, to
store data. SSDs, also known as flash drives, deliver the ultra-high performance required by
performance-sensitive applications. These devices, unlike conventional mechanical disk drives,
contain no moving parts and therefore do not exhibit the latencies associated with read/write
head movement and disk rotation. Compared to other available storage devices, SSDs deliver a
relatively higher number of input/output operations per second (IOPS) with very low response
times. They also consume less power and typically have a longer lifetime as compared to
mechanical drives. However, flash drives do have the highest cost per gigabyte ($/GB) ratio.
(Contd)
49
A magnetic tape is a thin, long strip of plastic film that is coated with a magnetizable material,
such as barium ferrite. The tape is packed in plastic cassettes and cartridges. A tape drive is the
device to record and retrieve data on a magnetic tape. Tape drives provide linear sequential
read/write data access. A tape drive may be standalone or part of a tape library. A tape library
contains one or more tape drives and a storage area where a number of tape cartridges are held
in slots. Tape is a popular medium for long-term storage due to its relative low cost and
portability. Tape drives are typically used by organizations to store large amounts of data,
typically for backup, offsite archiving, and disaster recovery. The low access speed due to the
sequential access mechanism, the lack of simultaneous access by multiple applications, and the
degradation of the tape surface due to the continuous contact with the read/write head are some
of the key limitations of tape.
An optical disc is a flat, circular storage medium made of polycarbonate with one surface having a
special, reflective coating (such as aluminum). An optical disc drive uses a writing laser to record
data on the disc in the form of microscopic light and dark dots. A reading laser reads the dots,
and generates electrical signals representing the data. The common optical disc types are
compact disc (CD), digital versatile disc (DVD), and Blu-ray disc (BD). These discs may be
recordable or re-writable. Recordable or read-only memory (ROM) discs have Write Once and
Read Many (WORM) capability and are typically used as a distribution medium for applications or
as a means to transfer small amounts of data from one system to another. The limited capacity
and speed of optical discs constrain their use as a general-purpose enterprise data storage
solution. However, high-capacity optical discs are sometimes used as a storage solution for fixedcontent and archival data. Some cloud providers of Storage as a Service offer a facility wherein
they copy backup files on encrypted optical discs, if required, and ship them to the
consumers.
50
Storage virtualization is the technique of abstracting physical storage resources to create virtual
storage resources. Storage virtualization software has the ability to pool and abstract physical
storage resources, and present them as a logical storage resources, such as virtual volumes,
virtual disk files, and virtual storage systems. Storage virtualization software is either built into
the operating environment of a storage system, installed on an independent compute system, or
available as hypervisors capability. Storage virtualization will be covered in detail in the storage
modules of this course.
51
Connectivity refers to the communication paths between IT infrastructure components for

information exchange and resource sharing. The two primary types of connectivity include the
interconnection between compute systems, and between a compute system and storage.
52
Compute-to-compute connectivity typically uses protocols based on the Internet Protocol (IP).
Each physical compute system is connected to a network through one or more host interface
devices, called a network interface controller (NIC). Physical switches and routers are the
commonly-used interconnecting devices. A switch enables different compute systems in the
network to communicate with each other. A router is an OSI Layer-3 device that enables different
networks to communicate with each other. The commonly-used network cables are copper cables
and optical fiber cables. The figure on the slide shows a network (LAN or WAN) that provides
interconnections among the physical compute systems. It is necessary to ensure that appropriate
switches and routers, with adequate bandwidth and ports, are available to provide the required
network performance.
53
The discussion in this lesson focuses on the connectivity between compute systems and storage.
Storage may be connected directly to a compute system or over a SAN as discussed previously in
this lesson. Connectivity and communication between compute and storage are enabled through
physical components and interface protocols. The physical components that connect compute to
storage are host interface device, port, and cable.
Host bus adapter: A host bus adapter (HBA) is a host interface device that connects a compute
system to storage or to a SAN. It is an application-specific integrated circuit (ASIC) board that
performs I/O interface functions between a compute system and storage, relieving the processor
from additional I/O processing workload. A compute system typically contains multiple HBAs.
Port: A port is a specialized outlet that enables connectivity between the compute system and
storage. An HBA may contain one or more ports to connect the compute system to the storage.
Cables connect compute systems to internal or external devices using copper or fiber optic media.
Protocol: A protocol enables communication between the compute system and storage. Protocols
are implemented using interface devices (or controllers) at both the source and the destination
devices. The popular interface protocols used for compute-to-storage communication are
Integrated Device Electronics/Advanced Technology Attachment (IDE/ATA), Small Computer
System Interface (SCSI), Fibre Channel (FC) and Internet Protocol (IP).
54
Integrated Device Electronics (IDE)/Advanced Technology Attachment (ATA) is a popular interface

protocol standard used for connecting storage devices, such as disk drives and optical drives. This
protocol supports parallel transmission and therefore is also known as Parallel ATA (PATA) or
simply ATA. IDE/ATA has a variety of standards and names. The Ultra DMA/133 version of ATA
supports a throughput of 133 MB/s. In a master-slave configuration, an ATA interface supports
two storage devices per connector. However, if the performance of the drive is important, sharing
a port between two devices is not recommended.
The serial version of this protocol supports single bit serial transmission and is known as Serial
ATA (SATA). High performance and low cost SATA has largely replaced PATA in the newer
systems. SATA revision 3.2 provides a data transfer rate up to 16 Gb/s.
SCSI has emerged as a preferred connectivity protocol in high-end compute systems. This
protocol supports parallel transmission and offers improved performance, scalability, and
compatibility compared to ATA. However, the high cost associated with SCSI limits its popularity
among home or personal desktop users. Over the years, SCSI has been enhanced and now
includes a wide variety of related technologies and standards. SCSI supports up to 16 devices on
a single bus and provides data transfer rates up to 640 MB/s (for the Ultra-640 version).
Serial attached SCSI (SAS) is a point-to-point serial protocol that provides an alternative to
parallel SCSI. A newer version (SAS 3.0) of serial SCSI supports a data transfer rate up to 12
Gb/s.
(Contd)
55
Fibre Channel is a widely-used protocol for high-speed communication to the storage device. The
Fibre Channel interface provides gigabit network speed. It provides a serial data transmission that
operates over copper wire and optical fiber. The latest version of the FC interface 16FC allows
transmission of data up to 16 Gb/s. The FC protocol and its features are covered in more detail in
Module 9, Fibre Channel (FC) SAN.
IP is a network protocol that has been traditionally used for compute-to-compute traffic. With the
emergence of new technologies, an IP network has become a viable option for compute-tostorage communication. IP offers several advantages in terms of cost and maturity and enables
organizations to leverage their existing IP-based network. iSCSI and FCIP protocols are common
examples that leverage IP for compute-to-storage communication. These protocols are detailed in
Module 10, Internet Protocol (IP) SAN.
56
Network virtualization is the technique of abstracting physical network resources to create virtual
network resources. Network virtualization software is either built into the operating environment
of a network device, installed on an independent compute system or available as hypervisors
capability. Network virtualization software has the ability to abstract the physical network
resources such as switches and routers to create virtual resources such as virtual switches. It also
has the ability to divide a physical network into multiple virtual networks, such as virtual LANs
and virtual SANs. Network virtualization available as a hypervisors capability can emulate the
network connectivity between virtual machines (VMs) on a physical compute system. It also
enables creating virtual switches that appear to the VMs as physical switches. Network
virtualization will be covered later in Module 9, Fibre Channel (FC) SAN, 10, Internet Protocol
(IP) SAN, and 11, FC over Ethernet (FCoE) SAN of this course.
57
This lesson covered evolution of storage architecture and the types of storage devices. This lesson
also covered compute-to-compute and compute-to-storage connectivity. Further, this lesson
covered different storage connectivity protocols.
58
This lesson covers software-defined data center and its architecture. This lesson also covers
software-defined controller and the benefits of software-defined architecture.
59
Software-defined data center (SDDC) is an architectural approach to IT infrastructure that

extends virtualization concepts such as abstraction, pooling, and automation to all of the data
centers resources and services to achieve IT as a service (ITaaS). In an SDDC, compute, storage,
networking, security, and availability services are pooled, aggregated, and delivered as a service.
SDDC services are managed by intelligent, policy-driven software.
SDDC is a vision that can be interpreted in many ways and can be implemented by
numerous concrete architectures. Typically, an SDDC is viewed as a conglomeration of
virtual infrastructure components, among which are software-defined compute (compute
virtualization), software-defined network (SDN), and software-defined storage (SDS).
SDDC is viewed as an important step in the progress towards a complete virtualized data
center (VDC), and is regarded as the necessary foundational infrastructure for third platform
transformation.
60
The software-defined approach separates the control or management functions from the
underlying components and provides it to external software. The external software takes over the
control operations and enables the management of multi-vendor infrastructure components
centrally.
Principally, a physical infrastructure component (compute, network, and storage) has a control
path and a data path. The control path sets and manages the policies for the resources, and the
data path performs the actual transmission of data. The software-defined approach decouples the
control path from the data path. By abstracting the control path, resource management function
operates at the control layer. This gives the ability to partition the resource pools, and manage
them uniquely by policy. This decoupling of the control path and data path enables the
centralization of data provisioning and management tasks through software that is external to the
infrastructure components. The software runs on a centralized compute system or a standalone
device, called the software-defined controller. The figure on the slide illustrates the softwaredefined architecture, where the management function is abstracted from the underlying
infrastructure components using controller.
61
A software-defined controller is software with built-in intelligence that automates provisioning and
configuration based on the defined policies. It enables organizations to dynamically, uniformly,
and easily modify and manage their infrastructure. The controller discovers the available
underlying resources and provides an aggregated view of resources. It abstracts the
underlying hardware resources (compute, storage, and network) and pools them. This enables the
rapid provisioning of resources from the pool based on pre-defined policies that align to the
service level agreements for different consumers.
The controller provides a single control point to the entire infrastructure enabling policybased infrastructure management. The controller enables an administrator to use a
software interface to manage the resources, node connectivity, and traffic flow; control
behavior of underlying components; apply policies uniformly across the infrastructure
components; and enforce security. The controller also provides interfaces that enable
applications, external to the controller, to request resources and access these resources
as services.
62
By extending virtualization throughout the data center, SDDC provides several benefits
to the organizations. Some key benefits are described below.
Agility: SDDC enables faster provisioning of resources based on workload policies. Consumers
provision infrastructure resources via self-service portal. These significantly improve business
agility.
Cost efficiency: SDDC enables organizations to use commodity hardware and existing
infrastructure, which significantly lowers CAPEX.
Improved control: SDDC provides improved control over application availability and
security through policy-based governance. SDDC provides automated business continuity

and disaster recovery features. It also has virtualization-aware security and compliance
features, and provides support for performing operational analytics.
Centralized management: An SDDC is automated and managed by intelligent, policy-
based data center management software, vastly simplifying governance and operations.
A single, unified management platform allows central monitoring and administration of all
heterogeneous physical and virtual resources across geographies and hybrid clouds.
Flexibility: SDDC enables organizations to use heterogeneous commodity hardware and
the latest advanced hardware technologies as suitable. Lower-value workloads can run on
commodity hardware, while software-based services and mission-critical applications can
run on advanced, more-intelligent infrastructure. SDDC also supports adoption of the
hybrid cloud model through the use of standard protocols and APIs.
63
This lesson covered software-defined data center and its architecture. This lesson also covered
software-defined controller and the benefits of software-defined architecture.
64
The Concepts in Practice section covers VCE Vblock, EMC VSPEX, VMware ESXi, VMware ThinApp,
VMware Horizon, and VMware NSX.
Note:
For the latest information on VCE products, visit www.vce.com.
For the latest information on EMC products, visit www.emc.com.
65
VCE Vblock is a completely integrated IT infrastructure offering from Virtual Computing

Environment Company (VCE), and includes compute, storage, network, and virtualization
products. These products are provided by EMC (storage solution provider), VMware (virtualization
solution provider), and Cisco (networking and compute solution provider), who have formed a
coalition to deliver Vblocks. Vblock is an integrated IT infrastructure solution that combines
compute, storage, network, virtualization, security, and management software into a single
package. This solution is a self-contained unit that accelerates deployment of a data center or a
third platform infrastructure. Vblocks are pre-architected, preconfigured, pretested and have
defined performance and availability attributes. Rather than the customers buying and assembling
the individual IT infrastructure components, Vblock provides a validated solution and is factoryready for deployment and production. This saves significant cost and deployment time associated
with building a data center infrastructure.
EMC VSPEX is an end-to-end virtualized infrastructure solution, which includes compute, storage,
network, virtualization, and backup products. The product vendors include EMC, Brocade, Cisco,
Citrix, Intel, Microsoft, and VMware. VSPEX offers choice to the customers in terms of the
hypervisor, compute systems, and networking components. Therefore, customers have the
flexibility to choose the infrastructure components that fit their existing IT infrastructures. EMC
VSPEX is a complete virtualization solution that accelerates the deployment of data center
infrastructures. It provides customers the flexibility to choose the hypervisor, compute system,
and network technology they prefer along with EMCs VNX and VNXe unified storage, and EMCs
backup and recovery solutions. Regardless of customers choice of hypervisor, compute system,
and network technologies, validation of VSPEX ensures fast and low-risk deployment. VSPEX
significantly reduces the planning, sizing, and configuration burdens that typically come with
designing, integrating, and deploying a best-of-breed solution. VSPEX, unlike Vblock, does not
offer unified management. It comes with element management tools such as Microsoft System
Center, VMware vCenter Operations Management Suite, and EMC Unisphere. But, it offers
customers the choice of service elements that make up the solution.
66
VMware ESXi is a bare-metal hypervisor. ESXi has a compact architecture that is designed for
integration directly into virtualization-optimized compute system hardware, enabling rapid
installation, configuration, and deployment. ESXi abstracts processor, memory, storage, and
network resources into multiple VMs that run unmodified operating systems and applications. The
ESXi architecture comprises underlying operating system called VMkernel, that provides a means
to run management applications and VMs. VMkernel controls all hardware resources on the
compute system and manages resources for the applications. It provides core OS functionality,
such as process management, file system, resource scheduling, and device drivers.
VMware ThinApp is an application virtualization solution. ThinApp encapsulates an application,
along with the assets it requires, into a single package that can be deployed, managed, and
updated independently from the underlying OS. A ThinApp application is a single executable file
that can be directly executed on a compute system. The application package can be distributed
across an organizations environment from a centralized location, such as a cloud. ThinApp
applications can be executed in two modes: deployed mode or streaming mode. In either case,
agent software is not required on the end-point device. In deployed mode, ThinApp packages are
copied to the compute system, which then executes the package locally. In deployed mode, an
application can run regardless of the availability of the network connectivity. In streaming mode,
the application is streamed from the centralized location, over a network, to the compute system.
67
VMware Horizon is a VDI solution for delivering virtualized or hosted desktops and applications
through a single platform to the end users. These desktop and application servicesincluding
RDS, hosted apps, packaged apps with VMware ThinApp, and SaaS appscan all be accessed
from one unified workspace across devices and locations. Horizon provides IT with a streamlined
approach to deliver, protect, and manage desktops and applications while containing costs and
ensuring that end users can work anytime, anywhere, on any device. Horizon supports both
Windows as well as Linux-based desktops.
VMware NSX is a network virtualization platform for the Software-defined Data Center (SDDC).
Similar to virtual machines for compute, virtual networks are programmatically provisioned and
managed independent of underlying hardware. NSX reproduces the entire network model in
software, enabling any network topologyfrom simple to complex multi-tier networksto be
created and provisioned. It enables a library of logical networking elements and services, such as
logical switches, routers, firewalls, load balancers, VPN, and workload security. Users can create
isolated virtual networks through custom combinations of these capabilities. NSX is ideal for data
centers with more than 500 virtual machines, for multi-tenant clouds, large enterprise private and
R&D clouds, and multi-hypervisor cloud environments.
68
This module covered the building blocks of a data center environment. This module also
covered compute system, its components, and its types. Additionally, this module
covered compute virtualization, application virtualization, and desktop virtualization.
Further, this module covered an overview of storage and connectivity in a data center.
Finally, this module covered an overview of software-defined data center.
69
70
This module focuses on the key components of an intelligent storage system. This module also
focuses on storage subsystems and provides details on components, addressing, and performance
parameters of a hard disk drive (HDD) and solid state drive (SSD). Then, this module focuses on
RAID and its use to improve performance and protection. It focuses on various RAID
implementations, techniques, and levels commonly used. Further, this module focuses on the
impact of RAID on performance and compares the commonly used RAID levels. Finally, this
module focuses on the types of intelligent storage systems and their architectures.
Module 4: Intelligent Storage Systems (ISS)
The slide provides a list of key requirements for an effective storage infrastructure.
The slide lists technology solutions that can meet the third platform requirements for the storage
infrastructure.
This lesson covers components of intelligent storage systems. This lesson also covers
components, addressing, and performance of hard disk drives.
Intelligent storage systems are feature-rich RAID arrays that provide highly optimized I/O
processing capabilities. These intelligent storage systems have the capability to meet the
requirements of todays I/O intensive third platform applications. These applications require high
levels of performance, availability, security, and scalability. Therefore, to meet the requirements
of the applications many vendors of intelligent storage systems now support SSDs, encryption,
compression, deduplication, and scale-out architecture. The use of SSDs and scale-out
architecture enable to service massive number of IOPS. These storage systems also support
connectivity to heterogeneous compute systems.
The storage systems have an operating environment that intelligently and optimally handles the
management, provisioning, and utilization of storage resources. The storage systems are
configured with a large amount of memory (called cache) and multiple I/O paths and use
sophisticated algorithms to meet the requirements of performance-sensitive applications.
The storage systems also support various technologies, such as deduplication, compression,
encryption, automated storage tiering, and virtual storage provisioning. These capabilities have
added a new dimension to storage system performance, scalability, and availability. Further, the
intelligent storage systems support APIs to enable integration with SDDC and cloud
environments.
An intelligent storage system has two key components, controller and storage. A controller is a
compute system that runs a purpose-built operating system that is responsible for performing
several key functions for the storage system. Examples of such functions are serving I/Os from
the application servers, storage management, RAID protection, local and remote replication,
provisioning storage, automated tiering, data compression, data encryption, and intelligent cache
management.
An intelligent storage system typically has more than one controller for redundancy. Each
controller consists of one or more processors and a certain amount of cache memory to process a
large number of I/O requests. These controllers are connected to the compute system either
directly or via a storage network. The controllers receive I/O requests from the compute systems
that are read or written from/to the storage by the controller. Depending on the type of the data
access method (discussed later in this module) used for a storage system, the controller can
either be classified as block-based, file-based, object-based, or unified.
An intelligent storage system can have all hard disk drives, all solid state drives, or a combination
of both.
A hard disk drive is a persistent storage device that stores and retrieves data using rapidly
rotating disks (platters) coated with magnetic material. The key components of a hard disk drive
(HDD) are platter, spindle, read-write head, actuator arm assembly, and controller board. I/O
operations in an HDD are performed by rapidly moving the arm across the rotating flat platters
coated with magnetic material. Data is transferred between the disk controller and magnetic
platters through the read-write (R/W) head which is attached to the arm. Data can be recorded
and erased on magnetic platters any number of times.
Platter: A typical HDD consists of one or more flat circular disks called platters. The data is
recorded on these platters in binary codes (0s and 1s). The set of rotating platters is sealed in
a case, called Head Disk Assembly (HDA). A platter is a rigid, round disk coated with magnetic
material on both surfaces (top and bottom). The data is encoded by polarizing the magnetic
area or domains of the disk surface. Data can be written to or read from both surfaces of the
platter. The number of platters and the storage capacity of each platter determine the total
capacity of the drive.
Spindle: A spindle connects all the platters and is connected to a motor. The motor of the
spindle rotates with a constant speed. The disk platter spins at a speed of several thousands of
revolutions per minute (rpm). Common spindle speeds are 5,400 rpm, 7,200 rpm, 10,000 rpm,
and 15,000 rpm. The speed of the platter increases with the improvement in technology;
although the extent to which it can be improved is limited.
(Contd)
Read/write head: Read/write (R/W) heads, read and write data from or to the platters.
Drives have two R/W heads per platter, one for each surface of the platter. The R/W head
changes the magnetic polarization on the surface of the platter when writing data. While
reading data, the head detects the magnetic polarization on the surface of the platter. During
reads and writes, the R/W head senses the magnetic polarization and never touches the
surface of the platter. When the spindle rotates, a microscopic air gap is maintained between
the R/W heads and the platters, known as the head flying height. This air gap is removed when
the spindle stops rotating and the R/W head rests on a special area on the platter near the
spindle. This area is called the landing zone. The landing zone is coated with a lubricant to
reduce friction between the head and the platter. The logic on the disk drive ensures that
heads are moved to the landing zone before they touch the surface. If the drive malfunctions
and the R/W head accidentally touches the surface of the platter outside the landing zone, a
head crash occurs. In a head crash, the magnetic coating on the platter is scratched and may
cause damage to the R/W head. A head crash generally results in data loss.
Actuator arm assembly: R/W heads are mounted on the actuator arm assembly, which
positions the R/W head at the location on the platter where the data needs to be written or
read. The R/W heads for all platters on a drive are attached to one actuator arm assembly and
move across the platters simultaneously.
Drive controller board: The controller is a printed circuit board, mounted at the bottom of a
disk drive. It consists of a microprocessor, internal memory, circuitry, and firmware. The
firmware controls the power supplied to the spindle motor as well as controls the speed of the
motor. It also manages the communication between the drive and the compute system. In
addition, it controls the R/W operations by moving the actuator arm and switching between
different R/W heads, and performs the optimization of data access.
Data on the disk is recorded on tracks, which are concentric rings on the platter around the
spindle. The tracks are numbered, starting from zero, from the outer edge of the platter. The
number of tracks per inch (TPI) on the platter (or the track density) measures how tightly the
tracks are packed on a platter.
Each track is divided into smaller units called sectors. A sector is the smallest, individually
addressable unit of storage. The track and sector structure is written on the platter by the drive
manufacturer using a low-level formatting operation. The number of sectors per track varies
according to the drive type. There can be thousands of tracks on a platter, depending on the
physical dimensions and the recording density of the platter.
Typically, a sector holds 512 bytes of user data; although some disks can be formatted with larger
sector sizes. In addition to user data, a sector also stores other information, such as the sector
number, head number or platter number, and track number. This information helps the controller
to locate the data on the drive.
A cylinder is a set of identical tracks on both surfaces of each drive platter. The location of R/W
heads is referred to by the cylinder number, not by the track number.
The earlier drives used physical addresses consisting of cylinder, head, and sector (CHS) number
to refer to specific locations on the disk, and the OS had to be aware of the geometry of each disk
used. Logical block addressing (LBA) has simplified the addressing by using a linear address to
access physical blocks of data. The disk controller translates LBA to a CHS address, and the
compute system needs to know only the size of the disk drive in terms of the number of blocks.
The logical blocks are mapped to physical sectors on a 1:1 basis.
In the slide, the drive shows eight sectors per track, six heads, and four cylinders. This means a
total of 8 6 4 = 192 blocks; so the block number ranges from 0 to 191. Each block has its own
unique address.
Assuming that the sector holds 512 bytes, a 500 GB drive with a formatted capacity of 465.7 GB
has in excess of 976,000,000 blocks.
10
A disk drive is an electromechanical device that governs the overall performance of the storage
system environment. The various factors that affect the performance of disk drives are:
Seek time
Rotational latency
Data transfer rate
11
The seek time (also called access time) describes the time taken to position the R/W
heads across the platter with a radial movement (moving along the radius of the platter).
In other words, it is the time taken to position and settle the arm and the head over the
correct track. Therefore, the lower the seek time, the faster the I/O operation. Disk
vendors publish the following seek time specifications:
Full Stroke: It is the time taken by the R/W head to move across the entire width of
the disk, from the innermost track to the outermost track.
Average: It is the average time taken by the R/W head to move from one random
track to another, normally listed as the time for one-third of a full stroke.
Track-to-Track: It is the time taken by the R/W head to move between adjacent
tracks.
Each of these specifications is measured in milliseconds (ms). The seek time of a disk is
typically specified by the drive manufacturer. The average seek time on a modern disk is
typically in the range of 3 to 15 ms. Seek time has more impact on the I/O operation of
random tracks rather than the adjacent tracks. To minimize the seek time, data can be
written to only a subset of the available cylinders. This results in lower usable capacity
than the actual capacity of the drive. For example, a 500 GB disk drive is set up to use
only the first 40 percent of the cylinders and is effectively treated as a 200 GB drive. This
is known as short-stroking the drive.
12
To access data, the actuator arm moves the R/W head over the platter to a particular track while
the platter spins to position the requested sector under the R/W head. The time taken by the
platter to rotate and position the data under the R/W head is called rotational latency. This
latency depends on the rotation speed of the spindle and is measured in milliseconds. The
average rotational latency is one-half of the time taken for a full rotation. Similar to the seek
time, rotational latency has more impact on the reading/writing of random sectors on the disk
than on the same operations on adjacent sectors.
Average rotational latency is approximately 5.5 ms for a 5,400-rpm drive, and around 2 ms for a
15,000-rpm drive as shown below.
Average rotational latency for 15K rpm (30000/15000) drive is =
2 ms
13
The data transfer rate (also called transfer rate) refers to the average amount of data per unit
time that the drive can deliver to the HBA. In a read operation, the data first moves from disk
platters to R/W heads; then it moves to the drives internal buffer. Finally, data moves from the
buffer through the interface to the compute systems HBA. In a write operation, the data moves
from the HBA to the internal buffer of the disk drive through the drives interface. The data then
moves from the buffer to the R/W heads. Finally, it moves from the R/W heads to the platters.
The data transfer rates during the R/W operations are measured in terms of internal and external
transfer rates, as shown on the slide.
Internal transfer rate is the speed at which data moves from a platters surface to the internal
buffer (cache) of the disk. The internal transfer rate takes into account factors such as the seek
time and rotational latency. External transfer rate is the rate at which data can move through the
interface to the HBA. The external transfer rate is generally the advertised speed of the interface,
such as 133 MB/s for ATA.
14
The utilization of a disk I/O controller has a significant impact on the I/O response time. Consider
that a disk is viewed as a black box consisting of two elements: the queue and the disk I/O
controller. Queue is the location where an I/O request waits before it is processed by the I/O
controller and disk I/O controller processes I/Os waiting in the queue one by one.
The I/O requests arrive at the controller at the rate generated by the application. The I/O arrival
rate, the queue length, and the time taken by the I/O controller to process each request
determines the I/O response time. If the controller is busy or heavily utilized, the queue size will
be large and the response time will be high. Based on the fundamental laws of disk drive
performance, the relationship between controller utilization and average response time is given
as:
Average response time = Service time/(1 Utilization)
where, service time is the time taken by the controller to serve an I/O.
As the utilization reaches 100 percent, that is, as the I/O controller saturates, the response time
moves closer to infinity. In essence, the saturated component or the bottleneck forces the
serialization of I/O requests; meaning, each I/O request must wait for the completion of the I/O
requests that preceded it. The figure on the slide shows a graph plotted between utilization and
response time. The graph indicates that as the utilization increases, the response time changes
are nonlinear. When the average queue sizes are low, the response time remains low. The
response time increases slowly with added load on the queue and increases exponentially when
the utilization exceeds 70 percent. Therefore, for performance-sensitive applications, it is
common to utilize disks below their 70 percent of I/O serving capability.
15
Determining storage requirements for an application begins with determining the required storage
capacity and I/O performance. Capacity can be easily estimated by the size and number of file
systems and database components used by applications. The I/O size, I/O characteristics, and the
number of I/Os generated by the application at peak workload are other factors that affect
performance, I/O response time and design of storage system.
The disk service time (TS) for an I/O is a key measure of disk performance; TS, along with disk
utilization rate (U), determines the I/O response time for an application. As discussed earlier the
total disk service time is the sum of the seek time, rotational latency, and transfer time.
Note that transfer time is calculated based on the block size of the I/O and given data transfer
rate of a disk drive. For example, for an I/O with a block size of 32 KB and given disk data
transfer rate of 40MB/s; the transfer time will be 32 KB/40 MB.
TS determines the time taken by the I/O controller to serve an I/O, therefore, the maximum
number of I/Os serviced per second or IOPS is (1/TS).
The IOPS calculated above represents the IOPS that can be achieved at potentially high levels of
I/O controller utilization (close to 100 percent). If the application demands a faster response time,
then the utilization for the disks should be maintained below 70 percent.
Based on this discussion, the total number of disks required for an application is computed as:
Max (Disks required for meeting capacity, Disks required for meeting performance)
(Contd)
16
Consider an example in which the capacity requirement for an application is 1.46 TB. The number
of IOPS generated by the application at peak workload is estimated at 9,000 IOPS. The vendor
specifies that a 146 GB, 15,000-rpm drive is capable of doing a maximum of 180 IOPS.
In this example, the number of disks required to meet the capacity requirements will be 1.46
TB/146 GB = 10 disks.
To meet the application IOPS requirements, the number of disks required is 9,000 / 180 = 50.
However, if the application is response-time sensitive, the number of IOPS a disk drive can
perform should be calculated based on 70 percent disk utilization. Considering this, the number of
IOPS a disk can perform at 70 percent utilization is 180 x 0.7 = 126 IOPS. Therefore, the number
of disks required to meet the application IOPS requirement will be 9,000/126 = 72.
As a result, the number of disks required to meet the application requirements will be Max (10,
72) = 72 disks.
The preceding example indicates that from a capacity perspective, 10 disks are sufficient;
however, the number of disks required to meet application performance is 72. To optimize disk
requirements from a performance perspective, various solutions are deployed in a real-time
environment. Examples of these solutions are disk native command queuing, use of flash drives,
RAID, and the use of cache memory. RAID and cache are detailed in module 5, Block-based
Storage System.
17
This lesson covered the components of intelligent storage systems. This lesson also covered the
components, addressing, and performance of HDDs.
18
This lesson covers components, addressing, and performance of solid state drives.
19
Solid state drives (SSDs) are storage devices that contain non-volatile flash memory. Solid state
drives are superior to mechanical hard disk drives in terms of performance, power use, and
availability. These drives are especially well suited for low-latency applications that require
consistent, low (less than 1 ms) read/write response times. In a HDD servicing, small-block,
highly-concurrent, random workloads involve considerable rotational and seek latency, which
significantly reduces throughput.
Externally solid state drives have the same physical format and connectors as mechanical
hard disk drives. This maintains the compatibility in form and format with mechanical
hard disk drives, and allows easy replacement of a mechanical drive with a solid state
drive. Internally, a solid state drives hardware architecture consists of the following
components: I/O interface, controller, and mass storage.
The I/O interface enables connecting the power and data connectors to the solid state
drives. SSDs typically support standard connectors such as SATA, SAS, or FC.
The controller includes a drive controller, RAM, and non-volatile memory (NVRAM). The drive
controller manages all drive functions. The SSDs include many features such as encryption and
write coalescing. The non-volatile RAM (NVRAM) is used to store the SSDs operational software
and data. Not all SSDs have separate NVRAM. Some models store their programs and data to the
drives mass storage. The RAM is used in the management of data being read and written from
the SSD as a cache, and for the SSDs operational programs and data. The portion of the drives
RAM used for controller cache enhances the overall performance of the SSD. Mass storage, which
is made of flash memories, writes slower than it reads. The drives RAM is used to minimize the
number of writes to mass storage and improve the response time of the drive.
(Contd)
20
Write coalescing is one of the techniques employed within the RAM. This is the process of
grouping write I/Os and writing them in a single internal operation versus many smaller-sized
write operations. In addition to caching, the RAM contains the drive controllers operational
software and mapping tables. Mapping tables correlate the internal data structure of the SSD to
the file system data structure of the compute system.
The mass storage is an array of non-volatile memory chips. They retain their contents when
powered off. These chips are commonly called Flash memory. The number and capacity of the
individual chips vary directly in relationship to the SSDs capacity. The larger the capacity of the
SSD, the larger is the capacity and the greater is the number of the Flash memory chips.
The Flash memory chips that make up the drives mass storage come from numerous
manufacturers. Two types of Flash memory chip are used in commercially available SSDs: SingleLevel Cell (SLC) and Multi-Level Cell (MLC). SLC-type Flash is typically used in enterprise-rated
SSDs for its increased memory speed and longevity. MLC is slower but has the advantage of
greater capacity per chip. Although SLC type Flash memory offers a lower density, it also provides
a higher level of performance in the form of faster reads and writes. In addition, SLC Flash
memory has higher reliability. As SLC Flash memory stores only one bit per cell, the likelihood for
error is reduced. SLC also allows for higher write/erase cycle endurance. For these reasons, SLC
Flash memory is preferred for use in applications requiring higher reliability, and increased
endurance and viability in multi-year product life cycles.
SSDs consume less power compared to hard disk drives. Because SSDs do not have moving
parts, they generate less heat compared to HDDs. Therefore, it further reduces the need for
cooling in storage enclosure, which further reduces the overall system power consumption.
SSDs have multiple parallel I/O channels from its drive controller to the flash memory storage
chips. Generally, the larger the number of flash memory chips in the drive, the larger is the
number of channels. The larger the number of channels, the greater is the SSDs internal
bandwidth. The drives controller uses native command queuing to efficiently distribute read and
write operations across all available channels. Bandwidth performance scales upward with parallel
use of all available channels. Note that the drives with the same capacity, but from different
vendors, can have a different number of channels. These drives will have different levels of
performance. The drive with more channels will outperform the drive with a fewer under some
circumstances.
21
Solid state memory chips have different capacities, for example a solid state memory chip
can be 32 GB or 4 GB per chip. However, all memory chips share the same logical
organization, that is pages and blocks.
At the lowest level, a solid state drive stores bits. Eight bits make up a byte, and while on the
typical mechanical hard drive 512 bytes would make up a sector, solid state drives do not have
sectors. Solid state drives have a similar physical data object called a page. Like a mechanical
hard drive sector, the page is the smallest object that can be read or written on a solid state
drive. Unlike mechanical hard drives, pages do not have a standard capacity. A pages capacity
depends on the architecture of the solid state memory chip. Typical page capacities are 4 KB, 8
KB, and 16 KB.
A solid state drive block is made up of pages. A block may have 32, 64, or 128 pages. 32
is a common block size. The total capacity of a block is dependent on the solid state chips
page size. Only entire blocks may be written or erased on a solid state memory chip.
Individual pages may be read or invalidated (a logical function). For a block to be
written, pages are assembled into full blocks in the solid state drives cache RAM and then
written to the block storage object.
22
A page has three possible states, erased (empty), valid, and invalid. In order to write any
data to a page, its owning block location on the flash memory chip must be electrically
erased. This function is performed by the SSDs hardware. Once a page has been erased,
new data can be written to it. For example, when a 4 KB of data is written to a 4 KB
capacity page, the state of that page is changed to valid, as it is holding valid data. A
valid pages data can be read any number of times. If the drive receives a write request
to the valid page, the page is marked invalid and that write goes to another page. This is
because erasing blocks is time consuming and may increase the response time. Once a page is
marked invalid, its data can no longer be read. An invalid page needs to be erased before
it can once again be written with new data. Garbage collection handles this process.
Garage collection is the process of providing new erased blocks.
A block has three possible states, erased (empty), new, and used. Once a block is
erased, a blocks number of pages that have been assembled in the SSDs RAM may be
written to it. For example, thirty two 4 KB pages may be assembled into a block, and then
written to the erased block. This sets the blocks state to new, meaning it is holding
pages with valid data. A blocks valid pages can be read any number of times. There are
two mechanisms to invalidate a page, writes and deletes. If the drive receives a write
request to a valid block page, the page must be changed. The current page containing
the destination of the write is marked invalid. The blocks state changes to used,
because it contains invalid pages. These writes go to another page, on an erased block. A
delete invalidates a page without resulting in a subsequent write.
23
Solid state drives are semiconductor, random-access devices; these result in very low response
times compared to hard disk drives. This, combined with the multiple parallel I/O channels on the
back end, gives SSDs performance characteristics that are better than HDDs.
SSD performance is dependent on access type, drive state, and workload duration. SSD performs
random reads the best. In carefully tuned multi-threaded, small-block random I/O workload
storage environments, SSDs can deliver much lower response times and higher throughput than
HDDs. This is because random-read I/Os cannot usually be serviced by read-ahead algorithms on
a HDD or by read cache on the storage system. The latency of a random read operation is directly
related to the seek time of a HDD. For HDDs, this is the physical movement of the drives
read/write head to access the desired area. Because they are random access devices, SSDs pay
no penalty for retrieving I/O that is stored in more than one area; as a result their response time
is in an order of magnitude faster than the response time of HDDs.
For large block I/Os, SSDs tend to use all internal I/O channels in parallel. Since the singlethreaded sequential I/O streams on FC HDDs do not suffer seek and rotational latencies (because
of the storage system cache), single-threaded large-block sequential I/O streams will not show
major performance improvements with SSDs over FC HDDs. However, with the increased
application concurrency (as more threads are added), the load starts to resemble a large blockrandom workload. In this case, seek and rotational latencies are introduced that decrease the FC
HDD effectiveness but do not decrease SSD effectiveness.
A new SSD or an SSD with substantial unused capacity has the best performance. Drives with
substantial amounts of their capacity consumed will take longer to complete the read-modifywrite cycle. SSDs are best for workloads with short bursts of activity.
24
This lesson covered the components, addressing, and performance of solid state drives.
25
This lesson covers RAID and its use to improve performance and protection. It covers various
RAID implementations, techniques, and levels commonly used. This lesson also covers the impact
of RAID on performance and compares the commonly used RAID levels.
26
Redundant Array of Independent Disks (RAID) is a technique in which multiple disk drives are
combined into a logical unit called a RAID set and data is written in blocks across the disks in the
RAID set. RAID protects against data loss when a drive fails, through the use of redundant drives
and parity. RAID also helps in improving the storage system performance as read and write
operations are served simultaneously from multiple disk drives.
RAID is typically implemented by using a specialized hardware controller present either on the
compute system or on the storage system. The key functions of a RAID controller are
management and control of drive aggregations, translation of I/O requests between logical and
physical drives, and data regeneration in the event of drive failures.
(Contd)
27
There are two methods of RAID implementation, hardware and software. Both have their
advantages and disadvantages.
Software RAID uses compute system-based software to provide RAID functions and is
implemented at the operating-system level. Software RAID implementations offer cost and
simplicity benefits when compared with hardware RAID. However, they have the following
limitations:
Performance: Software RAID affects the overall system performance. This is due to
additional CPU cycles required to perform RAID calculations.
Supported features: Software RAID does not support all RAID levels.
Operating system compatibility: Software RAID is tied to the operating system; hence,
upgrades to software RAID or to the operating system should be validated for compatibility.
This leads to inflexibility in the data-processing environment.
In hardware RAID implementations, a specialized hardware controller is implemented either on

the compute system or on the storage system. Controller card RAID is a compute system-based
hardware RAID implementation in which a specialized RAID controller is installed in the compute
system, and disk drives are connected to it. Manufacturers also integrate RAID controllers on
motherboards. A compute system-based RAID controller is not an efficient solution in a data
center environment with a large number of compute systems. The external RAID controller is a
storage system-based hardware RAID. It acts as an interface between the compute system and
the disks. It presents storage volumes to the compute system, and the compute system manages
these volumes as physical drives. The key functions of the RAID controllers are as follows:
Management and control of disk aggregations
Translation of I/O requests between logical disks and physical disks
Data regeneration in the event of disk failures
28
A RAID array is an enclosure that contains a number of disk drives and supporting hardware to
implement RAID. A subset of disks within a RAID array can be grouped to form logical
associations called logical arrays, also known as a RAID set or a RAID group.
29
The three different RAID techniques that form the basis for defining various RAID levels are
striping, mirroring, and parity. These techniques determine the data availability and performance
of a RAID set as well as the relative cost of deploying a RAID level.
Striping is a technique of spreading data across multiple drives (more than one) in order to use
the drives in parallel. All the read-write heads work simultaneously, allowing more data to be
processed in a shorter time and increasing performance, compared to reading and writing from a
single disk. Within each disk in a RAID set, a predefined number of contiguously addressable disk
blocks are defined as strip. The set of aligned strips that spans across all the disks within the
RAID set is called a stripe. The figure 1 on the slide shows representations of a striped RAID set.
Strip size (also called stripe depth) describes the number of blocks in a strip (represented as A1,
A2, A3, and A4), and is the maximum amount of data that can be written to or read from a
single disk in the set, assuming that the accessed data starts at the beginning of the strip. All
strips in a stripe have the same number of blocks. Having a smaller strip size means that the data
is broken into smaller pieces while it is spread across the disks. Stripe size (represented as A) is a
multiple of strip size by the number of data disks in the RAID set. For example, in a four disk
striped RAID set with a strip size of 64KB, the stripe size is 256 KB (64KB x 4). In other words, A
= A1 +A2 + A3 + A4. Stripe width refers to the number of data strips in a stripe. Striped RAID
does not provide any data protection unless parity or mirroring is used.
(Contd)
30
Mirroring is a technique whereby the same data is stored on two different disk drives, yielding two
copies of the data. If one disk drive failure occurs, the data remains intact on the surviving disk
drive and the controller continues to service the compute systems data requests from the
surviving disk of a mirrored pair. When the failed disk is replaced with a new disk, the controller
copies the data from the surviving disk of the mirrored pair. This activity is transparent to the
compute system. In addition to providing complete data redundancy, mirroring enables fast
recovery from disk failure. However, disk mirroring provides only data protection and is not a
substitute for data backup. Mirroring constantly captures changes in the data, whereas a backup
captures point-in-time images of the data. Mirroring involves duplication of data the amount of
storage capacity needed is twice the amount of data being stored. Therefore, mirroring is
considered expensive and is preferred for mission-critical applications that cannot afford the risk
of any data loss. Mirroring improves read performance because read requests can be serviced by
both disks. However, write performance is slightly lower than that in a single disk because each
write request manifests as two writes on the disk drives. Mirroring does not deliver the same
levels of write performance as a striped RAID.
Parity is a method to protect striped data from disk drive failure without the cost of mirroring. An
additional disk drive is added to hold parity, a mathematical construct that allows re-creation of
the missing data. Parity is a redundancy technique that ensures protection of data without
maintaining a full set of duplicate data. Calculation of parity is a function of the RAID controller.
Parity information can be stored on separate, dedicated disk drives, or distributed across all the
drives in a RAID set. The first three disks in the figure, labeled D1 to D3, contain the data. The
fourth disk, labeled P, stores the parity information, which, in this case, is the sum of the
elements in each row. Now, if one of the data disks fails, the missing value can be calculated by
subtracting the sum of the rest of the elements from the parity value. Here, for simplicity, the
computation of parity is represented as an arithmetic sum of the data. However, parity calculation
is a bitwise XOR operation.
Compared to mirroring, parity implementation considerably reduces the cost associated with data
protection. Consider an example of a parity RAID configuration with four disks where three disks
hold data, and the fourth holds the parity information. In this example, parity requires only 33
percent extra disk space compared to mirroring, which requires 100 percent extra disk space.
However, there are some disadvantages of using parity. Parity information is generated from data
on the data disk. Therefore, parity is recalculated every time there is a change in data. This
recalculation is time-consuming and affects the performance of the RAID array.
For parity RAID, the stripe size calculation does not include the parity strip. For example in a four
(3 + 1) disk parity RAID set with a strip size of 64 KB, the stripe size will be 192 KB (64KB x 3).
31
The RAID level selection depends on the parameters such as application performance, data
availability requirements, and cost. These RAID levels are defined on the basis of striping,
mirroring, and parity techniques. Some RAID levels use a single technique, whereas others use a
combination of techniques. The commonly used RAID levels are listed on the slide.
32
RAID 0 configuration uses data striping techniques, where data is striped across all the disks
within a RAID set. Therefore it utilizes the full storage capacity of a RAID set. To read data, all the
strips are gathered by the controller. When the number of drives in the RAID set increases, the
performance improves because more data can be read or written simultaneously. RAID 0 is a
good option for applications that need high I/O throughput. However, if these applications require
high availability during drive failures, RAID 0 does not provide data protection and availability.
33
RAID 1 is based on the mirroring technique. In this RAID configuration, data is mirrored to
provide fault tolerance. A RAID 1 set consists of two disk drives and every write is written to both
disks. The mirroring is transparent to the compute system. During disk failure, the impact on data
recovery in RAID 1 is the least among all RAID implementations. This is because the RAID
controller uses the mirror drive for data recovery. RAID 1 is suitable for applications that require
high availability and cost is not a constraint.
34
Most data centers require data redundancy and performance from their RAID arrays. RAID 1+0
combines the performance benefits of RAID 0 with the redundancy benefits of RAID 1. It uses
mirroring and striping techniques and combines their benefits. This RAID type requires an even
number of disks, the minimum being four.
RAID 1+0 is also known as RAID 10 (Ten) or RAID 1/0. RAID 1+0 is also called striped mirror.
The basic element of RAID 1+0 is a mirrored pair, which means that data is first mirrored and
then both copies of the data are striped across multiple disk drive pairs in a RAID set. When
replacing a failed drive, only the mirror is rebuilt. In other words, the storage system controller
uses the surviving drive in the mirrored pair for data recovery and continuous operation. Data
from the surviving disk is copied to the replacement disk.
35
RAID 3 stripes data for performance and uses parity for fault tolerance. Parity information is
stored on a dedicated drive so that the data can be reconstructed if a drive fails in a RAID set. For
example, in a set of five disks, four are used for data and one for parity. Therefore, the total disk
space required is 1.25 times the size of the data disks. RAID 3 always reads and writes complete
stripes of data across all disks because the drives operate in parallel. There are no partial writes
that update one out of many strips in a stripe.
36
RAID 5 is a versatile RAID implementation. It is similar to RAID 4 because it uses striping. The
drives (strips) are also independently accessible. The difference between RAID 4 and RAID 5 is
the parity location. In RAID 4, parity is written to a dedicated drive, creating a write bottleneck
for the parity disk. In RAID 5, parity is distributed across all disks to overcome the write
bottleneck of a dedicated parity disk.
37
RAID 6 works the same way as RAID 5, except that RAID 6 includes a second parity element to
enable survival if two disk failures occur in a RAID set. Therefore, a RAID 6 implementation
requires at least four disks. RAID 6 distributes the parity across all the disks. The write penalty
(explained later in this module) in RAID 6 is more than that in RAID 5; therefore, RAID 5 writes
perform better than RAID 6. The rebuild operation in RAID 6 may take longer than that in RAID 5
due to the presence of two parity sets.
38
When choosing a RAID type, it is imperative to consider its impact on disk performance and
application IOPS. In both mirrored and parity RAID configurations, every write operation
translates into more I/O overhead for the disks, which is referred to as a write penalty. In a RAID
1 implementation, every write operation must be performed on two disks configured as a mirrored
pair, whereas in a RAID 5 implementation, a write operation may manifest as four I/O operations.
When performing I/Os to a disk configured with RAID 5, the controller has to read, recalculate,
and write a parity segment for every data write operation.
This slide illustrates a single write operation on RAID 5 that contains a group of five disks. The
parity (P) at the controller is calculated as follows:
Cp = C1 + C2 + C3 + C4 (XOR operations)
Whenever the controller performs a write I/O, parity must be computed by reading the old parity
(Cp old) and the old data (C4 old) from the disk, which means two read I/Os. Then, the new parity
(Cp new) is computed as follows:
Cp new = Cp old C4 old + C4 new (XOR operations)
After computing the new parity, the controller completes the write I/O by writing the new data
and the new parity onto the disks, amounting to two write I/Os. Therefore, the controller
performs two disk reads and two disk writes for every write operation, and the write penalty is 4.
In RAID 6, which maintains dual parity, a disk write requires three read operations: two parity
and one data. After calculating both the new parities, the controller performs three write
operations: two parity and an I/O. Therefore, in a RAID 6 implementation, the controller performs
six I/O operations for each write I/O, and the write penalty is 6.
39
The table on the slide compares different RAID levels.
40
A hot sparing refers to a process that temporarily replaces a failed disk drive with a spare drive in
a RAID array by taking the identity of the failed disk drive. With the hot spare, one of the
following methods of data recovery is performed depending on the RAID implementation:
If parity RAID is used, the data is rebuilt onto the hot spare from the parity and the data on
the surviving disk drives in the RAID set.
If mirroring is used, the data from the surviving mirror is used to copy the data onto the hot
spare.
When a new disk drive is added to the system, data from the hot spare is copied to it. The hot
spare returns to its idle state, ready to replace the next failed drive. Alternatively, the hot spare
replaces the failed disk drive permanently. This means that it is no longer a hot spare, and a new
hot spare must be configured on the storage system.
A hot spare should be large enough to accommodate data from a failed drive. Some systems
implement multiple hot spares to improve data availability.
A hot spare can be configured as automatic or user initiated, which specifies how it will be used in
the event of disk failure. In an automatic configuration, when the recoverable error rates for a
disk exceed a predetermined threshold, the disk subsystem tries to copy data from the failing disk
to the hot spare automatically. If this task is completed before the damaged disk fails, the
subsystem switches to the hot spare and marks the failing disk as unusable. Otherwise, it uses
parity or the mirrored disk to recover the data. In the case of a user-initiated configuration, the
administrator has control of the rebuild process. For example, the rebuild could occur overnight to
prevent any degradation of system performance. However, the system is at risk of data loss if
another disk failure occurs.
41
This lesson covered RAID and its use to improve performance and protection. It covered various
RAID implementations, techniques, and levels commonly used. This lesson also covered the
impact of RAID on performance and compared the commonly used RAID levels.
42
This lesson covers different types of data access methods. It also covers types of intelligent
storage systems. Finally, this lesson covers the scale-up and scale-out architectures.
43
Data is accessed and stored by applications using the underlying infrastructure. The key
components of this infrastructure are the OS (or file system), connectivity, and storage. The
compute system controller card accesses the storage devices using predefined protocols, such as
IDE/ATA, SCSI, or Fibre Channel (FC). IDE/ATA and SCSI are popularly used in small and
personal computing environments for accessing internal storage. FC and iSCSI protocols are used
for accessing data from an external storage device (or subsystems). External storage devices can
be connected to the compute system directly or through the storage network. When the storage
is connected directly to the compute system, it is referred as Direct-Attached Storage (DAS).
Data can be accessed over a network in one of the following ways: block level, file level, or object
level. In general, the application requests data from the file system (or operating system) by
specifying the filename and location. The file system has two components: user component and
storage component. The user component of the file system performs functions such as hierarchy
management, naming, and user access control. The storage component maps the files to the
physical location on the storage device. The file system maps the file attributes to the logical
block address of the data and sends the request to the storage device. The storage device
converts the logical block address (LBA) to a cylinder-head-sector (CHS) address and fetches the
data.
In a block-level access, the file system is created on a compute system, and data is accessed on a
network at the block level. In this case, raw disks or logical volumes are assigned to the compute
system for creating the file system.
In a file-level access, the file system is created on a separate file server or at the storage side,
and the file-level request is sent over a network. Because data is accessed at the file level, this
method has higher overhead, as compared to the data accessed at the block level.
Object-level access is an intelligent evolution, whereby data is accessed over a network in terms
of self-contained objects with a unique object identifier. In this type of access, the file systems
user component resides on the compute system and the storage component resides on the
storage system.
44
Based on the type of data access, a storage system can be classified as block-based storage
system, file-based storage system, object-based storage system, and unified storage system. A
unified storage system provides block-based, file-based, and object-based data access in a single
system.
Details on block-based, file-based, object-based, and unified storage systems are covered in the
following modules.
45
An intelligent storage system may be built either based on scale-up or scale-out architecture.
A scale-up storage architecture provides the capability to scale the capacity and performance of a
single storage system based on requirements. Scaling up a storage system involves upgrading or
adding controllers and storage. These systems have a fixed capacity ceiling, which limits their
scalability and the performance also starts degrading when reaching the capacity limit.
A scale-out storage architecture provides the capability to maximize its capacity by simply adding
nodes to the cluster. Nodes can be added quickly to the cluster, when more performance and
capacity is needed, without causing any downtime. This provides the flexibility to use many nodes
of moderate performance and availability characteristics to produce a total system that has better
aggregate performance and availability. Scale-out architecture pools the resources in the cluster
and distributes the workload across all the nodes. This results in linear performance
improvements as more nodes are added to the cluster.
46
This lesson covered different types of data access methods. This lesson also covered the types of
intelligent storage systems. Finally, the lesson covered the scale-up and scale-out architectures.
47
This module covered the key components of an intelligent storage system. This module also
covered the storage subsystems and detailed on components, addressing, and performance
parameters of a hard disk drive (HDD) and solid state drive (SSD). This module covered RAID and
its use to improve performance and protection. It detailed various RAID implementations,
techniques, and levels commonly used. Further, the module described the impact of RAID on
performance and compared the commonly used RAID levels. Finally, this module covered the
types of intelligent storage systems and their architectures.
48
Scenario:
An organization plans to deploy a new business application in their data center. The new
application requires 1.5 TB of storage capacity for business and application data. During peak
workload period, the application is expected to generate 5200 IOPS (I/Os per second) with a
typical I/O size of 4 KB. The available disk drive option is a 15,000 RPM drive with 250 GB
capacity. The disk drive has an average seek time of 4.2 milliseconds (ms) and a data transfer
rate of 80 MB/s.
(Contd)
Requirement:
The application is business critical and its response time must be within an acceptable range.
Deliverable:
Calculate the number of disk drives that is required to meet both the capacity and the
performance requirements of the application.
Solution:
Step 1: Calculate the time required to perform one I/O. This depends on the disk service time.
Disk service time = Average seek time + Rotational latency + Data transfer time
The average seek time = 4.2 ms (given).
The rotational latency is half the time taken for a full rotation. Since the rotation speed is
15,000 RPM, one rotation takes 60 / 15000 seconds. Therefore, the time taken for half a
rotation is 0.5 x (60 / 15000) = 2 ms.
The data transfer rate is 80 MB/s. Therefore, the transfer of one 4 KB I/O takes 4 KB / 80 MB/s
= 0.05 ms.
Therefore, disk service time = 4.2 ms + 2 ms + 0.05 ms = 6.25 ms.

Step 2: Calculate the maximum number of IOPS the disk drive can perform.
Maximum number of IOPS is 1 / 6.25 ms = 160 IOPS.
For an acceptable response time, the disk controller utilization must be less than 70%.
Therefore, the maximum number of IOPS at 70% utilization is 160 X 0.7 = 112 IOPS.
(Contd)
Step 3: Calculate the number of disk drives for the application:
Disk drives required to meet performance requirement = 5200 IOPS / 112 IOPS = 47
Disk drives required to meet capacity requirement = 1.5 TB / 250 GB = 6
The number of disk drives required for the application is:

Maximum (Capacity, Performance) = Maximum (6, 47) = 47 disk drives
Scenario:
An organization has multiple regional offices across the country. All financial activity is managed
and tracked by a single accounting application. The application is hosted on a dedicated compute
system with storage disks configured with RAID 0. The application generates around 15% random
write operations, and the remaining 85% operations are random reads. The accounting data is
currently stored on a 5-disk RAID 0 set. Each disk has an advertised formatted capacity of 250
GB, and the total size of the applications data is 900 GB. The company performs nightly backups
and removes old information. Therefore, the amount of data is unlikely to change significantly
over the next six months.
Requirements and challenges:
The organization wants the accounting data to be highly available. For this, the organization plans
to implement fault tolerance in the storage infrastructure. The organizations IT budget is
depleted as it is the end of the financial year. Therefore, it is not possible for the organization to
purchase any new disk drives.
(Contd)
Deliverables:
Recommend a RAID level that the organization can use to reconfigure their storage while
fulfilling their needs.
Justify the choice of RAID level based on cost, performance, and availability of the new
solution.
Solution:
Step 1: Calculation for the formatted capacity of the disks and the number of disks required for
the application:
Consider the formatted capacity of the disks. A 250 GB disk has an expected actual data
capacity of about 232.82 GB. This is because 1 GB is represented as 1000 MB, when in fact it
represents 1024 MB. Therefore, the actual capacity of the 250 GB disk can be calculated by
dividing 250,000,000,000 bytes by 1024 three times. This is equivalent to dividing 250 by
1.024 three times as given below:
250 / (1.024 * 1.024 * 1.024) = 232.83 GB
The total size of the application data is 900 GB. Therefore, the number of disks required is 900
/ 232.38 = 3.9 4 disks.
(Contd)
Solution:
Step 2: Choice of RAID level and its justification:
For fault tolerance, consider a solution that uses parity-based RAID. RAID 3 and RAID 5 are
the two solutions that will not require purchasing of any additional disks.
RAID 3 is used in environments that have large sequential access patterns. The environment
described in the scenario has random reads and writes and therefore the RAID 3 protection
schema is an unsuitable solution.
RAID 5 would perform reasonably well because the proportion of writes does not exceed 25%.
It is therefore recommended that the disks be reconfigured with RAID 5. The advantage is that
it provides low-cost data protection. The disadvantage is that if a disk fails, there will be a high
overhead due to the disk rebuild operation.
This module focuses on the key components of a block-based storage system. It details the
function of each component, including cache management and protection techniques. This module
also focuses on the two storage provisioning methods. Finally, this module focuses on the storage
tiering mechanisms.
Module 5: Block-based Storage System
This lesson covers block-based storage system components, intelligent cache algorithms, and
cache protection mechanisms.
A block-based storage system provides compute systems with block-level access to the storage
volumes. In this environment, the file system is created on the compute systems and data is
accessed on a network at the block level. These block-based storage systems can either be based
on scale-up or scale-out architecture. The block-based storage system consists of one or more
controller(s) and storage. Controllers and storage are discussed next.
A controller of a block-based storage system consists of three key components: front end, cache,
and back end. An I/O request received from the compute system at the front-end port is
processed through cache and back end, to enable storage and retrieval of data from the storage.
A read request can be serviced directly from cache if the requested data is found in the cache. In
modern intelligent storage systems, front end, cache, and back end are typically integrated on a
single board (referred as a storage processor or storage controller).
The front end provides the interface between the storage system and the compute system. It
consists of two components: front-end ports and front-end controllers. Typically, a front end has
redundant controllers for high availability, and each controller contains multiple ports that enable
large numbers of compute systems to connect to the intelligent storage system. Each front-end
controller has processing logic that executes the appropriate transport protocol, such as Fibre
Channel, iSCSI, FICON, or FCoE for storage connections. Front-end controllers route data to and
from cache via the internal data bus. When the cache receives the write data, the controller sends
an acknowledgment message back to the compute system.
Cache is semiconductor memory where data is placed temporarily to reduce the time required to
service I/O requests from the compute system. Cache improves storage system performance by
isolating compute systems from the storage (HDDs and SSDs). The performance improvement is
significantly higher when the back-end storage is hard disk drive (HDD). In this case, cache
improves storage system performance by isolating compute systems from the mechanical delays
associated with rotating disks or HDDs. Rotating disks are the slowest component of an intelligent
storage system. Data access on rotating disks usually takes several milliseconds because of seek
time and rotational latency. Accessing data from cache is fast and typically takes less than a
millisecond. On intelligent storage systems, write data is first placed in cache and then written to
the storage.
When a compute system issues a read request, the storage controller reads the tag RAM to
determine whether the required data is available in cache. If the requested data is found in the
cache, it is called a read cache hit or read hit and data is sent directly to the compute system,
without any back-end storage operation. This provides a fast response time to the compute
system (about a millisecond). If the requested data is not found in cache, it is called a cache miss
and the data must be read from the storage. The back end accesses the appropriate storage
device and retrieves the requested data. Data is then placed in cache and finally sent to the
compute system through the front end. Cache misses increase the I/O response time.
A prefetch or read-ahead algorithm is used when read requests are sequential. In a sequential
read request, a contiguous set of associated blocks is retrieved. Several other blocks that have
not yet been requested by the compute system can be read from the storage and placed into
cache in advance. When the compute system subsequently requests these blocks, the read
operations will be read hits. This process significantly improves the response time experienced by
the compute system. The intelligent storage system offers fixed and variable prefetch sizes. In
fixed prefetch, the intelligent storage system prefetches a fixed amount of data. It is most
suitable when compute system I/O sizes are uniform. In variable prefetch, the storage system
prefetches an amount of data in multiples of the size of the compute system request. Maximum
prefetch limits the number of data blocks that can be prefetched to prevent the storage from
being rendered busy with prefetch at the expense of other I/Os.
Read performance is measured in terms of the read hit ratio, or the hit rate, usually expressed as
a percentage. This ratio is the number of read hits with respect to the total number of read
requests. A higher read hit ratio improves the read performance.
Write operations with cache provide performance advantages over writing directly to storage.
When an I/O is written to cache and acknowledged, it is completed in less time (from the
compute systems perspective) than it would take to write directly to storage. Sequential writes
also offer opportunities for optimization because many smaller writes can be coalesced for larger
transfers to storage with the use of cache. A write operation with cache is implemented in the
following ways:
Write-through cache: Data is placed in the cache and immediately written to the storage,
and an acknowledgment is sent to the compute system. Because data is committed to storage
as it arrives, the risks of data loss are low, but the write-response time is longer because of the
storage operations.
Write-back cache: Data is placed in cache and an acknowledgment is sent to the compute
system immediately. Later, data from several writes are committed (de-staged) to the storage.
Write response times are much faster because the write operations are isolated from the
storage devices. However, uncommitted data is at risk of loss if cache failures occur.
Cache can be bypassed under certain conditions, such as large size write I/O. In this
implementation, if the size of an I/O request exceeds the predefined size, called write aside size,
writes are sent to the storage directly to reduce the impact of large writes consuming a large
cache space. This is particularly useful in an environment where cache resources are constrained
and cache is required for small random I/Os. Cache can be implemented as either dedicated
cache or global cache. With dedicated cache, separate sets of memory locations are reserved for
reads and writes. In global cache, both reads and writes can use any of the available memory
addresses. Cache management is more efficient in a global cache implementation because only
one global set of addresses has to be managed.
Global cache allows users to specify the percentages of cache available for reads and writes for
cache management. Typically, the read cache is small, but it should be increased if the application
being used is read-intensive. In other global cache implementations, the ratio of cache available
for reads versus writes is dynamically adjusted based on the workloads.
Cache is an expensive resource that needs proper management. Even though modern intelligent
storage systems come with a large amount of cache, when all cache pages are filled, some pages
have to be freed up to accommodate new data and avoid performance degradation. Various cache
management algorithms are implemented in intelligent storage systems to proactively maintain a
set of free pages and a list of pages that can be potentially freed up whenever required.
The most commonly used algorithms are discussed in the following list:
Least Recently Used (LRU): An algorithm that continuously monitors data access in cache
and identifies the cache pages that have not been accessed for a long time. LRU either frees up
these pages or marks them for reuse. This algorithm is based on the assumption that data that
has not been accessed for a while will not be requested by the compute system. However, if a
page contains write data that has not yet been committed to storage, the data is first written
to the storage before the page is reused.
Most Recently Used (MRU): This algorithm is the opposite of LRU, where the pages that
have been accessed most recently are freed up or marked for reuse. This algorithm is based on
the assumption that recently accessed data may not be required for a while.
As cache fills, the storage system must take action to flush dirty pages (data written into the
cache but not yet written to the storage) to manage space availability. Flushing is the process
that commits data from cache to the storage. On the basis of the I/O access rate and pattern,
high and low levels called watermarks are set in cache to manage the flushing process. High
watermark (HWM) is the cache utilization level at which the storage system starts high-speed
flushing of cache data. Low watermark (LWM) is the point at which the storage system stops
flushing data to the storage drives. The cache utilization level drives the mode of flushing to be
used:
Idle flushing: It occurs continuously, at a modest rate, when the cache utilization level is
between the high and the low watermark.
High watermark flushing: It is activated when cache utilization hits the high watermark. The
storage system dedicates some additional resources for flushing. This type of flushing has
some impact on I/O processing.
Forced flushing: It occurs in the event of a large I/O burst when cache reaches 100 percent
of its capacity, which significantly affects the I/O response time. In forced flushing, system
flushes the cache on priority by allocating more resources.
The rate of flushing and the rate of acceptance of compute system I/O into cache is managed
dynamically to optimize the storage system performance. The controller dynamically auto-adjusts
the incoming and outgoing I/O. The intelligent flushing algorithms provide the storage drives with
the time needed to process an increased flushing load by delaying compute system I/O
acknowledgments, thus slowing the rate of incoming writes. Throttling continues until the rate of
incoming data is equated with the abilities of the underlying storage drives to handle the
workload.
10
Cache is volatile memory; so a power failure or any kind of cache failure will cause loss of the
data that is not yet committed to the storage drive. This risk of losing uncommitted data held in
cache can be mitigated using cache mirroring and cache vaulting:
Cache mirroring: Each write to cache is held in two different memory locations on two
independent memory cards. If a cache failure occurs, the write data will still be safe in the
mirrored location and can be committed to the storage drive. Reads are staged from the
storage drive to the cache; therefore, if a cache failure occurs, the data can still be accessed
from the storage drives. Because only writes are mirrored, this method results in better
utilization of the available cache. In cache mirroring approaches, the problem of maintaining
cache coherency is introduced. Cache coherency means that data in two different cache
locations must be identical at all times. It is the responsibility of the storage system's operating
environment to ensure coherency.
Cache vaulting: The risk of data loss due to power failure can be addressed in various ways:
powering the memory with a battery until the AC power is restored or using battery power to
write the cache content to the storage drives. If an extended power failure occurs, using
batteries is not a viable option. This is because in intelligent storage systems, large amounts of
data might need to be committed to numerous storage drives, and batteries might not provide
power for sufficient time to write each piece of data to its intended storage drive. Therefore,
storage vendors use a set of physical storage drives to dump the contents of cache during
power failure. This is called cache vaulting and the storage drives are called vault drives. When
power is restored, data from these storage drives is written back to write cache and then
written to the intended drives.
11
The back end provides an interface between cache and the physical storage drives. It consists of
two components: back-end ports and back-end controllers. The back-end controls data transfers
between cache and the physical drives. From cache, data is sent to the back end and then routed
to the destination storage drives. Physical drives are connected to ports on the back end. The
back-end controller communicates with the storage drives when performing reads and writes and
also provides additional, but limited, temporary data storage. The algorithms implemented on
back-end controllers provide error detection and correction, along with RAID functionality.
For high data protection and high availability, storage systems are configured with dual controllers
with multiple ports. Such configurations provide an alternative path to physical storage drives if a
controller or port failure occurs. This reliability is further enhanced if the storage drives are also
dual-ported. In that case, each drive port can connect to a separate controller. Multiple controllers
also facilitate load balancing.
12
Physical storage drives are connected to the back-end storage controller and provide persistent
data storage. Modern intelligent storage systems provide support to a variety of storage drives
with different speeds and types, such as FC, SATA, SAS, and solid state drives. They also support
the use of a mix of SSD, FC, or SATA within the same storage system.
Workloads that have predictable access patterns typically work well with a combination of HDDs
and SSDs. If the workload changes, or constant high performance is required for all the storage
being presented, using a SSD can meet the desirable performance requirements.
13
This lesson covered block-based storage system components, intelligent cache algorithms, and
cache protection mechanisms.
14
This lesson covers traditional and virtual provisioning processes. This lesson also covers LUN
expansion and LUN masking mechanisms.
15
Storage provisioning is the process of assigning storage resources to compute systems based on
capacity, availability, and performance requirements. Storage provisioning can be performed in
two ways: traditional and virtual. Virtual provisioning leverages virtualization technology for
provisioning storage for applications.
16
In traditional storage provisioning, physical storage drives are logically grouped together on which
a required RAID level is applied to form a set, called RAID set. The number of drives in the RAID
set and the RAID level determine the availability, capacity, and performance of the RAID set. It is
highly recommend to create the RAID set from drives of the same type, speed, and capacity to
ensure maximum usable capacity, reliability, and consistency in performance. For example, if
drives of different capacities are mixed in a RAID set, the capacity of the smallest drive is used
from each drive in the set to make up the RAID sets overall capacity. The remaining capacity of
the larger drives remains unused. Likewise, mixing higher speed drives with lower speed drives
lowers the overall performance of the RAID set.
RAID sets usually have a large capacity because they combine the total capacity of individual
drives in the set. Logical units are created from the RAID sets by partitioning (seen as slices of
the RAID set) the available capacity into smaller units. These units are then assigned to the
compute system based on their storage requirements. Logical units are spread across all the
physical drives that belong to that set. Each logical unit created from the RAID set is assigned a
unique ID, called a logical unit number (LUN). LUNs hide the organization and composition of the
RAID set from the compute systems. LUNs created by traditional storage provisioning methods
are also referred to as thick LUNs to distinguish them from the LUNs created by virtual
provisioning methods.
The figure on the slide shows a RAID set consisting of five storage drives that have been sliced or
partitioned into two LUNs: LUN 0 and LUN 1. These LUNs are then assigned to Compute 1 and
Compute 2 for their storage requirements.
When a LUN is configured and assigned to a non-virtualized compute system, a bus scan is
required to identify the LUN. This LUN appears as a raw storage drive to the operating system. To
make this drive usable, it is formatted with a file system and then the file system is mounted.
(Contd)
17
In a virtualized compute system environment, the LUN is assigned to the hypervisor, which
recognizes it as a raw storage drive. This drive is configured with the hypervisor file system, and
then virtual storage drives are created on it. Virtual storage drives are files on the hypervisor file
system. The virtual storage drives are then assigned to virtual machines and appear as raw
storage drive to them. To make the virtual storage drive usable to the virtual machine, similar
steps are followed as in a non-virtualized environment. Here, the LUN space may be shared and
accessed simultaneously by multiple virtual machines.
Virtual machines can also access a LUN directly on the storage system. In this method the entire
LUN is allocated to a single virtual machine. Storing data in this way is recommended when the
applications running on the virtual machine are response-time sensitive, and sharing storage with
other virtual machines may impact their response time. The direct access method is also used
when a virtual machine is clustered with a physical machine. In this case, the virtual machine is
required to access the LUN that is being accessed by the physical machine.
18
MetaLUN is a method to expand LUNs that require additional capacity or performance. A metaLUN
can be created by combining two or more LUNs. A metaLUN consists of a base LUN and one or
more component LUNs. MetaLUNs can be either concatenated or striped.
Concatenated expansion simply adds additional capacity to the base LUN. In this expansion, the
component LUNs are not required to be of the same capacity as the base LUN. All LUNs in a
concatenated metaLUN must be either protected (parity or mirrored) or unprotected (RAID 0).
RAID types within a metaLUN can be mixed. For example, a RAID 1/0 LUN can be concatenated
with a RAID 5 LUN. However, a RAID 0 LUN can be concatenated only with another RAID 0 LUN.
Concatenated expansion is quick but does not provide any performance benefit.
Striped expansion restripes the base LUNs data across the base LUN and component LUNs. In
striped expansion, all LUNs must be of the same capacity and RAID level. Striped expansion
provides improved performance due to the increased number of drives being striped.
All LUNs in both concatenated and striped expansion must reside on the same storage drive type:
either all SSDs, Fibre Channel, or all ATA.
19
Virtual provisioning enables creating and presenting a LUN with more capacity than is physically
allocated to it on the storage system. The LUN created using virtual provisioning is called a thin
LUN to distinguish it from the traditional LUN.
Thin LUNs do not require physical storage to be completely allocated to them at the time they are
created and presented to a compute system. Physical storage is allocated to the compute system
on-demand from a shared pool of physical capacity. A shared pool consists of physical storage
drives. A shared pool in virtual provisioning is analogous to a RAID set, which is a collection of
drives on which LUNs are created. Similar to a RAID set, a shared pool supports a single RAID
protection level. However, unlike a RAID set, a shared pool might contain large numbers of
drives. Shared pools can be homogeneous (containing a single drive type) or heterogeneous
(containing mixed drive types, such as SSD, FC, SAS, and SATA drives).
Virtual provisioning enables more efficient allocation of storage to compute systems. Virtual
provisioning also enables oversubscription, where more capacity is presented to the compute
systems than is actually available on the storage system. Both the shared pool and the thin LUN
can be expanded non-disruptively as the storage requirements of the compute systems grow.
Multiple shared pools can be created within a storage system, and a shared pool may be shared
by multiple thin LUNs.
20
A storage pool comprises physical drives that provide the physical storage used by Thin LUNs. A
storage pool is created by specifying a set of drives and a RAID type for that pool. Thin LUNs are
then created out of that pool (similar to traditional LUN created on a RAID set). All the Thin LUNs
created from a pool share the storage resources of that pool. Adding drives to a storage pool
increases the available shared capacity for all the Thin LUNs in the pool. Drives can be added to a
storage pool while the pool is used in production. The allocated capacity is reclaimed by the pool
when Thin LUNs are destroyed.
When a storage pool is expanded, the sudden introduction of new empty drives combined with
relative full drives cause a data imbalance. This imbalance is resolved by automating a one-time
data relocation, referred to as rebalancing. Storage pool rebalancing is a technique that provides
the ability to automatically relocate extents (minimum amount of physical storage capacity that is
allocated to the thin LUN from the pool) on physical storage drives over the entire pool when new
drives are added to the pool. Storage pool rebalancing restripes data across all the drives( both
existing and new drives) in the storage pool. This enables spreading out the data equally on all
the physical drives within the storage pool, ensuring that the used capacity of each drive is
uniform across the pool. After the storage pool capacity is increased, the capacity of the existing
LUNs can be expanded.
21
Administrators typically allocate storage capacity based on anticipated storage requirements. This
generally results in the over provisioning of storage capacity, which then leads to higher costs and
lower capacity utilization. Administrators often over-provision storage to an application for various
reasons such as, to avoid frequent provisioning of storage if the LUN capacity is exhausted, and to
reduce disruption to application availability. Over provisioning of storage often leads to additional
storage acquisition and operational costs.
Virtual provisioning addresses these challenges. Virtual provisioning improves storage capacity
utilization and simplifies storage management. The figure on the slide illustrates an example,
comparing virtual provisioning with traditional storage provisioning.
With traditional provisioning, three LUNs are created and presented to one or more compute
systems. The total storage capacity of the storage system is 2 TB. The allocated capacity of LUN 1
is 500 GB, of which only 100 GB is consumed, and the remaining 400 GB is unused. The size of
LUN 2 is 550 GB, of which 50 GB is consumed, and 500 GB is unused. The size of LUN 3 is 800
GB, of which 200 GB is consumed, and 600 GB is unused. In total, the storage system has 350
GB of data, 1.5 TB of allocated but unused capacity, and only 150 GB of remaining capacity
available for other applications.
Now consider the same 2 TB storage system with virtual provisioning. Here, three thin LUNs of
the same sizes are created. However, there is no allocated unused capacity. In total, the storage
system with virtual provisioning has the same 350 GB of data, but 1.65 TB of capacity is available
for other applications, whereas only 150 GB is available in traditional storage provisioning.
(Contd)
22
Virtual provisioning and thin LUN offer many benefits, although in some cases traditional LUN is
better suited for an application. Thin LUNs are appropriate for applications that can tolerate
performance variations. In some cases, performance improvement is perceived when using a thin
LUN, due to striping across a large number of drives in the pool. However, when multiple thin
LUNs contend for shared storage resources in a given pool, and when utilization reaches higher
levels, the performance can degrade. Thin LUNs provide the best storage space efficiency and are
suitable for applications where space consumption is difficult to forecast. Using thin LUNs benefits
organizations in reducing power and acquisition costs and in simplifying their storage
management.
Traditional LUNs are suited for applications that require predictable performance. Traditional
LUNs provide full control for precise data placement and allow an administrator to create LUNs on
different RAID groups if there is any workload contention. Organizations that are not highly
concerned about storage space efficiency may still use traditional LUNs.
Both traditional and thin LUNs can coexist in the same storage system. Based on the requirement,
an administrator may migrate data between thin and traditional LUNs.
23
LUN masking is a process that provides data access control by defining which LUNs a compute
system can access. The LUN masking function is implemented on the storage system. This
ensures that volume access by compute system is controlled appropriately, preventing
unauthorized or accidental use in a shared environment.
For example, consider a storage system with two LUNs that store data of the sales and finance
departments. Without LUN masking, both departments can easily see and modify each others
data, posing a high risk to data integrity and security. With LUN masking, LUNs are accessible
only to the designated compute systems.
24
This lesson covered traditional provisioning, virtual provisioning, LUN expansion, and LUN
masking.
25
This lesson covers LUN and sub-LUN tiering, cache tiering, and server flash caching.
26
Storage tiering is a technique of establishing a hierarchy of different storage types (tiers). This
enables storing the right data to the right tier, based on service level requirements, at a minimal
cost. Each tier has different levels of protection, performance, and cost. For example, high
performance solid-state drives (SSDs) or FC drives can be configured as tier 1 storage to keep
frequently accessed data and low cost SATA drives as tier 2 storage to keep the less frequently
accessed data. Keeping frequently used data in SSD or FC improves application performance.
Moving less-frequently accessed data to SATA can free up storage capacity in high performance
drives and reduce the cost of storage. This movement of data happens based on defined tiering
policies. The tiering policy might be based on parameters, such as frequency of access. For
example, if a policy states move the data that are not accessed for the last 30 mins to the lower
tier, then all the data matching this condition are moved to the lower tier.
The process of moving the data from one type of tier to another is typically automated. In
automated storage tiering, the application workload is proactively monitored; the active data is
automatically moved to a higher performance tier and the inactive data is moved to higher
capacity, lower performance tier. The data movement between the tiers is performed nondisruptively.
There are three techniques of storage tiering implemented in a block-based storage system: LUN
and sub-LUN tiering, cache tiering, and server flash-caching.
27
The process of storage tiering within a storage system is called intra-array storage tiering. It
enables the efficient use of SSD, FC, and SATA drives within a system and provides performance
and cost optimization. The goal is to keep the SSDs busy by storing the most frequently accessed
data on them, while moving out the less frequently accessed data to the SATA drives. Data
movements executed between tiers can be performed at the LUN level or at the sub-LUN level.
The performance can be further improved by implementing tiered cache.
Traditionally, storage tiering is operated at the LUN level that moves an entire LUN from one tier
of storage to another. This movement includes both active and inactive data in that LUN. This
method does not give effective cost and performance benefits. Today, storage tiering can be
implemented at the sub-LUN level. In sub-LUN level tiering, a LUN is broken down into smaller
segments and tiered at that level. Movement of data with much finer granularity, for example 8
MB, greatly enhances the value proposition of automated storage tiering. Tiering at the sub-LUN
level effectively moves active data to faster drives and less active data to slower drives.
28
Tiering is also implemented at the cache level. A large cache in a storage system improves
performance by retaining large amount of frequently accessed data in a cache; so most reads are
served directly from the cache. However, configuring a large cache in the storage system involves
more cost. An alternative way to increase the size of the cache is by utilizing the SSDs on the
storage system. In cache tiering, SSDs are used to create a large capacity secondary cache and
to enable tiering between DRAM (primary cache) and SSDs (secondary cache). Server flashcaching is another tier of cache in which flash-cache card is installed in the server to further
enhance the application performance.
29
Server flash-caching technology uses intelligent caching software and a PCI Express-based (PCIe)
flash card installed on the compute system. This dramatically improves application performance
by reducing latency, and accelerates throughput. Server flash-caching technology works in both
physical and virtual environments and provides performance acceleration for read-intensive
workloads. This technology uses minimal CPU and memory resources from the compute system
by offloading flash management onto the PCIe card.
It intelligently determines which data would benefit by sitting in the compute system on PCIe
flash and closer to the application. This avoids the latencies associated with I/O access over the
network to the storage system. With this, the processing power required for an applications most
frequently referenced data is offloaded from the back-end storage to the PCIe card. Therefore, the
storage system can allocate greater processing power to other applications.
30
To gain cost advantage, organizations may move their application to cloud. To ensure proper
functioning of the application and provide acceptable performance, service providers offer blockbased storage in cloud. The service providers enable the consumers to create a block-based
storage volumes and attach them to the virtual machine instances. After the volumes are
attached, the consumers can create the file system on these volumes and run applications the
way they would on an on-premise data center.
31
This lesson covered LUN and Sub-LUN tiering, cache tiering, server flash caching, and block-based
storage use case.
32
The Concepts in Practice section covers EMC XtremIO, EMC VNX-F, EMC FAST VP, EMC XtremSF,
and EMC XtremCache.
Note:
33
EMC XtremIO is an all-flash, block-based, scale-out enterprise storage system that provides
substantial improvements to I/O performance. It is purpose-built to leverage flash media and
delivers new levels of real-world performance, administrative ease, and advanced data services
for applications. It uses a scale-out clustered design that grows capacity and performance linearly
to meet any requirement. XtremIO storage systems are created from building blocks called "XBricks" that are each a high-availability, high-performance, fully active/active storage system with
no single point of failure. XtremIO's powerful operating system, XIOS, manages the XtremIO
storage cluster. XIOS ensures that the system remains balanced and always delivers the highest
levels of performance with no administrator intervention. XtremIO helps the administrators to
become more efficient by enabling system configuration in a few clicks, provisioning storage in
seconds, and monitoring the environment with real-time metrics.
EMC VNX-F is a block-based SSD only storage system. It provides consistent performance and
low latency for application environments such as OLTP, RDBMS, compute virtualization, and
desktop virtualization. The VNX-F supports Fibre Channel, FCoE, and iSCSI block protocols.
EMC FAST VP performs storage tiering at a sub-LUN level in a virtual provisioned environment.
FAST VP automatically moves more active data (data that is more frequently accessed) to the
best performing storage tier, and it moves less active data to a lower performance and less
expensive tier. Data movement between the tiers is based on user-defined policies, and is
executed automatically and non-disruptively by FAST VP.
34
EMC XtremSF is a PCIe flash card deployed in the compute system to dynamically improve
application performance by reducing latency and accelerating throughput. XtremSF is used as a
caching device in conjunction with server flash caching software, EMC XtremCache. XtremSF,
when deployed in a compute system, minimizes the overhead of the network storage access,
reducing response time.
EMC XtremCache is an intelligent caching software that leverages server-based flash technology
and write-through caching for accelerated application performance with data protection.
XtremCache accelerates reads and protects data using write-through cache to the networked
storage. It extends EMC FAST VP into the compute system, adding another tier of intelligence and
performance to the I/O stack.
35
This module covered various components of block-based storage system. This module also
covered traditional and virtual storage provisioning methods. Finally, this module covered storage
tiering mechanisms.
36
Scenario:
An organization plans to deploy an OLTP application and a back-office application in their
environment. The organization has a block-based storage system with SSD, FC drives, and SATA
drives. The OLTP application is mission-critical and will generate a large number of small-sized,
random I/Os with the number of reads expected to be around 75%. The back-office application
will be deployed for non-production operations.
Requirements and challenges:

The organization is concerned about the throughput and response time of the OLTP application.
The OLTP application requires high performance with low latency. The storage administrator
wants full control of the storage provisioned for this application for precise data placement on
physical drives. The back-office application can tolerate performance variations. The capacity
requirement of the back-office application is unpredictable and may vary.
Deliverable:
Suggest a storage provisioning and tiering strategy to meet the organizations requirements.
Solution:
Use traditional storage provisioning for the OLTP application. This will enable meeting the
applications performance and response time requirements, while providing greater control to the
storage administrator. Since the OLTP application is read-heavy, use server flash caching on the
compute system hosting the application to improve the applications performance. Use virtual
storage provisioning for the back-office application. This enables oversubscription, dynamic
capacity expansion, and improved storage capacity utilization.
This module focuses on the components and architectures of a file-based storage system. This
module also focuses on various file sharing protocols supported by a file-based storage system.
Finally, this module focuses on file-level virtualization and tiering.
Module 6: File-based Storage System (NAS)
This lesson covers components of NAS, NAS architectures, NAS file access methods, and NAS I/O
operations.
File sharing, as the name implies, enables users to share files with other users. In a file-sharing
environment, a user who creates the file (the creator or owner of a file) determines the type of
access (such as read, write, execute, append, delete) to be given to other users. When multiple
users try to access a shared file at the same time, a locking scheme is required to maintain data
integrity and at the same time make this sharing possible.
Some examples of file-sharing methods are the peer-to-peer (P2P) model, File Transfer Protocol
(FTP), client-server models that use file-sharing protocols such as NFS and CIFS, and Distributed
File System (DFS).
FTP is a client-server protocol that enables data transfer over a network. An FTP server and an
FTP client communicate with each other using TCP as the transport protocol.
A peer-to-peer (P2P) file sharing model uses peer-to-peer network. P2P enables client machines
to directly share files with each other over a network. Clients use a file sharing software that
searches for other peer clients. This differs from client-server model that uses file servers to store
files for sharing.
The standard client-server file-sharing protocols, such as NFS and CIFS, enable the owner of a file
to set the required type of access, such as read-only or read-write, for a particular user or group
of users. Using this protocol, the clients mount remote file systems that are available on
dedicated file servers.
A distributed file system (DFS) is a file system that is distributed across several compute systems.
A DFS can provide compute systems with direct access to the entire file system, while ensuring
efficient management and data security. Hadoop Distributed File System (HDFS) is an example of
distributed file system which will be detailed later in this module. Vendors now support HDFS on
their NAS systems to support the scale-out architecture. The scale-out architecture helps to meet
the big data analytics requirements.
NAS is a dedicated, high-performance file sharing and storage device. NAS enables its
clients to share files over an IP network. NAS provides the advantages of server
consolidation by eliminating the need for multiple file servers. It also consolidates the
storage used by the clients onto a single system, making it easier to manage the
storage. NAS uses network and file-sharing protocols to provide access to the file data.
These protocols include TCP/IP for data transfer and Common Internet File System
(CIFS) and Network File System (NFS) for network file service. Apart from these
protocols, the NAS systems may also use HDFS and its associated protocols (discussed
later in the module) over TCP/IP to access files. NAS enables both UNIX and Microsoft
Windows users to share the same data seamlessly.
A NAS device uses its own operating system and integrated hardware and software
components to meet specific file-service needs. Its operating system is optimized for file
I/O and, therefore, performs file I/O better than a general-purpose server. As a result, a
NAS device can serve more clients than general-purpose servers and provide the benefit
of server consolidation.
A NAS device is optimized for file-serving functions such as storing, retrieving, and
accessing files for applications and clients. As shown on the slide, a general-purpose
server can be used to host any application because it runs a general-purpose operating
system. Unlike a general-purpose server, a NAS device is dedicated to file-serving. It has
a specialized operating system dedicated for file serving by using industry standard
protocols. NAS vendors also support features, such as clustering for high availability,
scalability, and performance. The clustering feature enables multiple NAS
controllers/heads/nodes to function as a single entity. The workload can be distributed
across all the available nodes. Therefore, NAS devices support massive workloads.
A NAS system consists of two components, controller and storage. A controller is a compute
system that contains components such as network, memory, and CPU resources. A specialized
operating system optimized for file serving is installed on the controller. Each controller may
connect to all storage in the system. The controllers can be active/active, with all controllers
accessing the storage, or active/passive with some controllers performing all the I/O processing
while others act as spares. A spare will only be used for I/O processing if an active controller fails.
The controller is responsible for configuration of RAID set, creating LUNs, installing file system,
and exporting the file share on the network.
Storage is used to persistently store data. The NAS system may have different types of storage
devices to support different requirements. The NAS system may support SSD, SAS, and SATA in a
single system.
The extent to which the components, such as CPU, memory, network adaptors, and storage, can
be scaled depends upon the type of NAS architecture used. There are two types of NAS
architectures; scale-up and scale-out. Both these architectures are detailed in the next few slides.
A scale-up NAS architecture provides the capability to scale the capacity and performance of a
single NAS system based on requirements. Scaling up a NAS system involves upgrading or adding
NAS heads and storage. These NAS systems have a fixed capacity ceiling, which limits their
scalability. The performance of these systems starts degrading when reaching the capacity limit.
There are two types of scale-up NAS implementations, integrated NAS and gateway NAS. An
integrated NAS system contains one or more NAS heads and storage in a single system. NAS
heads are connected to the storage. The storage may consist of different drive types, such as
SAS, ATA, FC, and solid state drives, to meet different workload requirements. Each NAS head in
an integrated NAS has front-end Ethernet ports, which connect to the IP network. The front-end
ports provide connectivity to the clients. Each NAS head has back-end ports to provide
connectivity to the attached storage. Integrated NAS systems have NAS management software
that can be used to perform all the administrative tasks for the NAS head and storage.
A gateway NAS system consists of one or more NAS heads and uses external and
independently managed storage. In gateway NAS implementation, the NAS gateway

shares the storage from a block-based storage system. The management functions in
this type of solution are more complex than those in an integrated NAS environment
because there are separate administrative tasks for the NAS head and the storage. The
administrative tasks of the NAS gateway are performed by the NAS management
software and the storage system is managed with the management software of the
block-based storage system. A gateway solution can use the FC infrastructure, such as
switches and directors for accessing SAN-attached storage arrays or direct-attached
storage arrays.
The scale-out NAS implementation pools multiple NAS nodes together in a cluster. A node may
consist of either the NAS head or the storage or both. The cluster performs the NAS operation as
a single entity. A scale-out NAS provides the capability to scale its resources by simply adding
nodes to a clustered NAS architecture. The cluster works as a single NAS device and is managed
centrally. Nodes can be added to the cluster, when more performance or more capacity is needed,
without causing any downtime. Scale-out NAS provides the flexibility to use many nodes of
moderate performance and the availability characteristics to produce a total system that has
better aggregate performance and availability. It also provides ease of use, low cost, and
theoretically unlimited scalability.
Scale-out NAS uses a distributed clustered file system that runs on all nodes in the cluster. All
information is shared among nodes, so the entire file system is accessible by clients connecting to
any node in the cluster. Scale-out NAS stripes data across all nodes in a cluster along with mirror
or parity protection. As data is sent from clients to the cluster, the data is divided and allocated to
different nodes in parallel. When a client sends a request to read a file, the scale-out NAS
retrieves the appropriate blocks from multiple nodes, recombines the blocks into a file, and
presents the file to the client. As nodes are added, the file system grows dynamically and data is
evenly distributed to every node. Each node added to the cluster increases the aggregate storage,
memory, CPU, and network capacity. Hence, cluster performance is also increased.
(Contd.)
Scale-out NAS clusters use separate internal and external networks for back-end and front-end
connectivity respectively. An internal network provides connections for intra-cluster
communication, and an external network connection enables clients to access and share file data.
Each node in the cluster connects to the internal network. The internal network offers high
throughput and low latency and uses high-speed networking technology, such as InfiniBand or
Gigabit Ethernet. To enable clients to access a node, the node must be connected to the external
Ethernet network. Redundant internal or external networks may be used for high availability.
Note:
InfiniBand is a networking technology that provides a low-latency, high-bandwidth communication
link between hosts and peripherals. It provides serial connection and is often used for inter-server
communications in high-performance computing environments. InfiniBand enables remote direct
memory access (RDMA) that enables a device (host or peripheral) to access data directly from the
memory of a remote device. InfiniBand also enables a single physical link to carry multiple
channels of data simultaneously by using a multiplexing technique.
10
Different methods can be used to access files on a NAS system. The most common
methods of accessing NAS systems are by using Network File System (NFS), Common
Internet File System (CIFS), and Hadoop Distribution File System (HDFS).
11
Common Internet File System (CIFS) is a client-server application protocol that enables client
programs to make requests for files and services on remote computers over TCP/IP. It is a public
or open variation of Server Message Block (SMB) protocol.
The CIFS protocol enables remote clients to gain access to files on a server. CIFS enables file
sharing with other clients by using special locks. Filenames in CIFS are encoded using unicode
characters. CIFS provides the following features to ensure data integrity:
It uses file and record locking to prevent users from overwriting the work of another user on a
file or a record.
It supports fault tolerance and can automatically restore connections and reopen files that
were open prior to an interruption. The fault tolerance features of CIFS depend on whether an
application is written to take advantage of these features. Moreover, CIFS is a stateful
protocol because the CIFS server maintains connection information regarding every connected
client. If a network failure or CIFS server failure occurs, the client receives a disconnection
notification. User disruption is minimized if the application has the embedded intelligence to
restore the connection. However, if the embedded intelligence is missing, the user must take
steps to reestablish the CIFS connection.
Users refer to remote file systems with an easy-to-use file-naming scheme:

\\server\share or \\servername.domain.suffix\share.
12
Network File System (NFS) is a client-server protocol for file sharing that is commonly used on
UNIX systems. NFS was originally based on the connectionless User Datagram Protocol (UDP). It
uses a machine-independent model to represent user data. It also uses Remote Procedure Call
(RPC) as a method of inter-process communication between two computers. The NFS protocol
provides a set of RPCs to access a remote file system for the following operations:
Searching files and directories
Opening, reading, writing to, and closing a file
Changing file attributes
Modifying file links and directories
NFS creates a connection between the client and the remote system to transfer data. NFS (NFSv3
and earlier) is a stateless protocol, which means that it does not maintain any kind of table to
store information about open files and associated pointers. Therefore, each call provides a full set
of arguments to access files on the server. These arguments include a file handle reference to the
file, a particular position to read or write, and the versions of NFS.
Currently, three versions of NFS are in use:
NFS version 2 (NFSv2): NFSv2 uses UDP to provide a stateless network connection between
a client and a server. The feature such as locking is handled outside the protocol.
NFS version 3 (NFSv3): NFSv3 is the most commonly used version, which uses UDP or TCP,
and is based on the stateless protocol design. It includes some new features, such as a 64-bit
file size, asynchronous writes, and replies containing additional file attributes to avoid
subsequent calls.
NFS version 4 (NFSv4): NFSv4 uses TCP and is based on a stateful protocol design. It offers
enhanced security. The latest NFS version 4.1 is the enhancement of NFSv4 and includes
some new features, such as session model, parallel NFS (pNFS), and data retention.
13
Many of the scale-out NAS vendors support HDFS. HDFS is a file system that spans multiple
nodes in a cluster and allows user data to be stored in files. It presents a traditional hierarchical
file organization so that users or applications can manipulate (create, rename, move, or remove)
files and directories. It also presents a streaming interface to run any application of choice using
the MapReduce framework. HDFS requires programmatic access because the file system cannot
be mounted. All HDFS communication is layered on top of the TCP/IP protocol. HDFS has a
master/slave architecture. An HDFS cluster consists of a single NameNode that acts as a master
server and has in-memory maps of every file, file locations as well as all the blocks within the file
and which DataNodes they reside on. The NameNode is responsible for managing the file system
namespace and controlling the access to the files by clients. DataNodes act as slaves that serve
read/write requests as well as perform block creation, deletion, and replication as directed by the
NameNode.
The HDFS communication protocols, which are ClientProtocol and DataNode protocol, are
layered on top of the TCP/IP protocol. A client compute system and the NameNode
establish a connection to a configurable TCP port. The ClientProtocol is used for
communication between client compute systems and the NameNode. Further, the DataNode
Protocol is used for communication between the DataNodes and the NameNode.
14
The figure on the slide illustrates I/O operation in a scale-up NAS system. The process of
handling I/Os in a scale-up NAS environment is as follows:
1. The requestor (client) packages an I/O request into TCP/IP and forwards it through
the network stack. The NAS system receives this request from the network.
2. The NAS system converts the I/O request into an appropriate physical storage
request, which is a block-level I/O, and then performs the operation on the physical
storage.
3. When the NAS system receives data from the storage, it processes and repackages
the data into an appropriate file protocol response.
4. The NAS system packages this response into TCP/IP again and forwards it to the
client through the network.
15
The figure on the slide illustrates I/O operation in a scale-out NAS system. A scale-out NAS
consists of multiple NAS nodes and each of these nodes has the functionality similar to a
NameNode or a DataNode. In some proprietary scale-out NAS implementations, each node may
function as both a NameNode and DataNode, typically to provide Hadoop integration. All the NAS
nodes in scale-out NAS are clustered. New nodes can be added as required. As new nodes are
added, the file system grows dynamically and is evenly distributed to each node. As the client
sends a file to store to the NAS system, the file is evenly striped across the nodes. When a client
writes data, even though that client is connected to only one node, the write operation occurs in
multiple nodes in the cluster. This is also true for read operations. A client is connected to only
one node at a time. However, when that client requests a file from the cluster, the node to which
the client is connected will not have the entire file locally on its drives. The node to which the
client is connected retrieves and rebuilds the file using the back-end InfiniBand network.
16
This lesson covered components of NAS, NAS architectures, NAS file access methods, and NAS
I/O operations.
17
This lesson covers file-level virtualization, storage tiering, and NAS use case.
18
A network-based file sharing environment is composed of multiple file servers or NAS

devices. It might be required to move the files from one device to another due to reasons
such as cost or performance. File-level virtualization, implemented in NAS or the file
server environment, provides a simple, non-disruptive file-mobility solution.
File-level virtualization eliminates the dependencies between the data accessed at the file
level and the location where the files are physically stored. It creates a logical pool of
storage, enabling users to use a logical path rather than a physical path, to access files.
A global namespace is used to map the logical path of a file to the physical path names.
File-level virtualization enables the movement of files across NAS devices, even if the
files are being accessed by the clients.
19
Before virtualization, each client knows exactly where its file resources are located. This
environment leads to underutilized storage resources and capacity problems because
files are bound to a specific NAS device or file server. It may be required to move the
files from one server to another because of performance reasons or when the file server
fills up. Moving files across the environment is not easy and may make files inaccessible
during file movement. Moreover, hosts and applications need to be reconfigured to
access the file at the new location. This makes it difficult for storage administrators to
improve storage efficiency while maintaining the required service level.
File-level virtualization simplifies file mobility. It provides user or application
independence from the location where the files are stored. File-level virtualization
facilitates the movement of files across online file servers or NAS devices. This means
that while the files are being moved, clients can access their files non-disruptively.
Clients can also read their files from the old location and write them back to the new
location without realizing that the physical location has changed.
20
As the unstructured data in the NAS environment grows, organizations deploy a tiered
storage environment. This environment optimizes the primary storage for performance and the
secondary storage for capacity and cost.
Storage tiering works on the principle of Hierarchical Storage Management (HSM). HSM is
a file mobility concept where a policy-engine, which can be software or hardware where
policies are configured, facilitates moving files from the primary tiered storage to the secondary
tiered storage that meets the predefined policies. In HSM, a hierarchy of storage tier is defined
based on parameters such as cost, performance, and/or availability of storage.
Some prevalent reasons to tier data across storage systems or between storage system and cloud
is archival or to meet compliance requirements. As an example, the policy engine might be
configured to relocate all the files in the primary storage tier that have not been accessed in one
month and archive those files to the secondary storage. For each archived file, the policy engine
creates a small space-saving stub file in the primary storage that points to the data on the
secondary storage. When a user tries to access the file from its original location on the primary
storage, the user is transparently provided with the actual file from the secondary storage.
21
The figure on the slide illustrates the file-level storage tiering. In a file-level storage tiering
environment, a file can be moved to a secondary storage tier or to the cloud. Before moving a file
from primary NAS to secondary NAS or from primary NAS to cloud, the policy engine scans the
primary NAS to identify files that meet the predefined policies. After identifying the data files, the
stub files are created and the data files are moved to the destination storage tier.
22
The data lake represents a paradigm shift from the linear data flow model. As data and the
insights gathered from it increase in value, the enterprise-wide consolidated storage is
transformed into a hub around which the ingestion and consumption systems work (see figure on
the slide). This enables enterprises to bring analytics to data and avoid expensive cost of multiple
systems, storage, and time for ingestion and analysis.
By eliminating a number of parallel linear data flow, enterprises can consolidate vast amounts of
their data into a single store, a data lake, through a native and simple ingestion process. Analytics
can be performed on this data which provides insight. Actions can be taken based on this insight
in an iterative manner, as the organization and technology matures. Enterprises can thus
eliminate the cost of having silos or islands of information spread across their enterprises.
The key characteristics of a scale-out data lake are that it:
Accepts data from a variety of sources like file shares, archives, web applications, devices, and
the cloud, in both streaming and batch processes
Enables access to this data for a variety of uses from conventional purpose to mobile,
analytics, and cloud applications
Scales to meet the demands of future consolidation and growth as technology evolves and new
possibilities emerge for applying data to gain competitive advantage in the market place
Provides a tiering ability that enables organizations to manage their costs without setting up
specialized infrastructures for cost optimization
Scale-out NAS has the ability to provide the storage platform to this data lake. The scale-out NAS
enhances this paradigm by providing scaling capabilities in terms of capacity, performance,
security, and protection.
23
This lesson covered file-level virtualization, storage tiering, and NAS use case.
24
The Concepts in Practice section covers the products such as EMC Isilon, EMC VNX Gateway, and
EMC Cloud Tiering Appliance.
Note:
25
EMC Isilon is a scale-out NAS product powered by the OneFS operating environment. It
enables pooling multiple nodes together to construct a clustered NAS system. OneFS creates a
single file system that spans across all nodes in an Isilon cluster. Isilon provides the capability to
manage and store large (petabyte-scale), high-growth data in a single system with the flexibility
to meet a broad range of performance requirements.
EMC VNX Gateway is a gateway NAS product. It contains one or more NAS heads,
called X-Blades, that access external storage arrays, such as Symmetrix and block-based
VNX via SAN. X-Blades run the VNX operating environment that is optimized for highperformance and multiprotocol network file system access. All the X-Blades in a VNX
gateway system are managed by Control Station, which provides a single point for
configuring VNX Gateway.
Cloud Tiering Appliance (CTA) provides the policy engine functionality in the HSM
process. By interacting with a NAS share or export, the CTA identifies files that fit the
predefined criteria and initiates movement to a lower storage tier. This allows more
efficient use of the most expensive, highest-performing NAS storage. Inactive data can be
archived onto secondary storage where file retention requirements can be enforced. CTA also
enables the use of cloud as one of the tiers. CTA is integrated with storage platform APIs, so
native functionality and existing processes can be maintained.
26
This module covered the components and architectures of a file-based storage system. This
module also covered various file sharing protocols supported by a file-based storage system.
Finally, this module covered file-level virtualization and tiering.
27
28
This module focuses on the key OSD components. This module also focuses on the key features of
OSD and OSD operations. Finally, this module focuses on unified storage architecture.
Module 7: Object-based and Unified Storage
This lesson covers the drivers for object-based storage. This lesson also compares hierarchical file
system with flat address space. This lesson further covers key components and features of OSD.
The amount of data created each year is growing exponentially and the recent studies have
shown that more than 90 percent of data generated is unstructured (e-mail, instant messages,
graphics, images, and videos). The rapid adoption of third platform technologies like social, big
data analytics, cloud, and mobile application leads to significant data growth. Today,
organizations not only have to store and protect petabytes of data, but they also have to retain
the data over longer periods of time, for regulation and compliance reasons. They have also
recognized that data can help gain competitive advantages and even support new revenue
streams. In addition to increasing amounts of data, there has also been a significant shift in how
people want and expect to access their data. The rising adoption rate of smartphones, tablets,
and other mobile devices by consumers, combined with increasing acceptance of these devices in
enterprise workplaces, has resulted in an expectation for on-demand access to data from
anywhere on any device.
Traditional storage solutions like NAS, which is a dominant solution for storing unstructured data,
cannot scale to the capacities required or provide universal access across geographically
dispersed locations. Data growth adds high overhead to the NAS in terms of managing large
number of permission and nested directories. File systems require more management as they
scale and are limited in size. Their performance degrades as file system size increases, and do not
accommodate metadata beyond file properties which is a requirement of many new applications.
These challenges demand a smarter approach (object storage) that allows to manage data growth
at low cost, provides extensive metadata capabilities, and also provides massive scalability to
keep up with the rapidly growing data storage and access demands.
Object-based storage device stores data in the form of objects on flat address space based on its
content and other attributes rather than the name and the location. An object is the fundamental
unit of object-based storage that contains user data, related metadata (size, date, ownership,
etc.), and user defined attributes of data (retention, access pattern, and other business-relevant
attributes). The additional metadata or attributes enable optimized search, retention and deletion
of objects. For example, when an MRI scan of a patient is stored as a file in a NAS system, the
metadata is basic and may include information such as file name, date of creation, owner, and file
type. When stored as an object, the metadata component of the object may include additional
information such as patient name, ID, and attending physicians name, apart from the basic
metadata.
Each object stored in the object-based storage system is identified by a unique identifier called
the object ID. The object ID allows easy access to objects without the need to specify the storage
location. The object ID is generated using specialized algorithms (such as a hash function) on the
data and guarantees that every object is uniquely identified. Any changes in the object, like userbased edits to the file, results in a new object ID. Most of the object storage system supports
APIs to integrate it with software-defined data center and cloud environments.
File-based storage systems (NAS) are based on file hierarchies that are complex in
structure. Most file systems have restrictions on the number of files, directories and
levels of hierarchy that can be supported, which limits the amount of data that can be
stored.
OSD stores data using flat address space where the objects exist at the same level and one object
cannot be placed inside another object. Therefore, there is no hierarchy of directories and files,
and as a result, billions of objects are to be stored in a single namespace. This enables the OSD
to meet the scale-out storage requirement of third platform.
The OSD system is typically composed of three key components: nodes (controllers), internal
network, and storage.
The OSD system is composed of one or more nodes. A node is a server that runs the OSD
operating environment and provides services to store, retrieve, and manage data in the
system. Typically OSD systems are architected to work with inexpensive x86-based nodes,
each node provides both compute and storage resources, and scales linearly in capacity and
performance by simply adding nodes. The OSD node has two key services: metadata service
and storage service. The metadata service is responsible for generating the object ID from the
contents (may also include other attributes of data) of a file. It also maintains the mapping of
the object IDs and the file system namespace. In some implementations, the metadata service
runs inside an application server. The storage service manages a set of disks on which the user
data is stored.
The OSD nodes connect to the storage via an internal network. The internal network provides
node-to-node connectivity and node-to-storage connectivity. The application server accesses
the node to store and retrieve data over an external network.
OSD typically uses low-cost and high-density disk drives to store the objects. As more capacity
is required, more disk drives can be added to the system.
Typically, the object-based storage device has the following features:

Scale-out architecture: Scalability has always been the most important characteristic of enterprise storage
systems, since the rationale of consolidating storage assumes that the system can easily grow with
aggregate demand. OSD is based on distributed scale-out architecture where each node in the cluster
contributes with its resources to the total amount of space and performance. Nodes are independently added
to the cluster that provides massive scaling to support petabytes and even exabytes of capacity with billions
of objects that make it suitable for cloud environment.
Multi-tenancy: Enables multiple applications to be securely served from the same infrastructure. Each
application is securely partitioned and data is neither co-mingled nor accessible by other tenants. This
feature is ideal for businesses providing cloud services for multiple customers or departments within an
enterprise.
Metadata-driven policy: Metadata and policy-based information management capabilities combine to
intelligently (automate) drive data placement, data protection, and other data services (compression,
deduplication, retention, and deletion) based on the service requirements. For example, when an object is
created, it is created on one node and subsequently copied to one or more additional nodes, depending on
the policies in place. The nodes can be within the same data center or geographically dispersed.
Global namespace: Another significant value of object storage is that it presents a single global namespace
to the clients. A global namespace abstracts storage from the application and provides a common view,
independent of location and making scaling seamless. This unburdens client applications from the need to
keep track of where data is stored. The global namespace provides the ability to transparently spread data
across storage systems for greater performance, load balancing, and non-disruptive operation. The global
namespace is especially important when the infrastructure spans multiple sites and geographies.
(Contd)
Flexible data access method: OSD supports REST/SOAP APIs for web/mobile access, and
file sharing protocols (CIFS and NFS) for file service access. Some OSD storage systems
support HDFS interface for big data analytics.
Automated system management: OSD provides self-configuring and auto-healing capabilities

to reduce administrative complexity and downtime. With respect to services or processes running
in the OSD, there is no single point of failure. If one of the services goes down, and if the node
becomes unavailable, or site becomes unavailable, there are redundant components and services
that will facilitate normal operations.
Data protection: The objects stored in an OSD are protected using two methods: replication and
erasure coding. The replication provides data redundancy by creating an exact copy of an object.
The replica requires the same storage space as the source object. Based on the policy configured
for the object, one or more replicas are created and distributed across different locations. Erasure
coding technique is discussed in the next slide.
Object storage systems support erasure coding technique that provides space-optimal data
redundancy to protect data loss against multiple drive failures. In storage systems, erasure
coding can also ensure data integrity without using RAID. This avoids the capacity overhead of
keeping multiple copies and the processing overhead of running RAID calculations on very large
data sets. The result is data protection for very large storage systems without the risk of very
long RAID rebuild cycles. In general, erasure coding technique breaks the data into fragments,
encoded with redundant data and stored across a set of different locations, such as disks, storage
nodes, or geographic locations. In a typical erasure coded storage system, a set of n disks is
divided into m disks to hold data and k disks to hold coding information, where n, m, and k are
integers. The coding information is calculated from the data. If up to k of the n disks fail, their
contents can be recomputed from the surviving disks.
The figure on the slide illustrates an example of dividing a data into nine data segments (m = 9)
and three coding fragments (k = 3). The maximum number of drive failure supported in this
example is three. Erasure coding offers higher fault tolerance (tolerates k faults) than replication
with less storage cost. The additional storage requirement for storing coding segments increases
as the value of k/m increases.
This lesson covered the comparison of hierarchical file system with flat address space. This lesson
also covered the key components and key features of OSD.
10
This lesson covers object storage device implementation and OSD operations (storage and
retrieval process). This lesson also covers object storage gateway.
11
Object-based storage has two common implementation options: software-based and hardwarebased.
Software-based: In this option, the object storage software is installed on any compatible
hardware (compute system). It provides the flexibility to reuse the existing IT infrastructure
including compute systems and file-based or SAN-based storage. Some vendors offer a solution
where the object storage software can also be installed on virtual machines that acts as nodes
and enables to leverage the existing storage.
Hardware-based: In this option, the object storage software is installed on purpose-built
hardware, and typically pre-configured and pre-tested by the vendor. Hardware-based object
storage provides better performance.
12
The process of storing objects in OSD is illustrated on the slide. The data storage process in an
OSD system is as follows:
1. The application server presents the data to be stored to the OSD node.
2. The OSD node divides the data into two parts: user data and metadata.
3. The OSD node generates the object ID using a specialized algorithm. The algorithm is
executed against the contents of the user data to derive an ID unique to this data.
4. For future access, the OSD node stores the metadata and object ID using the metadata
service.
5. The OSD node stores the user data (objects) in the storage device using the storage service.
6. An acknowledgment is sent to the application server stating that the object is stored. In some
implementations, where the metadata service is running inside an application server, the
object ID is returned back to the application server.
13
The process of retrieving objects in OSD is illustrated on the slide. The process of data retrieval
from OSD is as follows:
1. The application server sends a read request to the OSD system.
2. The metadata service retrieves the object ID for the requested data.
3. The metadata service sends the object ID to the application server.
4. The application server sends the object ID to the OSD storage service for object retrieval.
5. The OSD storage service retrieves the object from the storage device.
6. The OSD storage service sends the data to the application server.
14
The capabilities or features of OSD such as multi-tenancy, scalability, geographical distribution of

data, and data sharing across heterogeneous platforms or tenants while ensuring integrity of
data, make it a strong option for cloud-based storage. Enterprise end-users and cloud subscribers
are also interested in the cloud storage offerings because it provides better agility, on-demand
scalability, lower cost, and operational efficiency compared to traditional storage solution. Cloud
storage provides unified and universal access, policy-based data placement, and massive
scalability. It also enables data access through web service or file access protocols and provides
automated data protection and efficiency to manage large amount of data. With the growing
adoption of cloud computing, cloud service providers can leverage OSD to offer storage-as-aservice, backup-as-a-service, and archive-as-a-service to their consumers.
15
The lack of standardized cloud storage APIs has made gateway appliance a crucial component for
cloud adoption. Typically service providers offer cloud-based object storage with interfaces such
as REST or SOAP, but most of the business applications expect storage resources with blockbased iSCSI or FC interfaces or file-based interfaces, such as NFS or CIFS. The cloud-based object
storage gateways provide a translation layer between these standard interfaces and service
provider's REST API. The gateway device is a physical or virtual appliance that sits in a data
center and presents file and block-based storage interfaces to the applications. It performs
protocol conversion so that data can be sent directly to cloud storage. To provide security for the
data sent to the cloud, most gateways automatically encrypt the data before it is sent. To speed
up data transmission times (as well as to minimize cloud storage costs), most gateways support
data deduplication and compression. Cloud-based object storage gateway provides a local cache
to reduce latency associated with having the storage capacity far away from the data center. The
gateway appliances offer not only an interface to the cloud, but also provide a layer of
management that can even help to determine what data should be sent to the cloud and
what data should be held locally.
16
This lesson covered object storage device implementation and OSD operations (storage and
retrieval process). This lesson also covered object storage gateway.
17
This lesson covers unified storage architecture.
18
In an enterprise data center, typically different storage systems (block-based, file-based, and
object-based storage) are deployed to meet the needs of different applications. In many cases,
this situation has been complicated by mergers and acquisitions that bring together disparate
storage infrastructures. The resulting silos of storage have increased the overall cost because of
complex management, low storage utilization, and direct data center costs for power, space, and
cooling.
An ideal solution would be to have an integrated storage solution that supports block, file, and
object access. Unified storage is a single integrated storage infrastructure that simultaneously
supports Fibre Channel (FC), Fibre-Channel-over-Ethernet (FCoE), IP Storage Area Networks
(iSCSI), Network Attached Storage (NAS) data protocols, along with REST and SOAP protocols.
There are numerous benefits associated with deploying unified storage systems:
Creates a single pool of storage resources that can be managed with a single management
interface.
Sharing of pooled storage capacity for multiple business workloads should lead to a lower
overall system cost and administrative time, thus reducing the total cost of ownership (TCO).
Provides the capability to plan the overall storage capacity consumption. Deploying a unified
storage system takes away the guesswork associated with planning for file and block storage
capacity separately.
Increased utilization, with no stranded capacity. Unified storage eliminates the capacity
utilization penalty associated with planning for block and file storage support separately.
Provides the capability to integrate with software-defined storage environment to provide next
generation storage solutions for mobile, cloud, big data, and social computing needs.
19
A unified storage architecture enables the creation of a common storage pool that can be shared
across a diverse set of applications with a common set of management processes. The key
component of a unified storage architecture is unified controller. The unified controller provides
the functionalities of block storage, file storage, and object storage. It contains iSCSI, FC, FCoE,
and IP front-end ports for direct block access to application servers and file access to NAS clients.
For block-level access, the controller configures LUNs and presents them to application servers
and the LUNs presented to the application server appear as local physical disks. A file system is
configured on these LUNs at the server and is made available to applications for storing data. For
NAS clients, the controller configures LUNs and creates a file system on these LUNs and creates a
NFS, CIFS, or mixed share, and exports the share to the clients. Some storage vendors offer
REST API to enable object-level access for storing data from the web/cloud applications.
In some implementation, there are dedicated or separate controllers for block functionality, NAS
functionality, and object functionality.
20
This lesson covered unified storage architecture.
21
The Concepts in Practice section covers the product example of object-based and unified storage.
It covers products such as EMC Atmos, EMC ECS Appliance, EMC VNX, EMC VMAX3, and EMC
CloudArray.
Note:
22
EMC Atmos is a cloud storage platform for enterprises and service providers to deploy public,
private, or hybrid cloud storage. It enables to store, manage, and protect globally distributed,
unstructured content at scale. Atmos is a scale-out object architecture that stores data as objects
with the associated metadata. It enables storage to be scaled out without the need to rewrite
applications. Some of the key cloud features of Atmos include a global namespace, REST APIdriven storage, multi-tenancy, self-service, and metering and chargeback.
EMC ECS Appliance provides a hyper-scale storage infrastructure that is specifically designed to
support modern applications with unparalleled availability, protection, simplicity, and scale. It
provides universal accessibility with support for object, and HDFS. ECS Appliance enables cloud
service providers to deliver competitive cloud storage services at scale. ECS provides a single
platform for all web, mobile, Big Data, and social media applications.
23
EMC VNX family is a group of products that provide a unified storage platform that consolidates
block, file, and object access into one solution. It enables organizations to dynamically grow,
share, and manage multi-protocol file systems and multi-protocol block storage access. The VNX
operating environment enables Windows and UNIX/Linux users to share files using NFS and CIFS.
It also supports FC, iSCSI, and FCoE access. EMC VNX delivers file access (NAS) functionality via
X-Blades (Data Movers) and block access functionality via storage processors. Optionally it offers
object access to the storage using EMC Atmos Virtual Edition (Atmos VE). It is ideally suited for
applications with predictable workloads that require moderate-to-high throughput.
EMC VMAX3 architecture is purpose built for hybrid cloud. It delivers mission-critical storage with
the scale, performance, availability, and agility to meet the high demands of extreme data growth
in cloud environment. VMAX3 unified storage provides embedded file data services (eNAS) that
enable to consolidate islands of block and file storage. The VMAX3 series is an innovative platform
built around a scalable Dynamic Virtual Matrix architecture to support the storage growth
demands of cloud environments. It also supports multiple protocols for host connectivity. VMAX3
storage systems provide business continuity solution by supporting various local and remote
replications.
EMC CloudArray is a middleware system that rests between the application layer and the cloud
storage. CloudArray enables organizations to scale their SAN and NAS with on-demand cloud
capacity. It moves infrequently accessed data to lower cost cloud storage tiers. It introduces a
caching system to reduce cloud access latency and several other features such as encryption and
snapshots. CloudArray provides a robust, multi-layered security to ensure data is
protected with two layers of AES 256-bit military grade encryption.
24
This module covered the key OSD components. This module also covered the key features of
OSD, the process of storage and retrieval of data in an object-based storage. Finally, this module
covered unified storage architecture.
25
26
This module focuses on software-defined storage and its attributes. This module also focuses on
software-defined storage architecture. Further, this module focuses on the functions of the control
layer of software-defined storage. Finally, this module focuses on software-defined storage
extensibility.
Module 8: Software-Defined Storage
This lesson covers the drivers for software-defined storage. This lesson also covers the definition
and attributes of software-defined storage. Further, this lesson covers software-defined storage
architecture. Finally, this lesson covers the benefits of software-defined storage.
In a traditional data center, there are several challenges in provisioning and managing storage in
an efficient and cost-effective manner. Some key challenges are described here.
In a traditional environment, each application type normally has its own vertical stack of compute,
networking, storage, and security. This leads to the creation of a loose collection of IT silos, which
increases the infrastructures complexity. This creates management overhead and increases
operating expenses. It also leads to poor resource utilization because capacity cannot be shared
across stacks.
Data centers have multi-vendor, heterogeneous storage systems, and each type of storage
system (block-based, file-based, and object-based) has its own unique value. However, critical
functionality is often tied to specific storage types, and each storage system commonly has its
own monitoring and management tools. There is limited resource sharing, no centralized
management, a little automation, and a lack of standards in this environment.
Application workload complexities and higher SLA demands pose a further challenge to IT. IT
finds it difficult to allocate storage to satisfy the capacity requirements of applications in real time.
There are also new requirements and expectations for continuous access and delivery of
resources as in a cloud environment.
Traditional environments are not architected for third platform technologies such as cloud
computing, Big Data analytics, and mobile applications. Therefore, there are several challenges in
managing massive data growth, cost-effective scaling, and providing self-service access to
storage.
These challenges have led to the advent of the software-defined storage model, which is covered
next.
Software-defined storage (SDS) is a storage infrastructure that is managed and automated by

software. SDS abstracts heterogeneous storage systems and their underlying capabilities, and
pools the storage resources. Storage capacity is dynamically and automatically allocated from the
storage pools based on policies to match the needs of applications.
In general, SDS software abstracts the physical details of storage (media, formats, location, lowlevel hardware configuration), and delivers storage as software. A storage system is a
combination of hardware and software. The software stack exposes the data access method such
as block, file, or object, and uses persistent media such as HDD or SSD to store the data. SDS
software separates the software layer of a storage system from the hardware. It supports
combinations of multiple storage types and access methods, such as block, file, and object. It
enables storing data on both storage systems and commodity disks, while providing a unified
external view of storage. This allows organizations to reuse existing storage assets, and mix and
match them with commodity resources, while serving data through a single namespace and
storage system spread across these different assets. For example, in a data center that contains
several distinct file servers, SDS can provide a global file system, spanning the file servers and
allowing location-independent file access. This is similar to how the local file system on a compute
system hides the underlying disk block structure and manifests file access interface to the storage
media.
SDS enables organizations to build modern, hyperscale storage infrastructure in a cost-effective
manner using standardized, commercial off-the-shelf components. The components individually
provide lower performance. However, at sufficient scale and with the use of SDS software, the
pool of components provides greater capacity and performance characteristics.
SDS transforms existing heterogeneous physical storage into a simple, extensible, and open
virtual storage platform. The key attributes of software-defined storage are as follows:
Storage abstraction and pooling: SDS abstracts and pools storage resources across
heterogeneous storage infrastructure. SDS software creates a single large storage pool with
the underlying storage resources, from which several virtual storage pools are created. SDS
decouples the storage control path from the data path. Applications connect to storage via the
data path.
Automated, policy-driven storage provisioning: A storage service is some combination

of capacity, performance, protection, encryption, and replication. In the SDS model, storage
services are dynamically composed from available resources. SDS uses application policies to
create a just-in-time model for storage service delivery. Storage assets and capabilities are
configured and assigned to specific applications only when they are needed. If the policy
changes, the storage environment dynamically and automatically responds with the new
requested service level.
Unified management: SDS provides a unified storage management interface that provides an
abstract view of the storage infrastructure. Unified management provides a single control point
for the entire infrastructure across all physical and virtual resources.
Self-service: Resource pooling enables multi-tenancy, and automated storage provisioning

enables self-service access to storage resources. Users select storage services from a selfservice catalog and self provision them.
Open and extensible: An SDS environment is open and easy to extend enabling new
capabilities to be added. An extensible architecture enables integrating multi-vendor storage,
and external management interfaces and applications into the SDS environment through the
use of application programming interfaces (APIs).
The figure on the slide depicts the generic architecture of a software-defined storage
environment. Although the physical storage devices themselves are central to SDS, they are not a
part of the SDS environment. Physical storage may be block-based, file-based, or object-based
storage systems or commodity hardware.
The fundamental component of the SDS environment is the policy-driven control plane, which
manages and provisions storage. The control plane is implemented through software called SDS
controller, which is also termed as a storage engine in some SDS products. The SDS controller
is software that manages, abstracts, pools, and automates the physical storage systems into
policy-based virtual storage pools. Through the use of automation and orchestration, the
controller enables self-service access to a catalog of storage resources. Users provision storage
using data services, which may be block, file, or object services. The functions of the control
plane are covered in the next lesson.
An SDS controller may provide either all or a part of the features and services shown in the
architecture. For example, an SDS controller may only support file and block data services. Some
controllers may also support the Hadoop Distributed File System (HDFS). Some SDS products
provide the feature of creating a block-based storage pool from the local direct-attached storage
(DAS) of x86-based commodity servers in a compute cluster. The storage pool is then shared
among the servers in the cluster. This is covered in the next slide.
The REST API is the core interface to the SDS controller. All underlying resources managed by the
controller are accessible through the API. The REST API makes the SDS environment open and
extensible, which enables integration of multi-vendor storage, external management tools, and
written applications. The API also integrates with monitoring and reporting tools. Further, the API
provides access to external cloud/object storage. APIs and extensibility are covered in Lesson 3,
Software-Defined Storage Extensibility of this module.
A compute-based storage area network is a software-defined virtual SAN created from the directattached storage located locally on the compute systems in a cluster. A compute-based SAN
software creates a large pool of block-based storage that can be shared among the compute
systems (or nodes) in the cluster. This creates a large-scale SAN without storage systems, and
enables leveraging the local storage of existing compute systems. The convergence of storage
and compute ensures that the local storage on compute systems, which often goes unused, is not
wasted.
A compute system that requires access to the block storage volumes, runs a client program. The
client program is a block device driver that exposes shared block volumes to an application on the
compute system. The blocks that the client exposes can be blocks from anywhere within the
compute-based SAN. This enables the application to issue an I/O request and the client fulfills it
regardless of where the particular blocks reside. The client communicates with other compute
systems either over Ethernet (ETH) or Infiniband (IB) a high-speed, low latency communication
standard for compute networking. The compute systems that contribute their local storage to the
shared storage pool within the virtual SAN, run an instance of a server program. The server
program owns the local storage and performs I/O operations as requested by a client from a
compute system within the cluster.
A compute-based SANs control component, known as the metadata manager, serves as the
monitoring and configuration agent. It holds cluster-wide mapping information and monitors
capacity, performance, and load balancing . It is also responsible for decisions regarding
migration, rebuilds, and all system-related functions. The metadata manager is not on the virtual
SAN data path, and reads and writes do not traverse the metadata manager. The metadata
manager may communicate with other compute-based SAN components within the cluster in
order to perform system maintenance and management operations but not data operations. The
metadata manager may run on a compute system within the compute-based SAN, or on an
external compute system.
The key benefits of software-defined storage are described below:

Simplified storage environment: SDS breaks down storage silos and their
associated complexity. It also provides centralized management across all physical and
virtual storage environments. This simplifies management by enabling administrators to
centralize storage management and provisioning tasks.
Operational efficiency: Automated policy-driven storage provisioning improves

quality of services, reduces errors, and lowers operational cost. It provides faster
streamlined storage provisioning, which enables new requirements to be satisfied more rapidly.
Agility: The ability to deliver self-service access to storage via a service catalog
provides agility and reduces time-to-market.
Reusing existing infrastructure: SDS supports multi-vendor storage systems and
commodity hardware, which enables organizations to work with their existing
infrastructure. It also enables the SDS environment to be managed through external
management interfaces. This protects the current investments of organizations.
Cloud support: SDS enables an enterprise data center to connect to external cloud
storage services for consuming services such as cloud-based backup, and disaster
recovery. In addition, SDS facilitates extending object storage to existing file and
block-based storage, which enables organizations to deploy mobile and cloud
applications on their existing infrastructure.
This lesson covered the drivers for software-defined storage. This lesson also covered the
definition and attributes of software-defined storage. Further, this lesson covered softwaredefined storage architecture. Finally, this lesson covered the benefits of software-defined storage.
This lesson covers the functions of the control plane of software-defined storage. This lesson
covers asset discovery. This lesson also covers resource abstraction and pooling. Further, this
lesson covers resource provisioning for services.
10
The key functions of the control plane include asset discovery, resource abstraction and pooling,
provisioning resources for services, and providing support for data protection. These functions are
covered in the subsequent slides. The control plane in software-defined storage is implemented
by SDS controller software, which enables storage management and provisioning.
An SDS controller commonly provides two native user interfaces: a command-line interface (CLI)
and a graphical user interface (GUI). Both the interfaces may either be integrated into the
controller, or may be external to it. If the native user interfaces are external then they leverage
the REST API to interact with the controller.
The CLI provides granular access to the controllers functions and more control over controller
operations as compared to the GUI. The GUI is a browser-based interface that can be used with a
supported web browser. The GUI may be used by both storage administrators and by end users.
For this, the GUI has two views: an administrator view and a user view. The administrator view
enables an administrator to carry out tasks such as managing the infrastructure, creating service
catalogs, and defining storage services. The user view enables an end user to access the service
catalog and self-provision storage services.
11
An SDS controller automatically detects an asset when it is added to the SDS environment. The
controller uses the assets credentials to connect to it over the network, and either obtains or
confirms its configuration information. This process is called discovery. Discovery can also be
initiated manually to verify the status of an asset. Examples of assets are storage systems,
storage networks, compute systems and clusters, and data protection solutions. If the asset is a
storage system, the controller collects information about the storage ports and the pools that it
provides. If the asset is a compute system, the controller discovers its initiator ports. Clusters can
also be discovered, allowing volumes to be provisioned to the compute systems in the cluster.
The controller can also discover the storage area networks within a data center. Asset discovery is
covered in detail in Module 16, Managing the Storage Infrastructure.
12
Data centers commonly contain many physical storage systems of different types and often from
multiple manufacturers. Each physical storage system must also be individually managed, which
is time consuming and error prone.
An SDS controller exposes the storage infrastructure through a simplified model, hiding and
handling details such as storage system and disk selection, LUN creation, LUN masking, and the
differences between the storage systems. The SDS controller leverages the intelligence of
individual storage systems. It abstracts storage across the physical storage systems and manages
individual components. This allows administrators and users to treat storage as a large resource.
It enables focusing just on the amount of storage needed, and the performance and protection
characteristics required.
13
A virtual storage system is an abstraction of physical storage systems, and the network
connectivity between compute systems and the storage systems. A virtual storage system, also
called as a virtual array, provides a more abstract view of the storage environment for
provisioning and applying policy. In a typical physical environment there are multiple storage
systems, each with their own management tools, processes, and best practices. With the virtual
storage system, all of the unique capabilities of the physical storage systems are available, but an
SDS controller automates the operations of the tools, processes, and best practices. This
simplifies storage provisioning across a heterogeneous storage infrastructure. With the physical
storage systems configured into virtual storage systems, an administrator can build policies that
are automatically applied across heterogeneous storage.
All the physical storage systems in a virtual storage system have equivalent network connectivity.
When a storage administrator adds physical storage systems to the SDS environment, the SDS
controller discovers their storage pools, ports, and configuration. Similarly, the controller also
automatically discovers and maps the networks. After examining the connectivity between hosts
and storage systems, the administrator can build the virtual storage systems. When populating a
virtual storage system with physical storage systems and networks, the administrator ensures
that the storage that is presented from the virtual storage system to a compute, is physically
reachable from the compute. When all compute systems can reach all storage systems, the entire
storage infrastructure can be grouped into a single virtual storage system. However, physical
storage systems may have to be placed into separate virtual storage systems to accommodate
different physical configurations and different requirements for fault tolerance, network isolation,
or tenant isolation.
14
During the configuration of the SDS environment, the physical storage system pools are mapped
into virtual storage pools. A virtual storage pool is an abstraction that represents a standardized
storage service offering out of which storage may be provisioned. There are three main types of
virtual storage pools: block, file, and object. Block virtual storage pools and file storage virtual
pools are sets of block and file storage capabilities that meet various storage performance and
cost needs. Object virtual storage pools store object data on underlying NAS systems or
commodity hardware.
Virtual storage pools expose performance and protection levels from the storage to the users.
When defining a virtual storage pool, the administrator separates the types of pools, and then
selects the tiers of performance characteristics that are to be exposed to users. The administrator
also identifies the level of data protection available to each pool. The SDS controller automatically
matches existing physical pools on the storage systems to the virtual storage pool characteristics
that an administrator specifies. The administrator can enable automatic assigning of the matching
physical pools to the virtual storage pool, or carry out the process manually.
Users consume storage from virtual storage pools that an administrator makes available to them,
which relieves storage administrators from provisioning tasks. End users only need to know the
type of storage (virtual storage pool) and the compute system/cluster to which the storage should
be attached. They do not have to know the details of the underlying physical storage
infrastructure. When storage is provisioned, users identify which virtual storage pool they want to
use. The SDS solution applies built-in best practices to select the best physical storage system
and the storage pool that meet the provisioning request.
There is generally no exclusive relationship between virtual storage systems and virtual storage
pools. Virtual storage pools can have physical storage pools from multiple virtual storage
systems; additionally these physical storage pools can belong to multiple virtual pools. A virtual
storage pool can reside in a single virtual data center, or it can span multiple virtual data centers.
15
In software-defined storage, a virtual data center is a collection of storage infrastructure that can
be managed as a cohesive unit by data center administrators. The virtual data center enables
an administrator to discover physical storage and abstract it into virtual storage systems and
virtual storage pools. Geographical co-location of storage systems in a virtual data center is not
necessary. An SDS controller can scale to manage one data center or multiple data centers
across geographies. However, high bandwidth and low latency are assumed in the virtual data
center. The data centers are connected using business continuity solutions for high availability
(HA) and disaster recovery (DR). HA and DR are covered in Module 12, Introduction to Business
Continuity.
The virtual data center may also have a multisite configuration, where multiple SDS controllers
manage multiple data centers in different locations. In this type of configuration, the
SDS controllers behave as a loosely-coupled federation of autonomous virtual data centers.
16
An administrator can configure the SDS environment with multiple tenants, where each tenant is
a group of multiple users. Each tenant can be configured with an authentication system, and
users with specific assigned roles can be mapped into tenants. The block, file and object storage
resources can be grouped logically into collections, which can be assigned to the tenants. Having
a tenant-specific configuration enables an administrator to restrict access to resources, based on
tenant.
Each tenant has its own custom environment for creating and managing storage which cannot be
accessed by users from other tenants. For example, certain virtual storage systems and/or virtual
pools might only be accessible to specific tenants. Multi-tenant configuration allows each tenant to
manage its own version of the service catalog and restrict access for tenant members to specific
services and service categories.
17
After configuring the storage abstractions, an administrator customizes and exposes storage
services by creating service catalogs for tenants. The administrator uses the GUIs administrator
view to create storage services and organize them into categories in a service catalog. The service
catalog provides the tenant users with access to the set of predefined storage services. An
administrator can create different categories of services such as block service, file service, and
object service. The administrator can configure the different services within each category, and
also restrict them to specific users or user groups. The subsequent slides describe these services.
The user view of the GUI provides users within a tenant with access to their service catalog. The
user view presents all the services and categories that are available for provisioning for a specific
user. Users can request a service by simply clicking the service and placing a request to run it.
Some SDS platforms may not provide an interface for users to request services, and require the
use of external client software.
An SDS controller automates the provisioning of resources when a user requests for a service. It
employs a policy-based placement algorithm to find the best fit in the infrastructure to fulfill user
requests for data services. The SDS controller uses orchestration for automating the provisioning
process. Orchestration uses workflows to automate the arrangement, coordination, and
management of various functions required to provision resources. As a result, provisioning does
not require administrator or user interaction. Orchestration and workflows are covered in detail in
Module 16, Managing the Storage Infrastructure.
The administrator can view the details and the progress of placed requests in real time. The
details include which service was requested, which parameters were specified in the service
request, who requested it, the outcome of the request submission, and the affected resources and
volumes.
18
The block data service provides a block volume of required size, performance level, and protection
level to a user. Examples of the services that an administrator defines in this service category are
as follows:
Create a block volume: A user can create a block storage volume by selecting a virtual
storage system and virtual pool. On receiving the request, the SDS controller chooses the
physical pool from the selected virtual pool and storage system. It creates a block volume,
which corresponds to a LUN on the storage system.
Delete a block volume: A user can delete an existing volume. On receiving the request, the
SDS controller destroys the volume from the physical storage pool.
Bind a block volume to compute: A user can assign a block volume to a selected compute
system/cluster. On receiving this request, the SDS controller binds the block volume to the
specified compute system/cluster. However, the volume cannot be written to or read from
unless it is mounted.
Unbind block volume from compute: A user can unbind a volume from a compute
system/cluster. This simply makes the block volume invisible to the compute.
Mount a block volume: A user can mount a block volume on a compute system/cluster. The
SDS controller sends commands to the OS to mount the volume. This operation is specific to
the type of OS on the compute system such as Windows, Linux, and ESXi.
Unmount block volume: A user can unmount a block volume from a compute system/cluster.
On receiving the request, the SDS controller sends commands to the compute to unmount the
volume.
Expand block volume: A user can expand/extend a block volume by combining it either with
a newly created volume or with an existing volume. On receiving the request to expand a
volume, the SDS controller commands the storage system to expand the LUN.
19
The file data service enables users to create NFS and CIFS file shares and export them to
compute systems. Examples of the services that an administrator defines in this service category
are as follows:
Create file share: A user can create a new file share. On receiving the request, the SDS
controller chooses a physical pool from the selected virtual pool and storage system. For a
Windows compute, it creates a new file system from the selected virtual storage system based
on the criteria of the selected virtual pool, and creates a CIFS share. For a Linux compute, the
SDS controller does the same to create an NFS share.
Expand file share: A user can expand a file system. The SDS controller treats all file storage
expansion the same without differentiating based on the type of compute. It simply selects the
file system and expands it by commanding the storage system to provision more storage.
Delete file share: A user can remove a file system. On receiving the request, the SDS
controller sends a command to unmount it if it is mounted on a compute, and reclaims the
storage on the storage system. When deleting file storage, an SDS controller does not
differentiate between Linux or Windows. The file share can be NFS or CIFS.
20
The object data service enables data to be stored, accessed, and manipulated as objects on
object-based, file-based, and commodity storage platforms managed by an SDS controller.
Objects are stored in the object virtual pool within containers called buckets. A bucket is a
logical grouping of objects and is similar to a directory of files. Buckets can be used to control
access to objects and to set properties that define attributes for all contained objects, such as
retention periods and quotas.
A namespace connects the object data service to an object virtual pool. There may be a single
namespace or an administrator may create a namespace for each tenant. Object buckets within
object virtual pools are segregated based on the namespace of the tenant. Users and applications
access the object virtual pool through the configured namespace. A namespace restricts access to
buckets and the objects to users that have the appropriate permissions.
Examples of the services that an administrator defines in the object data service category are as
follows:
Create an object bucket: A user can create a new object data service bucket to write and
read objects. The user can specify attributes for the bucket such as name, owner, namespace,
capacity, and access permissions.
Edit an object bucket: A user can edit the attributes of an existing object bucket.
Delete an object bucket: A user can delete an existing bucket.
Ingest file share into an object bucket: A user can perform ingestion of a file share into an
existing bucket. This imports data in bulk from an existing file system into the bucket.
21
An SDS controller ensures data availability and protection against data corruption,
hardware failures, and data center disasters. The controller itself may not implement data
protection features such as high availability, backup, and replication. It usually leverages the
protection technology that is either natively built into the underlying storage systems, or offered
by external solutions. After a data protection solution is added to the controller, an administrator
can select it to provide data protection when virtual pools are created. The controller uses this
technology to compose data services that precisely meet performance, protection, and availability
requirements. Data protection is covered in Module 12, Introduction to Business Continuity,
Module 13, Backup and Archive, and Module 14, Replication.
Block protection provides features such as configuring failover, creating snapshots of block
volumes, and continuous data protection. A snapshot is a point-in-time virtual copy of a block
volume. Snapshots and continuous data protection are covered in Module 14, Replication. File
protection provides features such as creating snapshots of file systems, restoring snapshots, and
removing snapshots. A file system snapshot is a point-in-time virtual copy of a set of files. File
system snapshots are covered in Module 14, Replication. Object protection provides features
such as replication and erasure coding for data protection. Erasure coding was introduced in
Module 7, Object-based and Unified Storage, and will be further discussed in Module 14,
Replication.
22
This lesson covered the functions of the control plane of software-defined storage. This lesson
covered asset discovery. This lesson also covered resource abstraction and pooling. Further, this
lesson covered resource provisioning for services.
23
This lesson covers extensibility of software-defined storage. This lesson covers application
programming interface (API) and RESTful API. This lesson also covers integrating external
management tools and applications with software-defined storage.
24
An application programming interface (API) is a set of programmatic instructions and

specifications that provides an interface for software components to communicate with each
other. It specifies a set of routines (operations), input parameters, outputs/responses, data
types, and errors. The routines can be called from a software component allowing it to interact
with the software providing the API. Thus, an API provides a programmable interface, which is a
means for communicating with an application without understanding its underlying architecture.
This allows programmers to use the component-based approach to build software systems. APIs
may be pre-compiled code that is leveraged in programming languages, and can also be webbased.
In the context of web development, a web-based API is typically defined as a set of Hypertext
Transfer Protocol (HTTP) request messages and the structure of response messages. The
response messages are usually in an Extensible Markup Language (XML) or JavaScript Object
Notation (JSON) format. Web-based APIs may be implemented as Simple Object Access Protocol
(SOAP) based web services or as representational state transfer (REST) APIs. A detailed
description of APIs is beyond the scope of this course. This lesson provides an overview of APIs,
and focuses primarily on REST APIs.
25
As third platform technologies become more prevalent, the ability to dynamically adapt to
variations in application workloads and storage requirements is becoming increasingly important.
The next-generation software-defined data centers and cloud stacks are powered by APIs. With
advancements in technology, APIs are providing improving communication and connectivity
between IT systems, and increasing agility through automation.
APIs provide a flexible, easy-to-use means for integrating third-party applications and capabilities
into existing infrastructure. This integration also provides a layer of security between public
(external) and private (internal) business capabilities, allowing organizations to provide services in
the way they see fit while offering end-users a variety of services. For example, a public cloud
storage provider may provide an API that allows a consumer-written application to access and use
cloud storage as regular storage. Similarly, online social networks may provide APIs that enable
developers to access to the feeds of their users. Further, with the advent of the Internet of
Things, devices enabled with web-based APIs are becoming common. APIs enable the smart
devices to communicate with each other and with applications.
In a software-defined data center, APIs enable automated provisioning of resources from
compute, storage, and networking pools to ensure that SLAs are met. The use of APIs is enabling
software-defined storage to be easily managed and provisioned. In SDS, the REST API provides
the interface to all underlying resources. Management interfaces use the API to provision,
manage, monitor, and meter logical storage resources. The API also provides a means to
integrate with multi-vendor storage systems and external storage platforms. It also offers a
programmable environment enabling developers and users to extend SDS functionality.
26
Representational State Transfer (REST) is a client-server software architecture approach

that was originally introduced for building large-scale, distributed hypermedia (for
example, hypertext, audio, video, image, and text) systems. REST is not a standard but
rather an architectural style that has become a popular choice for developing HTTP-based
APIs called RESTful APIs. It leverages HTTP methods such as GET, POST, PUT, DELETE
for client-server interaction. It supports the resource-oriented architecture for the
development of scalable and lightweight web applications while adhering to a set of
constraints.
REST-based communication provides simple, human-readable data access methods.
RESTful APIs do not require XML-based web service protocols such as SOAP to support
their light-weight interfaces. However, they still support XML-based and JSON data
formats. These services provide an easy means to consume services, and support the
combination of multiple web resources into new applications. Recent trends reveal
increasing adoption of REST for developing APIs to provide simple and cost-effective requestbased services, and support the demand for real-time data.
27
The implementation of RESTful APIs follows four basic design principles:

1. Resource identification using URI: In REST, data and functionality are treated as
resources. Each resource is accessed using a Uniform Resource Identifier (URI), which is a
string of characters that uniquely identifies a resource. URIs typically have a directory-like
structure. URIs provide a global addressing space for resource and service discovery. There
can be almost limitless set of valid URIs that can be used to access resources to the finest
levels of granularity.
2. Use of standard HTTP methods: Resources are manipulated using a fixed set of four create,
read, update, delete operations: PUT, GET, POST, and DELETE. PUT creates a new resource,
which can then be deleted by using DELETE. GET retrieves the current state of a resource in
some representation. POST transfers a new state onto a resource.
3. Self-descriptive resource representation: A resource representation typically reflects the
current state of a resource, and its attributes, at the time an application requests it. A service
requester performs actions on a resource by using a representation of that resource. A
representation has enough information to manipulate a resource. Resources are decoupled
from their representation so that their content can be accessed in different formats, such as
HTML, XML, plain text, and JSON. The use of standard formats allows RESTful APIs to be used
by applications written in different languages and running on different computing platforms.
4. Stateless design: REST architecture is based on stateless interaction with resources. This
includes the use of self-contained request messages. A service requester includes all of the
data needed by the service provider to generate a response within the HTTP request. This
eliminates the need to store application state at the service provider between the requests
and to retrieve the state while processing a request. Statelessness improves API performance,
because it offloads the responsibility of maintaining the application state to the requesting
application, reducing server-side resource utilization.
28
The REST API enables the extensibility of the SDS functionality through integration with written
applications, and external management tools and cloud stacks such as VMware, Microsoft, and
OpenStack. This provides an alternative to provisioning storage from the native management
interface. The open platform enables users and developers to write new data services. This
enables building an open development community around the platform. The API also integrates
with tools for monitoring and reporting system utilization, performance, and health. This also
enables generating chargeback/showback reports. The API may also support cloud/object storage
platforms such as EMC Atmos, Amazon S3, and OpenStack Swift. Further, the API may also
support integration with HDFS for running Hadoop applications.
The REST API describes the programmatic interfaces that allow users to create, read, update, and
delete resources through the HTTP methods PUT, GET, POST, and DELETE respectively. The REST
API is accessible using any web browser or programming platform that can issue HTTP requests.
The browser may require a special plugin such as httpAnalyzer for Internet Explorer, Poster for
Firefox, and PostMan for Chrome. The REST API may also be accessed using scripting platforms
such as Perl. Vendors may also provide class libraries that enable developers to write applications
that access the SDS data services.
29
The example on the slide depicts a sample HTTP POST request to the REST API to provision a
block volume. An application can use HTTP GET commands to retrieve information about virtual
pools and virtual storage systems in the SDS environment. The user receives the unique
identification for virtual pools (Virtual Pool ID) and virtual storage systems (Virtual Storage
System ID) that are used to identify them. The request to create a block volume contains the
information necessary to provision the volume. The body of the request specifies the name and
size for the volume along with the Virtual Pool ID and Virtual Storage System ID where the block
volume should be created. After the volume is created, the response would include the volume
information including its unique identifier.
30
This lesson covered extensibility of software-defined storage. This lesson covered application
programming interface (API) and RESTful API. This lesson also covered integrating external
management tools and applications with software-defined storage.
31
The Concepts in Practice section covers EMC ViPR Controller, EMC ECS, and EMC ScaleIO.
Note:
32
EMC ViPR Controller is a storage-defined storage platform that abstracts, pools, and automates
a data centers physical storage infrastructure. It delivers block and file storage services on
demand through a self-service catalog. It supports data protection across geographicallydispersed data centers. It provides a single control plane to manage heterogeneous storage
environments, including EMC and non-EMC block and file storage. ViPR Controller also provides a
REST-based API making the storage architecture extensible. It supports multiple vendors enabling
organizations to choose storage platforms from either EMC or third-party. It also supports
different cloud stacks such as VMware, Microsoft, and OpenStack. ViPR Controller development is
driven by the open source community, which enables expanding its features and functionalities.
EMC ECS is a software-defined cloud storage platform that supports the storage of unstructured
data on a massive scale on commodity hardware. ECS supports object and HDFS data services
currently, with file services to be added in the future. It delivers a storage platform with features
such as multi-tenancy, metering, and self-service provisioning. It protects data against node,
disk, and site failures through replication and erasure coding. ECS provides a management REST
API that enables the object store to be configured and managed. Organizations can either buy the
EMC ECS Appliance which is a complete storage appliance with bundled hardware and software, or
ECS can be bought as a software product. ECS can be installed on servers or EMC-certified
commodity disks.
EMC ScaleIO is software that creates a compute-based SAN from local server storage. It
converges storage and compute resources of commodity hardware into a single-layer
architecture, aggregating capacity and performance, simplifying management, and scaling to
thousands of nodes. As an alternative to a traditional SAN infrastructure, ScaleIO combines HDDs,
SSDs, and PCIe flash cards to create a virtual pool of block storage with varying performance
tiers. ScaleIOs architecture allows for non-disruptive elastic scaling out by simply adding nodes to
the environment. It is hardware-agnostic and supports physical and/or virtual servers. All servers
participate in servicing I/O requests using massively parallel processing.
33
This module covered software-defined storage and its attributes. This module also covered
software-defined storage architecture. Further, this module covered the functions of the control
layer of software-defined storage. Finally, this module covered software-defined storage
extensibility.
34
This module focuses on the third platform requirements for SAN, software-defined networking, FC
SAN components, FC architecture, FC SAN topologies, link aggregation, and zoning. This module
also focuses on virtualization in FC SAN environment.
Module 9: Fibre Channel (FC) SAN
This lesson covers the definition and benefits of SAN, third platform requirements for SAN,
technology solutions for SAN, and software-defined networking.
Storage area network (SAN) is a network that primarily connects the storage systems
with the compute systems and also connects the storage systems with each other. It
enables multiple compute systems to access and share storage resources. It also enables
data transfer between the storage systems. With long-distance SAN, the data transfer
over SAN can be extended across geographic locations. A SAN usually provides access to
block-based storage systems.
SANs address the limitations of direct-attached storage (DAS) environment. Unlike a DAS
environment, where the compute systems own the storage, SANs enable both consolidation and
sharing of storage resources across multiple compute systems. This improves the utilization of
storage resources compared to a DAS environment and reduces the total amount of storage that
an organization needs to purchase and manage. With consolidation, storage management
becomes centralized and less complex, which further reduces the cost of managing information.
A SAN may span over wide locations. This enables organizations to connect
geographically dispersed compute systems and storage systems. The long-distance SAN
connectivity enables the compute systems across locations to access shared data. The
long-distance connectivity also enables the replication of data between storage systems
that reside in separate locations. The replication over long-distances helps in protecting
data against local and regional disaster. Further, the long-distance SAN connectivity
facilitates remote backup of application data. Backup data can be transferred through a
SAN to a backup device that may reside at a remote location. This avoids having to ship
tapes (backup media) from the primary site to the remote site and the associated pitfalls
such as packing and shipping expenses and lost tapes in transit.
The IT industry is in the middle of a massive technological and structural shift toward what the
industry analyst IDC calls the third platform. Third platform is built on social networking, mobile
computing, cloud services, and big data analytics technologies. Applications that support these
technologies require significantly higher performance, scalability, and availability compared to the
traditional applications. Similar to the compute and storage infrastructure, the SAN infrastructure
must also be ready to support the requirements of third platform applications. Therefore, it is
necessary to establish how the third platform application requirements are translated into the
SAN requirements. This slide provides a list of key requirements for an effective SAN
infrastructure.
Technology solutions that can meet the third platform requirements for the SANs are:
Software-defined networking
SAN implementations:
Fibre Channel (FC) SAN
Internet Protocol (IP) SAN
Fibre Channel over Ethernet (FCoE) SAN
Virtualization in SAN
Both the software-defined networking and the FC SAN are covered in this module. Module 10,
Internet Protocol (IP) SAN and module 11, FC over Ethernet (FCoE) SAN describe IP SAN and
FCoE SAN respectively. Virtualization in SAN is described along with each of the SAN
implementations.
Traditionally, a network component such as a switch or a router consists of a data plane and a
control plane. These planes are implemented in the firmware of the network components. The
function of the data plane is to transfer the network traffic from one physical port to another port
by following rules that are programmed into the component. The function of the control plane is
to provide the programming logic that the data plane follows for switching or routing of the
network traffic. Software-defined networking is an approach to abstract and separate the control
plane functions from the data plane functions. Instead of the built-in control functions at the
network components level, the software external to the components takes over the control
functions. The software runs on a compute system or a standalone device and is called network
controller. The network controller interacts with the network components to gather configuration
information and to provide instructions for data plane in order to handle the network traffic.
Software-defined networking in the SAN provides several benefits. These are described below:
Centralized control: The software-defined approach provides a single point of control for the
entire SAN infrastructure that may span across data centers. The centralized control plane
provides that programming logic for transferring the SAN traffic, which can be uniformly and
quickly applied across the SAN infrastructure. The programming logic can be upgraded
centrally to add new features and based on application requirements.
Policy-based automation: With software-defined approach, many hardware-based SAN
management operations such as zoning (described later) can be automated.

Management operations may be programmed in the network controller based on
business policies and best practices. This reduces the need for manual operations that
are repetitive, error-prone, and time-consuming. Policy-based automation also helps
to standardize the management operations.
Simplified, agile management: The network controller usually provides a management

interface that includes a limited and standardized set of management functions. With policybased automation in place, these management functions are available in simplified form,
abstracting the underlying operational complexity. This makes it easy to configure a SAN
infrastructure. This also helps promptly modifying the SAN configuration to respond to the
changing application requirements.
This lesson covered the definition and benefits of SAN, third platform requirements for SAN,
technology solutions for SAN, and software-defined networking.
This lesson covers the components of FC SAN, three FC interconnectivity options, and FC port
types.
10
Fibre Channel SAN (FC SAN) uses Fibre Channel (FC) protocol for communication. FC protocol
(FCP) is used to transport data, commands, and status information between the compute systems
and the storage systems. It is also used to transfer data between the storage systems.
FC is a high-speed network technology that runs on high-speed optical fiber cables and serial
copper cables. The FC technology was developed to meet the demand for the increased speed of
data transfer between compute systems and mass storage systems. In comparison with UltraSmall Computer System Interface (Ultra-SCSI) that is commonly used in the DAS environments,
FC is a significant leap in storage networking technology. The latest FC implementations of 16
GFC offer a throughput of 3200 MB/s (raw bit rates of 16 Gb/s), whereas Ultra640 SCSI is
available with a throughput of 640 MB/s. FC is expected to come with 6400 MB/s (raw bit rates of
32 Gb/s) and 25600 MB/s (raw bit rates of 128 Gb/s) throughput in 2016. Technical Committee
T11, which is the committee within International Committee for Information Technology
Standards (INCITS), is responsible for FC interface standards.
FC supports credit-based flow control mechanism (described later). The flow control mechanism
delivers data as fast as the destination buffer is able to receive it, without dropping frames. FC
also has very little transmission overhead. The FC architecture is highly scalable, and
theoretically, a single FC SAN can accommodate approximately 15 million devices.
Note: FibRE refers to the protocol, whereas fibER refers to a media.
11
The key FC SAN components are network adapters, cables, and interconnecting devices. These
components are described below.
Network adapters: In an FC SAN, the end devices, such as compute systems and storage
systems are all referred to as nodes. Each node is a source or destination of information. Each
node requires one or more network adapters to provide a physical interface for communicating
with other nodes. Examples of network adapters are FC host bus adapters (HBAs) and storage
system front-end adapters. An FC HBA has SCSI-to-FC processing capability. It encapsulates
OS or hypervisor storage I/Os (usually SCSI I/O) into FC frames before sending the frames to
the FC storage systems over an FC SAN.
Cables: FC SAN implementations primarily use optical fiber cabling. Copper cables may be
used for shorter distances because it provides acceptable signal-to-noise ratio for distances up
to 30 meters. Optical fiber cables carry data in the form of light. There are two types of optical
cables: multimode and single-mode. Multimode fiber (MMF) cable carries multiple beams of
light projected at different angles simultaneously onto the core of the cable. In an MMF
transmission, multiple light beams traveling inside the cable tend to disperse and collide. This
collision weakens the signal strength after it travels a certain distance a process known as
modal dispersion. Due to modal dispersion, an MMF cable is typically used for short distances,
commonly within a data center. Single-mode fiber (SMF) carries a single ray of light projected
at the center of the core. The small core and the single light wave help to limit modal
dispersion. Single-mode provides minimum signal attenuation over maximum distance (up to
10 km). A single-mode cable is used for long-distance cable runs, and the distance usually
depends on the power of the laser at the transmitter and the sensitivity of the receiver. A
connector is attached at the end of a cable to enable swift connection and disconnection of the
cable to and from a port. A standard connector (SC) and a lucent connector (LC) are two
commonly used connectors for fiber optic cables.
Interconnecting devices: The commonly used interconnecting devices in FC SANs are FC

hubs, FC switches, and FC directors. These devices are further detailed in the next slide.
12
FC hubs are used as communication devices in Fibre Channel Arbitrated Loop (FC-AL)
implementations (discussed later). Hubs physically connect nodes in a logical loop or a physical
star topology. All the nodes must share the loop because data travels through all the connection
points. Because of the availability of low-cost and high-performance switches, the FC switches are
preferred over the FC hubs in FC SAN deployments.
FC switches are more intelligent than FC hubs and directly route data from one physical port to
another. Therefore, the nodes do not share the data path. Instead, each node has a dedicated
communication path. The FC switches are commonly available with a fixed port count. Some of
the ports can be active for operational purpose and the rest remain unused. The number of active
ports can be scaled-up non-disruptively. Some of the components of a switch such as power
supplies and fans are redundant and hot-swappable (components can be replaced while a device
is powered-on and remains in operation).
FC directors are high-end switches with a higher port count. A director has a modular architecture
and its port count is scaled-up by inserting additional line cards or blades to the directors chassis.
Directors contain redundant components with automated failover capability. Its key components
such as switch controllers, blades, power supplies, and fan modules are all hot-swappable. These
insure high availability for business critical applications.
13
The FC architecture supports three basic interconnectivity options: point-to-point, fibre channel
arbitrated loop (FC-AL), and fibre channel switched fabric (FC-SW). These interconnectivity
options are described below:
Point-to-point: In this configuration, two nodes are connected directly to each other. This
configuration provides a dedicated connection for data transmission between nodes. However,
the point-to-point configuration offers limited connectivity and scalability and is used in a DAS
environment.
FC-AL: In this configuration, the devices are attached to a shared loop. Each device contends
with other devices to perform I/O operations. The devices on the loop must arbitrate to gain
control of the loop. At any given time, only one device can perform I/O operations on the loop.
Because each device in a loop must wait for its turn to process an I/O request, the overall
performance in FC-AL environments is low. Further, adding or removing a device results in loop
re-initialization, which can cause a momentary pause in loop traffic. As a loop configuration,
FC-AL can be implemented without any interconnecting devices by directly connecting one
device to another two devices in a ring through cables. However, FC-AL implementations may
also use FC hubs through which the arbitrated loop is physically connected in a star topology.
FC-SW: It involves a single FC switch or a network of FC switches (including FC directors) to

interconnect the nodes. It is also referred to as fabric connect. A fabric is a logical space in
which all nodes communicate with one another in a network. In a fabric, the link between any
two switches is called an interswitch link (ISL). ISLs enable switches to be connected together
to form a single, larger fabric. They enable the transfer of both storage traffic and fabric
management traffic from one switch to another. In FC-SW, nodes do not share a loop; instead,
data is transferred through a dedicated path between the nodes. Unlike a loop configuration,
an FC-SW configuration provides high scalability. The addition or removal of a node in a
switched fabric is minimally disruptive; it does not affect the ongoing traffic between other
nodes.
14
The ports in a switched fabric can be one of the following types:
N_Port: It is an end point in the fabric. This port is also known as the node port. Typically, it is
a compute system port (FC HBA port) or a storage system port that is connected to a switch in
a switched fabric.
E_Port: It is a port that forms the connection between two FC switches. This port is also
known as the expansion port. The E_Port on an FC switch connects to the E_Port of another FC
switch in the fabric ISLs.
F_Port: It is a port on a switch that connects an N_Port. It is also known as a fabric port.
G_Port: It is a generic port on a switch that can operate as an E_Port or an F_Port and
determines its functionality automatically during initialization.
15
This lesson covered FC SAN components, three FC interconnectivity options, and various FC port
types.
16
This lesson covers FC protocol stack, FC and WWN addressing, and structure and organization of
FC data. This lesson also covers fabric services, fabric login types, and flow control.
17
Traditionally, compute operating systems have communicated with peripheral devices over
channel connections, such as Enterprise Systems Connection (ESCON) and SCSI. Channel
technologies provide high levels of performance with low protocol overheads. Such performance is
achievable due to the static nature of channels and the high level of hardware and software
integration provided by the channel technologies. However, these technologies suffer from
inherent limitations in terms of the number of devices that can be connected and the distance
between these devices.
In contrast to channel technology, network technologies are more flexible and provide greater
distance capabilities. Network connectivity provides greater scalability and uses shared bandwidth
for communication. This flexibility results in greater protocol overhead and reduced performance.
The FC architecture represents true channel and network integration and captures some of the
benefits of both channel and network technology. FC protocol provides both the channel speed for
data transfer with low protocol overhead and the scalability of network technology. FC provides a
serial data transfer interface that operates over copper wire and optical fiber.
FC protocol forms the fundamental construct of the FC SAN infrastructure. FC protocol
predominantly is the implementation of SCSI over an FC network. SCSI data is encapsulated and
transported within FC frames. SCSI over FC overcomes the distance and the scalability limitations
associated with traditional direct-attached storage. Storage devices attached to the FC SAN
appear as locally attached devices to the operating system (OS) or hypervisor running on
the compute system.
18
It is easier to understand a communication protocol by viewing it as a structure of independent

layers. FCP defines the communication protocol in five layers: FC-0 through FC-4 (except FC-3
layer, which is not implemented).
FC-4 Layer: It is the uppermost layer in the FCP stack. This layer defines the application
interfaces and the way Upper Layer Protocols (ULPs) are mapped to the lower FC layers. The FC
standard defines several protocols that can operate on the FC-4 layer. Some of the protocols
include SCSI, High Performance Parallel Interface (HIPPI) Framing Protocol, ESCON,
Asynchronous Transfer Mode (ATM), and IP.
FC-2 Layer: It provides FC addressing, structure, and organization of data (frames, sequences,
and exchanges). It also defines fabric services, classes of service, flow control, and routing.
FC-1 Layer: It defines how data is encoded prior to transmission and decoded upon receipt. At
the transmitter node, an 8-bit character is encoded into a 10-bit transmission character. This
character is then transmitted to the receiver node. At the receiver node, the 10-bit character is
passed to the FC-1 layer, which decodes the 10-bit character into the original 8-bit character. FC
links, with a speed of 10 Gbps and above, use 64-bit to 66-bit encoding algorithm. This layer also
defines the transmission words such as FC frame delimiters, which identify the start and the end
of a frame and the primitive signals that indicate events at a transmitting port. In addition to
these, the FC-1 layer performs link initialization and error recovery.
FC-0 Layer: It is the lowest layer in the FCP stack. This layer defines the physical interface,
media, and transmission of bits. The FC-0 specification includes cables, connectors, and optical
and electrical parameters for a variety of data rates. The FC transmission can use both electrical
and optical media.
19
An FC address is dynamically assigned when a node port logs on to the fabric. The FC
address has a distinct format, as shown on the slide. The first field of the FC address
contains the domain ID of the switch. A domain ID is a unique number provided to each
switch in the fabric. Although this is an 8-bit field, there are only 239 available addresses
for domain ID because some addresses are deemed special and reserved for fabric
services. For example, FFFFFC is reserved for the name server, and FFFFFE is reserved
for the fabric login service. The area ID is used to identify a group of switch ports used
for connecting nodes. An example of a group of ports with common area ID is a port card
on the switch. The last field, the port ID, identifies the port within the group.
Therefore, the maximum possible number of node ports in a switched fabric is calculated
as:
239 domains X 256 areas X 256 ports = 15,663,104 ports.
20
Each device in the FC environment is assigned a 64-bit unique identifier called the World Wide
Name (WWN). The FC environment uses two types of WWNs: World Wide Node Name (WWNN)
and World Wide Port Name (WWPN). WWNN is used to physically identify FC network adapters
and WWPN is used to physically identify FC adapter ports or node ports. For example, a dual-port
FC HBA has one WWNN and two WWPNs.
Unlike an FC address, which is assigned dynamically, a WWN is a static name for each device on
an FC network. WWNs are similar to the Media Access Control (MAC) addresses used in IP
networking. WWNs are burned into the hardware or assigned through software. Several
configuration definitions in an FC SAN use WWN for identifying storage systems and FC HBAs.
WWNs are critical for FC SAN configuration as each node port has to be registered by its WWN
before the FC SAN recognizes it. The name server in an FC SAN environment keeps the
association of WWNs to the dynamically created FC addresses for node ports. The figure on the
slide illustrates the WWN structure examples for a storage system and an HBA.
21
In an FC network, data transport is analogous to a conversation between two people,

whereby a frame represents a word, a sequence represents a sentence, and an exchange
represents a conversation.
Exchange: An exchange operation enables two node ports to identify and manage a set
of information units. Each upper layer protocol (ULP) has its protocol-specific information
that must be sent to another port to perform certain operations. This protocol-specific
information is called an information unit. The structure of these information units is
defined in the FC-4 layer. This unit maps to a sequence. An exchange is composed of one
or more sequences.
Sequence: A sequence refers to a contiguous set of frames that are sent from one port
to another. A sequence corresponds to an information unit, as defined by the ULP.
Frame: A frame is the fundamental unit of data transfer at FC-2 layer. An FC frame
consists of five parts: start of frame (SOF), frame header, data field, cyclic redundancy
check (CRC), and end of frame (EOF). The SOF and EOF act as delimiters. The frame
header is 24 bytes long and contains addressing information for the frame. The data field
in an FC frame contains the data payload, up to 2,112 bytes of actual data in most
cases the SCSI data. The CRC checksum facilitates error detection for the content of the
frame. This checksum verifies data integrity by checking whether the content of the
frames are received correctly. The CRC checksum is calculated by the sender before
encoding at the FC-1 layer. Similarly, it is calculated by the receiver after decoding at the
FC-1 layer.
22
All FC switches, regardless of the manufacturer, provide a common set of services as

defined in the FC standards. These services are available at certain predefined addresses.
Some of these services are Fabric Login Server, Fabric Controller, Name Server, and
Management Server.
Fabric Login Server: It is located at the predefined address of FFFFFE and is used
during the initial part of the nodes fabric login process.
Name Server (formally known as Distributed Name Server): It is located at the
predefined address FFFFFC and is responsible for name registration and management
of node ports. Each switch exchanges its Name Server information with other switches
in the fabric to maintain a synchronized, distributed name service.
Fabric Controller: Each switch has a Fabric Controller located at the predefined
address FFFFFD. The Fabric Controller provides services to both node ports and other
switches. The Fabric Controller is responsible for managing and distributing Registered
State Change Notifications (RSCNs) to the node ports registered with the Fabric
Controller. If there is a change in the fabric, RSCNs are sent out by a switch to the
attached node ports. The Fabric Controller also generates Switch Registered State
Change Notifications (SW-RSCNs) to every other domain (switch) in the fabric. These
RSCNs keep the name server up-to-date on all switches in the fabric.
Management Server: FFFFFA is the FC address for the Management Server. The
Management Server is distributed to every switch within the fabric. The Management
Server enables the FC SAN management software to retrieve information and
administer the fabric.
23
Fabric services define three login types:
Fabric login (FLOGI): It is performed between an N_Port and an F_Port. To log on to the
fabric, a node sends a FLOGI frame with the WWNN and WWPN parameters to the login service
at the predefined FC address FFFFFE (Fabric Login Server). In turn, the switch accepts the
login and returns an Accept (ACC) frame with the assigned FC address for the node.
Immediately after the FLOGI, the N_Port registers itself with the local Name Server on the
switch, indicating its WWNN, WWPN, port type, class of service, assigned FC address, and so
on. After the N_Port has logged in, it can query the name server database for information
about all other logged in ports.
Port login (PLOGI): It is performed between two N_Ports to establish a session. The initiator
N_Port sends a PLOGI request frame to the target N_Port, which accepts it. The target N_Port
returns an ACC to the initiator N_Port. Next, the N_Ports exchange service parameters relevant
to the session.
Process login (PRLI): It is also performed between two N_Ports. This login relates to the FC4 ULPs, such as SCSI. If the ULP is SCSI, N_Ports exchange SCSI-related service parameters.
24
Flow control is the process to regulate the data transmission rate between two devices so that a
transmitting device does not overflow a receiving device with data. A fabric uses the buffer-tobuffer credit (BB_Credit) mechanism for flow control. The BB_Credit management may occur
between any two FC ports.
In a fabric, an FC frame is received and stored in a receive buffer where it is processed by the
receiving FC port. If another frame arrives while the receiving port is processing the first frame, a
second receive buffer is needed to hold the new frame. If all the receive buffers are filled up with
received frames and the transmitting port sends another frame, then the receiving port would not
have a receive buffer available to hold the new frame and the frame would be dropped. BB_Credit
mechanism ensures that the FC ports do not run out of buffers and do not drop frames.
With BB_Credit mechanism, the transmitting and receiving ports agree on the number of buffers
available or BB_Credits during the port login process. The credit value is decremented when a
frame is transmitted and incremented upon receiving a response. A receiver ready (R_RDY) is
sent from the receiving port to the transmitting port for every free buffer on the receiving side.
The transmitting port increments the credit value per R_RDY it receives from the receiving port.
The transmitting port maintains a count of available credits and continues to send frames if the
count is greater than zero. If the available credits reaches zero, then further transmission of
frames is suspended until the credit count becomes a nonzero value.
In the figure shown on the slide, a data frame is sent from the N_Port of a compute system to the
F_Port of an FC switch. Once the frame is processed and passes the F_Port, the buffer is released
and an R_RDY is sent to the transmitting N_Port, allowing the buffer to be marked as available.
Each port-to-port data transmission occurs in this manner throughout the fabric.
25
This lesson covered FC protocol stack, FC and WWN addressing, and structure and organization of
FC data. This lesson also covered fabric services, fabric login types, and flow control.
26
This lesson covers FC SAN topologies such as single-switch, mesh, and core-edge. This lesson
also covers link aggregation and types of zoning.
27
FC switches (including FC directors) may be connected in a number of ways to form different

fabric topologies. Each topology provides certain benefits.
In a single-switch topology, the fabric consists of only a single switch. Both the compute
systems and the storage systems are connected to the same switch. A key advantage of
a single-switch fabric is that it does not need to use any switch port for ISLs. Therefore, every
switch port is usable for compute system or storage system connectivity. Further, this topology
helps eliminate FC frames traveling over the ISLs and consequently eliminates the ISL delays.
A typical implementation of a single-switch fabric would involve the deployment of an FC director.

FC directors are high-end switches with a high port count. When additional switch ports are
needed over time, new ports can be added via add-on line cards (blades) in spare slots available
on the director chassis. To some extent, a bladed solution alleviates the port count scalability
problem inherent in a single-switch topology.
28
A mesh topology may be one of the two types: full mesh or partial mesh. In a full mesh, every
switch is connected to every other switch in the topology. A full mesh topology may be
appropriate when the number of switches involved is small. A typical deployment would involve
up to four switches or directors, with each of them servicing highly localized compute-to-storage
traffic. In a full mesh topology, a maximum of one ISL or hop is required for compute-to-storage
traffic. However, with the increase in the number of switches, the number of switch ports used for
ISL also increases. This reduces the available switch ports for node connectivity.
In a partial mesh topology, not all the switches are connected to every other switch. In this
topology, several hops or ISLs may be required for the traffic to reach its destination. Partial
mesh offers more scalability than full mesh topology. However, without proper placement of
compute and storage systems, traffic management in a partial mesh fabric might be complicated
and ISLs could become overloaded due to excessive traffic aggregation.
29
The core-edge topology has two types of switch tiers: edge and core.
The edge tier is usually composed of switches and offers an inexpensive approach to adding more
compute systems in a fabric. The edge-tier switches are not connected to each other. Each switch
at the edge tier is attached to a switch at the core tier through ISLs.
The core tier is usually composed of directors that ensure high fabric availability. In addition,
typically all traffic must either traverse this tier or terminate at this tier. In this configuration, all
storage systems are connected to the core tier, enabling compute-to-storage traffic to traverse
only one ISL. Compute systems that require high performance may be connected directly to the
core tier and consequently avoid ISL delays.
The core-edge topology increases connectivity within the FC SAN while conserving the overall port
utilization. It eliminates the need to connect edge switches to other edge switches over ISLs.
Reduction of ISLs can greatly increase the number of node ports that can be connected to the
fabric. If fabric expansion is required, then administrators would need to connect additional edge
switches to the core. The core of the fabric is also extended by adding more switches or directors
at the core tier. Based on the number of core-tier switches, this topology has different variations,
such as single-core topology and dual-core topology. To transform a single-core topology to dualcore, new ISLs are created to connect each edge switch to the new core switch in the fabric.
30
Link aggregation combines two or more parallel ISLs into a single logical ISL, called a portchannel, yielding higher throughput than a single ISL could provide. For example, the aggregation
of 10 ISLs into a single port-channel provides up to 160 Gb/s throughput assuming the bandwidth
of an ISL is 16 Gb/s. Link aggregation optimizes fabric performance by distributing network traffic
across the shared bandwidth of all the ISLs in a port-channel. This allows the network traffic for a
pair of node ports to flow through all the available ISLs in the port-channel rather than restricting
the traffic to a specific, potentially congested ISL. The number of ISLs in a port channel can be
scaled depending on applications performance requirement.
This slide illustrates two examples. The example on the left is based on an FC SAN infrastructure
with no link aggregation enabled. Four HBA ports H1, H2, H3, and H4 have been configured to
generate I/O activity to four storage system ports S1, S2, S3, and S4 respectively. The HBAs and
the storage systems are connected to two separate FC switches with three ISLs between the
switches. Let us assume that the bandwidth of each ISL is 8 Gb/s and the data transmission rate
for the port-pairs {H1,S1}, {H2,S2}, {H3,S3}, and {H4,S4} are 5 Gb/s, 1.5 Gb/s, 2 Gb/s, and
4.5 Gb/s. Without link aggregation, the fabric typically assigns a particular ISL for each of the
port-pairs in a round-robin fashion. It is possible that port-pairs {H1,S1} and {H4,S4} are
assigned to the same ISL in their respective routes. The other two ISLs are assigned to the portpairs {H2,S2} and {H3,S3}. Two of the three ISLs are under-utilized, whereas the third ISL is
saturated and becomes a performance bottleneck for the port-pairs assigned to it. The example
on the right has aggregated the three ISLs into a port-channel that provides throughput up to 24
Gb/s. Network traffic for all the port-pairs are distributed over the ISLs in the port-channel, which
ensures even ISL utilization.
31
Zoning is an FC switch function that enables node ports within the fabric to be logically
segmented into groups and communicate with each other within the group.
Whenever a change takes place in the name server database, the fabric controller sends
a Registered State Change Notification (RSCN) to all the nodes impacted by the change.
If zoning is not configured, the fabric controller sends the RSCN to all the nodes in the
fabric. Involving the nodes that are not impacted by the change increases the amount of
fabric-management traffic. For a large fabric, the amount of FC traffic generated due to
this process can be significant and might impact the compute-to-storage data traffic.
Zoning helps to limit the number of RSCNs in a fabric. In the presence of zoning, a fabric
sends the RSCN to only those nodes in a zone where the change has occurred.
Zoning also provides access control, along with other access control mechanisms, such
as LUN masking. Zoning provides control by allowing only the members in the same zone
to establish communication with each other.
Zone members, zones, and zone sets form the hierarchy defined in the zoning process. A
zone set is composed of a group of zones that can be activated or deactivated as a single
entity in a fabric. Multiple zone sets may be defined in a fabric, but only one zone set can
be active at a time. Members are the nodes within the FC SAN that can be included in a
zone. FC switch ports, FC HBA ports, and storage system ports can be members of a
zone. A port or node can be a member of multiple zones. Nodes distributed across
multiple switches in a switched fabric may also be grouped into the same zone. Zone sets
are also referred to as zone configurations.
32
Zoning can be categorized into three types:
WWN zoning: It uses World Wide Names to define zones. The zone members are the unique
WWN addresses of the FC HBA and its targets (storage systems). A major advantage of WWN
zoning is its flexibility. If an administrator moves a node to another switch port in the fabric,
the node maintains connectivity to its zone partners without having to modify the zone
configuration. This is possible because the WWN is static to the node port.
Port zoning: It uses the switch port ID to define zones. In port zoning, access to node is
determined by the physical switch port to which a node is connected. The zone members are
the port identifiers (switch domain ID and port number) to which FC HBA and its targets
(storage systems) are connected. If a node is moved to another switch port in the fabric, port
zoning must be modified to allow the node, in its new port, to participate in its original zone.
However, if an FC HBA or storage system port fails, an administrator just has to replace the
failed device without changing the zoning configuration.
Mixed zoning: It combines the qualities of both WWN zoning and port zoning. Using mixed
zoning enables a specific node port to be tied to the WWN of another node.
The figure on the slide shows three types of zoning on an FC network.
33
This lesson covered FC SAN topologies such as single-switch, mesh, and core-edge. This lesson
also covered link aggregation and types of zoning.
34
This lesson covers port virtualization that comprises both N_Port ID virtualization (NPIV) and
N_Port virtualization (NPV). This lesson also covers block-level storage virtualization and virtual
SAN.
35
N_Port ID virtualization (NPIV) enables a single N_Port (such as an FC HBA port) to

function as multiple virtual N_Ports. Each virtual N_Port has a unique WWPN identity in
the FC SAN. This allows a single physical N_Port to obtain multiple FC addresses.
Hypervisors leverage NPIV to create virtual N_Ports on the FC HBA and then assign the
virtual N_Ports to virtual machines (VMs). A virtual N_Port acts as a virtual FC HBA port.
This enables a VM to directly access LUNs assigned to it (see the figure on the slide).
NPIV enables an administrator to restrict access to specific LUNs to specific VMs using
security techniques like zoning and LUN masking; similarly to the assignment of a LUN to
a physical compute system. To enable NPIV, both the FC HBAs and the FC switches must
support NPIV. The physical FC HBAs on the compute system, using their own WWNs, must have
access to all LUNs that are to be accessed by VMs running on that compute system.
36
The proliferation of compute systems in a data center causes increased use of edge switches in a
fabric. As the edge switch population grows, the number of domain IDs may become a concern
because of the limitation on the number of domain IDs in a fabric. N_Port virtualization (NPV)
addresses this concern by reducing the number of domain IDs in a fabric. Edge switches
supporting NPV do not require a domain ID. They pass traffic between the core switch and the
compute systems. NPV-enabled edge switches do not perform any fabric services, and instead
forward all fabric activity, such as login and name server registration to the core switch.
All ports at the NPV edge switches that connect to the core switch are established as NP_Ports
(not E_Ports). The NP_Ports connect to an NPIV-enabled core director or switch. If the core
director or switch is not NPIV-capable, the NPV edge switches do not function. As the switch
enters or exits from NPV mode, the switch configuration is erased and it reboots. Therefore,
administrators should take care when enabling or disabling NPV on a switch. The figure on the
slide shows a core-edge fabric that comprises two edge switches in NPV mode and one core
switch (an FC director).
37
Block-level storage virtualization aggregates block storage devices (LUNs) and enables
provisioning of virtual storage volumes, independent of the underlying physical storage. A
virtualization layer, which exists at the SAN, abstracts the identity of block-based storage systems
and creates a storage pool by aggregating LUNs from the storage systems. Virtual volumes are
created from the storage pool and assigned to the compute systems. Instead of being directed to
the LUNs on the individual storage systems, the compute systems are directed to the virtual
volumes provided by the virtualization layer. The virtualization layer maps the virtual volumes to
the LUNs on the individual storage systems. The compute systems remain unaware of the
mapping operation and access the virtual volumes as if they were accessing the physical storage
attached to them. Typically, the virtualization layer is managed via a dedicated virtualization
appliance to which the compute systems and the storage systems are connected.
The figure on the slide shows two compute systems, each of which has one virtual volume
assigned. These virtual volumes are mapped to the LUNs in the storage systems. When an I/O is
sent to a virtual volume, it is redirected to the mapped LUNs through the virtualization layer at
the FC SAN. Depending on the capabilities of the virtualization appliance, the architecture may
allow for more complex mapping between the LUNs and the virtual volumes.
Block-level storage virtualization enables extending the virtual volumes non-disruptively to meet
applications capacity scaling requirements. It also provides the advantage of non-disruptive data
migration. In a traditional SAN environment, LUN migration from one storage system to another
is an offline event. After migration, the compute systems are updated to reflect the new storage
system configuration. In other instances, processor cycles at the compute system were required
to migrate data from one storage system to the other, especially in a multivendor environment.
With a block-level storage virtualization solution in place, the virtualization layer handles the
migration of data, which enables LUNs to remain online and accessible while data is migrating. No
physical changes are required because the compute system still points to the same virtual volume
on the virtualization layer. However, the mapping information on the virtualization layer should be
changed. These changes can be executed dynamically and are transparent to the end user.
38
The use case is based on the block-level storage virtualization across data centers that
allow compute systems at different locations to share and concurrently access same
virtual volume. According to the use case, two data centers are separated by geographic
distance (see the figure on the slide). Each data center includes an FC SAN infrastructure
that supports block-level storage virtualization through the deployment of the
virtualization appliance. The storage virtualization solution provides the capability to
connect the virtualization appliances at the two data centers. The connected
virtualization appliances are managed centrally and work as a single virtualization layer
stretched across the data centers. This enables the pooling of LUNs across data centers.
Virtual volumes are created from the storage pool. These virtual volumes are accessible
from both the data centers. Compute systems at both the data centers can perform I/O
operations to these shared volumes concurrently. This enables real-time data
collaboration over distance, which is a key requirement for big data applications.
39
Virtual SAN (also called virtual fabric) is a logical fabric on an FC SAN, which enables
communication among a group of nodes regardless of their physical location in the fabric. In a
VSAN, a group of node ports communicate with each other using a virtual topology defined on the
physical SAN. Multiple VSANs may be created on a single physical SAN. Each VSAN behaves and
is managed as an independent fabric. Each VSAN has its own fabric services, configuration, and
set of FC addresses. Fabric-related configurations in one VSAN do not affect the traffic in another
VSAN. A VSAN may be extended across sites, enabling communication among a group of nodes,
in either site with a common set of requirements.
VSANs improve SAN security, scalability, availability, and manageability. VSANs provide enhanced
security by isolating the sensitive data in a VSAN and by restricting the access to the resources
located within that VSAN. For example, a cloud provider typically isolates the storage pools for
multiple cloud services by creating multiple VSANs on an FC SAN. Further, the same FC address
can be assigned to nodes in different VSANs, thus increasing the fabric scalability. The events
causing traffic disruptions in one VSAN are contained within that VSAN and are not propagated to
other VSANs. VSANs facilitate an easy, flexible, and less expensive way to manage networks.
Configuring VSANs is easier and quicker compared to building separate physical FC SANs for
various node groups. To regroup nodes, an administrator simply changes the VSAN configurations
without moving nodes and recabling.
40
To configure VSANs on a fabric, an administrator first needs to define VSANs on fabric switches.
Each VSAN is identified with a specific number called VSAN ID. The next step is to assign a VSAN
ID to the F_Ports on the switch. By assigning a VSAN ID to an F_Port, the port is included in the
VSAN. In this manner, multiple F_Ports can be grouped into a VSAN. For example, an
administrator may group switch ports (F_Ports) 1 and 2 into VSAN 10 (ID) and ports 6 to 12 into
VSAN 20 (ID). If an N_Port connects to an F_Port that belongs to a VSAN, it becomes a member
of that VSAN. The switch transfers FC frames between switch ports that belong to the same
VSAN.
Note: VSAN versus Zone

Both VSANs and zones enable node ports within a fabric to be logically segmented into groups.
But they are not same and their purposes are different. There is a hierarchical relationship
between them. An administrator first assigns physical ports to VSANs and then configures
independent zones for each VSAN. A VSAN has its own independent fabric services, but the fabric
services are not available on a per-zone basis.
41
VSAN trunking allows network traffic from multiple VSANs to traverse a single ISL. It supports a
single ISL to permit traffic from multiple VSANs along the same path. The ISL through which
multiple VSAN traffic travels is called a trunk link. VSAN trunking enables a single E_Port to be
used for sending or receiving traffic from multiple VSANs over a trunk link. The E_Port capable of
transferring multiple VSAN traffic is called a trunk port. The sending and receiving switches must
have at least one trunk E_Port configured for all of or a subset of the VSANs defined on the
switches.
VSAN trunking eliminates the need to create dedicated ISL(s) for each VSAN. It reduces the
number of ISLs when the switches are configured with multiple VSANs. As the number of ISLs
between the switches decreases, the number of E_Ports used for the ISLs also reduces. By
eliminating needless ISLs, the utilization of the remaining ISLs increases. The complexity of
managing the FC SAN is also minimized with a reduced number of ISLs.
The figure on the slide shows a VSAN trunking configuration contrasted with a network
configuration without VSAN trunking. In both the cases, the switches have VSAN 10, VSAN 20,
and VSAN 30 configured. If VSAN trunking is not used, three ISLs are required to transfer traffic
between the three distinct VSANs. When trunking is configured, a single ISL is used to transfer all
VSAN traffic.
42
VSAN tagging is the process of adding or removing a marker or tag to the FC frames that contains
VSAN-specific information. Associated with VSAN trunking, it helps isolate FC frames from
multiple VSANs that travel through and share a trunk link. Whenever an FC frame enters an FC
switch, it is tagged with a VSAN header indicating the VSAN ID of the switch port (F_Port) before
sending the frame down to a trunk link. The receiving FC switch reads the tag and forwards the
frame to the destination port that corresponds to that VSAN ID. The tag is removed once the
frame leaves a trunk link to reach an N_Port.
43
This lesson covered N_Port ID virtualization (NPIV), N_Port virtualization (NPV), block-level
storage virtualization, and virtual SAN.
44
The Concepts in Practice section covers EMC Connectrix and EMC VPLEX.
Note:
45
EMC Connectrix family is a group of networked storage connectivity products. EMC offers the
following connectivity products under the Connectrix brand:
Enterprise directors: They are ideal for large enterprise connectivity. They offer high port
density and high component redundancy. They are deployed in high-availability or large-scale
environments.
Departmental switches: They are designed to meet workgroup-level, department-level, and

enterprise-level requirements. They provide high availability through features such as nondisruptive software and port upgrade, and redundant and hot-swappable components.
Multi-purpose switches: They support various protocols such as FC, iSCSI, FCIP, FCoE, and
FICON. They include FCoE switches, FCIP gateways, and iSCSI gateways. Multiprotocol
capabilities offer many benefits, including long-distance SAN extension, greater resource
sharing, and simplified management.
EMC VPLEX provides solution for block-level storage virtualization and data mobility both within
and across data centers. It forms a pool of distributed block storage resources and enables
creating virtual storage volumes from the pool. These virtual volumes are then allocated to the
compute systems. VPLEX provides non-disruptive data mobility among storage systems to
balance the application workload and to enable both local and remote data access. VPLEX also
provides the capability to mirror data of a virtual volume both within and across locations. It uses
a unique clustering architecture and advanced data caching techniques that enable multiple
compute systems located across two locations to access a single copy of data. VPLEX Virtual
Edition (VPLEX/VE) is deployed as a set of virtual appliances that implement VPLEX technology on
VMware ESXi infrastructure. VPLEX/VE stretches ESXi infrastructure over distance, allowing an
ESXi cluster to span across two physical sites.
46
This module covered third platform requirements for SAN and software-defined networking. It
also covered FC SAN components network adapters, cables, and interconnecting devices; FC
connectivity options point-to-point, FC-AL, FC-SW; and fabric port types such as N_Port, E_Port,
F_Port, and G_Port. It includes FC protocol stack and addressing, structure and organization of FC
data, fabric services, login types, and flow control. This module also covered fabric topologies
single-switch, mesh, and core-edge; link aggregation; types of zoning WWN, port, and mixed;
port virtualization NPIV and NPV; block-level storage virtualization; and virtual SAN.
47
48
The storage infrastructure of an IT organization includes three block-based storage

systems direct-attached to a heterogeneous mix of 45 compute systems. All compute
systems are dual-attached to the storage systems for high availability. Because each
storage system has 32 front-end ports, each could support a maximum of 16 compute
systems. However, each existing storage system has the storage drive capacity to
support a maximum of 32 compute systems.
The organization plans to purchase 45 more compute systems to meet its growth requirements. If
it continues using direct-attached storage, the organization needs to purchase additional storage
systems to connect these new compute systems. The organization realizes that its existing
storage systems are poorly utilized; therefore, it plans to implement FC SAN to overcome the
scalability and utilization challenges. The organization uses high-performance applications;
therefore, it wants to minimize the number of ISLs or hop count required for compute-to-storage
traffic.
Given that 72-port FC switches are available for interconnectivity, propose a fabric topology to
address organizations challenges and requirements. Justify your choice of the fabric topology. In
addition, determine the minimum number of switches required in the fabric.
Single-switch topology cannot be implemented because the available FC switch does not provide
the required number of ports for node connectivity. Full mesh topology is not suitable for an
environment that requires high scalability. Partial mesh although provides more scalability than
full mesh, but several hops or ISLs may be required for the network traffic to reach its
destination.
Therefore, the recommended solution is core-edge topology. The core-edge topology provides
higher scalability than mesh topology. It also provides a maximum of one-hop or one-ISL storage
access to all compute systems in the environment. As ISLs between the edge switches are not
needed, the core-edge topology conserves the overall switch port utilization which increases
connectivity within the fabric.
Total number of compute system ports = 90 compute systems 2 ports = 180 ports
Total number of storage system ports = 3 storage systems 32 ports = 96 ports
Number of switches required at the core = 96 storage system ports / 72 ports per switch 2
switches
The core switches provide 144 ports of which 96 ports will be used for storage system
connectivity. Remaining 48 ports can be used for ISLs and future growth.
Number of switches at the edge = 180 compute system ports / 72 ports per switch 3 switches
The edge switches provide 216 ports of which 180 ports will be used for compute system
connectivity. Remaining 36 ports can be used for ISLs and future growth.
So, at minimum, two core switches and three edge switches are required to implement the coreedge fabric.
This module focuses on IP SAN protocols such as Internet SCSI (iSCSI) and Fibre Channel over IP
(FCIP), components, and connectivity. It also focuses on link aggregation, switch aggregation,
and virtual LAN (VLAN).
Module 10: Internet Protocol (IP) SAN
IP SAN uses Internet Protocol (IP) for the transport of storage traffic. It transports block
I/O over an IP-based network. Two primary protocols that leverage IP as the transport
mechanism for block-level data transmission are Internet SCSI (iSCSI) and Fibre Channel
over IP (FCIP). These protocols are covered in this module.
Fibre Channel (FC) SAN provides high performance and scalability. These advantages of
FC SAN come with the additional cost of buying FC components, such as FC HBA and FC
switches. IP is a matured technology and using IP as a storage networking option
provides several advantages. These are listed below:
Most organizations have an existing IP-based network infrastructure, which could also
be used for storage networking and may be a more economical option than deploying
a new FC SAN infrastructure.
IP network has no distance limitation, which makes it possible to extend or connect
SANs over long distances. With IP SAN, organizations can extend the geographical
reach of their storage infrastructure and transfer data that are distributed over wide
locations.
Many long-distance disaster recovery (DR) solutions are already leveraging IP-based
networks. In addition, many robust and mature security options are available for IP
networks.
This lesson covers iSCSI network components and connectivity. It also covers iSCSI protocol
stack, iSCSI address and name, and iSCSI discovery.
iSCSI is an IP-based protocol that establishes and manages connections between

compute systems and storage systems over IP. iSCSI is an encapsulation of SCSI I/O
over IP. iSCSI encapsulates SCSI commands and data into IP packets and transports
them using TCP/IP. iSCSI is widely adopted for transferring SCSI data over IP between
compute systems and storage systems and among the storage systems. It is relatively
inexpensive and easy to implement, especially environments in which an FC SAN does
not exist.
Key components for iSCSI communication are:

iSCSI initiators such as an iSCSI HBA
iSCSI targets such as a storage system with an iSCSI port
IP-based network such as a Gigabit Ethernet LAN
An iSCSI initiator sends commands and associated data to a target and the target
returns data and responses to the initiator.
A standard NIC with software iSCSI adapter, a TCP offload engine (TOE) NIC with
software iSCSI adapter, and an iSCSI HBA are the three common types of iSCSI
initiators. These initiators are described below:
Standard NIC with software iSCSI adapter: The software iSCSI adapter is an
operating system (OS) or hypervisor kernel-resident software that uses an existing
NIC of the compute system to emulate an iSCSI initiator. It is least expensive and
easy to implement because most compute systems come with at least one, and in
many cases two embedded NICs. It requires only a software initiator for iSCSI
functionality. Because NICs provide standard networking function, both the TCP/IP
processing and the encapsulation of SCSI data into IP packets are carried out by the
CPU of the compute system. This places additional overhead on the CPU. If a standard
NIC is used in heavy I/O load situations, the CPU of the compute system might
become a bottleneck.
TOE NIC with software iSCSI adapter: A TOE NIC offloads the TCP/IP processing
from the CPU of a compute system and leaves only the iSCSI functionality to the CPU.
The compute system passes the iSCSI information to the TOE NIC and then the TOE
NIC sends the information to the destination using TCP/IP. Although this solution
improves performance, the iSCSI functionality is still handled by a software adapter
that requires CPU cycles of the compute system.
iSCSI HBA: An iSCSI HBA is a hardware adapter with built-in iSCSI functionality. It is
capable of providing performance benefits over software iSCSI adapters by offloading
the entire iSCSI and TCP/IP processing from the CPU of a compute system.
The iSCSI implementations support two types of connectivity: native and bridged. These
are described below:
Native iSCSI: In this type of connectivity, the compute systems with iSCSI initiators
may be either directly attached to the iSCSI targets (iSCSI-capable storage systems)
or connected through an IP-based network. FC components are not required for native
iSCSI connectivity. The figure on the left shows a native iSCSI implementation that
includes a storage system with an iSCSI port. The storage system is connected to an
IP network. After an iSCSI initiator is logged on to the network, it can access the
available LUNs on the storage system.
Bridged iSCSI: This type of connectivity allows the initiators to exist in an IP
environment while the storage systems remain in an FC SAN environment. It enables
the coexistence of FC with IP by providing iSCSI-to-FC bridging functionality. The
figure on the right illustrates a bridged iSCSI implementation. It shows connectivity
between a compute system with an iSCSI initiator and a storage system with an FC
port. As the storage system does not have any iSCSI port, a gateway or a
multiprotocol router is used. The gateway facilitates the communication between the
compute system with iSCSI ports and the storage system with only FC ports. The
gateway converts IP packets to FC frames and vice versa, thereby bridging the
connectivity between the IP and FC environments. The gateway contains both FC and
Ethernet ports to facilitate the communication between the FC and the IP
environments. The iSCSI initiator is configured with the gateways IP address as its
target destination. On the other side, the gateway is configured as an FC initiator to
the storage system.
Typically, a storage system comes with both FC and iSCSI ports. This enables both the
native iSCSI connectivity and the FC connectivity in the same environment, as shown on
the slide.
The figure on the slide displays a model of iSCSI protocol layers and depicts the encapsulation
order of the SCSI commands for their delivery through a physical carrier.
SCSI is the command protocol that works at the application layer of the Open System
Interconnection (OSI) model. The initiators and the targets use SCSI commands and responses to
talk to each other. The SCSI commands, data, and status messages are encapsulated into TCP/IP
and transmitted across the network between the initiators and the targets.
iSCSI is the session-layer protocol that initiates a reliable session between devices that recognize
SCSI commands and TCP/IP. The iSCSI session-layer interface is responsible for handling login,
authentication, target discovery, and session management.
TCP is used with iSCSI at the transport layer to provide reliable transmission. TCP controls
message flow, windowing, error recovery, and retransmission. It relies upon the network layer of
the OSI model to provide global addressing and connectivity. The OSI Layer 2 protocols at the
data link layer of this model enable node-to-node communication through a physical network.
10
Both the initiators and the targets in an iSCSI environment have iSCSI addresses that facilitate
communication between them. An iSCSI address is comprised of the location of an iSCSI initiator
or target on the network and the iSCSI name. The location is a combination of the host name or
IP address and the TCP port number. For iSCSI initiators, the TCP port number is omitted from
the address.
iSCSI name is a unique worldwide iSCSI identifier that is used to identify the initiators and targets
within an iSCSI network to facilitate communication. The unique identifier can be a combination of
the names of the department, application, manufacturer, serial number, asset number, or any tag
that can be used to recognize and manage the iSCSI nodes. The following are three types of
iSCSI names commonly used:
iSCSI Qualified Name (IQN): An organization must own a registered domain name to
generate iSCSI Qualified Names. This domain name does not need to be active or resolve to an
address. It just needs to be reserved to prevent other organizations from using the same
domain name to generate iSCSI names. A date is included in the name to avoid potential
conflicts caused by the transfer of domain names. An example of an IQN is iqn.201504.com.example:optional_string. The optional_string provides a serial number, an asset
number, or any other device identifiers. IQN enables storage administrators to assign
meaningful names to the iSCSI initiators and the iSCSI targets, and therefore, manages those
devices more easily.
Extended Unique Identifier (EUI): An EUI is a globally unique identifier based on the IEEE
EUI-64 naming standard. An EUI is composed of the eui prefix followed by a 16-character
hexadecimal name, such as eui.0300732A32598D26.
Network Address Authority (NAA): NAA is another worldwide unique naming format as
defined by the InterNational Committee for Information Technology Standards (INCITS) T11 Fibre Channel (FC) protocols and is used by Serial Attached SCSI (SAS). This format enables
the SCSI storage devices that contain both iSCSI ports and SAS ports to use the same NAAbased SCSI device name. An NAA is composed of the naa prefix followed by a hexadecimal
name, such as naa.52004567BA64678D. The hexadecimal representation has a maximum size
of 32 characters (128 bit identifier).
11
An iSCSI initiator must discover the location of its targets on the network and the names of the
targets available to it before it can establish a session. This discovery commonly takes place in
two ways: SendTargets discovery or internet Storage Name Service (iSNS).
SendTargets discovery: In SendTargets discovery, the initiator is manually configured with

the targets network portal (IP address and TCP port number) to establish a discovery session.
The initiator issues the SendTargets command, and thereby the target network portal responds
to the initiator with the location and name of the target.
iSNS: iSNS in the iSCSI SAN is equivalent in function to the Name Server in an FC SAN. It
enables automatic discovery of iSCSI devices on an IP-based network. The initiators and
targets can be configured to automatically register themselves with the iSNS server. Whenever
an initiator wants to know the targets that it can access, it can query the iSNS server for a list
of available targets.
12
iSNS discovery domains function in the same way as FC zones. Discovery domains provide
functional groupings of devices (including iSCSI initiators and targets) in an IP SAN. The iSNS
server is configured with discovery domains. For devices to communicate with one another, they
must be configured in the same discovery domain.
The iSNS server may send state change notifications (SCNs) to the registered devices. State
change notifications (SCNs) inform the registered devices about network events that affect the
operational state of devices such as the addition or removal of devices from a discovery domain.
13
This lesson covered the iSCSI network components and connectivity. It also covered iSCSI
protocol stack, iSCSI address and name, and iSCSI discovery methods.
14
This lesson covers link aggregation, switch aggregation, and self-forming network. It also covers
virtual LAN (VLAN) and stretched VLAN.
15
Like an FC environment, the link aggregation in an Ethernet network also combines two or more
parallel network links into a single logical link (port-channel). Link aggregation enables obtaining
higher throughput than a single link could provide. It also enables distribution of network traffic
across the links that ensure even link utilization. If a link in the aggregation is lost, all network
traffic on that link is redistributed across the remaining links. Link aggregation can be performed
for links between two switches and between a switch and a node. The figure on the left shows an
example of link aggregation between two Ethernet switches. In this example, four links between
the switches are aggregated into a single port-channel.
Switch aggregation combines two physical switches to make them appear as a single logical
switch. All network links from these physical switches appear as a single logical link. This enables
nodes to use a port-channel across two switches. The network traffic is also distributed across all
the links in the port-channel. Switch aggregation allows ports in both the switches to be active
and to forward network traffic simultaneously. Therefore, it provides more active paths and
throughput than a single switch or multiple non-aggregated switches under normal conditions,
resulting in improved node performance. With switch aggregation, if one switch in the aggregation
fails, network traffic will continue to flow through another switch. The figure on the right shows an
example of switch aggregation. In this example, four physical links to the aggregated switches
appear as a single logical link to the third switch.
16
Self-forming network allows an Ethernet switch to join an Ethernet network automatically. To join
an Ethernet network, the new switch is simply powered-up and cabled to an existing switch in the
network. The Ethernet network automatically detects the new switch and populates its routing
table to start forwarding network traffic immediately.
Self-forming link aggregation enables an Ethernet network to automatically include new
interswitch links into a port-channel and redistributes network traffic among all the links in the
port-channel. Instead of manually configuring individual links into a port-channel on multiple
Ethernet switches, port-channels are automatically formed when new connections (links) are
added between the switches. This enables an organization to scale network bandwidth between
the Ethernet switches quickly.
Both the self-forming network and the self-forming link aggregation help in meeting the third
platform requirements. They simplify the configuration and ongoing management operations of
network infrastructures by automating some of the configuration tasks. They also reduce manual,
repetitive, and error-prone network configuration tasks. Further, they provide the agility needed
to deploy, scale, and manage the network infrastructure.
17
Virtual LANs (VLANs) are logical networks created on a LAN. A VLAN enables communication
between a group of nodes (compute systems and storage systems) with a common set of
functional requirements independent of their physical location in the network. VLANs are
particularly well-suited for iSCSI deployments as they enable isolating the iSCSI traffic from other
network traffic (for example, compute-to-compute traffic) when a physical Ethernet network is
used to transfer different types of network traffic.
A VLAN conceptually functions in the same way as a VSAN. Each VLAN behaves and is managed
as an independent LAN. Two nodes connected to a VLAN can communicate between themselves
without routing of frames even if they are in different physical locations. VLAN traffic must be
forwarded via a router or OSI Layer-3 switching device when two nodes in different VLANs are
communicating even if they are connected to the same physical LAN. Network broadcasts within
a VLAN generally do not propagate to nodes that belong to a different VLAN, unless configured to
cross a VLAN boundary.
To configure VLANs, an administrator first defines the VLANs on the switches. Each VLAN is
identified by a unique 12-bit VLAN ID (as per IEEE 802.1Q standard). The next step is to
configure the VLAN membership based on an appropriate technique supported by the switches,
such as port-based, MAC-based, protocol-based, IP subnet address-based, and application-based.
In the port-based technique, membership in a VLAN is defined by assigning a VLAN ID to a switch
port. When a node connects to a switch port that belongs to a VLAN, the node becomes a
member of that VLAN. In the MAC-based technique, the membership in a VLAN is defined on the
basis of the MAC address of the node. In the protocol-based technique, different VLANs are
assigned to different protocols based on the protocol type field found in the OSI Layer 2 header.
In the IP subnet address-based technique, the VLAN membership is based on the IP subnet
address. All the nodes in an IP subnet are members of the same VLAN. In the application-based
technique, a specific application, for example, a file transfer protocol (FTP) application can be
configured to execute on one VLAN. A detailed discussion on these VLAN configuration techniques
is beyond the scope of this course.
18
Similar to the VSAN trunking, network traffic from multiple VLANs may traverse a trunk link. A
single network port, called trunk port, is used for sending or receiving traffic from multiple VLANs
over a trunk link. Both the sending and the receiving network components must have at least one
trunk port configured for all or a subset of the VLANs defined on the network component.
As with VSAN tagging, VLAN has its own tagging mechanism. The tagging is performed
by inserting a 4-byte tag field containing 12-bit VLAN ID into the Ethernet frame (as per
IEEE 802.1Q standard) before it is transmitted through a trunk link. The receiving network
component reads the tag and forwards the frame to the destination port(s) that
corresponds to that VLAN ID. The tag is removed once the frame leaves a trunk link to
reach a node port.
Note: IEEE 802.1ad Multi-tagging
IEEE 802.1ad is an amendment to IEEE 802.1Q and enables inserting multiple VLAN tags to an
Ethernet frame. IEEE 802.1Q mandates a single tag with a 12-bit VLAN ID field, which limits the
number of VLANs in an environment theoretically up to 4096. In a large environment such as a
cloud infrastructure, this limitation may restrict VLAN scalability. IEEE 802.1ad provides the
flexibility to accommodate a larger number of VLANs. For example, by using a double-tag,
theoretically 16777216 (40964096 ) VLANs may be configured.
19
A stretched VLAN is a VLAN that spans across multiple sites over a WAN connection. In a
typical multi-site environment, network traffic between sites is routed through an OSI
Layer 3 WAN connection. Because of the routing, it is not possible to transmit OSI Layer
2 traffic between the nodes in two sites. A stretched VLAN extends a VLAN across the
sites and enables nodes in two different sites to communicate over a WAN as if they are
connected to the same network.
Stretched VLANs also allow the movement of virtual machines (VMs) between sites
without the need to change their network configurations. This simplifies the creation of
high-availability clusters, VM migration, and application and workload mobility across
sites. The clustering across sites, for example, enables moving VMs to an alternate site in
the event of a disaster or during the maintenance of one site. Without a stretched VLAN,
the IP addresses of the VMs must be changed to match the addressing scheme at the
other site.
20
This lesson covered link aggregation, switch aggregation, and self-forming network. It also
covered VLAN configuration, VLAN trunking, VLAN tagging, and stretched VLAN.
21
This lesson covers FCIP connectivity, FCIP tunnel configuration, and FCIP protocol stack.
22
FC SAN provides a high-performance infrastructure for localized data movement.

Organizations are now looking for ways to transport data over a long distance between
their disparate FC SANs at multiple geographic locations. One of the best ways to achieve
this goal is to interconnect geographically dispersed FC SANs through reliable, highspeed links. This approach involves transporting the FC block data over the IP
infrastructure.
FCIP is an IP-based protocol that enables distributed FC SAN islands to be interconnected
over an existing IP network. In FCIP, FC frames are encapsulated onto the IP payload
and transported over an IP network. The FC frames are not altered while transferring
over the IP network. In this manner, FCIP creates virtual FC links over IP network to
transfer FC data between FC SANs. FCIP is a tunneling protocol in which FCIP entity such
as an FCIP gateway is used to tunnel FC fabrics through an IP network.
The FCIP standard has rapidly gained acceptance as a manageable, cost-effective way to
blend the best of the two worlds: FC SAN and the proven, widely deployed IP
infrastructure. As a result, organizations now have a better way to store, protect, and
move their data by leveraging investments in their existing IP infrastructure. FCIP is
extensively used in disaster recovery implementations in which data is replicated to the
storage located at a remote site. It also facilitates data sharing and data collaboration
over distance, which is a key requirement for third platform applications.
23
In an FCIP environment, FCIP entity such as an FCIP gateway is connected to each fabric via a
standard FC connection. The FCIP gateway at one end of the IP network encapsulates the FC
frames into IP packets. The gateway at the other end removes the IP wrapper and sends the FC
data to the adjoined fabric. The fabric treats these gateways as fabric switches. An IP address is
assigned to the port on the gateway, which is connected to an IP network. After the IP
connectivity is established, the nodes in the two independent fabrics can communicate with other.
An FCIP tunnel consists of one or more independent connections between two FCIP ports on
gateways (tunnel endpoints). Each tunnel transports encapsulated FC frames over a TCP/IP
network. The nodes in either fabric are unaware of the existence of the tunnel. Multiple tunnels
may be configured between the fabrics based on connectivity requirement. Some
implementations allow aggregating FCIP links (tunnels) to increase throughput and to provide link
redundancy and load balancing.
24
An FCIP tunnel may be configured to merge interconnected fabrics into a single large fabric. In
the merged fabric, FCIP transports existing fabric services across the IP network.
The figure on the slide illustrates a merged fabric deployment. In this deployment, the E_Port on
an FCIP gateway connects to the E_Port of an FC switch in the adjoined fabric. The FCIP gateway
is also configured with a VE_port that behaves like an E_Ports, except that the VE_Port is used to
transport data through an FCIP tunnel. The FCIP tunnel has VE_Ports on both ends. The VE_Ports
establish virtual ISLs through the FCIP tunnel, which enable fabrics on either side of the tunnel to
merge.
25
Frequently, only a small subset of nodes in either fabric requires connectivity across an FCIP
tunnel. Thus, an FCIP tunnel may also use vendor-specific features to route network traffic
between specific nodes without merging the fabrics.
The figure on the slide shows EMC Connectrix B-series solution for FC-FC routing. The figure
shows the same fabrics as shown previously but the FCIP tunnel is configured in a way that does
not merge the fabrics. In this deployment, EX_Port and VE_Port are configured on each FCIP
gateway. The EX_Port on the FCIP gateway connects to an E_Port on an FC switch in the adjoined
fabric. The EX_Port functions similarly to an E_Port, but does not propagate fabric services from
one fabric to another. The EX_Port allows FC-FC routing through the FCIP tunnel, but the fabrics
remain separate.
26
A VSAN, similar to a stretched VLAN, may be extended across sites. The FCIP tunnel may use
vendor-specific features to transfer multiple VSAN traffic through it. The FCIP tunnel functions as
a trunk link and carries tagged FC frames. This allows extending separate VSANseach with their
own fabric services, configuration, and set of FC addressesacross sites.
The figure on the slide shows the EMC MDSseries solution for VSAN over FCIP. The FCIP tunnel is
configured to transfer tagged FC frames belonging to VSAN 10 and VSAN 20. The FCIP tunnel has
virtual trunking E_Port (TE_Port) on both ends, which allows the VSAN tagged traffic to traverse
the FCIP tunnel.
27
The FCIP protocol stack is shown on the slide. Applications generate SCSI commands and
data, which are processed by various layers of the protocol stack. The upper layer
protocol SCSI includes the SCSI driver program that executes the read-and-write
commands. Below the SCSI layer is the FC protocol (FCP) layer, which is simply an FC
frame whose payload is SCSI. The FC frames can be encapsulated into the IP packet and
sent to a remote FC SAN over the IP. The FCIP layer encapsulates the FC frames onto the
IP payload and passes them to the TCP layer. TCP and IP are used for transporting the
encapsulated information across Ethernet, wireless, or other media that support the
TCP/IP traffic.
Encapsulation of FC frame on to IP packet could cause the IP packet to be fragmented
when the data link cannot support the maximum transmission unit (MTU) size of an IP
packet. When an IP packet is fragmented, the required parts of the header must be
copied by all fragments. When a TCP packet is segmented, normal TCP operations are
responsible for receiving and re-sequencing the data prior to passing it on to the FC
processing portion of the device.
28
This lesson covered the FCIP connectivity, FCIP tunnel configuration, and FCIP protocol stack.
29
This module covered IP SAN protocols such as iSCSI and FCIP, components, and connectivity. It
also covered link aggregation, switch aggregation, and VLAN.
30
This module focuses on FCoE SAN components and connectivity, Converged Enhanced Ethernet
functionalities, and FCoE architecture.
Module 11: FC over Ethernet (FCoE) SAN
This lesson covers the components of an FCoE SAN, FCoE SAN connectivity, VLAN and VSAN in
FCoE, and FCoE port types.
FCoE SAN is a Converged Enhanced Ethernet (CEE) network that is capable of transporting FC
data along with regular Ethernet traffic over high speed (such as 10 Gbps or higher) Ethernet
links. It uses FCoE protocol that encapsulates FC frames into Ethernet frames. FCoE protocol is
defined by the T11 standards committee. FCoE is based on an enhanced Ethernet standard that
supports Data Center Bridging (DCB) functionalities (also called CEE functionalities). DCB ensures
lossless transmission of FC traffic over Ethernet. The DCB functionalities are covered later in this
module.
Data centers typically have multiple networks to handle various types of network traffic
for example, an Ethernet LAN for TCP/IP communication and an FC SAN for FC
communication. TCP/IP is typically used for compute-to-compute communication, data
backup, infrastructure management communication, and so on. FC is typically used for
moving block-level data between storage systems and compute systems. To support
multiple networks, compute systems in a data center are equipped with multiple
redundant physical network interfaces for example, multiple Ethernet and FC network
adapters. In addition, to enable the communication, different types of networking
switches and physical cabling infrastructure are implemented in data centers. The need
for two different kinds of physical network infrastructure increases the overall cost and
complexity of data center operation.
FCoE SAN provides the flexibility to deploy the same network components for transferring
both compute-to-compute traffic and FC storage traffic. This helps to mitigate the
complexity of managing multiple discrete network infrastructures. FCoE SAN uses multifunctional network adapters and switches. Therefore, FCoE reduces the number of
network adapters, cables, and switches, along with power and space consumption
required in a data center.
The key FCoE SAN components are:
Network adapters such as Converged Network Adapter (CNA) and software FCoE adapter
Cables such as copper cables and fiber optical cables
FCoE switch
The CNA, software FCoE adapter, and FCoE switch are detailed in the following slides.
The CNA is a physical adapter that provides the functionality of both a standard NIC and
an FC HBA in a single device. It consolidates both FC traffic and regular Ethernet traffic
on a common Ethernet infrastructure. CNAs connect compute systems to the FCoE
switches. They are responsible for encapsulating FC traffic onto Ethernet frames and
forwarding them to FCoE switches over CEE links. They eliminate the need to deploy
separate adapters and cables for FC and Ethernet communications, thereby reducing the
required number of network adapters and switch ports. A CNA offloads the FCoE protocol
processing task from the compute system, thereby freeing the CPU resources of the
compute system for application processing. It contains separate modules for 10 Gigabit
Ethernet (GE), FC, and FCoE Application Specific Integrated Circuits (ASICs). The FCoE
ASIC encapsulates FC frames into Ethernet frames. One end of this ASIC is connected to
10 GE and FC ASICs for compute system connectivity, while the other end provides a 10
GE interface to connect to an FCoE switch.
Instead of a CNA, a software FCoE adapter may also be used. A software FCoE adapter is
OS or hypervisor kernel-resident software that performs FCoE processing. The FCoE
processing consumes compute system CPU cycles. With software FCoE adapters, the OS
or hypervisor implements FC protocol in software that handles SCSI to FC processing.
The software FCoE adapter performs FC to Ethernet encapsulation. Both FCoE traffic
(Ethernet traffic that carries FC data) and regular Ethernet traffic are transferred through
supported NICs on the compute system.
An FCoE switch has both Ethernet switch and FC switch functionalities. It has a Fibre
Channel Forwarder (FCF), an Ethernet Bridge, and a set of ports that can be used for FC
and Ethernet connectivity. FCF handles FCoE login requests, applies zoning, and provides
the fabric services typically associated with an FC switch. It also encapsulates the FC
frames received from the FC port into the Ethernet frames and decapsulates the Ethernet
frames received from the Ethernet Bridge to the FC frames.
Upon receiving the incoming Ethernet traffic, the FCoE switch inspects the Ethertype
(used to indicate which protocol is encapsulated in the payload of an Ethernet frame) of
the incoming frames and uses that to determine their destination. If the Ethertype of the
frame is FCoE, the switch recognizes that the frame contains an FC payload and then
forwards it to the FCF. From there, the FC frame is extracted from the Ethernet frame
and transmitted to the FC SAN over the FC ports. If the Ethertype is not FCoE, the switch
handles the traffic as usual Ethernet traffic and forwards it over the Ethernet ports.
The most common FCoE connectivity uses FCoE switches to interconnect a CEE network
containing compute systems with an FC SAN containing storage systems. The compute systems
have FCoE ports that provide connectivity to the FCoE switches. The FCoE switches enable the
consolidation of FC traffic and Ethernet traffic onto CEE links.
This type of FCoE connectivity is suitable when an organization has an existing FC SAN
environment. Connecting FCoE compute systems to the FC storage systems through FCoE
switches do not require any change in the FC environment.
Some vendors offer FCoE ports in their storage systems. These storage systems connect
directly to the FCoE switches. The FCoE switches form FCoE fabrics between compute
systems and storage systems and provide end-to-end FCoE support. The end-to-end FCoE
connectivity is suitable for a new FCoE deployment.
FCoE protocol enables transmission of FC SAN traffic through an Ethernet LAN that supports Data
Center Bridging (DCB) functionalities. If VLANs and VSANs are created on the LAN and FC SAN
respectively, a mapping is required between the VLANs and VSANs. The mapping determines
which VLAN will carry FC traffic that belongs to a VSAN. The VSAN to VLAN mapping is performed
at the FCoE switch. Multiple VSANs are not allowed to share a VLAN; hence a dedicated VLAN
must be configured at the FCoE switch for each VSAN. It is also recommended that VLANs that
carry regular LAN traffic should not be used for VSAN traffic.
The figure on the slide provides an example of mapping between VLANs and VSANs. In this
example, the FCoE switch is configured with four VLANs VLAN 10, VLAN 20, VLAN 30, and VLAN
40. The Ethernet switch is configured with two VLANs VLAN 10 and VLAN 20. Both VLAN 10 and
VLAN 20 transfer regular LAN traffic to enable compute-to-compute communication. The FC
switch has VSAN 100 and VSAN 200 configured. To allow data transfer between the FCoE
compute system and the FC storage systems through the FCoE switch, VSAN 100 and VSAN 200
must be mapped to VLANs configured on the FCoE switch. As VLAN 10 and VLAN 20 are already
being used for regular LAN traffic, VSAN 100 and VSAN 200 should be mapped to VLAN 30 and
VLAN 40 respectively.
10
FCoE utilizes virtual ports, which emulate the behavior of regular FC ports. FCoE virtual ports
establish virtual FC links between the devices in an FCoE SAN. The different FCoE port types are
listed below:
VN_Port: It is an end point in an FCoE SAN. Typically, it is a CNA port or an FCoE storage
system port that is connected to an FCoE switch in the FCoE SAN.
VF_Port: It is a port on an FCoE switch that connects a VN_Port.
VE_Port: It is a port that forms the connection between two FCoE switches. The VE_Port on an
FCoE switch connects to the VE_Ports of another FCoE switch in an FCoE SAN.
11
This lesson covered the components of an FCoE SAN, FCoE SAN connectivity, VLAN and VSAN in
FCoE, and FCoE port types.
12
This lesson covers Converged Enhanced Ethernet (CEE) and its functionalities priority-based
flow control, enhanced transmission selection, congestion notification, and data center bridging
exchange protocol.
13
Conventional Ethernet is lossy in nature, which means that frames might be dropped or lost under
congestion conditions. Converged Enhanced Ethernet (CEE) provides a new specification to the
existing Ethernet standard. It eliminates the lossy nature of Ethernet and enables convergence of
various types of network traffic on a common Ethernet infrastructure. CEE eliminates the dropping
of frames due to congestion and thereby ensures lossless transmission of FCoE traffic over an
Ethernet network.
The lossless Ethernet is required for the reliable transmission of FC data over an Ethernet
network. Unlike TCP/IP, the loss of a single FC frame typically requires the entire FC exchange to
be aborted and retransmitted, instead of just resending a particular missing frame. CEE makes a
high-speed (such as 10 Gbps or higher) Ethernet network a viable storage networking option,
similar to an FC SAN.
The CEE requires certain functionalities. These functionalities are defined and maintained by the
data center bridging (DCB) task group, which is a part of the IEEE 802.1 working group. These
functionalities are:
Priority-based flow control
Enhanced transmission selection
Congestion notification
Data center bridging exchange protocol
Each of these functionalities is described in the following slides.
14
Traditional FC manages congestion through the use of a link-level, credit-based flow

control that guarantees no loss of FC frames. Typical Ethernet, coupled with TCP/IP, uses
a packet drop flow control mechanism. The packet drop flow control is not lossless. This
challenge is eliminated by using an IEEE 802.3x Ethernet PAUSE control frame to create
a lossless Ethernet. A receiver can send a PAUSE request to a sender when the receivers
buffer is filling up. Upon receiving a PAUSE frame, the sender stops transmitting frames,
which guarantees no loss of frames. The downside of using the Ethernet PAUSE frame is
that it operates on the entire link, which might be carrying multiple traffic flows.
PFC provides a link-level flow control mechanism. PFC creates eight separate virtual links
on a single physical link and allows any of these links to be paused and restarted
independently. PFC enables the PAUSE mechanism based on user priorities or classes of
service. Enabling the PAUSE based on priority allows creating lossless links for network
traffic, such as FCoE traffic. This PAUSE mechanism is typically implemented for FCoE
while regular TCP/IP traffic continues to drop frames. The figure on the slide illustrates
how a physical Ethernet link is divided into eight virtual links and allows a PAUSE for a
single virtual link without affecting the traffic for the others.
15
Enhanced transmission selection (ETS) provides a common management framework for

the allocation of bandwidth to different traffic classes, such as LAN, SAN, and Inter
Process Communication (IPC). For example, an administrator may assign 40 percent of
network bandwidth to LAN traffic, 40 percent of bandwidth to SAN traffic, and 20 percent
of bandwidth to IPC traffic. When a particular class of traffic does not use its allocated
bandwidth, ETS enables other traffic classes to use the available bandwidth.
16
Congestion notification (CN) provides end-to-end congestion management for protocols,

such as FCoE, that do not have built-in congestion control mechanisms. Link level
congestion notification provides a mechanism for detecting congestion and notifying the
source to move the traffic flow away from the congested links. Link level congestion
notification enables a switch to send a signal to other ports that need to stop or slow
down their transmissions. The process of congestion notification and its management is
shown on the slide, which represents the communication between the nodes A (sender)
and B (receiver). If congestion at the receiving end occurs, the algorithm running on the
switch generates a congestion notification message to the sending node (Node A). In
response to the message, the sending end limits the rate of data transfer.
17
DCBX is a discovery and capability exchange protocol, which helps CEE devices to convey
and configure their features with the other CEE devices in the network. DCBX is used to
negotiate capabilities between the switches and the network adapters, which allows the
switch to distribute the configuration values to all the attached adapters. This helps to
ensure consistent configuration across the entire network.
18
This lesson covered Converged Enhanced Ethernet (CEE) and its functionalities priority-based
flow control, enhanced transmission selection, congestion notification, and data center bridging
exchange protocol.
19
This lesson covers FCoE frame structure, FCoE frame mapping, FCoE process, FCoE addressing,
and FCoE frame forwarding.
20
An FCoE frame is an Ethernet frame that contains an FCoE Protocol Data Unit (PDU). The figure
on the slide shows the FCoE frame structure. The Ethernet header includes the source and
destination MAC addresses, IEEE 802.1Q VLAN tag, and Ethertype field. FCoE has its own
Ethertype. The FCoE header includes a version field that identifies the version of FCoE being
implemented and some reserved bits. The Start of Frame (SOF) and the End of Frame (EOF) mark
the start and the end of the encapsulated FC frame respectively. The encapsulated FC frame
consists of the FC header and the data being transported (including the FC CRC). The FCoE frame
ends with the Frame Check Sequence (FCS) field that provides error detection for the Ethernet
frame. Notice that the FCoE frame, unlike iSCSI and FCIP, has no TCP/IP overhead.
Frame size is an important factor in FCoE. A typical FC data frame has a 2112-byte payload, a 24byte header, and an FCS. A standard Ethernet frame has a default payload capacity of 1500
bytes. To maintain good performance, FCoE must use jumbo frames to prevent an FC frame from
being split into two Ethernet frames.
Note: Jumbo frames

Jumbo frames are Ethernet frames with more than 1500 bytes of payload.
21
The encapsulation of the FC frames occurs through the mapping of the FC frames onto
Ethernet, as shown on the slide. FC and traditional networks have stacks of layers where
each layer in the stack represents a set of functionalities. The FC stack consists of five
layers: FC-0 through FC-4. Ethernet is typically considered as a set of protocols that
operates at the physical and data link layers in the seven-layer OSI stack. The FCoE
protocol specification replaces the FC-0 and FC-1 layers of the FC stack with Ethernet.
This provides the capability to carry the FC-2 to the FC-4 layer over the Ethernet layer.
22
To understand the FCoE communication, it is important to know the FCoE process. The FCoE
process includes three key phases: discovery, login, and data transfer. These phases are
described below:
Discovery phase: In this phase, the FCFs discover each other and form an FCoE fabric. The
FCoE nodes also find the available FCFs for login. Moreover, both the FCoE nodes and the FCFs
discover potential VN_Port to VF_Port pairing.
Login phase: In this phase, the virtual FC links are established between VN_Ports and
VF_Ports as well as between VE_Ports. VN_ports perform FC login (including FLOGI, PLOGI,
PRLI) to the discovered FCFs and obtain FC addresses. Each VN_Port also obtains a unique
MAC address.
Data transfer phase: After login, the VN_Ports can start transferring regular FC frames
(encapsulated) over the CEE network.
23
In an FCoE SAN, an FCoE node needs a discovery mechanism that allows it to discover the
available FCFs before it can perform FC login. The mechanism used for the discovery is the FCoE
Initialization Protocol (FIP). FIP is used for discovering the FCFs and establishing virtual links
between FCoE devices (FCoE nodes and FCoE switches). Unlike FCoE frames, FIP frames do not
transport FC data, but contain discovery and login/logout parameters. FIP frames are assigned a
unique EtherType code to distinguish them from the FCoE frames.
The FCoE node to FCF discovery and the login use the following FIP operations:
FCoE node sends multicast FIP Solicitation frame to find which FCFs are available for login.
Each FCF replies to the FCoE node by sending unicast FIP Advertisement frame.
After the FCoE node decides which FCF is appropriate, it sends FIP FLOGI request to the FCF.
The selected FCF sends FIP FLOGI Accept which contains both FC address and MAC address for
the VN_Port. The reason for using FIP for FLOGI instead of a regular FLOGI is that the FIP
FLOGI Accept has a field for the FCF to assign a MAC address to the VN_Port.
24
An FCoE SAN uses MAC address for frame forwarding. The MAC addresses are assigned to the
VN_Ports, VF_Ports, and VE_Ports. The destination and the source MAC addresses are used to
direct frames to their Ethernet destinations. Both the VF_Ports and the VE_Ports obtain MAC
addresses from the FCoE switch. FCoE supports two types of addressing for the VN_Ports: serverprovided MAC address (SPMA) and fabric-provided MAC address (FPMA). These addressing types
are described below:
SPMA: In this type of addressing, the compute systems provide MAC addresses to the
associated VN_Ports. The MAC addresses are issued in accordance with Ethernet standards.
These addresses are either burned-in by the manufacturers of the network adapters or are
configured by an administrator. SPMA can use a single MAC address exclusively for FCoE traffic
or it can have different MAC address for each VN_Port.
FPMA: In this type of addressing, the VN_Ports receive MAC addresses from the FCoE switches
dynamically during login. The VN_Ports then use their granted MAC addresses for
communication. This address is derived by concatenating the 24-bit FC MAC address prefix
(FC-MAP) and the 24-bit FC address assigned to the VN_Port by the FCoE switch. FC-MAP
identifies the fabric to which an FCoE switch belongs. The FPMA ensures that the MAC
addresses are unique within an FCoE SAN.
25
In an FCoE SAN, a node must know two different addresses to forward a frame to another node.
First, it must know the Ethernet MAC address of the FCoE switch port (VF_Port). Second, it must
know the FC address assigned to the destination node port (VN_Port or N_Port). The MAC address
is used to forward an Ethernet frame containing FC payload over a CEE network. The FC address
is used to send the FC frame, encapsulated into the Ethernet frame, to its FC destination.
The figure on the slide shows an example of frame forwarding in the FCoE with existing FC SAN
environment. In this example, an FCoE compute system sends an FC frame to an FC storage
system. The header part of the FC frame includes the standard FC destination ID (D_ID) and FC
source ID (S_ID). The FC frame is encapsulated into an Ethernet frame with a header that
includes the destination and source Ethernet MAC addresses. The Ethernet frame is sent to the
MAC destination address (MAC B) of the attached FCoE switch having FC domain ID 05. Upon
receiving the frame, the FCoE switch strips off the Ethernet frame to retrieve the FC frame. The
FC frame is sent without modification to the FC switch with Domain ID = 1, because 01 is the first
byte in the D_ID of the FC frame. The FC switch forwards the FC frame to the storage system.
26
This lesson covered FCoE frame structure, FCoE frame mapping, FCoE process, FCoE addressing,
and FCoE frame forwarding.
27
This module covered FCoE SAN components and connectivity and Converged Enhanced Ethernet
(CEE) functionalities priority-based flow control, enhanced transmission selection, congestion
notification, and data center bridging exchange protocol. It also covered FCoE architecture that
includes FCoE frame structure, FCoE frame mapping, FCoE process, FCoE addressing, and FCoE
frame forwarding.
28
This module focuses on business continuity and information availability. This module also focuses
on causes and impact of information unavailability. Further, this module focuses on BC planning
lifecycle, BC requirements for third platform, BC technology solutions, specifically on eliminating
single points of failure and application resiliency.
Module 12: Introduction to Business Continuity
This lesson covers the importance of business continuity to an organization, the factors that can
affect information availability and the consequences of information unavailability. This lesson also
covers information availability metrics namely mean time between failure (MTBF) and mean time
to repair (MTTR). Further this lesson covers business continuity planning lifecycle and business
impact analysis. Finally, this lesson covers the BC requirements for third platform environments.
Business continuity (BC) is a set of processes that includes all activities that a business
must perform to mitigate the impact of planned and unplanned downtime. BC entails
preparing for, responding to, and recovering from a system outage that adversely affect
business operations. It describes the processes and procedures an organization
establishes to ensure that essential functions can continue during and after a disaster.
Business continuity prevents interruption of mission-critical services, and reestablishes
the impacted services as swiftly and smoothly as possible by using an automated
process. BC involves proactive measures such as business impact analysis, risk
assessment, building resilient IT infrastructure, deploying data protection solutions
(backup and replication). It also involves reactive countermeasures such as disaster
recovery (discussed later in this module). In a software-defined data center, policy-based
services can be created that include data protection through the self-service portal.
Consumers can select the class of service that best meets their performance, cost, and
protection requirements on demand. Once the service is activated, the underlying data
protection solutions required to support the service is automatically invoked to meet the
required data protection. For example if a service requires VM backup for every six hours, then
backing up VM is scheduled automatically every six hours. The goal of a BC solution is to
ensure information availability required to conduct vital business operations.
Today, businesses rely on information more than ever. Continuous access to information is a must
for the smooth functioning of business operations for any organization. The organizations are
under pressure to deliver services to customers in accordance with service level agreements
(SLAs). The cost of unavailability of information is greater than ever, and outages in key
industries cost millions of dollars per hour. There are also compliance issues, especially if an
organization holds consumer data. The failure to meet industry or government regulations may
result in hefty fines; and loss of business-critical data could compound the financial impact
significantly.
There are many threats to business continuity, such as natural disasters, unplanned occurrences,
and planned occurrences that could result in the inaccessibility of information. IT organizations
are increasingly embracing bring-your-own-device (BYOD) to improve their employee
productivity, lower costs, and support flexible working conditions. However, they also bring
potential risks if they are not deployed smartly due to the fact that the business data (sensitive
and critical data) may also be stored on these devices. Therefore, it is critical for businesses to
define appropriate strategies that can help them to overcome these crises in order to provide
continuous access to information. Business continuity is an important process to define and
implement these strategies.
Information availability (IA) refers to the ability of an IT infrastructure to function according to

business requirements and customer expectations during its specified time of operation. IA
ensures that people (employees, customers, suppliers, and partners) can access information
whenever they need it. IT organizations need to design and build their infrastructure to maximize
the availability of the information, while minimizing the impact of an outage on consumers. IA can
be defined in terms of accessibility, reliability, and timeliness of information.
Accessibility: Information should be accessible to the right user when required.
Reliability: Information should be reliable and correct in all aspects. It is the same as what was
stored and there is no alternation or corruption to the information.
Timeliness: Defines the time window (a particular time of the day, week, month, and year as
specified) during which information must be accessible. For example, if online access to an
application is required between 8:00 am and 10:00 pm each day, any disruption to data
availability outside of this time slot is not considered to affect timeliness.
The slide lists some of the key causes of information unavailability. Data center failure due to
disaster (natural or man-made disasters such as flood, fire, earthquake, and so on) is not the only
cause of information unavailability. Poor application design or resource configuration errors can
also lead to information unavailability. For example, if the database server is down for some
reason, then the data is inaccessible to the consumers, which leads to IT service outage. Even the
unavailability of data due to several factors (data corruption and human error) leads to outage.
The IT department is routinely required to take on activities such as refreshing the data center
infrastructure, migration, running routine maintenance, or even relocating to a new data center.
Any of these activities can have its own significant and negative impact on information
availability.
Note:
In general, the outages can be broadly categorized into planned and unplanned outages. Planned
outages may include installation and maintenance of new hardware, software upgrades or
patches, performing application and data restores, facility operations (renovation and
construction), and migration. Unplanned outages include failure caused by human errors,
database corruption, failure of physical and virtual components, and natural or human-made
disasters.
IT service outage, due to information unavailability, results in loss of productivity, loss of revenue,
poor financial performance, and damages to reputation. The loss of revenue includes direct loss,
compensatory payments, future revenue loss, billing loss, and investment loss. The damages to
reputations may result in a loss of confidence or credibility with customers, suppliers, financial
markets, banks, and business partners. The other possible consequences of outage include the
cost of additional rented equipment, overtime, and extra shipping.
Information availability relies on the availability of both physical and virtual components of a data
center. The failure of these components might disrupt information availability. A failure is the
termination of a components ability to perform a required function. The components ability can
be restored by performing various external corrective actions, such as a manual reboot, a repair,
or replacement of the failed component(s). Proactive risk analysis, performed as part of the BC
planning process, considers the component failure rate and average repair time, which are
measured by MTBF and MTTR:
Mean Time Between Failure (MTBF): It is the average time available for a system or
component to perform its normal operations between failures. It is the measure of system or
component reliability and is usually expressed in hours.
Mean Time To Repair (MTTR): It is the average time required to repair a failed component.
MTTR includes the total time required to do the following activities: detect the fault, mobilize the
maintenance team, diagnose the fault, obtain the spare parts, repair, test, and restore the data.
IA can be expressed in terms of system uptime and downtime and measured as the amount or
percentage of system uptime:
IA = system uptime/(system uptime + system downtime), where system uptime is the period of
time during which the system is in an accessible state; when it is not accessible, it is termed as
system downtime. In terms of MTBF and MTTR, IA could also be expressed as: IA = MTBF/(MTBF
+ MTTR)
Note:
Uptime per year is based on the exact timeliness requirements of the service. This calculation
leads to the number of 9s representation for availability metrics. For example, a service that is
said to be five 9s available is available for 99.999 percent of the scheduled time in a year
(24365).
Disaster recovery (DR) is a part of BC process which involves a set of policies and procedures for
restoring IT infrastructure, including data that is required to support the ongoing IT services, after
a natural or human-induced disaster occurs. Disaster Recovery Plans (DRP) are generally part of a
larger, more extensive practice known as Business Continuity Planning. DR plans should be well
practiced so that the key people become familiar with the specific actions they will need to take
when a disaster occurs. DR plans must also be adaptable and routinely updated, e.g. if some new
people, a new branch office, or some new hardware or software are added to an organization,
they should promptly be incorporated into the organization's disaster recovery plan. The
companies must consider all these facets of their organization as well as update and practice their
plan if they want to maximize their recovery after a disaster. The basic underlying concept of DR
is to have a secondary data center or site (DR site) and at a pre-planned level of operational
readiness when an outage happens at the primary data center. Typically in a DR process, a
previous copy of the data is restored and logs are applied to that copy to bring it to a known point
of consistency. After all recovery efforts are completed, the data is validated to ensure that it is
correct.
The disaster recovery methods often require buying and maintaining a complete set of IT
resources at secondary data centers that matches the business-critical systems at the primary
data center. This includes sufficient storage to house a complete copy of all of the enterprises
business data by regularly copying production data on the mirror systems at secondary site. This
may be a complex process and expensive solution for a significant number of organizations.
Disaster Recovery-as-a-Service (DRaaS) has emerged as a solution to strengthen the portfolio of
a cloud service provider, while offering a viable DR solution to consumer organizations. Having DR
sites in the cloud reduces the need for data center space, IT infrastructure, and IT resources,
which lead to significant cost reductions to organizations. DRaaS is further discussed in module
14, Replication.
As more critical applications are virtualized and data centers move towards softwaredefined approach, it is important for organizations to know that not all applications have
the same recovery requirements. When designing a business continuity strategy,
businesses must consider the two important parameters that are closely associated with
recovery.
Recovery Point Objective (RPO): This is the point-in-time to which systems and data
must be recovered after an outage. It defines the amount of data loss that a business
can endure. Based on the RPO, organizations plan for the frequency with which a backup
or replica must be made. An organization can plan for an appropriate BC technology
solution on the basis of the RPO it sets. For example, if the RPO of a particular business
application is 24 hours, then backups are created every midnight. The corresponding
recovery strategy is to restore data from the set of last backup.
Recovery Time Objective (RTO): This is the time within which systems and
applications must be recovered after an outage. It defines the amount of downtime that
a business can endure and survive. For example, if the RTO is a few seconds, then
implementing global clustering would help to achieve the required RTO. The more critical
the application, the lower the RTO should be.
Both RPO and RTO are counted in minutes, hours, or days and are directly related to the
criticality of the IT service and data. The lower the number of RTO and RPO, the higher
will be the cost of a BC solution.
10
BC planning must follow a disciplined approach like any other planning process. Organizations
today dedicate specialized resources to develop and maintain BC plans. From the
conceptualization to the realization of the BC plan, a lifecycle of activities can be defined for the
BC process. The BC planning lifecycle includes five stages:
1. Establishing objectives
Determine BC requirements
Estimate the scope and budget to achieve requirements
Select a BC team that includes subject matter experts from all areas of business, whether
internal or external
Create BC policies
2. Analyzing
Collect information on data profiles, business processes, infrastructure support, dependencies,
and frequency of using business infrastructure
Conduct a business impact analysis
Identify critical business processes and assign recovery priorities
Perform risk analysis for critical functions and create mitigation strategies
Perform cost benefit analysis for available solutions based on the mitigation strategy
Evaluate options
(Contd)
11
3. Designing and developing

Define the team structure and assign individual roles and responsibilities; for example,
different teams are formed for activities such as emergency response and infrastructure
and application recovery
Design data protection strategies and develop infrastructure
Develop contingency solution and emergency response procedures
Detail recovery and restart procedures
4. Implementing
Implement risk management and mitigation procedures that include backup,
replication, and management of resources
Prepare the DR sites that can be utilized if a disaster affects the primary data center.
The DR site could be one of the organizations own data center or could be a cloud
Implement redundancy for every resource in a data center to avoid single points of
failure
5. Training, testing, assessing, and maintaining

Train the employees who are responsible for backup and replication of business-critical data on
a regular basis or whenever there is a modification in the BC plan
Train employees on emergency response procedures when disasters are declared
Train the recovery team on recovery procedures based on contingency scenarios
Perform damage-assessment processes and review recovery plans
Test the BC plan regularly to evaluate its performance and identify its limitations
Assess the performance reports and identify limitations
Update the BC plans and recovery/restart procedures to reflect regular changes within the data
center
12
A business impact analysis (BIA) identifies which business units, operations, and processes are
essential to the survival of the business. It evaluates the financial, operational, and service impact
of a disruption to essential business processes. The selected functional areas are evaluated to
determine resilience of the infrastructure to support information availability. The BIA process
leads to a report detailing the incidents and their impact over business functions. The impact may
be specified in terms of money or in terms of time. Based on the potential impact associated with
downtime, businesses can prioritize and implement countermeasures to mitigate the likelihood of
such disruptions. These are detailed in the BC plan. A BIA includes the following set of tasks:
Determine the business areas.
For each business area, identify the key business processes critical to its operation.
Determine the attributes of the business process in terms of applications, databases, and
hardware and software requirements.
Estimate the costs of failure for each business process.
Calculate the maximum tolerable outage and define RTO for each business process.
Establish the minimum resources required for the operation of business processes.
Determine the recovery strategies and the cost of implementing them.
Optimize the business recovery strategy based on business priorities.
Analyze the current state of BC readiness and optimize the future BC planning.
13
The rise of the third platform has reached a tipping point according to IDC, and with the
growth of investments in the cloud, big data, mobile, and social computing, there is, in
effect, no way of avoiding it by any organizations. Applications that support these
technologies require significantly higher performance, scalability, and availability compared to the
traditional applications. It has become increasingly important for IT organizations to deliver
services to their customers in accordance with the SLAs. Business continuity solutions must also
be ready to support the requirements of third platform applications. The slide provides a list of
key BC requirements.
14
With the aim of meeting the required availability, organizations should build a resilient IT
infrastructure. Building a resilient IT infrastructure requires the following high availability
solutions:
Deploying redundancy at both the component level and the site (data center) level to avoid
single points of failure
Deploying data protection solutions such as backup and replication
Implementing automated application failover
Architecting resilient applications
For example, as soon as a disaster occurs, the BC solution automatically triggers the DR
process. This process typically involves both operational personnel and automated procedure in
order to reactivate the service (application) at a functioning data center. This requires the transfer
of application users, data, and services to the new data center. This involves the use of redundant
infrastructure across different geographic locations, live migration, backup, and replication
solutions.
15
This lesson covered the importance of business continuity solutions to an organization, factors
that can affect information availability, and the consequences of information unavailability. This
lesson covered information availability metrics namely mean time between failure (MTBF) and
mean time to repair (MTTR). Further this lesson covered business continuity planning lifecycle and
business impact analysis. Finally, this lesson covered the BC requirements for third platform.
16
This lesson covers identifying and eliminating single points of failure. This lesson also covers the
key fault tolerance mechanisms and application resiliency.
17
Single point of failure refers to any individual component or aspect of an infrastructure whose
failure can make the entire system or service unavailable. Single point of failure may occur at
infrastructure component level and site level (data center). The figure on the slide illustrates an
example where various IT infrastructure components, including the compute system, VM instance,
network devices, storage, and site itself, become a single point of failure. Assume that a web
application runs on a VM instance and it uses a database server which runs on another VM to
store and retrieve application data. If the database server is down, then the application would not
be able to access the data and in turn would impact the availability of the service. Consider
another example where a group of compute systems is networked through a single FC switch. The
switch would present a single point of failure. If the switch failed, all of the compute systems
connected to that switch would become inaccessible and result in service unavailability. Therefore,
it is important for organizations to build a fault tolerance IT infrastructure that eliminates single
points of failure in the environment.
18
Highly available infrastructures are typically configured without single points of failure to ensure
that individual component failures do not result in service outages. The general method to avoid
single points of failure is to provide redundant components for each necessary resource, so that a
service can continue with the available resource even if a component fails. Organizations may also
create multiple availability zones (discussed later in the module) to avoid single points of failure
at data center level. Usually, each zone is isolated from others, so that the failure of one zone
would not impact the other zones. It is also important to have high availability mechanisms that
enable automated application/service failover within and across the zones in the event of
component failure or disaster.
Note:
N+1 redundancy is a common form of fault tolerance mechanism that ensures service availability
in the event of a component failure. A set of N components has at least one standby component.
This is typically implemented as an active/passive arrangement, as the additional component does
not actively participate in the service operations. The standby component is active only if any one
of the active components fails. N+1 redundancy with active/active component configuration is
also available. In such cases all the component remains active. For example, if an active/active
configuration is implemented at the site level, then a service is fully deployed in both the sites.
The load for this service is balanced between the sites. If one of the sites is down, the available
site would manage the service operations and manage the workload.
19
Organizations should follow stringent guidelines to implement fault tolerance in their data
centers for uninterrupted services. The underlying IT infrastructure components (compute,
storage, and network) should be highly available and the single points of failure at the component
level should be avoided. The example shown on the slide represents an infrastructure designed to
mitigate the single points of failure at component level. The single points of failure at the compute
level can be avoided by implementing redundant compute systems in a clustered configuration.
Single points of failure at the network level can be avoided via path and node redundancy and
various fault tolerance protocols. Multiple independent paths can be configured between nodes so
that if a component along the main path fails, traffic is rerouted along another path. The key
techniques for protecting storage from single points of failure are RAID, erasure coding
techniques, dynamic disk sparing, and configuring redundant storage system components. Many
storage systems also support redundant array independent nodes (RAIN) architecture to improve
the fault tolerance. The following slides will discuss the various fault tolerance mechanisms as
listed on the slide to avoid single points of failure at component level.
20
Compute clustering is one of the key fault tolerance mechanisms that provide continuous
availability of service even when a VM instance, physical compute systems, OS, or hypervisor
fails. Clustering is a technique where at least two compute systems (or nodes) work together and
are viewed as a single compute system to provide high availability and load balancing. If one of
the compute systems fails, the service running in the compute system can failover to another
compute system in the cluster to minimize or avoid any outage.
The two common cluster implementations are active/active and active/passive. In active/active
clustering, the nodes in a cluster are all active participants and run the same service of their
clients. The active/active cluster balances requests for service among the nodes. If one of the
nodes fails, the surviving nodes take the load of the failed one. This method enhances both the
performance and the availability of a service. The nodes in the cluster have access to shared
storage volumes. In active/active clustering only one node can write or update the data in a
shared file system or database at a given time. In active/passive clustering the service runs on
one or more nodes, and the passive node just waits for a failover. If the active node fails, the
service that had been running on the active node is failed over to the passive node.
Active/passive clustering does not provide performance improvement like active/active clustering.
Clustering uses a heartbeat mechanism to determine the health of each node in the cluster. The
exchange of heartbeat signals, usually happens over a private network, allows participating
cluster members to monitor one anothers status. Clustering can be implemented between
multiple physical compute systems, or between multiple VMs, or between VM and physical
compute system, or between multiple hypervisors.
21
The figure on the slide shows an example of clustering where multiple hypervisors running on
different compute systems are clustered and are accessing hypervisors native file system (a
clustered file system that allows multiple hypervisors, running on different physical compute
systems, to access the same shared storage resources concurrently). This method provides high
availability for services running on VMs by pooling the VMs and compute systems that reside on
into a cluster. If a physical compute system running a VM fails, the VM will be restarted on
another compute system in the cluster. This method provides rapid recovery of services running
on VMs in the event of compute system failure.
In some hypervisor cluster implementations, the hypervisor uses its native technique to provide
continuous availability of services running on VMs even if a physical compute system or a
hypervisor fails. In this implementation, a live instance (i.e., a secondary VM) of a primary VM is
created on another compute system. The primary and secondary VMs exchange heartbeats. If the
primary VM fails due to hardware failure, the clustering enables failover to the secondary VM
immediately. After a transparent failover occurs, a new secondary VM is created and redundancy
is reestablished. The hypervisor running the primary VM as shown in the figure on the slide
captures the sequence of events for the primary VM, including instructions from the virtual I/O
devices, virtual NICs, etc. Then it transfers these sequences to the hypervisor running on another
compute system. The hypervisor running the secondary VM receives these event sequences and
sends them to the secondary VM for execution. The primary and the secondary VMs share the
same storage, but all output operations are performed only by the primary VM. A locking
mechanism ensures that the secondary VM does not perform write operations on the shared
storage. The hypervisor posts all events to the secondary VM at the same execution point as they
occurred on the primary VM. This way, these VMs play exactly the same set of events and their
states are synchronized with each other.
22
A short-time network interruption could impact plenty of services running in a data center
environment. So, the network infrastructure must be fully redundant and highly available with no
single points of failure. The techniques such as link aggregation, NIC teaming, and multipathing
provide fault tolerance mechanism against link failure.
Link aggregation combines two or more network links into a single logical link, called portchannel, yielding higher bandwidth than a single link could provide. Link aggregation enables
distribution of network traffic across the links and traffic failover in the event of a link failure. If a
link in the aggregation is lost, all network traffic on that link is redistributed across the remaining
links.
NIC teaming groups NICs so that they appear as a single, logical NIC to the OS or hypervisor. NIC
teaming provides network traffic failover to prevent connectivity loss in the event of a NIC failure
or a network link outage. In some cases, NIC teaming enables aggregation of network bandwidth
of individual NICs. The bandwidth aggregation facilitates distribution of network traffic across
NICs in the team.
23
Multipathing enables organizations to meet aggressive availability and performance service levels.
It enables a compute system to use multiple paths for transferring data to a LUN on a storage
system. Multipathing enables automated path failover that eliminates the possibility of disrupting
an application or service due to the failure of an adapter, cable, port, and so on. In the event of a
path failover, all outstanding and subsequent I/O requests are automatically directed to
alternative paths.
Typically, a single path from a compute system to a LUN consists of a NIC or HBA port, switch
ports, connecting cables, and a storage controller (SC) port. To use multipathing, multiple paths
must exist between the compute and the storage systems. Each path can be configured as either
active or standby. Standby paths become active if one or more active paths fails. If an active path
fails, the multipathing process detects the failed path and then redirects I/Os of the failed path to
another active path. Multipathing can be a built-in OS and hypervisor function, or it can be
provided by a third-party as a software module that can be installed to the OS or hypervisor. The
figure on the slide shows a configuration where four paths between the physical compute system
(with dual-port HBAs) and the LUN enable multipathing. Multipathing can perform load balancing
by distributing I/O across all active paths.
24
Data centers comprise storage systems with a very large number of disk drives and solid state
drives to support the various applications and services running in the environment. The failure of
these drives could result in data loss and information unavailability. The greater the number of
drives in use, the greater is the probability of a drive failure. The following techniques provide
data protection in the event of drive failure.
RAID is a technique that combines multiple drives into a logical unit called a RAID set. Nearly all
RAID implementation models provide data protection against drive failures. The figure on the slide
illustrates an example of RAID 6 (dual distributed parity), where data is protected against two
disk failures.
Erasure coding provides space-optimal data redundancy to protect data loss against multiple drive
failures.
A dynamic disk sparing is a fault tolerance mechanism that refers to a spare drive which
automatically replaces a failed disk drive by taking the identity of it. A spare drive should be large
enough to accommodate data from a failed drive. Some systems implement multiple spare drives
to improve data availability. In a dynamic disk sparing, when the recoverable error rates for a disk
exceed a predetermined threshold, the disk subsystem tries to copy data from the failing disk to
the spare drive automatically. If this task is completed before the damaged disk fails, the
subsystem switches to the spare disk and marks the failing disk as unusable. Otherwise, it uses
parity or the mirrored disk to recover the data.
25
Storage resiliency can also be achieved by using a storage virtualization appliance. A virtualization
layer created at SAN using virtualization appliance abstracts the identity of physical storage
devices and creates a storage pool from heterogeneous storage systems. Virtual volume is
created from the storage pool and assigned to the compute system. Instead of being directed to
the LUNs on the individual storage systems, the compute systems are directed to the virtual
volume provided by the virtualization layer.
The figure on the slide provides an illustration of a virtual volume that is mirrored between LUNs
of two different storage systems. Each I/O to the virtual volume is mirrored to the underlying
LUNs on the storage systems. If one of the storage systems incurs an outage due to failure or
maintenance, the virtualization appliance will be able to continue processing I/O on the surviving
mirror leg. Upon restoration of the failed storage system, the data from the surviving LUN is
resynchronized to the recovered leg. This method provides protection and high availability for
critical services in the event of a storage system failure.
26
An important high availability design best practice is to create availability zones. An availability
zone is a location with its own set of resources and isolated from other zones. Therefore, a failure
in one zone will not impact other zones. A zone can be a part of a data center or may even be
comprised of the whole data center. This provides redundant computing facilities on which
applications or services can be deployed. Organizations can deploy multiple zones within a data
center (to run multiple instances of a service), so that if one of the zone incurs an outage due to
some reason, the service can be failed over to the other zone. For example, if two compute
systems are deployed, one in zone A and the other in zone B, then the probability that both go
down at the same time due to an external event is extremely small. This simple strategy allows
the organization to construct highly reliable web services by placing compute systems into
multiple zones such that the failure of one zone does not disrupt the service, or at the very least,
allow us to rapidly reconstruct the service in the second zone. Organizations also deploy multiple
zones across geographically dispersed data centers (to run multiple instances of a service), so
that the service can survive even if the failure is at the data center level. It is also important that
there should be a mechanism that allows seamless (automated) failover of services running in
one zone to another. Automated failover provides a reduced RTO when compared to the manual
process. A failover process also depends upon other capabilities, including replication and live
migration capabilities (discussed in Module 14, Replication), and reliable network infrastructure
between the zones.
27
High availability can be achieved by moving services across zones located in different locations
without user interruption. The services can be moved across zones by implementing stretched
cluster. A stretched cluster is a cluster with compute systems in different remote locations which
provides DR capability in the event of disaster in one of the data centers. Stretched clusters are
typically built as a way to create active/active zones in order to provide high availability and
enable dynamic workload balancing across zones. The figure on the slide also illustrates that a
virtual volume is created from the federated storage resources across zones. The virtualization
appliance has the ability to mirror the data of a virtual volume between the LUNs located in two
different storage systems at different locations. Each I/O from a host to the virtual volume is
mirrored to the underlying LUNs on the storage systems. If an outage occurs at one of the data
centers, for example at zone A, then the running VMs at zone A can be restarted at Zone B
without impacting the service availability. This setup also allows accessing the storage even if one
of the storage systems is unavailable. If storage system at zone A is unavailable, then the
hypervisor running there still accesses the virtual volume and in turn accesses the data from the
available storage system at zone B.
28
Today, organizations typically build their IT infrastructure using commodity systems to achieve
scalability and keep hardware costs down. In this environment, it is assumed that some
components will fail. Therefore, in the design of an application the failure of individual resources
often has to be anticipated to ensure an acceptable availability of the application. A reliable
application properly manages the failure of one or more modules and continues operating
properly. If a failed operation is retried a few milliseconds later, the operation may succeed.
These types of error conditions are called as transient faults. Fault resilient applications have logic
to detect and handle transient fault conditions in order to avoid application downtime. In the next
slide, we will discuss on the key application design strategies for improving availability.
29
Graceful degradation: It refers to the ability of an application to maintain limited functionality

even when some of the components, modules, or supporting services are not available. The
purpose of graceful degradation of application functionality is to prevent the complete failure of a
business application. For example, consider an e-commerce application that consists of modules
such as product catalog, shopping cart, order status, order submission, and order processing.
Assume that due to some problem the payment gateway is unavailable. It is impossible for the
order processing module of the application to continue. If the application is not designed to
handle this scenario, the entire application might go offline. However, in this same scenario, it is
still possible to make the product catalog module available to consumers, to view the product
catalog. The application could also allow one to place the order and move it into the shopping
cart. This provides the ability to process the orders when the payment gateway is available or
after failing over to a secondary payment gateway.
Fault detection and retry logic: A key mechanism in an application design is to implement retry
logic within a code in order to handle a service that is temporarily down. When applications use
other services, errors can occur because of temporary conditions such as intermittent service,
infrastructure-level faults, or network issues. Very often, this form of problem can be solved by
retrying the operation a few milliseconds later, and the operation may succeed. To implement the
retry logic in an application, it is important to detect and identify that particular exception which is
likely to be caused by a transient fault condition. A retry strategy must also be defined to state
how many retries can be attempted before deciding that the fault is not transient.
Persistent application state model: In a stateful application model, the session state
information of an application (for example user ID, selected products in a shopping cart, and so
on) is usually stored in compute system memory. However, the information stored in the memory
can be lost if there is an outage with the compute system where the application runs. In a
persistent application state model, the state information is stored out of the memory and is
usually stored in a repository (database). If a VM running the application instance fails, the state
information is still available in the repository. A new application instance is created on another VM
which can access the state information from the database and resume the processing.
30
This lesson covered identifying and eliminating single points of failure. This lesson also covered
the key fault tolerance mechanisms and application resiliency.
31
The Concepts in Practice section covers the product EMC PowerPath, VMware HA and FT.
Note:
32
EMC PowerPath is host-based multipathing software. Every I/O from the host to the array must
pass through the PowerPath software, which allows PowerPath to provide intelligent I/O path
management. PowerPath provides path failover and dynamic load balancing. PowerPath/VE
software allows optimizing virtual environments with PowerPath multipathing features.
VMware HA provides high availability for applications running in VMs. In the event of physical
compute system failure, the affected VMs are automatically restarted on other compute systems.
VMware HA minimizes unplanned downtime and IT service disruption while eliminating the need
for dedicated standby hardware and installation of additional software.
VMware FT provides continuous availability for applications in the event of server failures by
creating a live shadow instance of a VM that is in virtual lockstep with the primary VM instance.
VMware FT is used to prevent application disruption due to hardware failures. The downtime
associated with mission-critical applications can be very expensive and disruptive to businesses.
By allowing instantaneous failover between the two instances in the event of hardware failure, FT
eliminates even the smallest chance of data loss or disruption.
33
This module covered business continuity and information availability. This module also covered
BC planning lifecycle, BC technology solutions, specifically on eliminating single points of

failure and application resiliency.
34
A system has three components and requires all three to be operational for 24 hours from
Monday to Friday. Failure of component 1 occurs as follows:
Monday = No failure
Tuesday = 5 am to 7 am
Wednesday = No failure
Thursday = 4 pm to 8 pm
Friday = 8 am to 11 am
Calculate the MTBF and MTTR of component 1.
MTBF is calculated as: total uptime / number of failures

Total downtime = 2 hours on Tuesday + 4 hours on Thursday + 3 hours on Friday = 9 hours
Total uptime = (5 24) 9 = 111 hours
So, MTBF = 111 / 3 = 37 hours
MTTR is calculated as: total downtime / number of failures

Total downtime = 2 hours on Tuesday + 4 hours on Thursday + 3 hours on Friday = 9 hours
So, MTTR = 9 hours / 3 = 3 hours
A system has three components and requires all three to be operational from 8 am to 5 pm,
Monday to Friday. Failure of component 2 occurs as follows:
Monday = 8 am to 11 am
Tuesday = No failure
Wednesday = 4 pm to 7 pm
Thursday = 5 pm to 8 pm
Friday = 1 pm to 2 pm
Calculate the availability of component 2.
Availability is calculated as: system uptime / (system uptime + system downtime)

System downtime = 3 hours on Monday + 1 hour on Wednesday + 1 hour on Friday = 5 hours
System uptime = total operational time system downtime = 45 hours 5 hours = 40 hours
Availability (%) = (40 / 45) 100 = 88.9%
This module focuses on backup architecture and various backup targets. This module also focuses
on various backup methods and data deduplication technologies. This module further focuses on
cloud-based and mobile device backup. Finally, this module focuses on data archive.
Module 13: Backup and Archive
This lesson covers the primary purposes of backup and backup architecture including key
components in a backup environment. This lesson also covers various backup targets such as
tape library, disk library, and virtual tape library. This lesson further covers backup granularity
and key backup and recovery considerations.
Like protecting the infrastructure components (compute, storage, and network), it is also critical
for organizations to protect the data. Typically organizations implement data protection solution in
order to protect the data from accidentally deleting files, application crashes, data corruption, and
disaster. Data should be protected at local location and as well as to a remote location to ensure
the availability of service. For example, when a service is failed over to other zone (data center),
the data should be available at the destination in order to successfully failover the service to
minimize the impact to the service. One of the key data protection solutions widely implemented
is backup.
A backup is an additional copy of production data, created and retained for the sole purpose of
recovering the lost or corrupted data. With the growing business and the regulatory demands for
data storage, retention, and availability, organizations face the task of backing up an everincreasing amount of data. This task becomes more challenging with the growth of data, reduced
IT budgets, and less time available for taking backups. Moreover, organizations need fast backup
and recovery of data to meet their service level agreements. Most organizations spend a
considerable amount of time and money protecting their application data but give less attention
to protecting their server configurations. During disaster recovery, server configurations must be
re-created before the application and data are accessible to the user. The process of system
recovery involves reinstalling the operating system, applications, and server settings and then
recovering the data. So it is important to backup both application data and server configurations.
Evaluating backup technologies, recovery, and retention requirements for data and applications is
an essential step to ensure successful implementation of a backup and recovery solution.
Backups are performed for three primary purposes; disaster recovery, operational restores, and
long-term storage.
Disaster recovery addresses the requirement to restore all, or a large part of, an IT
infrastructure in the event of a major disaster. The backup copies are used for restoring data at
an alternate site when the primary site is incapacitated due to a disaster. Based on recovery
requirements, organizations use different backup strategies for disaster recovery.
Operational backup is a backup of data at a point-in-time (PIT) for the purpose of restoring
data in the event of data loss or logical corruptions that may occur during routine processing.
The majority of restore requests in an organization are classified in this category. An example
of an operational backup is a backup taken just before a major change to a production system
occurs. This ensures the availability of a clean copy of production data if the change corrupts
the production data.
Backups are also performed to address long-term storage requirements. For example, an
organization may require to keep transaction records and other business records required for
regulatory compliance.
In a backup environment, the common backup components are backup client, backup server,
storage node, and backup target.
The role of a backup client is to gather the data that is to be backed up and send it to the storage
node. The backup client can be installed on application servers, mobile clients, and desktops. It
also sends the tracking information to the backup server.
The backup server manages the backup operations and maintains the backup catalog, which
contains information about the backup configuration and backup metadata. The backup
configuration contains information about when to run backups, which client data to be backed up,
and so on. The backup metadata contains information about the backed up data.
The storage node is responsible for organizing the clients data and writing the data to a backup
device. A storage node controls one or more backup devices. In most of the implementation, the
storage node and the backup server run on the same system. Backup devices may be attached
directly or through a network to the storage node. The storage node sends the tracking
information about the data written to the backup device to the backup server. Typically this
information is used for recoveries.
A wide range of backup targets are currently available such as tape, disk, and virtual tape library.
Now, organization can also back up their data to the cloud storage. Many service providers offer
backup as a service that enables an organization to reduce its backup management overhead.
Backup as a service will be discussed later in this module.
A tape library contains one or more tape drives that records and retrieves data on a magnetic
tape. Tape is portable, and one of the primary reasons for the use of tape is long-term, off-site
storage. Backups implemented using tape devices involve several hidden costs. Tapes must be
stored in locations with a controlled environment to ensure preservation of the media and to
prevent data corruption. Physical transportation of the tapes to offsite locations also adds
management overhead and increases the possibility of loss of tapes during offsite shipment. The
traditional backup process, using tapes, is not optimized to recognize duplicate content. Due to its
sequential data access, both backing up of data and restoring it take more time with tape, and
this may impact the backup window (the period during which a production volume is available to
perform backup) and RTO. Data integrity and recoverability are also major issues with tape-based
backup media.
Disk density has increased dramatically over the past few years, lowering the cost per gigabyte to
the point where it became a viable backup target for organizations. When used in a highly
available configuration in a storage array, disks offer a reliable and fast backup target medium.
One way to implement a backup-to-disk system is by using it as a staging area, offloading backup
data to a secondary backup target such as tape after a period of time. Some vendors offer a
purpose-built, disk-based backup appliances that are emerged as the optimal backup target
solution. These systems are optimized for backup and recovery operations, offering extensive
integration with popular backup management applications. The built-in features such as
replication, compression, encryption, and data deduplication increase the value of purpose-built
backup appliances.
Virtual tape libraries use disks as backup media. Virtual tapes are disk drives that are emulated
and presented as tapes to the backup software. Compared to physical tapes, virtual tapes offer
better performance, better reliability, and random disk access. A virtual tape drive does not
require the usual maintenance tasks associated with a physical tape drive, such as periodic
cleaning and drive calibration. Compared to the disk library, a virtual tape library offers easy
installation and administration because it is preconfigured by the manufacturer. A key feature that
is usually available on virtual tape library appliances is replication.
The backup operation is typically initiated by a server, but it can also be initiated by a client. The
backup server initiates the backup process for different clients based on the backup schedule
configured for them. For example, the backup for a group of clients may be scheduled to start at
3:00 a.m. every day. The backup server coordinates the backup process with all the components
in a backup environment. The backup server maintains the information about backup clients to be
backed up and storage nodes to be used in a backup operation. The backup server retrieves the
backup-related information from the backup catalog and, based on this information, instructs the
storage node to load the appropriate backup media into the backup devices. Simultaneously, it
instructs the backup clients to gather the data to be backed up and sends it over the network to
the assigned storage node. After the backup data is sent to the storage node, the client sends
some backup metadata (the number of files, name of the files, storage node details, and so on) to
the backup server. The storage node receives the client data, organizes it, and sends it to the
backup device. The storage node then sends additional backup metadata (location of the data on
the backup device, time of backup, and so on) to the backup server. The backup server updates
the backup catalog with this information. The backup data from the client can be sent to the
backup device over a LAN or SAN network.
Hot backup and cold backup are the two methods deployed for backup. They are based on the
state of the application when the backup is performed. In a hot backup, the application is up-andrunning, with users accessing their data during the backup process. This method of backup is also
referred to as online backup. The hot backup of online production data is challenging because
data is actively being used and changed. If a file is open, it is normally not backed up during the
backup process. In such situations, an open file agent is required to back up the open file. These
agents interact directly with the operating system or application and enable the creation of
consistent copies of open files. The disadvantage associated with a hot backup is that the agents
usually affect the overall application performance. A cold backup requires the application to be
shutdown during the backup process. Hence, this method is also referred to as offline backup.
Consistent backups of databases can also be done by using a cold backup. The disadvantage of a
cold backup is that the database is inaccessible to users during the backup process.
After the data is backed up, it can be restored when required. A restore process can be manually
initiated from the client. A recovery operation restores data to its original state at a specific PIT.
Typically backup applications support restoring one or more individual files, directories, or VMs.
The figure on the slide depicts a restore operation.
Upon receiving a restore request, an administrator opens the restore application to view the list of
clients that have been backed up. While selecting the client for which a restore request has been
made, the administrator also needs to identify the client that will receive the restored data. Data
can be restored on the same client for whom the restore request has been made or on any other
client. The administrator then selects the data to be restored and the specified point in time to
which the data has to be restored based on the RPO. Because all this information comes from the
backup catalog, the restore application needs to communicate with the backup server. The
backup server instructs the appropriate storage node to mount the specific backup media onto the
backup device. Data is then read and sent to the client that has been identified to receive the
restored data.
Some restorations are successfully accomplished by recovering only the requested production
data. For example, the recovery process of a spreadsheet is completed when the specific file is
restored. In database restorations, additional data, such as log files, must be restored along with
the production data. This ensures consistency of the restored data. In these cases, the RTO is
extended due to the additional steps in the restore operation. It is also important for the backup
and recovery applications to have security mechanisms to avoid recovery of data by nonauthorized users.
Backup granularity depends on business needs and the required RTO/RPO. Based on the
granularity, backups can be categorized as full, incremental, cumulative (or differential),
synthetic, and incremental forever backup. Most organizations use a combination of these backup
types to meet their backup and recovery requirements. The figure on the slide depicts the
different backup granularity levels.
Full backup: As the name applies, it is a full copy of the entire data set. Organizations typically
use full backup on a periodic basis because it requires more storage space and also takes more
time to back up. The full backup provides a faster data recovery.
Incremental backup: It copies the data that has changed since the last backup. For example, a
full backup is created on Monday, and incremental backups are created for the rest of the week.
Tuesday's backup would only contain the data that has changed since Monday. Wednesday's
backup would only contain the data that has changed since Tuesday. The primary disadvantage to
incremental backups is that they can be time-consuming to restore. Suppose an administrator
wants to restore the backup from Wednesday. To do so, the administrator has to first restore
Monday's full backup. After that, the administrator has to restore Tuesday's copy, followed by
Wednesday's.
Cumulative (differential) backup: It copies the data that has changed since the last full
backup. Suppose for example the administrator wants to create a full backup on Monday and
differential backups for the rest of the week. Tuesday's backup would contain all of the data that
has changed since Monday. It would therefore be identical to an incremental backup at this point.
On Wednesday, however, the differential backup would backup any data that had changed since
Monday (full backup). The advantage that differential backups have over incremental is shorter
restore times. Restoring a differential backup never requires more than two copies. Of course the
tradeoff is that as time progresses, a differential backup can grow to contain much more data
than an incremental backup.
(Contd)
Synthetic backup: Another way to implement full backup is synthetic backup. This method is
used when the production volume resources cannot be exclusively reserved for a backup process
for extended periods to perform a full backup. A synthetic backup takes data from an existing full
backup and merges it with the data from any existing incrementals and cumulatives. This
effectively results in a new full backup of the data. This backup is called synthetic because the
backup is not created directly from production data. A synthetic full backup enables a full backup
copy to be created offline without disrupting the I/O operation on the production volume. This
also frees up network resources from the backup process, making them available for other
production uses.
Incremental forever backup: Rather than scheduling periodic full backups, this backup solution
requires only one initial full backup. Afterwards, an ongoing (forever) sequence of incremental
backups occurs. The real difference, however, is that the incremental backups are automatically
combined with the original in such a way that you never need to perform a full backup again. This
method reduces the amount of data that goes across the network and reduces the length of the
backup window.
10
Organizations need reliable and efficient backup and recovery solutions to meet the required
service levels. Organizations need to understand some of the key backup and recovery
considerations while implementing the solutions. Today, business applications typically run on
virtual machines (VMs). Multiple VMs are hosted on single or clustered physical compute systems.
The virtualized compute system environment is typically managed from a management server,
which provides a centralized management console for managing the environment. The integration
of backup applications with the management server of virtualized environment is required.
Advanced backup methods require the backup application to obtain a view of the virtualized
environment and send configuration commands related to backup to the management server. As
the organization moves towards third platform technologies, it is important for the backup
applications to integrate with various business applications along with third platform applications
(social, mobile, analytics, and cloud) to protect the business data.
It is also important to ensure that most of the backup and recovery operations need to be
automated. Typically, the BC processes are automated through orchestration to reduce the
manual intervention. With the ability to rapidly add VMs, the solution needs to automatically
provision backup to the new VMs. The solution should recognize when a new VM is created and
apply an appropriate protection policy automatically. Similarly, when a VM is removed from the
environment, the protection policy should be removed as well.
IT environment has large volume of redundant data. Backing up of redundant data would
significantly impact the backup window and increase the operating expenditure. Backup
applications should support deduplication techniques to overcome these challenges. Deduplication
also provides the extended retention of backup copies. Backup solutions should also support WAN
optimization techniques, to reduce the network bandwidth requirement, when taking remote
office backup.
11
The amount of data loss and downtime that a business can endure in terms of RPO and RTO are
the primary considerations in selecting and implementing a specific backup strategy. RPO
specifies the time interval between two backups. For example, if a service requires an RPO of four
hours, the data needs to be backed up every four hours. RTO relates to the time taken by the
recovery process. To meet the defined RTO, organizations should choose the appropriate backup
target, and backup granularity to minimize the recovery time. For example, a restore from tapes
takes longer to complete than a restore from disks. The organization can also implement a
strategy where the inactive data (fixed data) can be archived before performing backup. This
would reduce backup windows and also reduce backup infrastructure cost.
When planning a backup solution, it is important to evaluate the restore needs. Most common
restores involve application data, e-mail, and specific files, but sometimes the user or business
may require complete restore of a VM. To meet the various restore demand, the backup
application should support recovery of data at both the file and the VM-level. Organizations can
provide the capability to their consumers for self-service file restores.
Particularly, when an organization offers backup services, they may store the consumers backup
data on a same backup device. So it is important that the backup device should support secure
multi-tenancy. The backup management software should have single management interface to
monitor and manage the whole backup infrastructure centrally. The management software should
have the capability to provide chargeback or show-back reporting for backed up data by their
consumers.
12
This lesson covered the primary purposes of backup and its architecture including key
components in a backup environment. This lesson also covered various backup targets such as
tape library, disk library, and virtual tape library. This lesson further covered backup granularity
and key backup and recovery considerations.
13
This lesson covers agent-based backup and image-based backup approach. This lesson also
covers NDMP-based backup for NAS environment and direct backup from primary storage to
backup device.
14
In this approach, an agent or client is installed on a virtual machine or a physical compute

system. The agent streams the backup data to the backup device as shown in the figure on the
slide. This backup does not capture virtual machine configuration files. The agent running on the
compute system consumes CPU cycles and memory resources. If multiple VMs on a compute
system are backed up simultaneously, then the combined I/O and bandwidth demands placed on
the compute system by the various backup operations can deplete the compute system
resources. This may impact the performance of the services or applications running on the VMs.
To overcome these challenges, the backup process can be offloaded from the VMs to a proxy
server. This can be achieved by using the image-based backup approach.
15
Image-level backup makes a copy of the virtual disk and configuration associated with a particular
VM. The backup is saved as a single entity called as VM image. This type of backup is suitable for
restoring an entire VM in the event of a hardware failure or human error such as the accidental
deletion of the VM. It is also possible to restore individual files and folders/directories within a
virtual machine. In an image-level backup, the backup software can backup VMs without installing
backup agents inside the VMs or at the hypervisor-level. The backup processing is performed by a
proxy server that acts as the backup client, thereby offloading the backup processing from the
VMs. The proxy server communicates to the management server responsible for managing the
virtualized compute environment. It sends commands to create a snapshot of the VM to be
backed up and to mount the snapshot to the proxy server. A snapshot captures the configuration
and virtual disk data of the target VM and provides a point-in-time view of the VM. The proxy
server then performs backup by using the snapshot. The figure on the slide illustrates image-level
backup.
16
Changed block tracking for backup: To increase the efficiency of image-based backup, some
vendors support incremental backup through tracking changed blocks. This feature identifies and
tags any blocks that have changed since the last VM snapshot. This enables the backup
application to backup only the blocks that have changed, rather than backing up every block.
Changed block tracking technique dramatically reduces the amount of data copied before
additional data reduction technologies are applied, reduces the backup windows and the amount
of required storage for protecting VMs.
Changed block tracking for restoring: This technique reduces recovery time (RTO)
compared to full image restores by only restoring the delta of changed VM blocks. During
a restore process, it is determined which blocks have changed since the last backup. For
example, if a large database is corrupted, a changed block recovery would just restore
the parts of the database that has changed since the last backup was made.
17
Recovery-in-place (Instant VM recovery) is a term that refers to running a VM directly from the
backup device, using a backed up copy of the VM image instead of restoring that image file. One
of the primary benefits of recovery in place is that it eliminates the need to transfer the image
from the backup area to the primary storage area before it is restarted, so the application that are
running on those VMs can be accessed more quickly. This not only saves time for recovery, but
also reduces network bandwidth to restore files. When a VM is recovered in place it is dependent
on the storage I/O performance of the actual disk target (disk backup appliance).
18
As the amount of unstructured data continues to grow exponentially, organizations face the
daunting task of ensuring that critical data on NAS systems are protected. Most NAS heads run on
proprietary operating systems designed for serving files. To maintain its operational efficiency
generally it does not support the hosting of third-party applications such as backup clients. This
forced backup administrators to backup data from application server or mount each NAS volume
via CIFS or NFS from another server across the network, which hosted a backup agent. These
approaches may lead to performance degradation of application server and production network
during backup operations, due to overhead.
Further, security structures differ on the two network file systems, NFS and CIFS. Backups
implemented via one of the file system would not effectively backup any data security attributes
on the NAS head that was accessed via a different file system. For example, CIFS backup, when
restored, would not be able to restore NFS file attributes and vice-versa. These backup challenges
of the NAS environment can be addressed with the use of Network Data Management Protocol
(NDMP).
NDMP is an industry-standard TCP/IP-based protocol specifically designed for a backup in a NAS
environment. It communicates with several elements in the backup environment (NAS head,
backup devices, backup server, and so on) for data transfer and enables vendors to use a
common protocol for the backup architecture. Data can be backed up using NDMP regardless of
the operating system or platform. NDMP backs up and restores data without losing the data
integrity and file system structure (with respect to different rights and permission in different file
systems). Due to its flexibility, it is no longer necessary to transport data through the application
server, which reduces the load on the application server and improves the backup speed. NDMP
optimizes backup and restore by leveraging the high-speed connection between the backup
devices and the NAS head. In NDMP, backup data is sent directly from the NAS head to the
backup device, whereas metadata is sent to the backup server.
19
The key components of an NDMP infrastructure are NDMP client and NDMP server. NDMP client is
the NDMP enabled backup software installed as add-on software on backup server.
The NDMP server has two components: data server and media server. The data server is a
component on a NAS system that has access to the file systems containing the data to be backed
up. The media server is a component on a NAS system that has access to the backup device.
The backup operation occurs as follows:
1. Backup server uses NDMP client and instructs the NAS head to start the backup
2. The NAS head uses its data server to read the data from the storage
3. The NAS head then uses its media server to send the data read by the data server to the
backup device
In this backup operation, NDMP uses the production network only to transfer the metadata. The
actual backup data is either directly transferred to backup device (NDMP 2-way) or through
private backup network (NDMP 3-way), by the NAS head.
20
NDMP 2-way (Direct NDMP method): In this method, the backup server uses NDMP over the
LAN to instruct the NAS head to start the backup. The data to be backed up from the storage is
sent directly to the backup device. In this model, network traffic is minimized on the production
network by isolating backup data movement from the NAS head to a locally attached backup
device. During the backup, metadata is transferred via NDMP over the LAN to the backup server.
During a restore operation, the backup server uses NDMP over the LAN to instruct the
NAS to start restoring files. Data is restored from the locally attached backup device.
NDMP 3-way (Remote NDMP method): In this method, the backup server uses NDMP
over the LAN to instruct the NAS head (A) to start backing up data to the backup device
attached to NAS head (B). These NAS devices can be connected over a private backup
network to reduce the impact on the production LAN network. During the backup, the
metadata is sent via NDMP by the NAS head (A) to the backup server over the
production LAN network. In the figure shown on the slide, NAS head (A) performs the role of
data server and other NAS head performs the role of media server. NDMP 3-way is useful when
there are limited backup devices in the environment. It enables the NAS head to control the
backup device and share it with other NAS heads by receiving backup data through NDMP.
21
This backup approach backs up data directly from primary storage system to a backup target
without requiring additional backup software. Typically, an agent runs on the application servers
that control the backup process. This agent stores configuration data for mapping the LUNs on the
primary storage system to the backup device in order to orchestrate backup (the transfer of
changed blocks and creation of backup images) and recovery operations. This backup information
(metadata) is stored in a catalog which is local to the application server. When a backup is
triggered through the agent running on application server, the application momentarily pauses
simply to mark the point in time for that backup. The data blocks that have changed since the last
backup will be sent across the network to the backup device. The direct movement from primary
storage to backup device eliminates the LAN impact by isolating all backup traffic to the SAN. This
approach eliminates backup impact on application servers and provides faster backup and
recovery to meet the application protection SLAs.
For data recovery, the backup administrator triggers recovery operation; then the primary storage
reads the backup image from the backup device. The primary storage replaces production LUN
with the recovered copy.
22
This lesson covered agent-based backup and image-based backup approach. This lesson also
covered NDMP-based backup for NAS environment and direct backup from primary storage to
backup device.
23
This lesson covers drivers for deduplication and factors affecting deduplication ratio. This lesson
also covers file-level and sub-file level deduplication. Further this lesson covers global
deduplication and deduplication in primary storage.
24
With the growth of data and 24x7 service availability requirements, organizations are facing
challenges in protecting their data. Typically, a lot of redundant data is backed up that
significantly increases the backup window size and also results in unnecessary consumption of
resources, such as backup storage space and network bandwidth. There are also requirements to
preserve data for longer periods whether driven by the need of consumers or legal and
regulatory concerns. Backing up large amount of duplicate data at the remote site for DR purpose
is also very cumbersome and requires lots of bandwidth. Data deduplication provides the solution
for organizations to overcome these challenges in a backup environment.
25
Deduplication is the process of detecting and identifying the unique data segments (chunk) within
a given set of data to eliminate redundancy. The use of deduplication techniques significantly
reduces the amount of data to be backed up. Data deduplication operates by segmenting a
dataset into blocks and identifying redundant data and writing the unique blocks to a backup
target. To identify redundant blocks, the data deduplication system creates a hash value or digital
signaturelike a fingerprintfor each data block and an index of the signatures for a given
repository. The index provides the reference list to determine whether blocks already exist in a
repository. When the data deduplication system sees a block it has processed before, instead of
storing the block again, it inserts a pointer to the original block in the repository. It is important
to note that the data deduplication can be performed in backup as well as in production
environment. In production environment, the deduplication is implemented at primary storage
systems to eliminate redundant data in the production volume.
The effectiveness of data deduplication is expressed as a deduplication ratio, denoting the ratio of
data before deduplication to the amount of data after deduplication. This ratio is typically
depicted as ratio:1 or ratio X, (10:1 or 10 X). For example, if 200 GB of data

consumes 20 GB of storage capacity after data deduplication, the space reduction ratio is
10:1. Every data deduplication vendor claims that their product offers a certain ratio of data
reduction. However, the actual data deduplication ratio varies, based on many factors. These
factors are discussed next.
26
Data deduplication performance (or ratio) is tied to the following factors:
Retention period: Thisistheperiodoftimethatdefineshowlongthebackupcopiesareretained.

The longer the retention, the greater is the chance of identical data existence in the backup set
which would increase the deduplication ratio and storage space savings.
Frequency of full backup: As more full backups are performed, it increases the amount of
same data being repeatedly backed up. So, it results in high deduplication ratio.
Change rate: This is the rate at which the data received from the backup application changes
from backup to backup. Client data with a few changes between backups produces higher
deduplication ratios.
Data type: Backups of user data such as text documents, PowerPoint presentations,
spreadsheets, and e-mails are known to contain redundant data and are good deduplication
candidates. Other data such as audio, video, and scanned images are highly unique and
typically do not yield good deduplication ratio.
Deduplication method: Deduplication method also determines the effective deduplication

ratio. Variablelength,subfilededuplication(discussedlater)discoverthehighestamount
ofdeduplicationofdata.
27
The level at which data is identified as duplicate affects the amount of redundancy or
commonality. The operational levels of deduplication include file-level deduplication and sub-file
deduplication. File-level deduplication (also called single instance storage) detects and removes
redundant copies of identical files in a backup environment. Only one copy of the file is stored;
the subsequent copies are replaced with a pointer to the original file. By removing all of the
subsequent copies of a file, a significant amount of space savings can be achieved. File-level
deduplication is simple but does not address the problem of duplicate content inside the files. A
change in any part of a file also results in classifying that as a new file and saving it as a separate
copy. For example, two 10-MB presentations with a difference in just the title page are not
considered as duplicate files, and each file is stored separately.
Sub-file deduplication breaks the file into smaller blocks and then uses a standard hash algorithm
to detect redundant data within and across the file. As a result, sub-file deduplication eliminates
duplicate data across files. There are two forms of sub-file deduplication, fixed-length and
variable-length. The fixed-length block deduplication divides the files into fixed length blocks and
uses a hash algorithm to find duplicate data. Although simple in design, the fixed-length block
may miss opportunities to discover redundant data because the block boundaries of similar data
may be different. For example, the addition of a persons name to a documents title page may
shift the whole document, and make all blocks appear to have changed, causing the failure of the
deduplication method to detect equivalencies. In variable-length block deduplication, if there is a
change in the block, then the boundary for that block only is adjusted, leaving the remaining
blocks unchanged. Consequently, more data is identified as common data, and there is less
backup data to store as only the unique data is backed up. Variable-length block deduplication
yields a greater granularity in identifying duplicate data, improving upon the limitations of filelevel and fixed-length block level deduplication.
28
Source-based data deduplication eliminates redundant data at the source (backup clients) before
it transmits to the backup device. The deduplication system consists of multiple backup clients
and deduplication server. The deduplication agent is installed in the backup client to perform
deduplication. The deduplication server maintains a hash index of the deduplicated data.
The deduplication agent running on the clients checks each file for duplicate content. It creates
the hash value for each chunk of the file and checks the hash value with the deduplication server
whether the hash is present on the server. If there is no match on the server, the client will send
the hash and the corresponding chunk to the deduplication server to store the backup data. If the
chunk is already been backed up, then the chunk will not be sent to the deduplication server by
the client, which ensures redundant backup data is eliminated at the client. The deduplication
server can be deployed in different ways. The deduplication server software can be installed on a
general purpose server or on VMs that access the backup target available in the environment (as
shown in the figure). Some vendors offer deduplication server along with backup device as an
appliance. The deduplication server would support encryption for secure backup data transmission
and would also support replication for disaster recovery purpose.
Source-based deduplication reduces the amount of data that is transmitted over a network from
the source to the backup device, thus requiring less network bandwidth. There is also a
substantial reduction in the capacity required to store the backup data. Backing up only unique
data from the backup client reduces the backup window. However, a deduplication agent running
on the client may impact the backup performance, especially when a large amount of data needs
to be backed up. When image-level backup is implemented, the backup workload is moved to a
proxy server. The deduplication agent is installed on the proxy server to perform deduplication,
without impacting the VMs running applications. Organization can implement source-based
deduplication when performing remote office branch office (ROBO) backup to their centralized
data center. Cloud service providers can also implement source-based deduplication when
performing backup (backup as a service) from consumers location to their location.
29
Target-based data deduplication occurs at the backup device or at the backup appliance, which
offloads the backup client from the deduplication process. The figure on the slide illustrates
target-based data deduplication. In this case, the backup client sends the data to the backup
device and the data is deduplicated at the backup target, either immediately (inline) or at a
scheduled time (post-process).
Inline deduplication performs deduplication on the backup data before it is stored on the backup
device. With inline data deduplication, the incoming backup stream is divided into smaller chunks,
and then compared to data that has already been deduplicated. The inline deduplication method
requires less storage space than the post process approach. However, inline deduplication may
slow down the overall backup process. Some vendors inline deduplication systems leverage the
continued advancement of CPU technology to increase the performance of the inline deduplication
by minimizing disk accesses required to deduplicate data. Such inline deduplication systems
identify duplicate data segments in memory, which minimizes the disk usage.
In post-processing deduplication, the backup data is first stored to the disk in its native backup
format and deduplicated after the backup is complete. In this approach, the deduplication process
is separated from the backup process and the deduplication happens outside the backup window.
However, the full backup data set is transmitted across the network to the storage target before
the redundancies are eliminated. So, this approach requires adequate storage capacity and
network bandwidth to accommodate the full backup data set. Organizations can consider
implementing target-based deduplication when their backup application does not have built-in
deduplication capabilities.
In some implementations, part of the deduplication functionality is moved to the backup host or
backup server. This reduces the burden on the target backup device for performing deduplication
and improves the overall backup performance.
30
In a global data deduplication, a single hash is shared among the appliances (nodes) to ensure
that the data is backed up only once across the backup environment. Global data deduplication
provides more effective data deduplication and increases the data deduplication ratio. Users with
large amounts of backup data across multiple locations are most benefited from this approach.
Global deduplication provides the following benefits:
Creates smaller storage footprints and reduces storage costs
Decreases the network bandwidth requirements for data replication
Eliminates data silos in a backup environment
Simplifies and centralizes the management of deduplication appliances
31
Today, organizations usually deploy primary storage system for their production environment in
order to meet the required service levels. The costs of these storage resources are very
expensive. So, it is important for organizations to effectively utilize and manage these storage
resources. Typically a lot of duplicate data is found in the production environment that
unnecessarily consumes more storage resources that leads to more total cost of ownership (TCO).
To avoid this situation, now most of the primary storage system (block-based storage and NAS)
supports deduplication technology to eliminate the duplicate data. This slide focuses on blockbased storage system.
The block-based storage system processes the data in blocks as it enters the storage controller.
All incoming data writes are chunked into blocks, and each block is fingerprinted (hash value)
based on the data content within the block. The fingerprinting methodology provides a
uniform distribution of values. Even a single bit of difference between any two blocks
results in completely uncorrelated fingerprint values. An advantage of fingerprinting each
block is that before a block is written, its fingerprint is compared to existing blocks in the
storage system. If the block already exists in the system, the data will not be written to
the disk. By eliminating the redundant data on the primary storage, the organization
could save cost on storage. Running deduplication consumes resources in the primary
storage and may impact the performance of the storage system.
32
This lesson covered drivers for deduplication and factors affecting deduplication ratio. This lesson
also covered file-level and sub-file level deduplication. Further this lesson covered global
deduplication and deduplication in primary storage.
33
This lesson covers cloud-based backup and backup service deployment options. This lesson also
covers mobile device backup and its challenges.
34
Data is important for businesses of all sizes. Organizations need to regularly back up data to avoid
losses, stay compliant, and preserve data integrity. IT organizations today are dealing with the
explosion of data, particularly with the development of third platform technologies. Data explosion
poses the challenge of data backup and quick data restore. It strains the backup windows, IT
budget, and IT management. The growth and complexity of the data environment, added with
proliferation of virtual machines and mobile devices constantly outpaces the existing data backup
plans. Deployment of a new backup solution takes weeks of planning, justification, procurement,
and setup. However, technology and data protection requirements change quickly. Enterprises
must also comply with regulatory and litigation requirements. These challenges can be addressed
with the emergence of cloud-based backup (backup as a service).
35
Backup as a service enables organizations to procure backup services on-demand in the cloud.
The backup service is offered by a service provider to consumers. Organizations can build their
own cloud infrastructure and provide backup services on demand to their employees/users. Some
organizations prefer hybrid cloud option for their backup strategy, keeping a local backup copy in
their private cloud and using public cloud for keeping their remote copy for DR purpose. For
providing backup as a service, the organizations and service providers should have necessary
backup technologies in place in order to meet the required service levels.
Backup as a service enables individual consumers or organizations to reduce their backup
management overhead. It also enables the individual consumer/user to perform backup and
recovery anytime, from anywhere, using a network connection. Consumers do not need to invest
in capital equipment in order to implement and manage their backup infrastructure. These
infrastructure resources are rented without obtaining ownership of the resources. Based on the
consumer demand, backups can be scheduled and infrastructure resources can be allocated with a
metering service. This will help to monitor and report resource consumption. Many organizations
remote and branch offices have limited or no backup in place. Mobile workers represent a
particular risk because of the increased possibility of lost or stolen devices. Backing up to cloud
ensures regular and automated backup of data. Cloud computing gives consumers the flexibility
to select a backup technology, based on their requirement, and quickly move to a different
technology when their backup requirement changes.
Data can be restored from the cloud using two methods namely web-based restore and mediabased restore. In web-based restore, the requested data is gathered and sent to the server,
running cloud backup agent. The agent software restores data on the server. This method is
considered if sufficient bandwidth is available. If a large amount of data needs to be restored and
sufficient bandwidth is not available, then the consumer may request for data restoration using
backup media such as DVD or disk drives. In this option, the service provider gathers the data to
restore, stores data to a set of backup media, and ships it to the consumer.
36
There are three common backup service deployment options in a cloud-based backup.
Local backup service (managed backup service): This option is suitable when a cloud
service provider is already providing some form of cloud services (example: compute services,
SaaS) to the consumers. The service provider may choose to offer backup services to the
consumers, helping protect consumers data that is being hosted in the cloud. In this approach,
the backup operation is completely managed by the service provider.
Remote backup service: In this option, consumers do not perform any backup at their local
site. Instead, their data is transferred over a network to a backup infrastructure managed by
the cloud service provider. To perform backup to the cloud, typically the cloud backup agent
software is installed on the servers that need to be backed up. After installation, this software
establishes a connection between the server and the cloud where the data will be stored. The
backup data transferred between the server and the cloud is typically encrypted to make the
data unreadable to an unauthorized person or system. Deduplication can also be implemented
to reduce the amount of data to be sent over the network (bandwidth reduction) and reduce
the cost of backup storage.
Replicated backup service: This is an option where a consumer performs backup at their
local site but does not want to either own or manage or incur the expense of a remote site for
disaster recovery purposes. For such consumers, they choose replicated backup service, where
the backup data in their site is replicated to the cloud (remote disaster recovery site).
Note: Cloud-to-Cloud Backup

It allows consumers to backup cloud-hosted applications data (SaaS applications) to other cloud.
37
The proliferation of mobile devices, such as smartphone and tablet, along with the growing
adoption of BYOD, creates a new dimension in backup. Employees and users started using these
mobile devices to create and store critical data. The key considerations around these are the
potential loss of corporate data if the device is lost or stolen. Hence, it is important to backup and
restore the data on these devices. Typically, organizations are subject to compliance regulations
that require IT to back up data wherever it exists. So, IT will have to assume the responsibility of
adding smartphones and tablets to their list of endpoint devices that must be backed up.
Typically, to start performing backup for mobile devices, backup client application needs to be
installed on the mobile device. The backup client periodically backs up the data either to the
enterprise data center or to the cloud. Organizations generally consider implementing
deduplication, compression, and incremental backup to minimize the network bandwidth
requirement and reduce the backup storage cost. For security reasons, data that are backed up
from the mobile devices are encrypted.
Note: Mobile Device Management (MDM)

MDM solution is used by an IT department to monitor, manage, protect (backup), and secure
(remote password locks, full data wipes) employees' mobile devices that are deployed across
multiple mobile service providers and across multiple mobile operating systems being used in the
organization. The Gartner research firm defines mobile device management as "a range of
products and services that enables organizations to deploy and support corporate applications to
mobile devices, such as smartphones and tablets, possibly for personal use enforcing policies
and maintaining the desired level of IT control across multiple platforms. MDM software also
reduces the overhead on IT administration associated with deploying and updating applications on
mobile devices.
38
Backing up mobile devices can be challenging due to intermittent (and sometimes poor)
connectivity. A key challenge for backing up mobile devices is that they are not always connected
to a corporate network. So, backups occur over the Internet, which can be troublesome. Another
challenge is that smartphone and tablet operating systems have security features built in them
which limit the access to the data stored on the devices. Some of the mobile devices, particularly
tablet and smartphone, may not allow traditional backup applications to access data. To overcome
these challenges, organizations must adopt new policies, strategies, and techniques to protect the
data residing on mobile devices.
Note: File sync-and-share application

Since capacity is limited on mobile devices, many users store data remotely rather than on the
device itself. Storing data remotely is the best way for users to make their data available across
all devices desktops, laptops, tablets and smartphones. This decreases the need to back up
mobile devices and laptops, because data stored remotely can be backed up using traditional
methods. One option is to use enterprise file sync-and-share software. File sync-and-share
applications automatically establish two-way synchronization between the device and a

designated network location, either enterprise data center or cloud. Any data a user
creates or modifies on the mobile device is automatically synchronized with on-premises
servers or cloud, where it is then backed up using traditional methods. File sync and share also
improves employee productivity by allowing users to access data from any device, anywhere, at
any time.
39
This lesson covered cloud-based backup and backup service deployment options. This lesson also
covered mobile device backup and its challenges.
40
This lesson covers key requirements for data archiving solutions and data archiving architecture.
This lesson also covers content addressed storage. Further, this lesson covers cloud-based
archiving and its considerations.
41
In the information lifecycle, data is actively created, accessed, and changed. As data ages, it is
less likely to be changed and eventually becomes fixed but continues to be accessed by
applications and users. This data is called fixed content. Assets such as X-rays, MRIs, CAD/CAM
designs, surveillance video, MP3s, and financial documents are just a few examples of fixed data
that is growing at over 90% annually. Data archiving is the process of moving data (fixed
content) that is no longer actively accessed to a separate low cost archival storage tier for long
term retention and future reference. Data archive is a storage repository that is used to store
these data. Organizations set their own policies for qualifying data to be moved into archives.
These policy settings are used to automate the process of identifying and moving the appropriate
data into the archive system. Organizations implement archiving processes and technologies to
reduce primary storage cost. With archiving, the capacity on expensive primary storage can be
reclaimed by moving infrequently-accessed data to lower-cost archive tier. Archiving fixed content
before taking backup helps to reduce the backup window and backup storage acquisition costs.
Government regulations and legal/contractual obligations mandate organizations to retain their
data for an extended period of time. The key to determine how long to retain an organization's
archives is to understand which regulations apply to the particular industry and which retention
rules apply to that regulation. For instance, all publicly traded companies are subject to the
Sarbanes-Oxley (SOX) Act which defines e-mail retention requirements, among other things
related to data storage and security. Archiving helps organizations to adhere to these
compliances. Archiving can help organizations use growing volumes of information in potentially
new and unanticipated ways. For example, new product innovation can be fostered if engineers
can access archived project materials such as designs, test results, and requirement documents.
In addition to meeting governance and compliance requirements, organizations retain data for
business intelligence and competitive advantage. Both active and archived information can help
data scientists drive new innovations or help to improve current business processes.
42
Archiving solutions should meet an organizations compliance requirements through automated

policy-driven data retention and deletion. It should provide the features such as scalability,
authenticity, immutability, availability, and security. The archiving solution should be able to
authenticate the creation and integrity of files in the archive storage. Long-term reliability is key
for archiving solutions because failure of an archive system could have disastrous consequences.
These systems hold critical documents, and any failure could have compliance, legal, and
business consequences.
In order to manage the large volumes of data, an important technical requirement of an archiving
solution is single instance storage (storage system that enables to keep one copy of content
and eliminates duplicate data). The archiving solution should support variety of online storage
options such as disk-based storage and cloud-based storage. Another key factor is to provide
support for variety of data types including e-mails, databases, pdfs, images, audios, videos,
binary files, and HTML files.
Powerful indexing and searching capability on archiving solutions speeds up the data retrieval. An
effective archival system needs to support complex searches of content within documents.
Archiving solutions should enable electronic discovery (eDiscovery) and sharing of data for
litigation purposes in a timely and compliant manner. Reporting capabilities are required to
process huge volumes of data and deliver customized reports for compliance requirements.
43
Archiving solution architecture consists of three key components: archiving agent, archiving
server, and archiving storage device. An archiving agent is a software installed on the application
servers (example: File servers and E-mail servers). The agent is responsible for scanning the data
and archiving it, based on the policy defined on the archiving server (policy engine). After the
data is identified for archiving, the data will be moved to the archiving storage device. From a
client perspective, this movement is completely transparent. Then, the original data on the
primary storage is replaced with a stub file. The stub file contains the address of the archived
data. The size of this file is small and significantly saves space on primary storage. When the
client is trying to access the files from the application servers, the stub file is used to retrieve the
file from the archive storage device. An archiving server is software installed on a server that
enables administrators to configure the policies for archiving data. Policies can be defined based
on file size, file type, or creation/modification/access time. Once the data is identified for
archiving, the archiving server creates the index for the data to be moved. By utilizing the index,
users may also search and retrieve their data with the web search tool.
Note: Converge Backup and Archive Storage

Backup is driven by the need for recoverability and disaster protection while archiving is driven by
the need for improved efficiency and to address compliance challenges. Real cost savings can be
realized by adopting a strategy for the physical storage of both backup and archiving workloads.
To accomplish this, a common storage target must be able to handle the throughput and inline
deduplication requirements of backup workloads and secure and long-term retention
requirements of archive workloads. In addition, the storage target should provide built-in
capabilities for network-efficient replication for disaster recovery needs, enterprise features such
as encryption, and allow for easy integration with existing application infrastructure. By leveraging
a common infrastructure for both, organizations can greatly ease the burden of eDiscovery, data
recovery, business continuity, and compliance and achieve these goals in the most cost-efficient
manner.
44
Data integrity, scalability, and protection are the primary requirement for any data archiving
solution. Traditional archival solutionsCD, DVD-ROM, tapedo not provide the required
scalability, availability, security, and performance. Content addressed storage (CAS) is a special
type of object-based storage device purposely built for storing and managing fixed content. CAS
stores user data and its attributes as an object. The stored object is assigned a globally unique
address, known as a content address (CA). This address is derived from the objects binary
representation. This content addressing eliminates the need for applications to understand and
manage the physical location of object on storage system. This content address (digital fingerprint
of the content) not only simplifies the task of managing huge numbers of objects, but also
ensures content authenticity. Data access in CAS may differ from other object-based storage
devices (OSD). In CAS, the application server can access the CAS device only via the CAS API
running on the application server. However, the way CAS stores data may be similar to other OSD
systems.
45
The key features of CAS are as follows:

Content integrity: It provides assurance that the stored content has not been altered. If the
fixed content is altered, CAS generates a new address for the altered content, rather than
overwriting the original fixed content.
Content authenticity: It assures the genuineness of stored content. This is achieved by
generating a unique content address for each object and validating the content address for stored
objects at regular intervals. Content authenticity is assured because the address assigned to each
object is as unique as a fingerprint. Every time an object is read, CAS uses a hashing algorithm to
recalculate the objects content address as a validation step and compares the result to its
original content address. If the object validation fails, CAS rebuilds the object using protection
scheme.
Single-instance storage: CAS uses a unique content address to guarantee the storage of only a
single instance of an object. When a new object is written, the CAS system is polled to see
whether an object is already available with the same content address. If the object is available in
the system, it is not stored; instead, only a pointer to that object is created.
Retention enforcement: Protecting and retaining objects is a core requirement of an archive
storage system. After an object is stored in the CAS system and the retention policy is defined,
CAS does not make the object available for deletion until the policy expires.
Scalability: CAS allows the addition of more nodes to the cluster to scale without any
interruption to data access.
(Contd)
46
Location independence: CAS uses a unique content address, rather than directory path names
or URLs, to retrieve data. This makes the physical location of the stored data irrelevant to the
application that requests the data.
Data protection: CAS ensures that the content stored on the CAS system is available even if a
disk or a node fails. CAS provides both local and remote protection to the data objects stored on
it. In the local protection option, data objects are either mirrored or parity protected. In mirror
protection, two copies of the data object are stored on two different nodes in the same cluster.
This decreases the total available capacity by 50 percent. In parity protection, the data object is
split in multiple parts and parity is generated from them. Each part of the data and its parity are
stored on a different node. This method consumes less capacity to protect the stored data, but
takes slightly longer to regenerate the data if corruption of data occurs. In the remote replication
option, data objects are copied to a secondary CAS at the remote location. In this case, the
objects remain accessible from the secondary CAS if the primary CAS system fails.
Performance: CAS stores all objects on disks, which provides faster access to the objects
compared to tapes and optical discs.
Self-healing: CAS automatically detects and repairs corrupted objects and alerts the
administrator about the potential problem. CAS systems can be configured to alert remote
support teams who can diagnose and repair the system remotely.
Audit trails: CAS keeps track of management activities and any access or disposition of data.
Audit trails are mandated by compliance requirements.
47
Healthcare centers typically examine hundreds of patients every day and generate huge volumes
of patient (medical) records. Each record might be composed of one or more images such as MRI,
CT scan, and x-ray that range in size from approximately 15 MB to more than 1 GB. Healthcare
providers are faced with a proliferation of compliance requirements related to patient information
security, operational practices, service delivery procedures, and electronic health record
management. Regulatory compliance requirements and industry standards include HIPAA, SOX,
and other national and state regulations for patient safety. So, it is important for healthcare
centers to securely store and retain the archived records to meet compliance requirements, and
also make the records accessible in real time when it is required. CAS facilitates long-term
storage of medical records and at the same time provides immediate access to data, when
needed. The patient records are stored on primary storage for a specific period of time for
immediate use by the attending physicians. For example, patients records are retained on the
primary storage for 60 days after which they are moved to the CAS system. Even if a patients
record is no longer needed, compliance requirements might stipulate that the records be kept in
the original format for several years.
48
E-mail is an example of an application that benefits most by an archival solution. Typically, a

system administrator configures small mailboxes that store a limited number of e-mails. This is
because large mailboxes with a large number of e-mails can make management difficult, increase
primary storage cost, and degrade system performance. When an e-mail server is configured with
a large number of mailboxes, the system administrator typically configures a quota on each
mailbox to limit its size. Configuring a fixed quota on mailboxes impacts end users. A fixed quota
for a mailbox forces users to delete e-mails as they approach the quota size. End users often
need to access e-mails that are weeks, months, or even years old.
E-mail archiving provides an excellent solution that overcomes the preceding challenges.
Archiving solutions move e-mails that have been identified as candidates for archive from primary
storage to the archive storage device based on a policy for example, e-mails that are 90 days
old should be archived. After the e-mail is archived, it is retained for years, based on the
retention policy. This considerably saves space on primary storage and enables organizations to
meet regulatory requirements. The implementation of an archiving solution gives end users
virtually unlimited mailbox space.
49
In a traditional in-house data archiving model, archiving systems and underlying infrastructure
are deployed and managed within an organizations data center. Due to exponential data growth,
organizations are facing challenges with the increased cost and complexity in their archiving
environment. Often an existing infrastructure is siloed by architecture and policy. The
organizations are looking for new ways to improve the agility and the scalability of their archiving
environments. Cloud computing provides highly scalable and flexible computing that is available
on demand. It empowers self-service requesting through a fully automated request-fulfillment
process in the background. It provides capital cost savings and agility to organizations. With
cloud-based archiving, organizations are required to pay as they use and can scale the usage as
needed. It also enables the organization to access their data from any device and any location.
Typically cloud-based archiving service is designed to classify, index, search, and retrieve
data in a security-rich manner while automating regulatory monitoring and reporting. It
also enables the organizations to consistently enforce the policies for the centralized cloud archive
repository. Hybrid cloud archiving is one step toward the cloud from the traditional in-house
approach. Archived data that may require high-speed access is retained internally while lowerpriority archive data is moved to low-cost, public cloud-based archive storage.
50
Some of the key considerations for cloud-based archiving are as follows:

Service Level Agreement (SLA): In a cloud, the assurance for quality and reliability of services
become an important consideration. As the demands of the consumers vary significantly, it may
not be possible for service provider to meet all the expectations. Therefore, a balance needs to be
made via a negotiation process and documented as service level agreement. The SLAs between
organization and service provider must reflect cost, availability, performance, data privacy, data
ownership, retention and disposition policies, regulatory requirements, search and data access,
compensation for data loss and misuse, and penalty or recourse (for not meeting the agreement)
as parameters of the agreement. The SLA should also include clauses related to the termination
of the service by both the consumer and the provider.
Vendor lock-in: Refers to a situation where a consumer is unable to move readily from the
current provider to another. This condition may result from various causes such as high cost of
migration, significant re-engineering effort requirement for an application migration, lack of open
standards, or restrictions imposed by the current provider. The use of widely accepted open
standard tools, APIs, and file formats can prevent vendor lock-in.
Compliance: A consumer should assess their compliance requirements and convey it to the
provider. An organizations compliance requirements may include internal policies and legal
requirements. Legal requirements may include data privacy laws imposed by different countries.
These legal requirements may also specify geographical locations to store consumer data and
disallow modification or deletion of data during its retention period. The service provider should
meet these compliance requirements for the consumer.
(Contd)
51
Data Security: There are various mechanisms (secure multi-tenancy, encryption, shredding,
access and identity management) that should be deployed by service provider to ensure
security for data stored in the cloud archive. Secured storage multi-tenancy is a key requirement
for any shared IT infrastructure such as cloud. Secure storage multi-tenancy ensures that no
tenant (consumer) can access another tenants data. Encryption is a technique to enable the
confidentiality and integrity of data. Ensure that the data should be encrypted both at rest and in
flight. Even when deleted, the data in the cloud may still be vulnerable to exposure. Data
shredding enables removing all traces of deleted data. Access and identity management is an
administrative process that deals with identifying users of an information system. Additionally,
identity management also controls access to system resources by placing restrictions using user
identities.
Pricing: There are various cloud pricing models that are available such as pay-as-you-go,
subscription-based, and user-based. Consumer should consider various factors and decide which
model is best suited to their needs.
52
This lesson covered the key requirements for data archiving solutions and data archiving
architecture. This lesson also covered content addressed storage. Further, this lesson covered
cloud-based archiving and its considerations.
53
The Concepts in Practice section covers various EMC backup and archive products.
Note:
54
EMC NetWorker is a backup and recovery software, which centralizes, automates, and
accelerates data backup and recovery operations. Following are the key features of NetWorker:
Supports heterogeneous platforms such as Windows, UNIX, Linux, and also virtual
environments.
Supports different backup targets tapes, disks, Data Domain purpose-built backup appliances
and virtual tapes.
Supports multiplexing (or multi-streaming) of data.
Provides both source-based and target-based deduplication capabilities by integrating with EMC
Avamar and EMC Data Domain respectively.
The cloud-backup option in NetWorker enables backing up data to public cloud configurations
EMC Avamar is a disk-based backup and recovery solution that provides inherent source-based
data deduplication. With its unique global data deduplication feature, Avamar differs from
traditional backup and recovery solutions by identifying and storing only unique sub-file data. EMC
Avamar provides a variety of options for backup, including guest OS-level backup and image-level
backup. The three major components of an Avamar system include Avamar server, Avamar
backup clients, and Avamar administrator. Avamar server provides the essential processes and
services required for client access and remote system administration. The Avamar client software
runs on each compute system that is being backed up. Avamar administrator is a user
management console application that is used to remotely administer an Avamar system.
55
EMC Data Domain deduplication storage system is a target-based data deduplication solution.
Using high-speed, inline deduplication technology, the Data Domain system provides a storage
footprint that is significantly smaller on average than that of the original data set. Data Domain
Data Invulnerability Architecture provides defense against data integrity issues. EMC Data Domain
Boost software significantly increases backup performance by distributing the parts of the
deduplication process to the backup server. With Data Domain Boost, only unique, compressed
data segments are sent to a Data Domain system. For archiving and compliance solutions, Data
Domain systems allow customers to cost-effectively archive non-changing data while keeping it
online for fast reliable access and recovery. EMC Data Domain Extended Retention is a solution for
long-term retention of backup data. It is designed with an internal tiering approach to enable
cost-effective, long-term retention of data on disk by implementing deduplication technology.
Data Domain provides secure multi-tenancy that enables data protection-as-a-service for large
enterprises and service providers who are looking to offer services based on Data Domain in a
private or public cloud. With secure multi-tenancy, a Data Domain system will logically isolate
tenant data, ensuring that each tenants data is only visible and accessible to them.
EMC ProtectPoint is a data protection solution which enables direct backup from primary storage
(EMC VMAX) to Data Domain system. It eliminates the backup impact on application server.
Unlike a traditional backup, ProtectPoint will only pause the application to mark the point in time
for an application consistent backup, and then the application can quickly return to normal
operations. By leveraging the primary storage change block tracking technology, only unique
blocks are sent from primary storage, but are stored as full independent backups on Data Domain
system.
56
EMC Mozy is a solution that provides a secure cloud-based online backup and recovery through
Software as a Service. Mozy provides protection against risks like file corruption, unintended
deletion, and hardware failure for compute and mobile systems. It is built on highly scalable and
available back-end storage architecture. Mozys web-based management console enables
consumers to specify the data to be backed up and when to perform backups. Backups are
encrypted and may be automatic or scheduled periodically. Mozy has three main products:
MozyHome, MozyPro, and MozyEnterprise. MozyHome is for the individual consumer, MozyPro is
for small businesses, and MozyEnterprise is for larger organizations. Mozy services are available
at a monthly subscription fee. Mozy does not require consumers to purchase any new hardware
and requires minimal IT resources to manage.
Spanning: Spanning, an EMC company and a leading provider of backup and recovery for SaaS
applications, helps organizations to protect and manage their information in the cloud. Spanning
solutions provide powerful, enterprise-class data protection for Google Apps, Salesforce, and
Office 365. Spanning backup is the most trusted cloud-to-cloud backup solution for organizations.
It allows administrators and end users to search for, restore, and export data.
57
EMC Centera is a simple, affordable, and secure repository for information archiving. EMC
Centera is designed and optimized specifically to deal with the storage and retrieval of fixed
content by meeting performance, compliance, and regulatory requirements. Compared to
traditional archive storage, EMC Centera provides faster record retrieval, Single instance storage
(SIS), guaranteed content authenticity, self-healing, and support for numerous industry and
regulatory standards. EMC Centera virtual archive enables the aggregation of a set of EMC
Centera clusters. This set of clusters forms a virtual, tamper-proof archive with the capacity of all
clusters aggregated and available to applications. Capacity and processing power can be added
beyond the existing boundaries of a single EMC Centera system.
EMC SourceOne: The EMC SourceOne Archiving family of products helps organizations to reduce
the burden of aging email, files, and Microsoft SharePoint content by archiving them to the
appropriate storage tiers. EMC SourceOne provides the ability to enable good information
governance and compliance by managing e-mail, files, and SharePoint content as a business
record and consistently enforce retention/disposition policies. The EMC SourceOne family of
products includes:
EMC SourceOne Email Management for archiving e-mail messages and other items.
EMC SourceOne for Microsoft SharePoint for archiving SharePoint site content.
EMC SourceOne for File Systems for archiving files from file servers.
EMC SourceOne Discovery Manager for discovering, collecting, preserving, reviewing, and
exporting relevant content.
EMC SourceOne Supervisor for monitoring corporate policy compliance.
58
EMC InfoArchive: A unified enterprise archiving platform that stores related structured
data and unstructured content in a single, consolidated repository. This product enables
corporations to preserve the value of enterprise information in a single, easily accessible,
unified archive. It provides a long-term, compliant archive, meeting retention
requirements and ensuring auditability, defensibility, and easy accessibility when needed.
It provides the ability to audit and preserve data and content to meet a variety of
regulatory and governance mandates. It stores information in an open, industry-standard
format for long term retention and easy access. It supports both active archiving and
application decommissioning.
VMware vSphere Data Protection Advanced: vSphere Data Protection Advanced is a

backup and recovery solution designed for vSphere environments and supported by EMC
Avamar. It provides agentless, image-level backups to disk, as well as guest-level,
application-consistent protection. vSphere Data Protection Advanced provides networkefficient, encrypted replication to replicate backups to one or more DR sites.
59
This module covered backup architecture and various backup targets. This module also covered
various backup methods and data deduplication. This module further covered cloud-based and
mobile device backup. Finally, this module covered data archive.
60
Scenario:
A major multinational bank runs business-critical applications in a virtualized data center.
It has over a million of customers and multiple remote/branch offices (ROBO) across
different geographic locations. The bank currently uses tape as their primary backup
storage media for backing up virtual machines (VMs) and application data. It uses an
agent-based backup solution for backing up data. It currently performs a full backup
every Sunday, and an incremental backup on other days. It also has a file-sharing
environment in which multiple NAS systems serve all the users. During NAS backup, the
data is backed up from application servers to backup device. Approximately 25% of data
in the production environment is inactive data (fixed content).
Challenges:
The backup operations consume resources on the compute systems that are running multiple
VMs. This is significantly impacting the applications deployed on the VMs. During NAS backup, the
application servers are impacted because the data is backed up from these servers to the backup
device.
The organizations backup environment has a huge amount of redundant data. This has led to an
increase in the infrastructure cost and is also impacting the backup window (backup takes more
time to complete).
(Contd)
Recovering data or VMs also takes more time and affects the SLAs. The branch offices also have
limited IT resources for managing backup. Backing up data from branch offices to a centralized
data center was restricted due to the time and cost involved in sending huge volumes of data
over the WAN.
The organization incurs a huge investment and operational expense in managing an offsite
backup infrastructure at remote site for disaster recovery purpose.
The slide listed the requirements of the organization.
The slide listed the requirements of the organization.
Deliverables:
Recommend the solutions that will meet the organizations requirements.
1. Implement disk-based backup solution to improve the backup and recovery performance.
2. Implement deduplication solution to eliminate the redundant copies of data.
3. Disk-based backup solutions along with source-based deduplication eliminate the challenges
associated with centrally backing up remote office data. Deduplication considerably reduces
the required network bandwidth and enables remote-office data backup using the existing
network.
4. Implement image-based backup that helps to offload backup operation from VMs to a proxy
server. No backup agent is required inside the VM to backup.
5. Deploy NDMP-based backup solution for NAS environment. In NDMP-based backup, data is
sent directly from the NAS head to the backup device without impacting application servers.
6. Organization can implement data archiving solutions that archive fixed content from the
production environment and also reduce the amount of data to be backed up.
7. Organization can choose backup as a service to replicate the offsite backup copy to the cloud.
This saves CAPEX and reduces the management overhead to the organization.
This module focuses on the primary uses of replication and its characteristics. This module also
focuses on various replication technologies implemented at compute, storage, and network.
Finally, this module focuses on data migration solutions and disaster recovery as a service.
Module 14: Replication
This lesson covers the primary uses of replication and its characteristics. This lesson also covers
various methods to ensure consistency during replication. Finally, this lesson covers different
types of replication.
In todays business situation, it is imperative for an organization to protect mission-critical data

across physical, virtual, and cloud environments to minimize the risk of business disruption. If a
disaster occurs, fast data restore and restart capabilities are essential to ensure business
continuity (BC). Replication is one of the ways to ensure BC. It is the process of creating an exact
copy (replica) of data. These replicas are used to restore and restart operations if data loss
occurs. For example, if a production VM goes down, then the replica VM can be used to restart the
production operations with minimum disruption. The replicas can also be assigned to other
compute systems to perform various business operations, such as backup, reporting, and testing.
Based on availability requirements, data can be replicated to one or more locations. For
example, data can be replicated within a data center, between data centers, from a data
center to a cloud, or between clouds. In a software-defined data center environment,
organizations have policy-based automation of replication process. For example, this policy-based
automation enables to decide the number of replicas to be created and the location where the
data should be resident (to comply with regulatory requirements) based on the criticality of the
application or the class of a service (Gold, Silver, Bronze). Typically, organizations providing cloud
services have multiple data centers across different locations and they may provide options to
consumers for choosing the location to which the data is to be replicated.
In a replication environment, a compute system accessing the production data from one or more
LUNs on storage system(s) is called a production compute system. These LUNs are known as
source LUNs, production LUNs, or simply the source. A LUN on which the production data is
replicated to is called the target LUN or simply the target or replica.
Replicas are created for various purposes, including the following:
Alternative source for backup: Under normal backup operations, data is read from the
production LUNs and written to the backup device. This places an additional burden on the
production infrastructure because production LUNs are simultaneously involved in production
operations and servicing data for backup operations. To avoid this situation, a replica can be
created from production LUN and it can be used as a source to perform backup operations. This
alleviates the backup I/O workload on the production LUNs.
Fast recovery and restart: For critical applications, replicas can be taken at short, regular
intervals. This allows easy and fast recovery from data loss. If a complete failure of the source
(production) LUN occurs, the replication solution enables one to restart the production
operation on the replica to reduce the RTO.
Decision-support activities, such as reporting: Running reports using the data on the
replicas greatly reduces the I/O burden placed on the production device.
Testing platform: Replicas are also used for testing new applications or upgrades. For
example, an organization may use the replica to test the production application upgrade; if the
test is successful, the upgrade may be implemented on the production environment.
Data migration: Another use for a replica is data migration. Data migrations are performed
for various reasons, such as migrating from a smaller capacity LUN to one of a larger capacity
for newer versions of the application.
A replica should have the following characteristics:
Recoverability: Enables restoration of data from the replicas to the source if data loss occurs.
Restartability: Enables restarting business operations using the replicas.
Consistency: Replica must be consistent with the source so that it is usable for both recovery
and restart operations. For example, if a service running on a primary data center is to failover
to remote site due to disaster, there must be a consistent replica available at that site. So,
ensuring consistency is the primary requirement for all the replication technologies.
Replicas can either be point-in-time (PIT) or continuous and the choice of replica ties back into
RPO.
PIT replica: The data on the replica is an identical image of the production at some specific
timestamp. For example, a replica of a file system is created at 4:00 PM on Monday. This
replica would then be referred to as the Monday 4:00 PM PIT copy. The RPO will map to the
time when the PIT was created to the time when any kind of failure on the production
occurred. If there is a failure on the production at 8:00 PM and there is a 4:00 PM PIT
available, the RPO would be 4 hours (8 4 = 4). To minimize RPO, take periodic PITs.
Continuous replica: The data on the replica is in-sync with the production data at all times.
The objective with any continuous replication is to reduce the RPO to zero or near-zero.
Consistency is a primary requirement to ensure the usability of replica device. In case of file
systems (FS), consistency can be achieved either by taking FS offline i.e. by un-mounting FS or
by keeping FS online by flushing compute system buffers before creating replica. File systems
buffer the data in the compute system memory to improve the application response time.
Compute system memory buffers must be flushed to disks to ensure data consistency on the
replica, prior to its creation. If the memory buffers are not flushed to the disk, the data on the
replica will not contain the information that was buffered in the compute system.
Similarly in case of databases, consistency can be achieved either by taking database offline for
creating consistent replica or by keeping online. If the database is online, it is available for I/O
operations, and transactions to the database update the data continuously. When a database is
replicated while it is online, changes made to the database at this time must be applied to the
replica to make it consistent. A consistent replica of an online database is created by using the
dependent write I/O principle or by holding I/Os momentarily to the source before creating the
replica. A dependent write I/O principle is inherent in many applications and database
management systems (DBMS) to ensure consistency. According to this principle, a write I/O is not
issued by an application until a prior related write I/O has completed.
Replication can be classified into two major categories: local and remote replication. Local
replication refers to replicating data within the same storage system or the same data center.
Local replicas help to restore the data in the event of data loss or enable restarting the application
immediately to ensure BC. Local replication can be implemented at compute, storage, and
network. Remote replication refers to replicating data to remote locations (locations can be
geographically dispersed). Remote replication helps organizations to mitigate the risks associated
with regional outages resulting from natural or human-made disasters. During disasters, the
services can be moved (failover) to a remote location to ensure continuous business operation.
Remote replication also allows organizations to replicate their data to the cloud for DR purpose. In
a remote replication, data can be synchronously or asynchronously replicated. Like local
replication, the remote replication can also be implemented at compute, storage, and network.
Organizations could choose the right replication solutions based on their business requirements.
This lesson covered primary uses of replication and its characteristics. This lesson also covered
various methods to ensure consistency during replication. Finally, this lesson covered types of
replication namely local and remote replication.
This lesson covers compute-based local replication solutions such as file system snapshot, VM
snapshot, and VM clone. This lesson also covers compute-based remote replication solutions such
as log shipping and VM replication across sites.
The compute-based replication uses compute system resources such as CPU and memory to
perform and manage the replication operation. The key compute-based local and remote
replication techniques are listed on the slide.
10
A snapshot is a virtual copy of a set of files, VM, or LUN as they appear at a specific PIT.
Snapshots can establish recovery points in just a small fraction of a time and can significantly
reduce RPO by supporting more frequent recovery points. If a file is lost or corrupted, it can
typically be restored from the latest snapshot data within a few seconds.
A file system (FS) snapshot creates a copy of a file system at a specific point-in-time, even while
the original file system continues to be updated and used normally. FS snapshot is a pointerbased replica that requires a fraction of the space used by the production FS. It uses the Copy on
First Write (CoFW) principle to create snapshots. When a snapshot is created, bitmap and
blockmap are created in the metadata of the snapshot FS. The bitmap is used to keep track of
blocks that are changed on the production FS after the snapshot creation. The blockmap is used
to indicate the exact address from which the data is to be read when the data is accessed from
the snapshot FS. Immediately after the creation of the FS snapshot, all reads from the snapshot
are actually served by reading the production FS. In a CoFW mechanism, if a write I/O is issued to
the production FS for the first time after the creation of a snapshot, the I/O is held and the
original data of production FS corresponding to that location is moved to the snapshot FS. Then,
the write is allowed to the production FS. The bitmap and blockmap are updated accordingly.
Subsequent writes to the same location will not initiate the CoFW activity. To read from the
snapshot FS, the bitmap is consulted. If the bit is 0, then the read is directed to the production
FS. If the bit is 1, then the block address is obtained from the blockmap and the data is read from
that address on the snapshot FS. Read requests from the production FS work as normal.
Typically read-only snapshots are created to preserve the state of the production FS at some PIT,
but sometimes writeable FS snapshots are also created for some business operations such as
testing and decision support.
11
A VM snapshot preserves the state and data of a VM at a specific PIT. The state includes the VMs
power state (for example, powered-on, powered-off, or suspended). The data includes all of the
files that make up the VM. This includes disks, memory, and other devices, such as virtual
network interface cards. This VM snapshot is useful for quick restore of a VM. For example, an
administrator can create a snapshot of a VM, make changes such as applying patches and
software upgrades to the VM. If anything goes wrong, the administrator can simply restore the
VM to its previous state using the VM snapshot. The hypervisor provides an option to create and
manage multiple snapshots. Taking multiple snapshots provide several restore points for a VM.
While more snapshots will improve the resiliency of the infrastructure, it is important to consider
the storage space they consume.
When a snapshot is created for a VM, a child virtual disk (delta disk file) is created from the base
image or parent virtual disk. The snapshot mechanism prevents the guest operating system from
writing to the base image or parent virtual disk and instead directs all writes to the delta disk file.
Successive snapshots generate a new child virtual disk from the last child virtual disk in the chain.
Snapshots hold only changed blocks. Consider an example in which three snapshots of a VM are
created as shown on the slide. In this example, child virtual disk 1 stores all the changes made to
the parent VM after snapshot 1 is created. Similarly, child virtual disk 2 and child virtual disk 3
store all the changes after snapshot 2 and snapshot 3 are created respectively. When committing
snapshot 3 for the VM, the data on child virtual disk file 1 and 2 are committed prior to
committing data on child virtual disk 3 to the parent virtual disk file. After committing the data,
the child virtual disk 1, child virtual disk 2, and child virtual disk 3 are deleted. However, while
rolling back to the snapshot 1(PIT), child disk file 1 is retained and the snapshots 2 and 3 are
discarded.
Sometimes it may be required to retain a snapshot for longer period, but it must be noted that
larger snapshots take longer time to commit and may impact the performance. Source (parent
VM) must be healthy in order to use snapshot for rollback.
12
A clone is a copy of an existing VM. The existing VM is called the parent of the clone. When the
cloning operation completes, the clone becomes a separate VM. The changes made to a clone do
not affect the parent VM. Changes made to the parent VM do not appear in a clone. A clone's MAC
address is different from those of the parent VM.
In general, installing a guest OS and applications on a VM is a time consuming task. With clones,
administrators can make many copies of a virtual machine from a single installation and
configuration process. For example, in a VDI environment, the administrator can clone a VM for
each new employee, with a suite of preconfigured software applications.
In general, snapshot is used to save the current state of the virtual machine, so that it allows to
revert to that state in case of any error. But clone is used when a copy of a VM is required for
separate use.
There are two types of clone namely full clone and linked clone. A full clone is an independent
copy of a VM that shares nothing with the parent VM. Because a full clone needs to have its own
independent copy of the virtual disks, the cloning process may take a relatively longer time. A
linked clone is made from a snapshot of the parent VM. The snapshot is given a separate network
identity and assigned to the hypervisor to run as an independent VM. However, all files available
on the parent at the moment of the snapshot creation continue to remain available to the linked
clone VM in read-only mode. Ongoing changes (writes) to the virtual disk of the parent do not
affect the linked clone and changes to the virtual disk of the linked clone do not affect the parent.
All the writes by the linked clone are captured in a delta disk.
Note: Instant Clone
Instant Clone enables a running VM to be cloned, such that the new VM is exactly identical to the
original. This enables to get a new, running, booted up VM in less than a second. Moreover, the
forked VM is small from a resource perspective as it shares all its memory with the original. This
capability is very useful in quickly scaling up a large number of VMs to meet the needs of spiking
workload in a cloud environment.
13
Log shipping is a replication solution that allows one to bring a standby database online in case
the primary database is down due to some reasons. In a log shipping technique, the transactions
to the source database are captured in logs, which are periodically transmitted by the source
compute system to the remote compute system. The remote compute system receives the logs
and applies them to the remote database. Prior to the starting of production work and replication
of the log files, all relevant components of the source database are replicated to the remote site.
After this step, the production work is started on the source database. The remote database is
started in a standby mode. All DBMSs switch log files at preconfigured time intervals or when a
log file is filled. This process ensures that the standby database is consistent up to the last
committed log. The log shipping allows to use the existing standard IP network for replicating log
files. The compute-based log shipping requires low network bandwidth because it transmits only
the log files at regular intervals. In log shipping, RPO at the remote site is finite and depends on
the size of the log and the frequency of log switching.
14
Although compute-based replication within a data center is useful, organizations often replicate
virtual machines (VM data and configuration files) between a primary site and a remote site. By
doing this, if one site is down as the result of a disaster, the remote site is available to take over
the operations. The replication is non- intrusive and independent of the guest operating system
within the VM. Hypervisor-based remote replication employs an agent installed on the compute
system. From a conceptual perspective, the agent is responsible for sending changed data from a
running VM at source (production) site to the disk files for the virtual machine at the remote site.
The process begins with initial full synchronization of the source and its replica copy. The time
required to complete this initial synchronization depends primarily on the amount of data to be
replicated and the available network bandwidth. After the initial full synchronization, changes to
the protected VM are tracked and replicated on a regular basis. Their frequency is determined by
the RPO that was configured for the VM. A lower RPO requires more-frequent replication. In a
hypervisor-based replication, even if multiple VMs share the same volume, specific VMs can be
selected for replication, greatly simplifying storage setup and minimizing storage capacity
requirements at the remote site.
Hypervisor-based replication supports both synchronous and asynchronous replication. In
synchronous replication, the write or update to the source VM is replicated and committed to the
target disk file of the VM. This may increase the application response time, but provides near zero
RPO. Because latency is a concern, synchronous replication is typically considered to be suitable
only when a high bandwidth connection is available between two sites that are separated by
about 100 miles or less. In asynchronous replication, the write or update to the source VM is
committed and acknowledgment is immediately sent. These updates are replicated to the target
VM at regular intervals. The challenge with this type of replication is that data is not replicated in
real time. Therefore, if a failure occurs in the source site then there is a chance that the replica in
the remote site might not be completely up to date. The amount of data that could be lost varies
depending upon the replication frequency. In hypervisor-based remote replication VMs can also be
replicated to more than two sites to meet the required SLAs.
15
This lesson covered compute-based location replication solutions such as file system snapshot, VM
snapshot, and VM clone. This lesson also covered compute-based remote replication solutions
such as log shipping and VM replication across sites.
16
This lesson covers storage system-based local replication solutions such as full volume replication
(clone) and pointer-based virtual replication (snapshot). This lesson also covers storage-based
remote replication and multi-site replication.
17
In storage system-based replication, the storage system operating environment performs the
replication process. Compute system resources, such as the CPU and memory, are not used in the
replication process. Consequently, the compute system running multiple VMs is not burdened by
the replication operations. Storage system-based replication supports both local and remote
replication.
In storage system-based local replication, the replication is performed within the storage system.
In other words, the source and the target LUNs reside on the same storage system. Typically,
local replication enables one to perform operational recovery in the event of data loss and also
provides the support for other business operations such as backup. The storage system-based
local replication can be implemented as full volume replication (clone), and pointer-based
virtual replication (snapshot).
In storage system-based remote replication, the replication is performed between storage

systems. Typically one of the storage systems is in source site and the other system is in remote
site for DR purpose. Data can be transmitted from the source storage system to the target
system over a shared or a dedicated network. Replication between storage systems may be
performed in synchronous or asynchronous modes.
18
Full volume replication provides the ability to create fully populated point-in-time copies of LUNs
within a storage system. When the replication session is started, an initial synchronization is
performed between the source LUN and the replica (clone). Synchronization is the process of
copying data from the source LUN to the clone. During synchronization process, the replica is not
available for any compute system access. Once the synchronization is completed, the replica is
exactly same as source LUN. The replica can be detached from the source LUN and it can be made
available to another compute system for business operations. Subsequent synchronizations
involve only a copy of any data that has changed on the source LUN since the previous
synchronization.
Typically after detachment, changes made to both the source and replica can be tracked at some
predefined granularity. This enables incremental resynchronization (source to target) or
incremental restore (target to source). The clone must be the same size as the source LUN.
19
Pointer-based virtual replication (storage system-based snapshot) is a space optimal solution

when compared to full volume replica. At the time of replication session activation, the target
(snapshot) contains pointers to the location of the data on the source. The snapshot does not
contain data at any time. Therefore, the snapshot is known as a virtual replica. Snapshot is
immediately accessible after the replication session activation. This replication method either uses
Copy on First Write (CoFW) or Redirect on Write (RoW) mechanism. Snapshot is typically
recommended when the changes to the source are less than 30 percent. Multiple snapshots can
be created from the same source LUN for various business requirements. Some snapshot software
provides the capability of automatic termination of a snapshot upon reaching the expiration date.
This is very useful where a rolling snapshot might be taken and then automatically removed after
its time of usefulness has passed. The unavailability of the source device invalidates the data on
the target.
20
In a copy on first write mechanism, when a write is issued to the source for the first time after the
replication session activation, the original data at that address is copied to a predefined area in
the storage system. This area is generally known as the save location. The pointer in the target is
updated to point to this data in the save location. After this, the new write is updated on the
source.
When a write is issued to the target for the first time after replication session activation, the data
is copied from the source to the save location, and the pointer is updated to the data in the save
location. Another copy of the original data is created in the save location before the new write is
updated on the save location. Subsequent writes to the same data block on the source or target
do not trigger a copy operation.
When reads are issued to the target, unchanged data blocks since the session activation are read
from the source, whereas data blocks that have changed are read from the save location.
Data on the target is a combined view of unchanged data on the source and data on the save
location. The unavailability of the source device invalidates the data on the target. The target
contains only pointers to the data, and therefore, the physical capacity required for the target is a
fraction of the source device. The capacity required for the save location depends on the amount
of the expected data change.
21
Some pointer-based virtual replication implementation uses redirect on write technology (RoW).
RoW redirects new writes destined for the source LUN to a reserved LUN in the storage pool. Such
implementation is different from CoFW, where the writes to the source LUN are held until the
original data is copied to the save location to preserve the point-in-time replica. In RoW, a new
write from source compute system is simply written to a new location (redirected) inside the pool.
The original data remains where it is, and is untouched by the RoW process. To read a data from
snapshot in CoFW implementation, the data which has not yet been modified is read from the
source LUN, while data that has been modified since the start of the replication session is read
from the save location. There is always a need to perform a lookup to determine whether data is
on the source LUN or save location, which causes snapshot reads to be slower than source LUN
reads. In the case of a RoW snapshot, the original data remains where it is, and is therefore read
from the original location on the source LUN.
22
Storage-based remote replication solution can avoid downtime by enabling business operations at
remote sites. Storage-based synchronous remote replication provides near zero RPO where the
target is identical to the source at all times. In synchronous replication, writes must be committed
to the source and the remote target prior to acknowledging write complete to the production
compute system. Additional writes on the source cannot occur until each preceding write has been
completed and acknowledged. This ensures that data is identical on the source and the target at
all times. Further, writes are transmitted to the remote site exactly in the order in which they are
received at the source. Therefore, write ordering is maintained and it ensures transactional
consistency when the applications are restarted at the remote location. Most of the storage
systems support consistency group, which allows all LUNs belonging to a given application,
usually a database, to be treated as a single entity and managed as a whole. This helps to ensure
that the remote images are consistent. As a result, the remote images are always restartable
copies. The figure on the slide illustrates an example of synchronous remote replication. If the
source site is unavailable due to disaster, then the service can be restarted immediately in the
remote site to meet the required SLA.
Note:
Application response time is increased with synchronous remote replication because writes must
be committed on both the source and the target before sending the write complete
acknowledgment to the compute system. The degree of impact on response time depends
primarily on the distance and the network bandwidth between sites. If the bandwidth provided for
synchronous remote replication is less than the maximum write workload, there will be times
during the day when the response time might be excessively elongated, causing applications to
time out. The distances over which synchronous replication can be deployed depend on the
applications capability to tolerate the extensions in response time. Typically synchronous remote
replication is deployed for distances less than 200 kilometers (125 miles) between the two sites.
23
It is important for an organization to replicate data across geographical locations in order to

mitigate the risk involved during disaster. If the data is replicated (synchronously) between sites
and the disaster strikes, then there would be a chance that both the sites may be impacted. This
leads to data loss and service outage. Replicating data across sites which are 1000s of kilometers
apart would help organization to face any disaster. If a disaster strikes at one of the regions then
the data would still be available in another region and the service could move to the location.
Asynchronous replication enables to replicate data across sites which are 1000s of kilometers
apart.
In asynchronous remote replication, a write from a production compute system is committed to
the source and immediately acknowledged to the compute system. Asynchronous replication also
mitigates the impact to the applications response time because the writes are acknowledged
immediately to the compute system. This enables to replicate data over distances of up to several
thousand kilometers between the source site and the secondary site (remote locations). In this
replication, the required bandwidth can be provisioned equal to or greater than the average write
workload. In asynchronous replication, compute system writes are collected into buffer (delta set)
at the source. This delta set is transferred to the remote site in regular intervals. Therefore,
adequate buffer capacity should be provisioned to perform asynchronous replication. Some
storage vendors offer a feature called delta set extension, which allows to offload delta set from
buffer (cache) to specially configured drives. This feature makes asynchronous replication resilient
to the temporary increase in write workload or loss of network link. In asynchronous replication,
RPO depends on the size of the buffer, the available network bandwidth, and the write workload
to the source. This replication can take advantage of locality of reference (repeated writes to the
same location). If the same location is written multiple times in the buffer prior to transmission to
the remote site, only the final version of the data is transmitted. This feature conserves link
bandwidth.
24
In a two-site synchronous replication, the source and target sites are usually within a short
distance. Therefore, if a regional disaster occurs, both the source and the target sites might
become unavailable. This can lead to extended RPO and RTO because the last known good copy of
data would need to come from another source, such as an offsite tape. A regional disaster will not
affect the target site in a two-site asynchronous replication because the sites are typically several
hundred or several thousand kilometers apart. If the source site fails, production can be shifted to
the target site, but there is no further remote protection of data until the failure is resolved.
Multi-site replication mitigates the risks identified in two-site replication. In a multi-site
replication, data from the source site is replicated to two or more remote sites. The example
shown in the figure is a three-site remote replication solution. In this approach, data at the
source is replicated to two different storage systems at two different sites. The source-to-bunker
site (target 1) replication is synchronous with a near-zero RPO. The source-to-remote site (target
2) replication is asynchronous with an RPO in the order of minutes. The key benefit of this
replication is the ability to failover to either of the two remote sites in the case of source-site
failure, with disaster recovery (asynchronous) protection between the bunker and remote sites.
Disaster recovery protection is always available if any one-site failure occurs. During normal
operations, all three sites are available and the production workload is at the source site. At any
given instance, the data at the bunker and the source is identical. The data at the remote site is
behind the data at the source and the bunker. The replication network links between the bunker
and the remote sites will be in place but will not be in use. The difference in the data between the
bunker and the remote sites is tracked, so that if a source site disaster occurs, operations can be
resumed at the bunker or the remote sites with incremental resynchronization between these two
sites.
25
This lesson covered storage system-based local replication solutions such as full volume
replication (clone) and pointer-based virtual replication (snapshot). This lesson also covered
storage-based remote replication (synchronous and asynchronous) and multi-site replication.
26
This lesson covers the key components of continuous data protection (CDP). This lesson also
covers local and remote CDP replication. Finally, this lesson covers hypervisor-based CDP
replication.
27
In network-based replication, the replication occurs at the network layer between the compute
and the storage systems. By offloading replication from compute and storage systems, networkbased replication can work across a large number of compute platforms and storage systems,
making it ideal for highly heterogeneous environments.
Continuous data protection (CDP) is a network-based replication solution that provides the
capability to restore data and VMs to any previous PIT. Traditional data protection technologies
offer a limited number of recovery points. If a data loss occurs, the system can be rolled back
only to the last available recovery point. CDP tracks all the changes to the production volumes
and maintains consistent point-in-time images. This makes the CDP to restore data to any
previous PIT. CDP supports both local and remote replication of data and VMs to meet operational
and disaster recovery respectively. In a CDP implementation, data can be replicated to more than
two sites using synchronous and asynchronous replication. CDP supports various WAN
optimization techniques (deduplication, compression) to reduce bandwidth requirements, and also
optimally utilizes the available bandwidth.
28
CDP uses a journal volume to store all the data that has changed on the production volume from
the time the replication session started. The journal contains the metadata and data that will
allow rollback to any recovery points. The amount of space that is configured for the journal
determines how far back the recovery points can go. CDP also uses an appliance and a write
splitter. A CDP appliance is an intelligent hardware platform that runs the CDP software and
manages local and remote data replications. Some vendors offer virtual appliance where the CDP
software is running inside VMs. Write splitters intercept writes to the production volume from the
compute system and split each write into two copies. Write splitting can be performed at the
compute, fabric, or storage system.
29
The figure on the slide portrays CDP local and remote replication operations where the write
splitter is deployed at the compute system. Typically the replica is synchronized with the source,
and then the replication process starts. After the replication starts, all the writes from the
compute system to the source (production volume) are split into two copies. One copy is sent to
the local CDP appliance at the source site, and the other copy is sent to the production volume.
Then the local appliance writes the data to the journal at the source site and the data in turn is
written to the local replica. If a file is accidently deleted, or the file is corrupted, the local journal
enables to recover the application data to any PIT.
In case of remote replication, the local appliance at the source site sends the received write I/O to
the appliance at the remote (DR) site. Then, the write is applied to the journal volume at the
remote site. As a next step, data from the journal volume is sent to the remote replica at
predefined intervals. CDP operates in either synchronous or asynchronous mode. In the
synchronous replication mode, the application waits for an acknowledgment from the CDP
appliance at the remote site before initiating the next write. In the case of asynchronous mode,
the local CDP appliance acknowledges a write as soon as it is received. In case of any disaster at
the source site, data can be recovered to the required PIT and the service can be restarted at the
DR site.
30
Some vendors offer continuous data protection for VMs through hypervisor-based CDP
implementation. In this deployment, the specialized hardware-based appliance is replaced with
virtual appliance which is running on a hypervisor. The write splitter is embedded in the
hypervisor. This option protects single or multiple VMs locally or remotely and enables to restore
VMs to any PIT. The local and remote replication operations are as similar as network-based CDP
replication. The figure on the slide shows a CDP local replication implementation.
31
This lesson covered the key components of continuous data protection (CDP). This lesson also
covered local and remote CDP replication. Finally, this lesson covered hypervisor-based CDP
replication.
32
This lesson covers storage system-based and virtualization appliance-based data migration. This
lesson also covers VM live migration and VM storage migration. Finally, this lesson covers disaster
recovery as a service (DRaaS).
33
To meet the business challenges presented by todays on-demand 24x7 world, data must be
highly available in the right place, at the right time, and at the right cost to the enterprise. Data
migration provides solution to these challenges. Data migration is a specialized replication
technique that enables to move data from one system to another within a data center or between
data centers. Data migration solutions provide the following benefits:
Data center maintenance without downtime: Typically in an IT environment, scheduled
maintenance of compute systems, storage systems, or data center is performed. During
maintenance the systems are down, which may impact the availability of applications running on
those systems. Data migration solutions enable to move the applications and data to other
systems or data center without impacting the downtime.
Disaster avoidance: Data centers in the path of natural calamities (such as hurricanes) can
proactively migrate the applications to another data center without impacting the business.
Technology refresh: As the technology keeps changing, there is a requirement to purchase a
new hardware (for example, storage system) for meeting the business requirements. IT
organizations have to migrate their data and applications to the new system from the old one.
Data center migration or consolidation: Sometimes, IT organization may require data center
migration or consolidation. Data migration solutions enable to move applications from one data
center to another as part of a data center migration or consolidation effort without downtime.
(Contd)
34
Storage system-based migration moves data between heterogeneous storage systems. This
technology is application and server-operating-system independent because the migration
operations are performed by one of the storage systems. The storage system performing the
migration operations is called the control storage system. Data can be moved from/to the devices
in the control storage system to/from a remote storage system. Data migration solutions perform
push and pull operations for data movement. These terms are defined from the perspective of the
control storage system. In the push operation, data is moved from the control storage system to
the remote storage system. In the pull operation, data is moved from the remote storage system
to the control storage system.
When a push or pull operation is initiated, the control storage system creates a protection bitmap
to track the migration process. Each bit in the protection bitmap represents a data chunk on the
control device. When the migration operation is initiated, all the bits are set to one, indicating
that all the contents of the source device need to be copied to the target device. As the migration
process copies data, the bits are changed to zero, indicating that a particular chunk has been
copied. During the push and pull operations, compute systems access to the remote device is not
allowed because the control storage system has no control over the remote storage and cannot
track any change on the remote device. Data integrity cannot be guaranteed if changes are made
to the remote device during the push and pull operations. The push/pull operations can be either
hot or cold. These terms apply to the control devices only. In a cold operation, the control device
is inaccessible to the compute system during migration. Cold operations guarantee data
consistency because both the control and the remote devices are offline. In a hot operation, the
control device is online for compute system operations. During hot push/pull operations, changes
can be made to the control device because the control storage system can keep track of all
changes, and thus ensure data integrity.
35
Virtualization simplifies resource management by pooling and sharing resources for

maximum utilization, and making them appear as logical resources with enhanced
capabilities. One of the key benefits provided by virtualization is non-disruptive data
migration. Typically for data migration in a virtualization environment, a virtualization
appliance (controller) is deployed that provides a translation layer in the SAN, between
the compute systems and the storage systems. The figure on the slide shows an example of
data migration from storage system A to storage system B using virtualization appliance. In this
example, LUNs created at the storage systems are assigned to the appliance. The appliance
abstracts the identity of these LUNs and creates a storage pool by aggregating LUNs from the
storage systems. A virtual volume is created from the storage pool and assigned to the compute
system. When an I/O is sent to a virtual volume, it is redirected through the virtualization layer at
the SAN to the mapped LUNs. For example, an administrator wants to perform a data migration
from storage system A to system B, the virtualization layer handles the migration of data, which
enables LUNs to remain online and accessible while data is migrating. No physical changes are
required because the compute system still points to the same virtual volume on the virtualization
layer. However, the mapping information on the virtualization layer should be changed. These
changes can be executed dynamically and are transparent to the end user. The key advantage of
using virtualization appliance is to support data migration between multi-vendor heterogeneous
storage systems. In a cloud environment, the service provider could also implement
virtualization-based data migration to migrate the customer data from their storage system to a
shared storage used by the service provider. This enables the customer to migrate without
causing downtime to their applications and end users during the migration process. The providers
themselves perform this data migration without the need to go for a third-party data migration
specialist.
36
Organizations using a virtualized infrastructure have many reasons to move running VMs from
one physical compute system to another within a data center or across data centers. The
migration can be used for routine maintenance, VM distribution across sites to balance system
load, disaster recovery, or consolidating VMs onto fewer physical compute systems during nonpeak hours to conserve resources. The ideal virtual infrastructure platform should enable to move
the running VMs as quickly as possible and with minimal impact on the end users. This can be
achieved with the help of implementing VM live migrations.
In a VM live migration, as shown in the figure on the slide, the entire active state of a VM is
moved from one hypervisor to another. The state information includes memory contents and all
other information that identifies the VM. This method involves copying the contents of VM
memory from the source hypervisor to the target and then transferring the control of the VMs
disk files to the target hypervisor. Next, the VM is suspended on the source hypervisor, and the
VM is resumed on the target hypervisor. Performing VM live migration requires a high speed
network connection. It is important to ensure that even after the migration, the VM network
identity and network connections are preserved.
VM live Migration with stretched cluster provides the ability to move VMs across data centers.
Stretched cluster configurations are typically deployed using stretched VLANs (discussed in
module 10, Internet Protocol (IP) SAN. The stretched VLANs allow movement of VMs from a
compute system at one location to another compute system at other location, without the need to
change the network configuration of VMs. This solution is suitable for cloud environment, where
consumers of a given application are spread across the globe, working in different time zones, the
productivity is enhanced to a great extent if an application is closer to the consumers. Live
migration with stretched cluster provides the ability to move VMs and applications to a location
that is closest to the consumer for faster/reliable access.
37
In a VM storage migration, VM disk files are moved from one storage system to another system
without any downtime or service disruption. This approach enables the administrator to move VM
disk files across dissimilar storage systems. Non-disruptive VM disk file migration to different
classes of storage enables cost-effective management of VM disks as part of a tiered storage
strategy. VM storage migration optimizes storage I/O performance by non-disruptively moving
disk files to alternative LUNs that are better architected to deliver the required performance.
VM storage migration starts by copying the metadata about the VM from the source system to the
target storage system. The metadata essentially consists of configuration, swap, and log files.
After the metadata is copied, the VM disk file is moved to the new location. During migration,
there might be a chance that the source is updated; therefore, it is necessary to track the
changes on the source to maintain data integrity. After the migration is completed, the blocks
that have changed since the migration started are transferred to the new location. Some vendors
offer mirroring capabilities to migrate data from the source system to the target storage system.
Mirroring enables a single-pass block copy of the source disk to the destination disk by mirroring
I/Os of copied blocks.
38
Facing an increased reliance on IT and the ever-present threat of natural or man-made disasters,
organizations need to rely on business continuity processes to mitigate the impact of service
disruptions. Whether it is online trading, or insurance-document imaging, or airline reservations,
financial databases, websites, or other computing systems, the fortune of business is inextricably
linked to the continuous availability of these services and data. Traditional disaster recovery
methods often require buying and maintaining a complete set of IT resources at secondary data
centers (DR site) that matches the business-critical systems at the primary data center. This
includes sufficient storage to house a complete copy of all of the organizations business data by
regularly replicating production data on the mirror systems at the secondary site. This may be a
complex process and expensive solution for organizations. Disaster Recovery-as-a-Service
(DRaaS) has emerged as a solution that offers a viable DR solution to organizations.
DRaaS allows organizations to have a DR site in the cloud. The cloud service provider assumes
the responsibility for providing IT resources to enable organizations to continue running their IT
services in the event of a disaster. Resources at the service providers location may either be
dedicated to the consumer or they can be shared. From organizations (consumers) perspective,
having a DR site in the cloud reduces the need for data center space and IT infrastructure, which
leads to significant cost reductions, and eliminates the need for upfront capital expenditure.
DRaaS is gaining popularity among organizations mainly due to its pay-as-you-go pricing model
that can lower costs, and the use of automated virtual platforms that can minimize the recovery
time after a failure.
During normal production operations, IT services run at the organizations production data center.
Replication of data occurs from the organizations production environment to the cloud over the
network. Typically during normal operating conditions, a DRaaS implementation may only need a
small share of resources to synchronize the application data and VM configurations from the
consumers site to the cloud. The full set of resources required to run the application in the cloud
is consumed only if a disaster occurs. In the event of a business disruption or disaster, the
business operations will failover to the providers infrastructure.
39
This lesson covered storage system-based and virtualization appliance-based data migration. This
lesson also covered VM live migration and VM storage migration. Finally, this lesson covered
disaster recovery as a service (DRaaS).
40
The Concepts in Practice section covers various replication and migration products.
Note:
41
EMC TimeFinder SnapVX is an EMC VMAX3 storage system-based local replication solution that
allows the user to create target-less snapshots. A single source volume can have up to 256
snapshots and each snapshot has four linked targets. To access a PIT copy, a link must be
created from the snapshot to a host-mapped target device. Linked targets can have one of the
two modes, Nocopy mode and Copy mode. Nocopy mode does not copy data to the linked target
volume but still makes the PIT accessible via pointers to the snapshot. The PIT image will not be
available after the target is unlinked because some target data may no longer be associated with
the PIT. Copy mode copies all relevant tracks from the snapshots PIT to the linked target volume
to create a complete copy of the PIT that will remain available after the target is unlinked.
EMC VNX Snapshot is a local replication solution that provides point-in-time copy of a source
LUN using redirect on first write methodology. This functionality differs significantly from copy on
first write used by SnapView. Redirect on first write technology increases write performance. VNX
Snapshots are limited to pool-based provisioned LUNs. It uses space for Snapshots in the same
storage pool as their base LUN. It provides instant restore which means when a LUN level restore
is initiated, the restored view is available immediately. VNX Snapshot provides point-in-time data
copies for backups, testing, decision support, and data recovery.
42
EMC VNX SnapSure saves disk space and time by allowing multiple snapshot versions of a file
system. Snapshots (called checkpoints) can be read-only or read-write. SnapSure is not a mirror.
It maintains pointers to track changes to the primary file system and reads data from either the
primary file system or a copy area.
EMC RecoverPoint is a high-performance, single product that provides both local and remote
continuous data protection. RecoverPoint family includes RecoverPoint and RecoverPoint for VMs.
RecoverPoint provides fast recovery of data and enables users restoring the data to any previous
point-in-time. RecoverPoint uses lightweight splitting technology to mirror a write. RecoverPointintegrated WAN bandwidth reduction technology uses compression to optimize network resource
utilization during remote replication. RecoverPoint for VMs is a hypervisor-based software data
protection tool that protects VMware VMs with VM level granularity. It protects VMs with its builtin automated provisioning and orchestration capabilities for disaster and operational recovery and
is fully integrated with VMware vCenter through a plug-in. It provides local and remote replication
over any distance with synchronous or asynchronous replication.
43
EMC SRDF remote replication solution provides disaster recovery and data mobility solutions for
the VMAX3 storage systems. SRDF products offer the ability to maintain multiple, hostindependent, remotely mirrored copies of data. SRDF allows VMAX3 storage systems to be in
the same room, different buildings, or hundreds to thousands of kilometers apart. SRDF
can integrate with other products such as Microsoft Cluster Enable and VMware SRM. SRDF modes
of operation address different service level requirements. Synchronous replication (SRDF/S)
provides real-time mirroring of data between the source VMAX3 and the target VMAX3 systems.
Data Mobility (SRDF/DM) transfers data from the source devices to the remote devices without
waiting for an acknowledgment, using Adaptive Copy mode. Asynchronous replication (SRDF/A)
places host writes into cycles and then transfers an entire chunk to the target system. SRDF/CG
- SRDF consistency groups preserve the dependent-write consistency of devices within a group by
monitoring data propagation from source devices to their corresponding target devices. SRDF/AR
is an automated remote replication solution that uses both SRDF and TimeFinder to provide a
periodic asynchronous remote replication of a re-startable data image for UNIX and Windows
environments. Concurrent SRDF and Cascaded SRDF are three-site DR solutions.
EMC MirrorView is a native VNX block remote replication for disaster recovery. It offers two
storage system-based remote mirroring products: MirrorView/Synchronous and
MirrorView/Asynchronous. These solutions provide end-to-end data protection by replicating the
contents of a primary volume to a secondary volume that resides on a different VNX storage
system. MirrorView/S offers a zero data loss option, while MirrorView/A offers an alternative when
minutes of data loss may be tolerable. MirrorView remote replication provides a disaster recovery
solution that is independent of the host server operating system, or applications that can replicate
over Fibre Channel or iSCSI connections.
44
EMC VNX Replicator is an IP-based replication solution that produces a read-only, point-in-time
copy of a source (production) file system. The VNX Replication service periodically updates this
copy, making it consistent with the production file system. Replicator uses internal checkpoints to
ensure availability of the most recent point-in-time copy. These internal checkpoints are based on
SnapSure technology. Replicator maintains a copy of a file system in an asynchronous manner for
a disaster recovery solution. When a replication session is first started, a full copy is performed.
After initial synchronization, Replicator only sends the changed data over IP.
EMC Open Replicator can be used to migrate data from older Symmetrix arrays, CLARiiON, or
VNX arrays, and certain third-party storage arrays to a VMAX3 storage system. Open Replicator
operations are controlled from a local host attached to a VMAX3 storage system. Data copying is
accomplished as part of the storage system process and does not require host resources. The
data can be copied online between the VMAX3 storage system array and remote devices, allowing
host applications, such as a database or file server, to remain operational during the copy
process.
EMC SAN Copy moves data to and from VNX storage systems on a per-logical-unit (LUN)
basis. A VNX storage system running SAN Copy can transfer data to and from other VNX
systems, Symmetrix VMAX systems, and supported third-party systems. SAN Copy is
designed as a multipurpose migration product for data mobility, migrations, content
distribution, and disaster recovery. With SAN Copy, all processing takes place at the
storage-system level, so the host is not impacted during migration. SAN Copy is also
useful in environments where allocated space is not actively utilized to store data. SAN
Copy can be used to migrate a thick LUN that contains a small amount of data to a Thin
LUN. The Thin LUN will only consume storage to which data is written, all the allocated
but unused storage is returned to the pool that can be used by other LUNs in the pool.
45
VMware vMotion performs live migration of a running virtual machine from one physical server
to another, without downtime. The virtual machine retains its network identity and connections,
ensuring a seamless migration process. Transferring the virtual machine's active memory and the
precise execution state over a high-speed network, allows the virtual machine to move from the
source vSphere host to the destination vSphere host. This entire process takes less than two
seconds on a gigabit Ethernet network. vMotion provides the following benefits:
Perform hardware maintenance without scheduling downtime or disrupting business
operations.
Move virtual machines away from failing or underperforming servers.
VMware Storage vMotion enables live migration of virtual machine disk files within and across
storage systems without service disruptions. Storage vMotion performs zero-downtime storage
migrations with complete transaction integrity. It migrates the disk files of virtual machines
running any supported operating system on any supported server hardware. It performs live
migration of virtual machine disk files across any Fibre Channel, iSCSI, FCoE, and NFS storage
system supported by VMware vSphere.
VMware vCloud Air Disaster Recovery is a DRaaS offering owned and operated by VMware,
built on vSphere Replication and vCloud Air a hybrid cloud platform for infrastructure-as-aservice (IaaS). Disaster Recovery leverages vSphere Replication to provide robust, asynchronous
replication capabilities at the hypervisor layer. This approach to replication allows virtual machines
in vSphere to be easily configured for disaster recovery without the traditional dependencies on
underlying infrastructure hardware or data center mirroring. Per-virtual-machine replication and
restore granularity further provide the ability to meet dynamic recovery objectives without
overshooting the actual business requirements for disaster recovery as they change over time.
46
This module covered the primary uses of replication and its characteristics. This module also
covered various replication technologies implemented at compute, storage, and network. Finally,
this module covered data migration solutions and disaster recovery as a service.
47
Scenario:
An organization runs a business-critical application that stores data in a LUN with RAID 1
configuration. The size of the LUN is 1 TB. The average data that changes in 24 hours is
60 GB. The organization has two data centers which are 1000 miles apart.
Organizations requirements:
Organization needs a solution to address logical corruption of data.
RPO requirement for the application is 1 hour.
In the event of a logical corruption, up to 8 hours of old data needs to be restored.
The amount of storage used for data protection should be minimal.
The organization also requires a disaster recovery (DR) solution that should not impact the
response time of the application.
Deliverables:
Suggest an appropriate storage system-based local replication solution to meet the RPO
requirement with minimum amount of storage. Estimate the storage required for the solution.
Recommend an appropriate remote replication solution for DR purpose.
1. A. Pointer-based virtual replication technology is the most obvious choice because the changes
to the source are less than 30% and the solution requires less storage space. The organization
requires to create a pointer-based virtual replica every hour and maintain each replica for
8 hours. Therefore, a total of 8 replicas need to be maintained to meet the organizations
requirement. If a logical corruption occurs, data can be restored from the latest replica.
At Hour 01:00 Create Replica 1

At Hour 09:00 Recreate Replica 1
1. B. If 60 GB of data changes in 24 hours, then on an average 20 GB of data changes

every 8 hours. Thus, each replica would require 20 GB, and 8 replicas need
approximately 160 GB of storage.
2. To meet the DR requirement, the organization can implement asynchronous remote replication.
This solution provides finite RPO and does not impact the application response time.
This module focuses on information security goals and key terminologies. This module also
focuses on the three storage security domains and key threats across the domains. Further, this
module focuses on the various security controls that enable an organization to mitigate these
threats. Finally, this module focuses on the governance, risk, and compliance (GRC) aspect in a
data center environment.
Module 15: Securing the Storage Infrastructure
This lesson covers goals of information security, security concepts and their relations, and
defense-in-depth strategy.
Information is an organizations most valuable asset. This information, including intellectual

property, personal identities, and financial transactions, is routinely processed and stored in
storage systems, which are accessed through the network. As a result, storage is now more
exposed to various security threats that can potentially damage business-critical data and disrupt
critical services. Organizations deploy various tools within their infrastructure to protect the asset.
These tools must be deployed on various infrastructure assets, such as compute (processes
information), storage (stores information), and network (carries information) to protect the
information.
As organizations are adopting third platform, in which cloud is a core element, one of the key
concerns they have is trust. Trust depends on the degree of control and visibility available to the
informations owner. Therefore, securing storage infrastructure has become an integral
component of the storage management process in second platform and third platform
environment. It is an intensive and necessary task, essential to manage and protect vital
information.
Information security includes a set of practices that protect information and information systems
from unauthorized disclosure, access, use, destruction, deletion, modification, and disruption.
Information security involves implementing various kinds of safeguards or controls, in order to
lessen the risk of an exploitation or a vulnerability in the information system which could
otherwise cause a significant impact to organizations business. From this perspective, security is
an ongoing process, not static, and requires continuous revalidation and modification. Securing
the storage infrastructure begins with understanding the goals of information security.
The goal of information security is to provide Confidentiality, Integrity, and Availability,

commonly referred to as the security triad, or CIA. Confidentiality provides the required
secrecy of information to ensure that only authorized users have access to data. Integrity
ensures that unauthorized changes to information are not allowed. The objective of
ensuring integrity is to detect and protect against unauthorized alteration or deletion of
information. Availability ensures that authorized users have reliable and timely access to
compute, storage, network, application, and data resources.
Ensuring confidentiality, integrity, and availability are the primary objective of any IT security
implementation. These are supported through the use of authentication, authorization, and
auditing processes.
Authentication is a process to ensure that users or assets are who they claim to be by verifying
their identity credentials. A user may be authenticated by a single-factor or multi-factor
method. Single-factor authentication involves the use of only one factor, such as a
password. Multi-factor authentication uses more than one factor to authenticate a user
(discussed later in this module).
Authorization refers to the process of determining whether and in what manner, a user,
device, application, or process is allowed to access a particular service or resource. For
example, a user with administrators privileges is authorized to access more services or
resources compared to a user with non-administrator (for example, read-only) privileges.
Authorization should be performed only if authentication is successful.
The most common authentication and authorization controls, used in a data center
environment are Windows Access Control List (ACL), UNIX permissions, Kerberos, and
Challenge-Handshake Authentication Protocol (CHAP). It is essential to verify the
effectiveness of security controls that are deployed with the help of auditing.
Auditing refers to the logging of all transactions for the purpose of assessing the effectiveness of
security controls. It helps to validate the behavior of the infrastructure components, and to
perform forensics, debugging, and monitoring activities.
The slide illustrates relationship among various security concepts in a data center environment.
An organization (owner of the asset) wants to safeguard the asset from threat agents (attackers)
who seek to abuse the assets. Risk arises when the likelihood of a threat agent (an attacker) to
exploit the vulnerability arises. Therefore, the organizations deploy various countermeasures to
minimize risk by reducing the vulnerabilities.
Risk assessment is the first step to determine the extent of potential threats and risks in an
infrastructure. The process assesses risk and helps to identify appropriate controls to mitigate or
eliminate risks. Organizations must apply their basic information security and risk-management
policies and standards to their infrastructure. Some of the key security areas that an organization
must focus on while building the infrastructure are: authentication, identity and access
management, data loss prevention and data breach notification, governance, risk, and compliance
(GRC), privacy, network monitoring and analysis, security information and event logging, incident
management, and security management. These security areas are covered later in this module.
Information is one of the most important assets for any organization. Other assets include
hardware, software, and other infrastructure components required to access the information. To
protect these assets, organizations deploy security controls. These security controls have two
objectives. The first objective is to ensure that the resources are easily accessible to authorized
users. The second objective is to make it difficult for potential attackers to access and
compromise the system. The effectiveness of a security control can be measured by two key
criteria. One, the cost of implementing the system should be a fraction of the value of the
protected data. Two, it should cost heavily to a potential attacker, in terms of money, effort, and
time, to compromise and access the assets.
Threats are the potential attacks that can be carried out on an IT infrastructure. These attacks
can be classified as active or passive. Passive attacks are attempts to gain unauthorized access
into the system. Passive attacks pose threats to confidentiality of information. Active attacks
include data modification, denial of service (DoS), and repudiation attacks. Active attacks pose
threats to data integrity, availability, and accountability.
Vulnerability is a weakness of any information system that an attacker exploits to carry out an
attack. The components that provide a path enabling access to information are vulnerable to
potential attacks. It is important to implement adequate security controls at all the access points
on these components.
Attack surface, attack vector, and work factor are the three factors to consider when assessing
the extent to which an environment is vulnerable to security threats. Attack surface refers to the
various entry points that an attacker can use to launch an attack, which includes people, process,
and technology. For example, each component of a storage infrastructure is a source of potential
vulnerability. An attacker can use all the external interfaces supported by that component, such
as the hardware and the management interfaces, to execute various attacks. These interfaces
form the attack surface for the attacker. Even unused network services, if enabled, can become a
part of the attack surface. An attack vector is a step or a series of steps necessary to complete an
attack. For example, an attacker might exploit a bug in the management interface to execute a
snoop attack. Work factor refers to the amount of time and effort required to exploit an attack
vector.
Having assessed the vulnerability of the environment, organizations can deploy specific control
measures. Any control measures should involve all the three aspects of infrastructure: people,
process, and technology, and their relationship. To secure people, the first step is to establish and
assure their identity. Based on their identity, selective controls can be implemented for their
access to data and resources. The effectiveness of any security measure is primarily governed by
the process and policies. The processes should be based on a thorough understanding of risks in
the environment, should enable recognizing the relative sensitivity of different types of data, and
help determine the needs of various stakeholders to access the data. Without an effective
process, the deployment of technology is neither cost-effective nor aligned to organizations
priorities.
(Contd)
An organization should deploy multiple layers of defense throughout the infrastructure to

mitigate the risk of security threats, in case one layer of the defense is compromised.
This strategy is referred to as defense-in-depth. This strategy may also be thought of as
a layered approach to security because there are multiple measures for security at
different levels. Defense-in-depth increases the barrier to exploitationan attacker must
breach each layer of defenses to be successfuland thereby provides additional time to
detect and respond to an attack. This potentially reduces the scope of a security breach.
However, the overall cost of deploying defense-in-depth is often higher compared to
single-layered security controls. An example of defense-in-depth could be a virtual
firewall installed on a hypervisor when there is already a network-based firewall deployed
within the same environment. This provides additional layer of security reducing the
chance of compromising hypervisors security if network-level firewall is compromised.
The fundamental requirements of information security and compliance, covered in this

lesson, pertain to both second platform and third platform infrastructures. However,
there are important additional considerations related to third platform (due to cloud
environment), arising from information ownership, responsibility and accountability for
information security, ability to access an application through mobile devices, and the
infrastructures multi-tenancy characteristic. Multi-tenancy refers to an architecture in
which a cloud service provider uses a common set of IT resources to provide services to
multiple consumers, or tenants of the infrastructure. Therefore, secure multi-tenancy is
a key requirement for third platform infrastructure.
Apart from multi-tenancy, third platform provides rapid elasticity, a feature rarely found in
traditional data centers. Therefore, the tools used to provide information security must have the
ability to detect newly provisioned resources and integrate with these scaled resources to provide
security. Without these capabilities, it is difficult to monitor and manage the security of such an
environment.
Some of the key security areas cloud service providers and consumers must focus on are:
authentication, identity and access management, data loss prevention and data breach
notification, governance, risk, and compliance (GRC), privacy, network monitoring and analysis,
security information and event logging, incident management, and security management.
10
This lesson covered the key security terminologies, confidentiality, integrity, and availability
(CIA); authentication, authorization, and auditing (AAA); assets, threats, vulnerabilities, security
controls, and defense-in-depth.
11
This lesson covers the storage security domains and the key security threats across domains.
12
The information made available on a network is exposed to security threats from a variety of
sources. Therefore, specific controls must be implemented to secure this information that is
stored on an organizations storage infrastructure. In order to deploy controls, it is important to
have a clear understanding of the access paths leading to storage resources. If each component
within the infrastructure is considered a potential access point, the attack surface of all these
access points must be analyzed to identify the associated vulnerabilities. To identify the threats
that apply to a storage infrastructure, access paths to data storage can be categorized into three
security domains: application access, management access, and backup, replication, and archive.
The figure on the slide depicts the three security domains of a storage environment. The first
security domain involves application access to the stored data through the storage network.
Application access domain may include only those applications that access the data through the
file system or a database interface.
The second security domain includes management access to storage and interconnecting devices
and to the data residing on those devices. Management access, whether monitoring, provisioning,
or managing storage resources, is associated with every device within the storage environment.
Most management software supports some form of CLI, system management console, or a webbased interface. Implementing appropriate controls for securing management applications is
important because the damage that can be caused by using these applications can be far more
extensive.
The third domain consists of backup, replication, and archive access. This domain is primarily
accessed by storage administrators who configure and manage the environment. Along with the
access points in this domain, the backup and replication media also needs to be secured.
To secure the storage environment, identify the attack surface and existing threats within each of
the security domains and classify the threats based on the security goals availability,
confidentiality, and integrity.
13
The slide lists some of the key security threats across domains.
14
Unauthorized access is an act of gaining access to the information systems, which includes
compute systems, network, storage, and management compute system of an organization
illegally. An attacker may gain unauthorized access to the organizations application, data, or
storage resources by various ways such as by bypassing the access control, exploiting a
vulnerability in the operating system, hardware, or application, by elevating the privileges,
spoofing identity, and device theft. The slide illustrates various ways in which an attacker may
gain access to organizations resources via the application access domain and management
access domain.
Many organizations allow their employees to access some of the applications through mobile
devices. This enables employees to access the application and data from any location. Mobile
device theft may increase the risk of exposure of data to an attacker.
An attacker may also gain unauthorized access to the organizations application or data by
bypassing the access controls. This may be accomplished by using the stolen laptop of an
employee.
Another way of unauthorized access to resources is by spoof user identity from compute A and
gain access to recourses accessible to that system and carry out an attack. Unauthorized access
to the resources may be gained by elevation of privileges of compute system A to gain access to
the resources of compute system B.
An attacker may also carry out an attack from management access domain by impersonating the
identity of an administrator or elevate his/her identity to administrator.
Some of the controls that may reduce the risks include strong authentication and authorization,
VLAN and VSAN, data encryption, and mobile device management.
15
A Denial of Service (DoS) attack prevents legitimate users from accessing resources or
services. DoS attacks can be targeted against compute systems, networks, or storage
resources in a storage environment. In all cases, the intent of DoS is to exhaust key
resources, such as network bandwidth or CPU cycles, thereby impacting production use.
For example, an attacker may send massive quantities of data over the network to the
storage system with the intention of consuming bandwidth. This prevents legitimate
users from using the bandwidth and the user may not be able to access the storage
system over the network. Such an attack may also be carried out by exploiting
weaknesses of a communication protocol. For example, an attacker may cause DoS to a
legitimate user by resetting TCP sessions. Apart from DoS attack, an attacker may also
carry out Distributed DoS attack.
A Distributed DoS (DDoS) attack is a variant of DoS attack in which several systems launch a
coordinated, simultaneous DoS attack on their target(s), thereby causing denial of service to the
users of the targeted system(s). In a DDoS attack, the attacker is able to multiply the
effectiveness of the DoS attack by harnessing the resources of multiple collaborating systems
which serve as attack platforms. Typically, a DDoS master program is installed on one compute
system. Then, at a designated time, the master program communicates to a number of "agent"
programs installed on compute systems. When the agents receive the command, they initiate the
attack.
The principal control that can minimize the impact of DoS and DDoS attack is to impose
restrictions and limits on the network resource consumption. For example, when it is identified
that the amount of data being sent from a given IP address exceeds the configured limits, the
traffic from that IP address may be blocked. This provides a first line of defense. Further,
restrictions and limits may be imposed on resources consumed by each compute system,
providing an additional line of defense.
16
Data loss can occur in a storage environment due to various reasons other than malicious attacks.
Some of the causes of data loss may include accidental deletion by an administrator or
destruction resulting from natural disasters. In order to prevent data loss, deploying appropriate
measures such as data backup or replication can reduce the impact of such events. Organizations
need to develop strategies that can avoid or at least minimize the data loss due to such events.
Examples of such strategies include choice of backup media, frequency of backup,
synchronous/asynchronous replication, and number of copies.
Further, if the organization is a cloud service provider then they must publish the protection
controls deployed to protect the data stored in cloud. The providers must also ensure appropriate
terms and conditions related to data loss and the associated penalties as part of the service
contract. The service contract should also include various BC/DR options, such as backup and
replication, offered to the consumers.
17
Today, most organizations are aware of the security threats posed by outsiders.
Countermeasures such as firewalls, malware protection software, and intrusion detection
systems can minimize the risk of attacks from outsiders. However, these measures do
not reduce the risk of attacks from malicious insiders. According to Computer Emergency
Response Team (CERT), a malicious insider could be an organizations current or former
employee, contractor, or other business partner who has or had authorized access to an
organizations compute systems, network, or storage. These malicious insiders may
intentionally misuse that access in ways that negatively impact the confidentiality,
integrity, or availability of the organizations information or resources.
For example, consider a former employee of an organization who had access to the
organizations storage resources. This malicious insider may be aware of security
weaknesses in that storage environment. This is a serious threat because the malicious
insider may exploit the security weakness. Control measures that can minimize the risk
due to malicious insiders include strict access control policies, disabling employee
accounts immediately after separation from the company, security audit, encryption, and
segregation of duties (role-based access control, which is discussed later in this module).
A background investigation of a candidate before hiring is another key measure that can
reduce the risk due to malicious insiders.
18
Account hijacking refers to a scenario in which an attacker gains access to an administrators or

users account(s) using methods such as phishing or installing keystroke-logging malware on
administrators or users compute systems.
Phishing is an example of a social engineering attack that is used to deceive users. Phishing
attacks are typically carried out by spoofing e-mail an e-mail with a fake but genuine-appearing
address, which provides a link to a website that masquerades as a legitimate website. After
opening the website, users are asked to enter details such as their login credentials. These details
are then captured by the attacker to take over the users account. For example, an employee of
an organization may receive an e-mail that is designed to appear as if the IT department of that
organization has sent it. This e-mail may ask the users to click the link provided in the e-mail and
update their details. After clicking the email, the user is directed to a malicious website where
their details are captured.
Another way to gain access to a users credentials is by installing keystroke-logging malware. In
this attack, the attacker installs malware in the storage administrators compute system which
captures user credentials and sends them to the attacker. After capturing the credentials, an
attacker can use them to gain access to the storage environment. The attacker may then
eavesdrop on the administrators activities and may also change the configuration of the storage
environment to negatively impact the environment.
A man-in-the-middle attack is another way to hack users credentials. In this attack, the
attacker eavesdropsoverhears the conversationon the network channel between two sites
when replication is occurring over the network. Use of multi-factor authentication and IPSec (a
suite of algorithms, protocols, and procedures used for securing IP communications by
authenticating and/or encrypting each packet in a data stream) can prevent this type of attack.
Intrusion detection and prevention systems and firewalls, which will be discussed later in this
module are additional controls that may reduce the risk of such attacks.
19
Application programming interfaces (APIs) are used extensively in software-defined and cloud
environment. It is used to integrate with management software to perform activities such as
resource provisioning, configuration, monitoring, management, and orchestration. These APIs
may be open or proprietary. The security of storage infrastructure depends upon the security of
these APIs. An attacker may exploit vulnerability in an API to breach a storage infrastructures
perimeter and carry out an attack. Therefore, APIs must be designed and developed following
security best practices such as requiring authentication and authorization, input validation of
APIs, and avoiding buffer overflows. Security review of the APIs must be performed by the
organizations. Access to the APIs must be restricted to authorized users. These practices provide
protection against both accidental and malicious attempts to bypass security.
20
Technologies that are used to build todays storage infrastructure provide a multi-tenant
environment enabling the sharing of resources. Multi-tenancy is achieved by using controls that
provide separation of resources such as memory and storage for each application. Failure of these
controls may expose the confidential data of one business unit to users of other business units,
raising security risks.
Compromising a hypervisor is a serious event because it exposes the entire environment to
potential attacks. Hyperjacking is an example of this type of attack in which the attacker installs a
rogue hypervisor that takes control of the compute system. The attacker now can use this
hypervisor to run unauthorized virtual machines in the environment and carry out further attacks.
Detecting this attack is difficult and involves examining components such as program memory
and the processor core registers for anomalies.
21
Backups and replications are essential business continuity processes of any data center. However,
inadequate security controls may expose organizations confidential information to an attacker.
There is a risk of a backup tape being lost, stolen, or misplaced, and the threat is even severe
especially if the tapes contain highly confidential information.
An attacker may gain access to an organizations confidential data by spoofing the identity of the
DR site. When the replication session is started, an attackers compute system may appear as the
DR site to the primary site and the replication data is sent to the attacker's compute system.
Some of the security controls that may reduce the risk due to these threats include physical
security, encrypting data-at-rest and data-in-flight, and strong authentication and authorization.
22
This lesson covered storage security domains and the key threats across these domains.
23
This lesson covers physical security and identity and access management deployed in cloud
environment.
24
Any security control should account for three aspects: people, process, and technology, and the
relationships among them. Security controls can be administrative or technical. Administrative
controls include security and personnel policies or standard procedures to direct the safe
execution of various operations. Technical controls are usually implemented through tools or
devices deployed on the IT infrastructure.
To protect a storage infrastructure, various technical security controls must be deployed at the
compute, network, and storage levels.
25
At the compute system level, security controls are deployed to secure hypervisors and hypervisor
management systems, virtual machines, guest operating systems, and applications. Security at
the network level commonly includes firewalls, demilitarized zones, intrusion detection and
prevention systems, virtual private networks, zoning and iSNS discovery domains, port binding
and fabric binding configurations, and VLAN and VSAN. At the storage level, security controls
include LUN masking, data shredding, and data encryption. Apart from these security controls,
the storage infrastructure also requires identity and access management, role-based access
control, and physical security arrangements.
26
Physical security is the foundation of any overall IT security strategy. Strict enforcement of
policies, processes, and procedures by an organization is critical element of successful physical
security. To secure the organizations storage infrastructure, the following physical security
measures may be deployed:
Disable all unused IT infrastructure devices and ports
24/7/365 onsite security
Biometric or security badge-based authentication to grant access to the facilities
Surveillance cameras [CCTV] to monitor activity throughout the facility
Sensors and alarms to detect motion and fire
27
Identity and access management is the process of managing users identifiers and their
authentication and authorization to access storage infrastructure resources. It also controls
access to resources by placing restrictions based on user identities. In todays environment, an
organization may collaborate with one or more cloud service providers to access various cloudbased storage services. This requires deploying multiple authentication systems to enable the
organization to authenticate employees and provide access to cloud-based storage services.
The key traditional authentication and authorization controls deployed in a storage environment
are Windows ACLs, UNIX permissions, Kerberos, and Challenge-Handshake Authentication
Protocol (CHAP). Alternatively, the organization can use Federated Identity Management (FIM) for
authentication. A federation is an association of organizations (referred to as trusted parties) that
come together to exchange information about their users and resources to enable collaboration.
Federation includes the process of managing the trust relationships among the trusted parties
beyond internal networks or administrative boundaries. FIM enables the organizations (especially
cloud service providers) to offer services without implementing their own authentication system.
The organization can choose an identity provider to authenticate their users. This involves
exchanging identity attributes between the organizations and the identity provider in a secure
way. The identity and access management controls used by organizations include OpenID and
OAuth.
28
Windows ACLs and UNIX Permissions form the first level of protection to compute
resources (application servers, file servers, and file sharing environment such as NAS) by
restricting accessibility and sharing. These permissions are deployed over and above the
default behaviors and attributes associated with files and folders. In addition, various
other authentication and authorization controls, such as Kerberos and directory services
are implemented to verify the identity of network users and define their privileges.
Windows supports two types of ACLs: discretionary access control lists (DACLs) and
system access control lists (SACLs). The DACL, commonly referred to as the ACL, is used
to determine access control. The SACL determines which accesses need to be audited if
auditing is enabled. In addition to these ACLs, Windows also supports the concept of
object ownership. The owner of an object has hard-coded rights to that object, and these
rights do not need to be explicitly granted in the SACL. The owner, SACL, and DACL are
all statically held as attributes of each object. Windows also offers the functionality to
inherit permissions, which allows the child objects existing within a parent object to
automatically inherit the ACLs of the parent object. ACLs are also applied to directory
objects known as security identifiers (SIDs). These are automatically generated by a
Windows server or domain when a user or group is created, and they are abstracted
from the user. In this way, though a user may identify his login ID as User1, it is
simply a textual representation of the true SID, which is used by the underlying
operating system. Internal processes in Windows refer to an accounts SID rather than
the accounts username or group name while granting access to an object. ACLs are set
by using the standard Windows Explorer GUI but can also be configured with CLI
commands or other third-party tools.
(Cont'd)
29
OAuth is an open authorization control that allows a client to access protected resources from a
resource server on behalf of a resource owner. This control can be used in the third platform
environment to secure application access domain.
There are four entities involved in the authorization control: resource owner, resource server,
client, and authorization server. A resource owner is an entity capable of granting access to a
protected resource. A resource server is the compute system hosting the protected resources,
capable of accepting and responding to protected resource requests using access tokens. A client
is an application making protected resource requests on behalf of the resource owner with the
resource owners authorization. An authorization server is the compute system issuing access
tokens to the client after successfully authenticating the resource owner and obtaining
authorization. The authorization server may be the same server as the resource server or a
separate entity.
The figure on the slide illustrates the steps involved in OAuth process as described in Request for
Comments (RFC) 6749 published by Internet Engineering Task Force (IETF):
1. The client requests authorization from the resource owner. The authorization request can be
made directly to the resource owner, or indirectly via the authorization server.
2. The client receives an authorization grant, which is a credential representing the resource
owner's authorization to access its protected resources. It is used by the client to obtain an
access token. Access tokens are credentials used to access protected resources. An access
token is a string representing an authorization issued to the client. The string is usually
opaque to the client. Tokens represent specific scopes and durations of access, granted by the
resource owner, and enforced by the resource server and authorization server.
(Cont'd)
30
Multi-factor authentication uses more than one factor to authenticate a user. A commonly
implemented two-factor authentication process requires the user to supply both

something he or she knows (such as a password) and also something he or she has
(such as a device). The second factor might also be a password generated by a physical
device (known as token), which is in the users possession. The password generated by
the token is valid for a pre-defined time. The token generates another password after the
pre-defined time is over. To further enhance the authentication process, additional
factors may also be considered. Examples of additional factors that may be used include
biometric identity. A multi-factor authentication technique may be deployed using any
combination of these factors. A users access to the environment is granted only when all
the required factors are validated.
31
Kerberos is a network authentication protocol, which is designed to provide strong authentication

for client/server applications by using secret-key cryptography. It uses cryptography so that a
client and server can prove their identity to each other across an insecure network connection.
After the client and server have proven their identities, they can choose to encrypt all their
communications to ensure privacy and data integrity. In Kerberos, authentications occur between
clients and servers. The client gets a ticket for a service and the server decrypts this ticket by
using its secret key. Any entity, user, or compute system that gets a service ticket for a Kerberos
service is called a Kerberos client. The term Kerberos server generally refers to the Key
Distribution Center (KDC). The KDC implements the Authentication Service (AS) and the Ticket
Granting Service (TGS). In Kerberos, users and servers for which a secret key is stored in the
KDC database are known as principals. The KDC has a copy of every password associated with
every principal, so it is absolutely vital that the KDC remains secure.
(Cont'd)
32
The Challenge-Handshake Authentication Protocol (CHAP) is a basic authentication

control that has been widely adopted by network devices and compute systems. CHAP
provides a method for initiators and targets to authenticate each other by utilizing a
secret code or password. CHAP secrets are usually random secrets of 12 to 128
characters. The secret is never exchanged directly over the communication channel;
rather, a one-way hash function converts it into a hash value, which is then exchanged.
A hash function, using the MD5 algorithm, transforms data in such a way that the result
is unique and cannot be changed back to its original form.
If the initiator requires reverse CHAP authentication, the initiator authenticates the target
by using the same procedure. The CHAP secret must be configured on the initiator and
the target. A CHAP entry, composed of the name of a node and the secret associated
with the node, is maintained by the target and the initiator. The same steps are executed
in a two-way CHAP authentication scenario.
After these steps are completed, the initiator authenticates the target. If both the
authentication steps succeed, then data access is allowed. CHAP is often used because it
is a fairly simple protocol to implement and can be implemented across a number of
disparate systems.
33
OpenID is an open standard for authentication in which an organization, known as the

relying party, uses authentication services from an OpenID provider, known as the
identity provider. An OpenID provider maintains users credentials on their authentication
system and enables relying parties to authenticate users requesting the use of the
relying partys services. This eliminates the need for the relying party to deploy their own
authentication systems. In the OpenID control, a user creates an ID with one of the
OpenID providers. This OpenID then can be used to sign-on to any organization (relying
party) that accepts Open ID authentication. This control can be used in the third platform
environment to secure application access domain.
The figure on the slide illustrates the OpenID concept by considering a user who requires
services from the relying party. For the user to use the services provided by the relying
party an identity (user ID and password) is required. The relying party does not provide
their own authentication control, however they support OpenID from one or more
OpenID providers. The user can create an ID with the identity provider and then use this
ID with the relying party. The relying party, after receiving the login request,
authenticates it with the help of identity provider and then grants access to the services.
34
This lesson covered physical security, Windows ACLs, UNIX permissions, OAuth, multi-factor
authentication, Kerberos, CHAP, and OpenID.
35
This lesson covers role-based access control, network monitoring and analysis, firewall, intrusion
detection and prevention system, adaptive security, port binding and fabric binding, virtual
private network, virtual LAN, virtual SAN, zoning, and iSNS discovery domain controls deployed in
a storage environment.
36
37
Role-based access control (RBAC) is an approach to restricting access to authorized users based
on their respective roles. A role may represent a job function, for example, a storage
administrator. Minimum privileges are assigned to a role that is required to perform the tasks
associated with that role.
It is advisable to consider administrative controls, such as separation of duties, when defining
data center security procedures. Clear separation of duties ensures that no single individual can
both specify an action and carry it out. For example, the person who authorizes the creation of
administrative accounts should not be the person who uses those accounts.
38
Monitoring and analyzing the network are important for the smooth and continuous operation of a
storage environment. Network availability can be compromised by DoS attacks and network
device failures. Proactive network monitoring and analysis can detect and prevent network
failures or performance problems.
Network monitoring can be performed in two ways: active monitoring and passive monitoring. In
active monitoring, the monitoring tools transmit data between the two endpoints that are
monitored. The measurement includes parameters such as availability, delay, loss, and
bandwidth. In passive monitoring, instead of transmitting data and then measuring, information
about a link or device is collected by probing the link or device. As the data passes through the
link or device, information is captured. This information is then used to analyze, detect, and
troubleshoot any issues related to performance, availability, or security. Some of the controls
used to monitor, detect, and prevent attacks are firewalls, intrusion detection systems, intrusion
prevention systems, and network analysis/forensics systems.
39
A firewall is a security control designed to examine data packets traversing a network and
compare them to a set of filtering rules. Packets that are not authorized by a filtering rule are
dropped and are not allowed to continue to the requested destination. A rule may use various
filtering parameters such as source address, destination address, port numbers, and protocols.
Some firewalls may support filtering parameters that enable packet inspection for content. These
rules can be set for both incoming and outgoing traffic. The effectiveness of a firewall depends on
how robustly and extensively the security rules are defined. Firewalls can be deployed at the
network, compute system, and hypervisor levels.
A network-level firewall is typically used as first line of defense for restricting certain type of
traffic from coming in and going out from a network. This type of firewall is typically deployed at
the entry point of an organizations network.
At the compute system-level, a firewall application is installed as second line of defense in a
defense-in-depth strategy. This type of firewall provides protection only to the compute system
on which it is installed.
In a virtualized environment, there is an added complexity of virtual machines running on a
smaller number of compute systems. When virtual machines on the same hypervisor
communicate with each other over a virtual switch, a network-level firewall cannot filter this
traffic. In such situations, a virtual firewall can be used to filter virtual machine traffic. A virtual
firewall is a software appliance that runs on a hypervisor to provide traffic filtering service. Virtual
firewalls give visibility and control over virtual machine traffic and enforce policies at the virtual
machine level.
(Cont'd)
40
Intrusion detection is the process of detecting events that can compromise the confidentiality,
integrity, or availability of IT resources. An intrusion detection system (IDS) is a security tool that
automates the detection process. An IDS generates alerts, in case anomalous activity is detected.
An intrusion prevention system (IPS) is a tool that has the capability to stop the events after they
have been detected by the IDS. These two controls usually work together and are generally
referred to as intrusion detection and prevention system (IDPS). The key techniques used by an
IDPS to identify intrusion in the environment are signature-based and anomaly-based detection.
In the signature-based detection technique, the IDPS relies on a database that contains known
attack patterns or signatures, and scans events against it. A signature can be an e-mail with a
specific subject or an e-mail attachment with a specific file name that is known to contain a virus.
This type of detection is effective only for known threats and is potentially circumvented if an
attacker changes the signature (the e-mail subject or the file name in the attachment, in this
example). In the anomaly-based detection technique, the IDPS scans and analyzes events to
determine whether they are statistically different from events normally occurring in the system.
This technique can detect various events such as multiple login failures, excessive process failure,
excessive network bandwidth consumed by an activity, or an unusual number of e-mails sent by a
user, which could signify an attack is taking place.
The IDPS can be deployed at the compute system, network, or hypervisor levels.
41
Security threats have evolved to the point that traditional security controls cannot respond to and
be effective as standalone controls. Sophisticated techniques such as phishing, Man in the Middle,
and others are used to gain unauthorized access to storage environment. To combat against such
sophisticated attacks, organizations require the use of adaptive security controls. Adaptive
security controls integrate with the organizations standalone controls such as IDPS and firewalls
and use heuristics to learn user behavior and detect fraudulent activity. Controls such as
behavioral profile, device-related profile, type of web browser, and plug-ins are used to establish
the normal operating profile of the environment. The intelligence in the adaptive security control
detects and identifies anomalies and blocks such anomalies capabilities that may not be possible
with traditional controls.
42
In the storage environment, a virtual private network (VPN) can be used to provide a
user a secure connection to the storage resources. VPN is also used to provide secure siteto-site connection between a primary site and a DR site when performing remote
replication. VPN can also be used to provide secure site-to-site connection between an
organizations data center and cloud.
A virtual private network extends an organizations private network across a public
network such as Internet. VPN establishes a point-to-point connection between two
networks over which encrypted data is transferred. VPN enables organizations to apply
the same security and management policies to the data transferred over the VPN
connection as applied to the data transferred over the organizations internal network.
When establishing a VPN connection, a user is authenticated before the security and
management policies are applied.
There are two methods in which a VPN connection can be established: remote access
VPN connection and site-to-site VPN connection. In a remote access VPN connection, a
remote client (typically client software installed on the users compute system) initiates a
remote VPN connection request. A VPN server authenticates and provides the user access
to the network. This method can be used by administrators to establish a secure
connection to data center and carry out management operations.
In a site-to-site VPN connection, the remote site initiates a site-to-site VPN connection.
The VPN server authenticates and provides access to internal network. One typical usage
scenario for this method is when deploying a remote replication or connecting the cloud.
43
In a storage environment, VLAN and VSAN ensure security by providing isolation over the
shared infrastructure. Each department of an organization may be provided VLANs and
VSANs to ensure their data is separated from other departments. A Virtual Local Area
Network (VLAN) is a virtual network created on a local area network (LAN) consisting of
virtual and/or physical switches. VLAN technology can divide a large LAN into smaller
virtual LANs or combine separate LANs into one or more virtual LANs. A VLAN enables
communication among a group of nodes based on the functional requirements of the
group, independent of the nodes location in the network. Similarly, Virtual Storage Area
Network (VSAN) enables the creation of multiple logical SANs over a common physical
SAN. They provide the capability to build larger consolidated fabrics and still maintain the
required security and isolation between them. Zoning should be done for each VSAN to
secure the entire physical SAN.
44
Zoning is a Fibre Channel switch control that enables node ports within a fabric to be
logically segmented into groups and to communicate with each other within the group.
There are three types of zoning. World Wide Port Name-based zoning is the most
commonly used to prevent unauthorized access when node ports are re-cabled to
different fabric ports. However, it is possible that a rogue compute system could join the
fabric, then spoof a legitimate WWPN and thereby gain access to resources in a zone. If
WWPN spoofing is a key concern, then port zoning and port binding can be used.
Internet Storage Name Service (iSNS) discovery domains function in the same way as FC
zones. Discovery domains provide functional groupings of devices in an IP SAN. For
devices to communicate with one another, they must be configured in the same
discovery domain. State change notifications inform the iSNS server when devices are
added to or removed from a discovery domain.
45
Port binding is a control used to limit the devices that can be attached to a specific switch port
and is supported in both FC SAN and Ethernet environments. In an FC SAN, port binding maps a
WWPN to a switch port. If a host tries to login to a port with a WWPN that is not allowed by the
port binding, the WWPN login is rejected. In an Ethernet network, port binding maps the MAC
address and IP address of a compute system to a specific switch port. A switch port will forward a
packet only if the MAC address and the IP address in the packet are mapped to that port. Port
binding mitigates but does not eliminate WWPN or MAC spoofing.
Fabric binding is another security control in an FC SAN environment that allows only authorized
switches to join an existing fabric. Inter-switch links are only enabled between specified switches
in the fabric. Each switch in the fabric obtains identical membership data that includes a list of
authorized switches in the fabric. The port security controls such as port locking and port-type
locking complement fabric binding by helping to prevent unauthorized access to a switch. Port
locking persistently (even after a switch reboot) prohibits an unused switch port from being used.
Port-type locking can be used to restrict how a specific switch port is used, such as preventing it
from being initialized as an inter-switch link.
46
This lesson covered role-based access control, network monitoring and analysis, firewall, intrusion
detection and prevention system, adaptive security, port binding and fabric binding, VPN, VLAN,
VSAN, zoning, and iSNS discovery domain.
47
This lesson covers securing hypervisor and management server, virtual machine hardening,
securing operating system and applications, LUN masking, data encryption, and data shredding
controls deployed in a storage environment.
48
49
The hypervisor and related management servers are critical components of the storage
infrastructure because they control the operation and management of the virtualized compute
environment. Compromising a hypervisor or management server places all VMs at a high risk of
attack. Hypervisors may be compromised by hyperjacking or performing other forms of attack.
Further, the management server may be compromised by exploiting vulnerabilities in the
management software or by an insecure configuration. For example, an administrator may have
configured a non-secured or non-encrypted remote access control. An attacker may take control
of the management server by exploiting a security loophole of the system. This enables the
attacker to perform unauthorized activities such as controlling all the existing VMs, creating new
VMs, deleting VMs, and modifying VM resources. These types of attacks impact security of
application access and management access domains. To protect against such attacks, securitycritical hypervisor updates should be installed when they are released by the hypervisor vendor.
Hypervisor hardening should be performed, using specifications provided by organizations such as
the Center for Internet Security (CIS) and Defense Information Systems Agency (DISA). Access
to the management server should be restricted to authorized administrators. Access to core levels
of functionality should be restricted to selected administrators. Network traffic should be
encrypted when management is performed remotely. A separate firewall with strong filtering rules
installed between the management system and the rest of the network can enhance security.
Virtual machines store troubleshooting information in a log file that is stored on the storage
presented to a hypervisor. An attacker may cause a virtual machine to abuse the logging function,
causing the size of the log file to grow rapidly. Over the period of time, the log file can consume
all the capacity of the storage presented to the hypervisor, effectively causing a denial of service.
This can be prevented by configuring the hypervisor to rotate or delete log files when they reach a
certain size. This option enables the administrator to configure the maximum size of the log file.
When this size is reached, the hypervisor makes an archive copy of the log file and starts storing
information in a new log file. Administrators can configure settings to maintain a specific number
of old log files. When the configured limit is reached, the hypervisor automatically deletes the
oldest file.
50
Virtual machine hardening is a key security control to protect virtual machines from various
attacks. Typically, a virtual machine is created with several default virtual components and
configurations. Some of the configurations and components may not be used by the operating
system and application running on it. These default configurations may be exploited by an
attacker to carry out an attack. Therefore, a virtual machine hardening process should be used in
which the default configuration is changed to achieve greater security. In this process, virtual
machines devices that are not required are removed or disabled. In this process the configuration
of VM features is tuned to operate in a secure manner such as changing default passwords,
setting permissions to VM files, and disallowing changes to the MAC address assigned to a virtual
NIC, mitigating spoofing attacks. Hardening is highly recommended when creating virtual machine
templates. This way, virtual machines created from the template start from a known security
baseline.
Operating system hardening typically includes deleting unused files and applications, installing
current operating system updates (patches), and configuring system and network components
following a hardening checklist. These hardening checklists are typically provided by operating
system vendors or organizations such as the Center for Internet Security (CIS) and Defense
Information Systems Agency (DISA), who also provide security best practices. Further,
vulnerability scanning and penetration testing can be performed to identify existing vulnerabilities
and to determine the feasibility of an attack. These controls assess the potential impact of an
attack on the business.
(Contd)
51
Malware protection software is typically installed on a compute system or on a mobile device to

provide protection for the operating system and applications. The malware protection software
detects, prevents, and removes malware and malicious programs such as viruses, worms, Trojan
horses, key loggers, and spyware. Malware protection software uses various techniques to detect
malware. One of the most common techniques used is signature-based detection. In this
technique, the malware protection software scans the files to identify a malware signature. A
signature is a specific bit pattern in a file. These signatures are cataloged by malware protection
software vendors and are made available to users as updates. The malware protection software
must be configured to regularly update these signatures to provide protection against new
malware programs. Another technique, called heuristics, can be used to detect malware by
examining suspicious characteristics of files. For example, malware protection software may scan
a file to determine the presence of rare instructions or code. Malware protection software may
also identify malware by examining the behavior of programs. For example, malware protection
software may observe program execution to identify inappropriate behavior such as keystroke
capture.
Malware protection software can also be used to protect operating system against attacks. A
common type of attack that is carried out on operating systems is by modifying its sensitive
areas, such as registry keys or configuration files, with the intention of causing the application to
function incorrectly or to fail. This can be prevented by disallowing the unauthorized modification
of sensitive areas by adjusting operating system configuration settings or via malware protection
software. In this case, when a modification is attempted, the operating system or the malware
protection software challenges the administrator for authorization.
52
Several organizations allow their employees to access organizations internal application and
resources via mobile devices. This introduces a potential threat to the organizations resources as
theft of these devices may expose resources to an attacker. This type of threat can be minimized
by the use of mobile device management. Mobile device management is a control that restricts
access to organizations resources only to authorized mobile devices.
The MDM solution consists of two components: the server component and the client component.
The server component is responsible for performing device enrollment, administration, and
management of mobile devices. The client component is installed on the mobile device that needs
access to the organizations resources. The client receives commands from the server component
which it executes on the mobile device.
To enroll the device, an MDM client is installed on the mobile device. The client component is used
to connect to the server component to receive administration and management commands. To
connect to the server component, the user is required to provide MDM authentication server and
user credentials details. Typically, the authentication server is placed in a DMZ. These credentials
are authenticated by the MDM authentication server. Devices that are successfully authenticated
are redirected to the MDM server. Now the authenticated mobile devices are enrolled and can be
managed. Further, these mobile devices can be granted access to the applications and other
resources.
MDM solution enables organizations to enforce organizations security policies on the users
mobile devices. The solution also provides the organizations the administrative and management
control to the users mobile device. With this control the organization will have the ability to
remotely wipe the data on the enrolled devices or brick the device when a threat is detected.
53
LUN masking refers to the assignment of LUNs to a specific host bus adapter world-wide names.
LUN masking is one of the basic SAN security controls used to protect against unauthorized
access to storage. LUN masking can be implemented at the host within the switch or at the
storage system. The standard implementations of LUN masking on storage arrays mask the LUNs
presented to a front-end storage port based on the WWPNs of the source HBAs. A stronger
variant of LUN masking may sometime be offered in which masking is done on the basis of the
source Fibre Channel address. The Fibre Channel address typically changes if the HBA is relocated
across ports in the fabric. To avoid this problem, most switch vendors offer a control to lock down
the Fibre Channel address of a given node port regardless of its location.
54
Data encryption is a cryptographic technique in which data is encoded and made indecipherable to
eavesdroppers or hackers. Data encryption is one of the most important controls for securing data
in-flight and at-rest. Data in-flight refers to data that is being transferred over a network and data
at-rest refers to data that is stored on a storage medium. Data encryption provides protection
from threats such as tampering with data which violates data integrity, media theft which
compromises data availability, and confidentiality and sniffing attacks which compromise
confidentiality.
Data should be encrypted as close to its origin as possible. If it is not possible to perform
encryption on the compute system, an encryption appliance can be used for encrypting data at
the point of entry into the storage network. Encryption devices can be implemented on the fabric
to encrypt data between the compute system and the storage media. These controls can protect
both the data at-rest on the destination device and data in-transit. Encryption can also be
deployed at the storage-level, which can encrypt data-at-rest.
Another way to encrypt network traffic is to use cryptographic protocols such as Transport Layer
Security (TLS) which is a successor to Secure Socket Layer (SSL). These are application layer
protocols and provide an encrypted connection for client-server communication. These protocols
are designed to prevent eavesdropping and tampering of data on the connection over which it is
being transmitted.
55
Data shredding is the process of deleting data or residual representations (sometimes called
remanence) of data and making it unrecoverable. Typically, when data is deleted, it is not made
unrecoverable from the storage and an attacker may use specialized tools to recover it. The
threat of unauthorized data recovery is greater when an organization discards the failed storage
media such as disk drive, solid state drive, or tape. After the organization discards the media, an
attacker may gain access to these media and may recover the data by using specialized tools.
Organizations can deploy data shredding controls in their storage infrastructure to protect from
loss of confidentiality of their data. Data may be stored on disks or on tapes. Techniques to shred
data stored on tape include overwriting it with invalid data, degaussing the media (a process of
decreasing or eliminating the magnetic field), and physically destroying the media. Data stored on
disk or flash drives can be shredded by using algorithms that overwrite the disks several times
with invalid data.
Organizations may create multiple copies (backups and replicas) of their data and store at
multiple locations as part of business continuity and disaster recovery strategy. Therefore,
organizations must deploy data shredding controls at all location to ensure that all the copies are
shred.
56
This lesson covered securing hypervisor and management server, virtual machine hardening,
securing operating system and applications, LUN masking, data encryption, and data shredding.
57
This lesson covers the focus areas of governance, key steps of risk management, types of
compliance that control IT operations, and the key auditing activities.
58
Governance, Risk, and Compliance (GRC) is a term encompassing processes that help an
organization to ensure that their acts are ethically correct and in accordance with their
risk appetite (the risk level an organization chooses to accept), internal policies, and
external regulations. This process should be integrated, holistic, and organization-wide.
All operations of an organization should be managed and supported through GRC.
Governance, risk management, and compliance management work together to enforce
policies and minimize potential risks. To better understand how these three components
work together, consider an example of how GRC is implemented in an IT organization.
Governance is the authority for making policies such as defining access rights to users
based on their roles and privileges. Risk management involves identifying resources that
should not be accessed by certain users in order to preserve confidentiality, integrity,
and availability. In this example, compliance management assures that the policies are
being enforced by implementing controls such as firewalls and identify management
systems.
GRC is an important component of data center infrastructure. Therefore, while building a third
platform infrastructure organizations must ensure that all aspects of GRC are deployed that
include cloud-related aspects such as ensuring secured multi-tenancy, the jurisdictions where
data should be stored, data privacy, and ownership.
59
Governance determines the purpose, strategy, and operational rules by which companies are
directed and managed. Enterprise Governance is based on the companys business
strategy and driven by the Board of Directors. It generally includes legal, HR, finance,
and the office of the CEO. Governance affects how the company addresses everything from
long-term strategies to day-to-day operations. The slide focuses on IT Governance, which is a
subset discipline of Enterprise Governance.
The objective of IT governance is to determine the desired behavior or results to achieve
the ITs strategic goals. Governance in IT is a system in which leaders monitor, evaluate,
and direct IT management to ensure IT effectiveness, accountability, and compliance. For
a governance system to work, it has to distribute the workload and decision making
process without losing value or gaining bias in the process. Roles and responsibilities
must be clearly defined, providing details such as who is responsible for directing,
controlling, and executing decisions, what information is required to make the decisions,
and how exceptions will be handled. The last step is to measure the outcome of the
governance process, finding areas for improvement, and instituting changes for
improvement.
60
Risk is the effect of uncertainty on business objectives. Risk management is a systematic process
of assessing its assets, placing a realistic valuation on each asset, and creating a risk profile that
is rationalized for each information asset across the business. Additionally, the organizations must
establish a risk threshold to measure against each asset. Risk management involves identification,
assessment, and prioritization of risks and institutes controls to minimize the impact of those
risks.
There are four key steps of risk management that an organization must perform before offering
resources or services to the users: risk identification, risk assessment, risk mitigation, and
monitoring. These steps are elaborated below:
Step 1: Risk identification points to the various sources of threats that give rise to risk. After
identifying risks in a data center environment, these risks and their sources need to be classified
into meaningful severity levels.
Step 2: Risk assessment determines the extent of potential threat and the risk associated with
data center resources. The output of this process helps organizations to identify appropriate
controls for reducing or eliminating risk during the risk mitigation process. All the assets at risk
(data, applications, and processes) must be carefully evaluated to assess their criticality to the
business. Critical assets are assets that are essential for the operation of the organizations
business and have high business value, such as intellectual property (IP), project plans, and
Personally Identifiable Information (PII). After the risks are assessed, the critical assets should be
associated with potential risks. For example, a companys IP records can be identified as critical
assets, and data disclosure could be a risk of high severity level.
Step 3: Risk mitigation involves planning and deploying various security controls (such as those
discussed in security controls lessons) that can either mitigate the risks or minimize the impact of
the risks.
(Cont'd)
61
Compliance is the act of adhering to, and demonstrating adherence to, external laws and
regulations as well as to corporate policies and procedures. While building and offering
resources and services to users, it is important to assess compliance against regulations and
demands (discussed earlier). It is also important to review the security and privacy controls that
are in place to ensure that appropriate controls are applied to the highest value and highest risk
assets.
There are primarily two types of policies controlling IT operations in an enterprise that
require compliance: internal policy compliance and external policy compliance. Internal policy
compliance controls the nature of IT operations within an organization. This requires clear
assessment of the potential difficulties in maintaining the compliance and processes to ensure
that this is effectively achieved. External policy compliance includes legal requirements,
legislation, and industry regulations. These external compliance policies control the nature of IT
operations related to the flow of data out of an organization. They may differ based upon the type
of information (for example, source code versus employee records), and business (for example,
medical services versus financial services).
In order to meet compliance requirements, organizations must have compliance management in

place. Compliance management ensures that an organization adheres to relevant policies and
legal requirements. Policies and regulations can be based on configuration best practices and
security rules. These include administrator roles and responsibilities, physical infrastructure
maintenance timelines, information backup schedules, and change control processes. Compliance
management activity includes periodically reviewing compliance enforcement in infrastructure
resources and services. If it identifies any deviation from compliance requirements, it initiates
corrective actions.
62
In the compliance context, audit is a process that determines the validity and reliability
of information about the enforcement of controls presented by an organization. Audit also
provides an assessment of the organizations security controls and their ability to provide
the organizations the logs required to verify the controls. Auditing of the data center
infrastructure can be performed by internal auditors (an auditing team within the
organization) or external auditors (from an external organization).
The auditor makes an independent assessment of the security controls in the information system
to determine if they meet the requirements and are running as originally intended. Key activities
that provide the basis for a security audit of a data center infrastructure include:
Review and evaluate the security controls to detect, prevent, and stop an attack in accordance
with an organizations internal policies. Additionally, review and evaluate physical security.
Determine how identity management is performed for accessing organizations resources and
services.
Determine whether adequate disaster recovery processes are available to provide

uninterrupted access to the users.
Review and evaluate whether appropriate governance processes are available to meet
organizations requirements.
63
This lesson covered governance, risk management, compliance management, and auditing.
64
The Concepts in Practice section covers RSA and VMware security products. These products are
RSA SecurID, RSA Security Analytics, RSA Archer eGRC, RSA Adaptive Authentication,
VMware vCloud Networking and Security, VMware AirWatch.
Note:
For the latest information on RSA products, visit www.emc.com.
65
RSA SecurID two-factor authentication provides an added layer of security to ensure that only
valid users have access to systems and data. RSA SecurID is based on something a user knows (a
password or PIN) and something a user has (an authenticator device). It provides a much more
reliable level of user authentication than reusable passwords. It generates a new, one-time token
code at pre-defined intervals, making it difficult for anyone other than the genuine user to input
the correct token code at any given time. To access their resources, users combine their secret
Personal Identification Number (PIN) with the token code that appears on their SecurID
authenticator device display at that given time. The result is a unique, one-time password used to
assure a users identity.
RSA Security Analytics helps security analysts detect and investigate threats often missed by
other security tools. Security Analytics provides converged network security monitoring and
centralized security information and event management (SIEM). Security Analytics combines big
data security collection, management, and analytics; full network and log-based visibility; and
automated threat intelligence enabling security analysts to better detect, investigate, and
understand threats they often could not easily see or understand before. It provides a single
platform for capturing and analyzing large amounts of network, log, and other data. It also
accelerates security investigations by enabling analysts to pivot through terabytes of metadata,
log data, and recreated network sessions. It archives and analyzes long-term security data
through a distributed computing architecture and provides built-in compliance reports covering a
multitude of regulatory regimes.
(Cont'd)
66
RSA Adaptive Authentication is a comprehensive authentication and fraud detection platform.

Adaptive Authentication is designed to measure the risk associated with a users login and postlogin activities by evaluating a variety of risk indicators. Using a risk and rules-based approach,
the system then requires additional identity assurance, such as out-of-band authentication, for
scenarios that are at high risk and violate a policy. This methodology provides transparent
authentication for organizations that want to protect users accessing websites and online portals,
mobile applications and browsers, Automated Teller Machines (ATMs), Secure Sockets Layer
(SSL), virtual private network (VPN) applications, web access management (WAM) applications,
and application delivery solutions.
VMware vCloud Networking and Security virtualizes networking and security to enable
greater agility, efficiency, and extensibility in the data center. vCloud Networking and Security
delivers software-defined networks and security with a broad range of services, which include a
virtual firewall, virtual private network, load balancing, and VXLAN extended networks.
These services are discussed below:
Virtual firewall: Stateful inspection firewall that can be applied either at the perimeter of the
virtual data center or at the virtual network interface card (vNIC) level directly in front of
specific workloads. The firewall-rule table is designed for ease of use and automation with
VMware vCenter objects for simple and reliable policy creation. Stateful failover enables high
availability for business-critical applications.
(Cont'd)
67
This module covered key security terminologies, key security threats, security controls, and
governance, risk, and compliance.
68
This module focuses on the key functions and processes of the storage infrastructure
management.
Module 16: Managing the Storage Infrastructure
This lesson covers the key characteristics of third platform-centric storage infrastructure
management and the key functions of storage infrastructure management.
The key storage infrastructure components are compute systems, storage systems, and storage
area networks (SANs). These components could be physical or virtual and are used to provide
services to the users. The storage infrastructure management includes all the storage
infrastructure-related functions that are necessary for the management of the infrastructure
components and services, and for the maintenance of data throughout its lifecycle. These
functions help IT organizations to align their storage operations and services to their strategic
business goal and service level requirements. They ensure that the storage infrastructure is
operated optimally by using as few resources as needed. They also ensure better utilization
of existing components, thereby limiting the need for excessive ongoing investment on
infrastructure.
As organizations are driving their IT infrastructure to support the workload of third platform
applications, the storage infrastructure management is also transformed to meet the application
requirements. Management functions are optimized to help an organization to become a social
networking, mobility, big data, or cloud service provider. This module describes the storage
infrastructure management from a service providers perspective.
Traditionally, storage infrastructure management is component specific. The management tools

only enable monitoring and management of specific components(s). This may cause management
complexity and system interoperability issues in a large environment that includes many multivendor components residing in world-wide locations. In addition, traditional management
operations such as provisioning LUNs and zoning are mostly manual. The provisioning tasks often
take days to weeks to complete, due to rigid resource acquisition process and long approval cycle.
Further, the traditional management processes and tools may not support a service oriented
infrastructure, especially if the requirement is to provide cloud services. They usually lack the
ability to execute management operations in agile manner, respond to adverse events quickly,
coordinate the functions of distributed infrastructure components, and meet sustained service
levels. This component specific, extremely manual, time consuming, and overly complex
management is simply not appropriate for the third platform-centric storage infrastructure.
The third platform-centric management functions are different in many ways from the traditional
management and have a set of distinctive characteristics. These are listed on the slide, and the
subsequent slides will describe these characteristics.
The storage infrastructure management in the third platform era has a service-based focus. It is
linked to the service requirements and service level agreement (SLA). Service requirements cover
the services to be created/upgraded, service features, service levels, and infrastructure
components that constitute a service. An SLA is a formalized contract document that describes
service level targets, service support guarantee, service location, and the responsibilities of the
service provider and the user. These parameters of a service determine how the storage
infrastructure will be managed. The slide lists some examples of management functions that are
linked to the service requirements and SLA.
In a third platform-centric environment, more value is given to the software-defined infrastructure

management over the traditional physical component-specific management. Management
functions are increasingly becoming decoupled from the physical infrastructure and moving to
external software controller. As a result of this shift, the infrastructure components are managed
through the software controller. The controller usually has a native management tool for
configuring components and creating services. Administrators may also use independent
management tools for managing the storage infrastructure. Management tools interact with the
controller commonly through the application programming interfaces (APIs).
Management through a software controller has changed the way a traditional storage
infrastructure is operated. The software controller automates and abstracts many common,
repeatable, and physical component-specific tasks, thereby reducing the operational complexity.
This allows the administrators to focus on strategic, value-driven activities such as aligning
services with the business goal, improving resource utilization, and ensuring SLA compliance.
Further, the software controller helps in centralizing the management operations. For example, an
administrator may set configuration settings related to automated storage tiering, thin
provisioning, backup, or replication from the management console. Thereafter, these settings are
automatically and uniformly applied across all the managed components that may be distributed
across wide locations. These components may also be proprietary or commodity hardware
manufactured by different vendors. But, the software controller ensures that the management
operations are independent of the underlying hardware.
Management in the third platform-centric environments provides end-to-end visibility into the
storage infrastructure components and deployed services. The end-to-end visibility of the storage
infrastructure enables comprehensive and centralized management. The administrators can view
the configuration, connectivity, capacity, performance, and interrelationships of all infrastructure
components centrally. Further, it helps in consolidating reports of capacity utilization, correlating
issues in multiple components, and tracking the movement of data and services across the
infrastructure.
Depending on the size of the storage infrastructure and the number of services involved, the
administrators may have to monitor information about hundreds or thousands of components
located in multiple data centers. In addition, the configuration, connectivity, and interrelationships
of components change as the storage infrastructure grows, applications scale, and services are
updated. Organizations typically deploy specialized monitoring tools that provide end-to-end
visibility of a storage infrastructure on a digital dashboard. In addition, they are capable of
reporting relevant information in a rapidly changing and varying workload environment.
Orchestration refers to the automated arrangement, coordination, and management of various

system or component functions in a storage infrastructure. Orchestration, unlike an automated
activity, is not associated with a specific infrastructure component. Instead, it may span multiple
components, located in different locations depending on the size of a storage infrastructure. In
order to sustain in a third platform environment, the storage infrastructure management must
rely on orchestration. Management operations should be orchestrated as much as possible to
provide business agility. Orchestration reduces the time to configure, update, and integrate a
group of infrastructure components that are required to provide and manage a service. By
automating the coordination of component functions, it also reduces the risk of manual errors and
the administration cost.
A purpose-built software, called orchestrator, is commonly used for orchestrating component
functions in a storage infrastructure. The orchestrator provides a library of predefined workflows
for executing various management operations. Workflow refers to a series of inter-related
component functions that are programmatically integrated and sequenced to accomplish a desired
outcome. The orchestrator also provides an interface for administrators or architects to define and
customize workflows. It triggers an appropriate workflow upon receiving a service provisioning or
management request. Thereafter, it interacts with the components as per the workflow to
coordinate and sequence the execution of functions by these components.
The example on the slide illustrates an orchestrated operation that creates a block volume for a
compute system. In this example, an administrator logs on to the management portal and
initiates the volume creation operation from the portal. The operation request is routed to the
orchestrator which triggers a workflow, as shown on the slide, to fulfill this request. The workflow
programmatically integrates and sequences the required compute, storage, and network
component functions to create the block volume. The orchestrator interacts with the softwaredefine storage (SDS) controller to let the controller to carry out the operation according to the
workflow. The SDS controller interacts with the infrastructure components to enable the execution
of component functions such as zoning, LUN creation, and bus rescan. Through the workflow, the
management portal receives the response on the outcome of the operation.
Storage infrastructure management performs two key functions: infrastructure discovery and
operations management. These functions are described next.
10
Infrastructure discovery creates an inventory of infrastructure components and provides

information about the components including their configuration, connectivity, functions,
performance, capacity, availability, utilization, and physical-to-virtual dependencies. It provides
the visibility needed to monitor and manage the infrastructure components. Discovery is
performed using a specialized tool that commonly interacts with infrastructure components
commonly through the native APIs of these components. Through the interaction, it collects
information from the infrastructure components. A discovery tool may be integrated with the
software-defined infrastructure controller, bundled with a management software, or an
independent software that passes discovered information to a management software. Discovery is
typically scheduled by setting an interval for its periodic occurrence. Discovery may also be
initiated by an administrator or be triggered by an orchestrator when a change occurs in the
storage infrastructure.
11
Operations management involves on-going management activities to maintain the storage

infrastructure and the deployed services. It ensures that the services and service levels are
delivered as committed. Operations management involves several management processes. The
slide lists the key processes that support operations management activities. The subsequent
lessons will describe these processes.
Ideally, operations management should be automated to ensure the operational agility.
Management tools are usually capable of automating many management operations. These
automated operations are described along with the management processes. Further, the
automated operations of management tools can also be logically integrated and sequenced
through orchestration.
12
This lesson covered the key characteristics of third platform-centric storage infrastructure
management and the key functions of storage infrastructure management.
13
This lesson covers monitoring, alerting, and reporting in a storage environment.
14
Monitoring forms the basis for performing management operations. Monitoring provides the
performance and availability status of various infrastructure components and services. It also
helps to measure the utilization and consumption of various storage infrastructure resources by
the services. This measurement facilitates the metering of services, capacity planning,
forecasting, and optimal use of these resources. Monitoring events in the storage infrastructure,
such as a change in the performance or availability state of a component or a service, may be
used to trigger automated routines or recovery procedures. Such procedures can reduce
downtime due to known infrastructure errors and the level of manual intervention needed to
recover from them. Further, monitoring helps in generating reports for service usage and trends.
It also helps to trigger alerts when thresholds are reached, security policies are violated, and
service performance deviates from SLA. Alerting and reporting are detailed later in this module.
Additionally, monitoring of the data center environment parameters such as heating, ventilating,
and air-conditioning (HVAC) helps in tracking any anomaly from their normal status.
15
A storage infrastructure is primarily monitored for configuration, availability, capacity,

performance and security. These monitoring parameters are described in the subsequent slides.
16
Monitoring configuration involves tracking configuration changes and deployment of storage

infrastructure components and services. It also detects configuration errors, non-compliance with
configuration policies, and unauthorized configuration changes.
The chart on the right lists configuration changes in the storage infrastructure shown on the left.
These configuration changes are captured and reported by a monitoring tool in real-time. In this
environment, a new zone was created to enable a compute system to access LUNs from one of
the storage systems. The changes were made on the FC switch (device).
17
Availability refers to the ability of a component or a service to perform its desired function during
its specified time of operation. Monitoring availability of hardware components (for example, a
port, an HBA, or a storage controller) or software component (for example, a database instance
or an orchestration software) involves checking their availability status by reviewing the alerts
generated from the system. For example, a port failure might result in a chain of availability
alerts.
A storage infrastructure commonly uses redundant components to avoid a single point of failure.
Failure of a component might cause an outage that affects service availability, or it might cause
performance degradation even though availability is not compromised. Continuous monitoring for
expected availability of each component and reporting any deviation help the administrator to
identify failing services and plan corrective action to maintain SLA requirements.
The slide illustrates an example of monitoring the availability of storage infrastructure
components. As shown on the slide, a storage infrastructure includes three compute systems (H1,
H2, and H3) that are running hypervisors. All the compute systems are configured with two FC
HBAs, each connected to the production storage system through two FC switches, SW1 and SW2.
All the compute systems share two storage ports on the storage system. Multipathing software
has also been installed on hypervisor running on all the three compute systems. If one of the
switches, SW1 fails, the multipathing software initiates a path failover, and all the compute
systems continue to access data through the other switch, SW2. However, due to absence of
redundant switch, a second switch failure could result in unavailability of the storage system.
Monitoring for availability enables detecting the switch failure and helps administrator to take
corrective action before another failure occurs. In most cases, the administrator receives
symptom alerts for a failing component and can initiate actions before the component fails.
18
Capacity refers to the total amount of storage infrastructure resources available. Inadequate
capacity leads to degraded performance or even service unavailability. Monitoring capacity
involves examining the amount of storage infrastructure resources used and usable such as the
free space available on a file system or a storage pool, the numbers of ports available on a switch,
or the utilization of allocated storage space to a service. Monitoring capacity helps an
administrator to ensure uninterrupted data availability and scalability by averting outages before
they occur. For example, if 90 percent of the ports are utilized in a particular SAN fabric, this
could indicate that a new switch might be required if more compute and storage systems need to
be attached to the same fabric. Monitoring usually leverages analytical tools to perform capacity
trend analysis. These trends help to understand future resource requirements and provide an
estimation of the time required to deploy them.
The slide provides an example that illustrates the importance of monitoring the NAS file system
capacity. If the file system is full and no space is available for applications to perform write I/O, it
may result in application/service outage. Monitoring tools can be configured to issue a notification
when thresholds are reached on the file system capacity. For example, when the file system
reaches 66 percent of its capacity, a warning message is issued, and a critical message is issued
when the file system reaches 80 percent of its capacity. This enables the administrator to take
actions to provision additional LUNs to the NAS and extend the NAS file system before it runs out
of capacity. Proactively monitoring the file system can prevent service outages caused due to lack
of file system space.
19
Performance monitoring evaluates how efficiently different storage infrastructure components and
services are performing and helps to identify bottlenecks. Performance monitoring measures and
analyzes behavior in terms of response time, throughput, and I/O wait time. It identifies whether
the behavior of infrastructure components and services meets the acceptable and agreed
performance level. This helps to identify performance bottlenecks. It also deals with the utilization
of resources, which affects the way resources behave and respond. For example, if a VM is
experiencing 80 percent of processor utilization continuously, it suggests that the VM may be
running out of processing power, which can lead to degraded performance and slower response
time. Similarly, if the cache and controllers of a storage system is consistently over utilized, it
may lead to performance degradation.
The slide provides an example that illustrates the importance of monitoring performance on iSCSI
storage systems. In this example, compute systems H1, H2, and H3 (with two iSCSI HBAs each)
are connected to the storage system through Ethernet switches SW1 and SW2. The three
compute systems share the same storage ports on the storage system to access LUNs. A new
compute system running an application with a high work load must be deployed to share the
same storage port as H1, H2, and H3.
Monitoring storage port utilization ensures that the new compute system does not adversely
affect the performance of the other compute systems. In this example, utilization of the shared
storage port is shown by the solid and dotted lines in the graph. If the port utilization prior to
deploying the new compute system is close to 100 percent, then deploying the new compute
system is not recommended because it might impact the performance of the other compute
systems. However, if the utilization of the port prior to deploying the new compute system is
closer to the dotted line, then there is room to add a new compute system.
20
Monitoring a storage infrastructure for security includes tracking unauthorized access, whether
accidental or malicious, and unauthorized configuration changes. For example, monitoring tracks
and reports the initial zoning configuration performed and all the subsequent changes. Another
example of monitoring security is to track login failures and unauthorized access to switches for
performing administrative changes. IT organizations typically comply with various information
security policies that may be specific to government regulations, organizational rules, or deployed
services. Monitoring detects all operations and data movement that deviate from predefined
security policies. Monitoring also detects unavailability of information and services to authorized
users due to security breach. Further, physical security of a storage infrastructure can also be
continuously monitored using badge readers, biometric scans, or video cameras.
This slide illustrates the importance of monitoring security in a storage system. In this example,
the storage system is shared between two workgroups, WG1 and WG2. The data of WG1 should
not be accessible by WG2 and vice versa. A user from WG1 might try to make a local replica of
the data that belongs to WG2. If this action is not monitored or recorded, it is difficult to track
such a violation of security protocols. Conversely, if this action is monitored, a warning message
can be sent to prompt a corrective action or at least enable discovery as part of regular auditing
operations.
21
An alert is a system-to-user notification that provides information about events or impending

threats or issues. Alerting of events is an integral part of monitoring. Alerting keeps
administrators informed about the status of various components and processes for example,
conditions such as failure of power, storage drives, memory, switches, or availability zone, which
can impact the availability of services and require immediate administrative attention. Other
conditions, such as a file system reaching a capacity threshold, an operation breaching a
configuration policy, or a soft media error on storage drives, are considered warning signs and
may also require administrative attention.
Monitoring tools enable administrators to define various alerted conditions and assign different
severity levels for these conditions based on the impact of the conditions. Whenever a condition
with a particular severity level occurs, an alert is sent to the administrator, an orchestrated
operation is triggered, or an incident ticket is opened to initiate a corrective action. Alert
classifications can range from information alerts to fatal alerts. Information alerts provide useful
information but do not require any intervention by the administrator. The creation of a zone or
LUN is an example of an information alert. Warning alerts require administrative attention so that
the alerted condition is contained and does not affect service availability. For example, if an alert
indicates that a storage pool is approaching a predefined threshold value, the administrator can
decide whether additional storage drives need to be added to the pool. Fatal alerts require
immediate attention because the condition might affect the overall performance or availability.
For example, if a service fails, the administrator must ensure that it is returned quickly.
As every IT environment is unique, most monitoring systems require initial set-up and
configuration, including defining what types of alerts should be classified as informational,
warning, and fatal. Whenever possible, an organization should limit the number of truly critical
alerts so that important events are not lost amidst informational messages. Continuous
monitoring, with automated alerting, enables administrators to respond to failures quickly and
proactively. Alerting provides information that helps administrators prioritize their response to
events.
22
Like alerting, reporting is also associated with monitoring. Reporting on a storage infrastructure
involves keeping track and gathering information from various components and processes that
are monitored. The gathered information is compiled to generate reports for trend analysis,
capacity planning, chargeback, performance, and security breaches. Capacity planning reports
contain current and historic information about the utilization of storage, file systems, database
tablespace, ports, etc. Configuration and asset management reports include details about device
allocation, local or remote replicas, and fabric configuration. This report also lists all the
equipment, with details, such as their purchase date, lease status, and maintenance records.
Chargeback reports contain information about the allocation or utilization of storage infrastructure
resources by various users or user groups. Performance reports provide current and historical
information about the performance of various storage infrastructure components and services as
well as their compliance with agreed service levels. Security breach reports provide details on the
security violations, duration of breach and its impact.
Reports are commonly displayed like a digital dashboard, which provide real time tabular or
graphical views of gathered information. Dashboard reporting helps administrators to make
instantaneous and informed decisions on resource procurement, plans for modifications in the
existing infrastructure, policy enforcement, and improvements in management processes.
23
Chargeback is the ability to measure storage resource consumption per business unit or user
group and charge them back accordingly. It aligns the cost of deployed storage services with
organizations business goals such as recovery of cost, making a profit, justifying new capital
spending, influencing consumption behaviors by the business units, and making IT more service
aware, cost conscious and accountable. To perform chargeback, the storage usage data is
collected by a billing system that generates chargeback report for each business unit or user
group. The billing system is responsible for accurate measurement of the number of units of
storage used and reports cost/charge for the consumed units. The example on the slide shows the
assignment of storage resource as services to two business units Payroll_1 and Engineering_1
and presents a sample chargeback report.
In this example, each business unit is using a set of compute systems that are running
hypervisor. The VMs hosted on these compute systems are used by the business units. LUNs are
assigned to the hypervisor from the production storage system. Storage system-based
replication technology is used to create both local and remote replicas. A chargeback report
documenting the exact amount of storage resources used by each business unit is created by a
billing system. If the unit for billing is GB of raw storage, the exact amount of raw space (usable
capacity plus protection provided) configured for each business unit must be reported.
(Contd)
24
Consider that the Payroll_1 unit has consumed two production LUNs, each 50 GB in size.
Therefore, the storage allocated to the hypervisor is 100 GB (50 + 50). The allocated storage for
local replication is 100 GB and for remote replication is also 100 GB. From the allocated storage,
the raw storage configured for the hypervisor is determined based on the RAID protection that is
used for various storage pools. If the Payroll_1 production LUNs are RAID 1-protected, the raw
space used by the production volumes is 200 GB. Assume that the local replicas are on
unprotected LUNs, and the remote replicas are protected with a RAID 5 configuration, then 100
GB of raw space is used by the local replica and 125 GB by the remote replica. Therefore, the
total raw capacity used by the Payroll_1 unit is 425 GB. The total cost of storage provisioned for
Payroll_1 unit will be $2,125 (assume cost per GB of raw storage is $5). The Engineering_1 unit
also uses two LUNs, but each 100 GB in size. Considering the same RAID protection and per unit
cost, the chargeback for the Engineering_1 unit will be $3,500. The slide provides a sample
chargeback report for both Payroll_1 and Engineering_1 units.
Chargeback reports can be extended to include a pre-established cost of other resources, such as
the number of switch ports, HBAs and storage system ports, and service level requested by the
users. Chargeback reports enable metering of storage services, providing transparency for both
the provider and the consumer of the utilized services.
25
This lesson covered monitoring, alerting, and reporting in a storage environment.
26
This lesson covers configuration management, change management, capacity management,

performance management, availability management, incident management, problem
management, and security management.
27
Configuration management is responsible for maintaining information about configuration items

(CI). CIs are components such as services, process documents, infrastructure components
including hardware and software, people, and SLAs that need to be managed in order to deliver
services. The information about CIs include their attributes, used and available capacity, history
of issues, and inter-relationships. Examples of CI attribute are the CIs name, manufacturer
name, serial number, license status, version, description of modification, location, and inventory
status (for example, on order, available, allocated, or retired). The inter-relationships among CIs
in a storage infrastructure commonly include service-to-user, storage pool-to-service, storage
volume-to-storage pool, storage system-to-storage pool, storage system-to-SAN switch, and data
center-to geographic location.
All information about CIs is usually collected and stored by the discovery tools in a single
database or in multiple autonomous databases mapped into a federated database called a
configuration management system (CMS). Discovery tools also update the CMS when new CIs are
deployed or when attributes of CIs change. CMS provides a consolidated view of CI attributes and
relationships, which is used by other management processes for their operations. For example,
CMS helps the security management process to examine the deployment of a security patch on
VMs, the problem management to resolve a connectivity issue, or the capacity management to
identify the CIs affected on expansion of a storage pool.
28
With the changing business requirements, the ongoing changes to the CIs become almost daily
task. Relevant changes could range from the introduction of a new service, to modification of an
existing services attributes, to retirement of a service; from replacing a SAN switch, to expansion
of a storage pool, to a software upgrade, and even to a change in process or procedural
documentation. Change management standardizes change-related procedure in a storage
infrastructure to respond to the changing business requirements in an agile way. It oversees all
changes to the CIs to minimize adverse impact of those changes to the business and the users of
services.
Change management typically uses an orchestrated approval process that helps making decision
on changes in an agile manner. Through an orchestration workflow, the change management
receives and processes the requests for changes. Changes that are at low risk, routine, and
compliant to predefined change policies go through the change management process only once to
determine that they can be exempted from change management review thereafter. After that,
these requests are typically treated as service requests and approved automatically. All other
changes are presented for review to the change management team. The change management
team assesses the potential risks of the changes, prioritizes, and makes a decision on the
requested changes.
29
Capacity management ensures adequate availability of storage infrastructure resources to provide

services and meet SLA requirements. It determines the optimal amount of storage required to
meet the needs of a service regardless of dynamic resource consumption and seasonal spikes in
storage demand. It also maximizes the utilization of available capacity and minimizes spare and
stranded capacity without compromising the service levels.
Capacity management tools are usually capable of gathering historical information on storage
usage over a specified period of time, establishing trends on capacity consumption, and
performing predictive analysis of future demand. This analysis serves as input to the capacity
planning activities and enables the procurement and provisioning of additional capacity in the
most cost effective and least disruptive manner.
Adding new nodes to a scale-out NAS cluster or an object-based storage system is an example of
capacity management. Addition of nodes increases the overall processing power, memory, or
storage capacity. Enforcing capacity quotas for users is another example of capacity
management. Provisioning a fixed amount of space for their files restricts users from exceeding
the allocated capacity. Other examples include creating and expanding a storage pool, setting a
threshold for the maximum utilization and amount of oversubscription allowed for each
storage pool, forecasting the usage of file system, LUN, and storage pool, and removing unused
resources from a service for their reassignment to another resource-crunched service.
Capacity management team uses several methods to maximize the utilization of capacity. Some
of the common methods are over-commitment of processing power and memory, data
deduplication and compression, automated storage tiering, and use of converged network such as
an FCoE SAN.
30
This example illustrates the expansion of a NAS file system using an orchestrated workflow. The
file system is expanded to meet the capacity requirement of a compute cluster that accesses the
file system.
In this example, an administrator initiates a file system expansion operation from the
management portal. The operation request is transferred to the orchestrator that triggers a
change approval and execution workflow. The orchestrator determines whether the request for
change needs to be reviewed by change management team. If the request is preapproved, it is
exempted from change management review. If not, the orchestrated workflow ensures that the
change management team reviews and approves/rejects the request.
If the file system expansion request is approved, the orchestrator interacts with the SDS
controller to invoke the expansion. Thereafter, the SDS controller interacts with the storage
infrastructure components to add the required capacity to the file system. The orchestrated
workflow also invokes the discovery operation which updates the CMS with information on the
modified file system size. The orchestrator responds by sending updates to the management
portal appropriately following completion or rejection of the expansion operation.
31
Performance management ensures the optimal operational efficiency of all infrastructure

components so that storage services can meet or exceed the required performance level.
Performance-related data such as response time and throughput of components are collected,
analyzed, and reported by specialized management tools. The performance analysis provides
information on whether a component meets the expected performance levels. These tools also
proactively alert administrators about potential performance issues and may prescribe a course of
action to improve a situation.
Performance management team carries out several activities to address performance-related
issues and improve the performance of the storage infrastructure components. For example, to
optimize the performance levels, activities on the compute system include fine-tuning the volume
configuration, database design or application layout, resource allocation to VMs, workload
balancing, and multipathing configuration. The performance management tasks on a SAN include
implementing new ISLs and aggregating links in a multiswitch fabric to eliminate performance
bottleneck. The storage system-related tasks include separating sequential and random I/Os to
different spindles, selecting an appropriate RAID type for a storage pool, and changing storage
tiering policy and cache configuration, when the performance management is concerned.
32
Availability management is responsible for establishing a proper guideline based on the defined
availability levels of services. The guideline includes the procedures and technical features
required to meet or exceed both current and future service availability needs at a justifiable cost.
Availability management also identifies all availability-related issues in a storage infrastructure
and areas where availability must be improved. The availability management team proactively
monitors whether the availability of existing services and components is maintained within
acceptable and agreed levels. The monitoring tools also help administrators to identify the gap
between the required availability and the achieved availability. With this information, the
administrators can quickly identify errors or faults in the infrastructure components that may
cause future service unavailability.
Based on the service availability requirements and areas found for improvement, the availability
management team may propose new business continuity (BC) solutions or changes in the existing
BC solutions. For example, when a set of compute systems is deployed to support a service or
any critical business function, it requires high availability. The availability management team
proposes redundancy at all levels, including components, data, or even site levels. This is
generally accomplished by deploying two or more HBAs per system, multipathing software, and
compute clustering. The compute systems must be connected to the storage systems using at
least two independent fabrics and switches that have built-in redundancy and hot-swappable
components. The VMs running on these compute systems must be protected from hardware
failure/unavailability through VM failover mechanisms. Deployed applications should have built-in
fault resiliency features. The storage systems should also have built-in redundancy for various
components and should support local and remote replication. RAID-protected LUNs should be
provisioned to the compute systems using at least two front-end ports. In addition, multiple
availability zones may be created to support fault tolerance at the site level.
33
The example on the slide illustrates a remote replication architecture for providing multi-site data
protection. The remote protection is provided using a continuous data protection (CDP) appliance
that is available at each site. During replication, all the write I/Os from the compute systems to
the storage system are duplicated at each site. The CDP appliance sends a copy of the write I/Os
to the remote site(s) over WAN connection. In this example, the design requires the data from
primary site A to be replicated to four other sites (site B, C, D, and E). This allows an organization
to maintain five copies of data in five different locations. Further, the site A should be used for
both recovering data and restarting business operations in the event of an outage at any of the
other sites. Hence, the design exhibits the replication of data from each of the site B, C, D, and E
to site A.
34
An incident is an unplanned event such as an HBA failure or an application error that may cause
an interruption to services or degrade the service quality. Incident management is responsible for
detecting and recording all incidents in a storage infrastructure. It investigates the incidents and
provides appropriate solutions to resolve the incidents. It also documents the incident history with
details of the incident symptoms, affected services, components and users, time to resolve the
incident, severity of the incident, description of the error, and the incident resolution data. The
incident history is used as an input for problem management (described next).
Incidents are commonly detected and logged by incident management tools. They also help
administrators to track, escalate, and respond to the incidents from their initiation to closure.
Incidents may also be registered by the users through a self-service portal, emails, or a service
desk. The service desk may consist of a call center to handle a large volume of telephone calls
and a help desk as the first line of service support. If the service desk is unsuccessful in providing
solutions against the incidents, they are escalated to other incident management support groups
or to problem management. The slide provides a sample list of incidents that are captured by an
incident management tool.
The incident management support groups investigate the incidents escalated by the incident
management tools or service desk. They provide solutions to bring back the services within an
agreed timeframe specified in the SLA. If the support groups are unable to determine and correct
the root cause of an incident, error-correction activity is transferred to problem management. In
this case, the incident management team provides a temporary solution (workaround) to the
incident; for example, migration of a storage service to a different storage pool in the same data
center or in a different data center. During the incident resolution process, the affected users are
kept apprised of the incident status.
35
A problem is recognized when multiple incidents exhibit one or more common symptoms.
Problems may also be identified from a single significant incident that is indicative of a single
error for which the cause is unknown, but the impact is high. Problem management reviews all
incidents and their history to detect problems in a storage infrastructure. It identifies the
underlying root cause that creates a problem and provides the most appropriate solution and/or
preventive remediation for the problem. If complete resolution is not available, problem
management provides solutions to reduce or eliminate the impact of a problem. In addition, the
problem management proactively analyzes errors and alerts in the storage infrastructure to
identify impending service failures or quality degradation. It solves errors before they turn out to
be an incident or a problem.
Incident and problem management, although separate management processes, require
automated interaction between them and use integrated incident and problem management tools.
These tools may help an administrator to track and mark specific incident(s) as a problem and
transfer the matter to problem management for further investigation. Alternatively, these tools
may automatically identify incidents that are most likely to require root cause analysis. Further,
these tools may have analytical ability to perform root cause analysis based on various alerts.
They search alerts that are indicative of problems and correlate these alerts to find the root
cause. This helps to resolve problems more quickly.
36
The example, shown on the slide, illustrates the resolution of a problem that causes intermittent
service availability. The problem is detected by an integrated incident and problem management
tool deployed in the storage infrastructure. The problem is recognized by correlating multiple
incidents that pertain to the same availability-related issue. The integrated incident and problem
management tool performs root cause analysis which reveals that a faulty line card of an Ethernet
director is the root cause of the problem. The tool also logs the problem for administrative action.
Administrators of the problem management team can view the problem details including the root
cause recorded by the integrated incident and problem management tool. They determine the
remedial steps to correct the root cause. In this case, the administrators decide to replace the
faulty line card. For that, they generate a request for change. Upon obtaining approval from the
change management, they ensure that the line card is replaced by the implementation engineers.
Thereafter, the problem management team closes the problem.
37
Security management ensures the confidentiality, integrity, and availability of information in a

storage infrastructure. It prevents the occurrence of security-related incidents or activities that
adversely affect the infrastructure components, management processes, information, and
services. It also meets regulatory or compliance requirements (both internal and external) for
protecting information at reasonable/acceptable costs. External compliance requirements include
adherence to the legal frameworks such as U.K. Data Protection Act 1998, U.K. Freedom of
Information Act 2000, U.S. Health Insurance Portability and Accountability Act 1996, and EU Data
Protection Regulation. Internal regulations are imposed based on an organizations information
security policies such as access control policy, bring-your-own-device (BYOD) policy, and policy on
the usage of cloud storage.
Security management is responsible for developing information security policies that govern the
organizations approach towards information security management. It establishes the security
architecture, processes, mechanisms, tools, user responsibilities, and standards needed to meet
the information security policies in a cost-effective manner. It also ensures that the required
security processes and mechanisms are properly implemented.
Security management team performs various activities to prevent unauthorized access and
security breaches in a storage infrastructure. For example, the security management team
manages the user accounts and access policies that authorize users to use a service. Further, the
access to data and services is controlled at multiple levels (defense in depth) reducing the risk of
a security breach if a protection mechanism at one level gets compromised. Applications and
databases are also scanned periodically to identify vulnerabilities and provide protection against
any threats. The security management activities in a SAN include configuration of zoning to
restrict an unauthorized HBA from accessing specific storage system ports and providing
mechanisms to transport encrypted data. Similarly, the security management task on a storage
system includes LUN masking that restricts a compute system from accessing a defined set of
LUNs.
38
This lesson covered configuration management, change management, capacity management,

performance management, availability management, incident management, problem
management, and security management.
39
The Concepts in Practice section covers five products that facilitate the storage infrastructure
management. These products are: EMC ViPR SRM, EMC Service Assurance Suite, EMC UIM,
VMware vRealize Operations, and VMware vRealize Orchestrator.
Note:
40
EMC ViPR SRM is a management software for automated monitoring and reporting of both
traditional and software-defined storage infrastructure. It provides visibility to the relationships
and topology from applications hosted on virtual or physical machines down to the LUNs. It also
enables administrators to analyze performance trends, capacity utilization, and configuration
compliance. With this insight, it helps administrators to optimize storage capacity through the
alignment of application workload to the right storage tier, capacity planning, and chargeback
reporting.
EMC Service Assurance Suite offers a combination of management tools, including Smarts and
M&R (formerly known as Watch4net), to perform IT operations in a software-defined data center.
It discovers infrastructure components and details information about each one, including
configuration and inter-relationship among components. It detects and correlates events related
to availability, performance, and configuration status of infrastructure components that may occur
due to problems. It also identifies the root causes of the problems and risk conditions. By quickly
finding the root causes and risks, it helps administrators to proactively resolve issues before they
impact the services levels.
EMC Unified Infrastructure Manager (UIM) is the management software for Vblock and
VSPEX systems (converged infrastructures). It enables unified management across compute,
storage, and network components. UIM discovers all infrastructure components and provides a
topology view of the components. UIM automatically configures and provisions resources to
support services based on predefined policy. Further, it helps identifying the root cause of a
problem quickly by displaying all the alerts in a centralized alert console and correlating these
alerts.
41
VMware vRealize Operations is a management tool that automates some of the key
management operations in a storage infrastructure. It identifies potential performance, capacity,
and configuration issues and helps remediate those issues before they become problems. It
optimizes the usage of capacity and performs capacity trend analysis. It also collects configuration
data, verifies configuration compliance with predefined policies, and recommends/triggers
necessary actions to remediate policy breaches. This enables organizations to enforce and
maintain the conformance with configuration standards, regulatory requirements, and security
hardening guidelines. Further, it provides end-to-end visibility across storage infrastructure
components including application-to-component mapping in a single console.
VMware vRealize Orchestrator is an orchestration software that helps to automate and
coordinate the service delivery and operational functions in a storage infrastructure. It comes with
a built-in library of pre-defined workflows as well as a drag-and-drop feature for linking actions
together to create customized workflows. These workflows can be launched from the VMware
vSphere client, from various components of VMware vCloud Suite, or through various triggering
mechanisms. vRealize Orchestrator can execute hundreds or thousands of workflows
concurrently.
42
This module covered the characteristics and functions of storage infrastructure management. This
module also covered the key storage infrastructure management processes.
43
44
This course covered the data center infrastructure and third platform technologies cloud, big
data, social, and mobile. It also covered various types of intelligent storage systems blockbased, file-based, object-based, and unified. In addition, the course covered software-defined
storage. Further, it covered various storage networking technologies such as FC SAN, IP SAN, and
FCoE SAN. Moreover, this course covered the business continuity and archiving solutions, security
threats and controls, and the key management processes in a storage infrastructure.
45
This concludes the Training. Thank you for your participation.
46

Student Guide

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Student Guide

Uploaded by

Copyright:

Available Formats

Information Storage and

Welcome to Information Storage and Management Version 3.

Revision Date: August 2015

Copyright 2015 EMC Corporation. All rights reserved.

Module: Course Introduction

Copyright 2015 EMC Corporation. All rights reserved.

Module: Course Introduction

Copyright 2015 EMC Corporation. All rights reserved.

Module: Course Introduction

Copyright 2015 EMC Corporation. All rights reserved.

Module: Course Introduction

Copyright 2015 EMC Corporation. All rights reserved.

Module: Course Introduction

Copyright 2015 EMC Corporation. All rights reserved.

Module: Course Introduction

Copyright 2015 EMC Corporation. All rights reserved.

Module: Course Introduction

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Organizations have become increasingly information-dependent in the twenty-first century, and

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Optical storage devices: For example, Blu-ray, DVD, and CD.

Storage is a core component in an organizations IT infrastructure. Various factors such as the

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

IT equipment: It includes equipment such as compute systems, storage systems, network

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Availability: Availability of information as and when required should be ensured. Unavailability

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Monitoring: It is a continuous process of gathering information on various resources in the

Reporting: It is a process of collating and presenting the monitored parameters such as

Planning: It is a process of estimating the amount of IT resources required to support

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 1: Introduction to Information Storage

Copyright 2015 EMC Corporation. All rights reserved.

Module 2: Third Platform Technologies

Copyright 2015 EMC Corporation. All rights reserved.