7/9/2012 3:51:00 AM

Cloud Testing
To ensure a successful cloud computing strategy, you must be able to: Manage performance and availability across the entire cloud service delivery chain Monitor cloud application performance from the end-user perspective Test your cloud applications prior to deployment Monitor your cloud applications after they go into production

The performance and availability of cloud applications can have a dramatic impact on user adoption and revenue. Monitoring and testing the performance of those applications requires uninterrupted visibility across the entire application delivery chain i.e. from your data center, through the Internet and cloud service providers to your end users own device and browser

Note: Traditional data center monitoring tools simply wont work in the cloud. You need to monitor and test your cloud applications from the only perspective that really matters: your end users. Cloud Testing has four major objectives: To assure the quality of cloud-based applications deployed in a cloud, including their functional services, business processes, and system performance as well as scalability based on a set of application-based system requirements in a cloud

To validate software as a service (SaaS) in a cloud environment, including software performance, scalability, security and measurement based on certain economic scales and pre-defined SLAs.

To check the provided automatic cloud-based functional services, for example auto-provisioned functions

To test cloud compatibility and inter-operation capability between SaaS and applications in a cloud infrastructure, for example, checking the APIs of SaaS and their cloud connectivity to others i.e. another SaaS with in the same/different cloud or end user interface .

Below Table shows the detailed tasks and comparative view among different parties: Test Type Testing focuses Cloud/SaaS-Oriented Testing inside a Cloud Online ApplicationBased Testing on a Cloud Service Function Testing GUI-based and API based service functions Testing SaaS/Cloud based service functions inside a cloud Testing onlinebased application service functions on a cloud Integration Testing SaaS interactions and Cloud connections Vendor-specific component and service integration inside a private/public cloud API and Connectivity Testing API interfaces and connectivity protocols (HTTPS, REST, SOAP, RMI) SaaS/Cloud API & connectivity testing in a cloud Integration between online clients and backend servers on a cloud Testing usercentered service APIs and connectivity on a cloud Performance & Scalability Testing Performance and scalability based on a SLA SaaS/Cloud performance and scalability testing in a cloud based on the given SLA Security SaaS/Application SaaS/Cloud security User-oriented System-level end-to-end User-oriented application performance and scalability testing on a cloud End-to-end system-level performance and scalability inside/on/over cloud based on a given SLA Testing application service APIs and connectivity over Clouds - End-to-end application integration over clouds - Integration with legacy systems over clouds Testing cloud-based application service functions over a cloud infrastructure Cloud-Based Application Testing over Clouds

Testing

data, processes, functions, and user privacy

features and user privacy in a cloud

security and privacy on a cloud

security over clouds

Interoperability & Compatibility Testing

Validate different client interfaces and technologies and diverse compatibilities on different platforms and browsers

Testing Cloud/ SaaS compatibility, connectivity protocols and UI/client technologies inside a cloud

Testing usercentered interoperability, compatibility of platforms/ OS/browsers, and client technologies on a cloud

Testing application compatibility, end-to-end interoperability and application connectivity to legacy systems over clouds

Regression Testing

Changed & impacted SaaS/Cloud service features and related APIs/ connectivity

Cloud/SaaS-oriented regression testing inside a cloud

User-centered revalidation on a cloud

End-to-end application system regression over clouds

1.3 Cloud Testing VS. Conventional Software Testing: Internet-Based Software Testing (i.e. Distributed/Web-Based System Infrastructure) Primary Testing Objectives - Assure the quality of system functions and performance based on the given specifications - Check usability, compatibility, interoperability. - Assure the quality of functions and performance of SaaS , Clouds, and applications by leveraging a cloud environment - Assure the quality of cloud elasticity & scalability based a SLA Testing as a service - In-house internal software testing as engineering tasks Real-time on-demand testing service offered by a third-party - Online testing service based on a predefined SLA Testing and Execution Time Testing Environment - A pre-fixed and configured test environment in a test lab. with purchased hardware and/or software - Offline test execution in a test lab. - Testing a product before its delivery - On-demand test execution by third-parties; - Online test execution in a public cloud; - Offline test execution in a private cloud - An open public test environment with diverse computing resources - A scalable private test environment in a test lab. Testing Costs - Required hardware costs and software (license) costs - Engineering costs in a test process - Based on a pre-defined service-levelagreement (SLA) - TaaS and Cloud testing service costs (payas-you-test) - Engineering costs in SaaS/Cloud/application vendors Test Simulation - Simulated online user access - Simulated online traffic data - Virtual/online user access simulation - Virtual/online traffic data simulation Cloud-Based Software Testing

Function Validation

- Validating component functions and system functions as well as service features

- SaaS/Cloud service functions, end-to-end application functions - Leveraged functions with legacy systems - SaaS-based integration in a cloud - SaaS integration between clouds

Integration Testing

- Function-based integration - Component-based integration

- Architecture-based integration - Interface/connection integration

- Application-oriented end-to-end integration over clouds - Enterprise-oriented application integration between SaaS/Cloud and with legacy systems

Security Testing

Aim to the following targets: Function-based security features User privacy Client/server access security Process access security Data/message integrity

Aim to the following targets: SaaS/Cloud security features, including monitor and measurement User privacy in diverse web clients End-to-end application security over clouds SaaS/Cloud API and connectivity security Security testing with virtual/real-time tests in a vendors cloud

Scalability & Performance Testing

- Performed a fixed test environment - Apply simulated user access, ,messages, and test data - Online monitor and evaluation

- Performed in a scalable test environment based on a SLA - Apply both virtual and real-time online test data - Online monitor, validation, and measurement

Characteristics of SaaS
Software as a Service (SaaS) is defined as software that is deployed over the internet With SaaS, a provider licenses an application to customers either as a service on demand, through a subscription, in a pay-as-you-go model, or (increasingly) at no charge when there is opportunity to generate revenue from streams other than the user, such as from advertisement or user list sales. SaaS applications are designed for end-users, delivered over the web. Following are the characteristics of SaaS: Web access to commercial software Software is managed from a central location Software delivered in a one to many model Users not required to handle software upgrades and patches Application Programming Interfaces (APIs) allow integration between different pieces of software

SaaS Attributes:
Integration with External Applications: Simple Object Access Protocol (SOAP)-based Service
Oriented Architecture (SOA), Extract Transform Load (ETL) and On Line Analytical Processing (OLAP) Application Programming Interfaces (APIs)

Manageability: Multi-tenant architecture to support clients from a single instance in order to reduce
the costs of infrastructure, hosting and management

Performance: Distributed data caching and code optimization tools for improving performance and
response time

Scalability: Meta-database and load balancing for scalability

Security: Multi-tiered, multi-layered, role-based security model. Typically improves due to centralization
of data, increased security-focused resources, etc., but raises concerns about loss of control over certain sensitive data. Security is often as good as or better than traditional systems, in part because providers

are able to devote resources to solving security issues that many customers cannot afford. Providers typically log accesses, but accessing the audit logs themselves can be difficult or impossible

Time-to-Market: Distributed Agile methodology and platform (GlobalLogic Velocity) to accelerate

time-to-market and provide shorter release cycles

Usability: AJAX-based APIs to provide interactive, professional-looking Graphical User Interfaces

(GUIs) supported by a dedicated team of usability experts

Compatibility: Portability experts to provide consistent support across a variety of browser platforms

Availability: 24/7 in-house support services to ensure uptime and continuous availability

Expertise on Open Source: Use of tools to reduce total cost of ownership

Reliability: Improves through the use of multiple redundant sites, which makes it suitable for business
continuity and disaster recovery. Nonetheless, most major cloud computing services have suffered outages and IT and business managers are able to do little when they are affected.

Sustainability: Comes about through improved resource utilization, more efficient systems, and
carbon neutrality. Nonetheless, computers and associated infrastructure are major consumers of energy

Maintainability: Usually this includes System/Integration testing, Performance testing, and User
Acceptance testing cycles. The client must be confident the new version of the software works in their environment AND with all of the interfacing applications. The process is significantly streamlined with SaaS. The client is relieved of the burden of testing the new software release in their environment, as the SaaS provider handles this for them. Note: If your implementation of a vendor's SaaS application is integrated with one or more external application (be they on-premise or SaaS), you must work closely with the vendor to ensure that no APIs upon which your integrations depend are being deprecated as part of this release. If you are dependent

upon deprecated APIs, you must re-write your interfaces to the new API or your intra-SaaS application business process will fail.

Interoperability: Cloud computing architectures are a heterogeneous blend of technologies and

platforms. The various software applications residing in the cloud do not exist in isolation. They must be able to communicate and exchange information transparently, irrespective of the technologies used to implement them. Thus, interoperability among the cloud SaaS is a relevant and significant issue in cloud computing. Interoperability between SaaS is possible using Web standards and middleware (possibly hosted in Cloud)

Adaptability: The entire way the software runs can be tailored for individual organizations
and to let any company define the hierarchies specific to them, and yet the overall software works out of a single code base

Customizable: In Software-as-a-Service (SaaS) delivery model a vendor maintains a single

application instance, which is used by multiple tenants. However, due to changing business requirements tenants expect customizations. Providing such customizations is trivial to retain tenants but a challenge to the vendor due to multi-tenancy.

The table below lists all the Attributes of SaaS Application:

User experience Data

Usability

Responsiveness Efficiency Performance Personalizable

State State full Stateless Stability Application constraints

o o o o

User interface

Database constraints Persistence Online/Offline Structure Unstructured Indexed Searchable Transaction management

o o o o o

Graphical Interactive Distributed Textural None

Interaction model

o o o

Device SaaS Online

Security

Maintainability

Emergency hot fix or breach management Security procedures Trust relationship with platform Applications security model Data flow Malicious code Access controls Remote access

Available skill sets Language support (dev) Application standards Technology implementation Application-code complexity and volume Configuration management Operational management Flexible

Identity Cryptography Auditing Authentication/Authorization model

Technology

Affordability

Scalability

Resource cost Development Available skills Software enhancements cost Licensing Postproduction hardware Decommissioning Initial hardware

Replication Caching Pooling Software load balancing Scale out Scale up Hardware load balancing

Conformability

Availability

Auditable Regulatory Standards

Technology/Configuration/ Implementation to support availability

Uptime requirement

Portability

Reliability

Cross-platform Within platform

Configuration management Startup and automatic recovery System performance Recovery procedures and methods Load balancing Fault tolerance

Distributability

Interoperability

Local Geo-distributed

Communications and data usage Integration impacts Architecture compatibility Ease integration (APIs)

Extensibility

Reusability

Meta-model Configurable

Distributable and reusable Modularity Hierarchy Code abstraction

How we test our SaaS QA Platform?

The cloud is defined by its service model, deployment model and usage: Mostly
cloud applications are based on SaaS. They are Software as a Service solutions that run completely on cloud infrastructure and platforms. Hence testing of SaaS applications is completely different from testing the traditional applications. They need to be tested on three levels: namely the infrastructure, the platform and the application itself. The usage of standard services of applications also means a change for system testing. In principle, its not different from testing any other application; it requires merging different techniques used in daily basis.

Cloud/SaaS-oriented testing: This type of testing activities usually is performed inside a cloud by
engineers of cloud/SaaS vendors. The primary objective is to assure the quality of the provided service functions offered in a cloud (or a SaaS program). These engineers must go through unit testing, integration, system function validation, regression testing and cross platform (compatibility) testing, as well as performance and scalability evaluation. Since clouds and SaaS usually provide certain service APIs and connectivity interfaces to their customers, it is required task for engineers to validate these APIs and connectivity in a cloud environment. In addition, testing cloud-based or SaaS-based security services and functional features must be tested. Furthermore, performance testing and scalability evaluation in a cloud is very important and critical to cloud/SaaS vendors because this assures the quality of cloud elasticity to support SaaS and cloud services inside a cloud.

Testing Categories: Following are list of testing techniques that can be used to test SAAS
platform at different phases:

Test Category
Business Testing

Testing techniques
Manual/Automation functional Testing Exploratory Testing End to End business workflow testing Manual/Automated regression testing Data integration and data migration testing Checklist validation

Security Testing

Application Security Testing Network Security testing User Access and Roles testing Data security integrity testing Compliance testing Identity Federation mechanism testing

Performance Testing

Scalability testing Volume Testing Availability testing Reliability testing Load testing for single instance Load testing in a instance loaded environment

Compatibility Testing

Multi-browser and OS compatibility Localization testing Accessibility testing from remote locations Internalization testing Interface backward compatibility testing

Live Testing

Disaster recovery testing Statefull scenario testing Live upgrade testing

Saas Attribute Testing

Multi-tenancy isolation testing Api Integration testing Billing mechanism testing

Functional Testing Checklist:

For any application we make sure that the functionality works as expected. This is the standard functional testing to validate if the app is doing what it is supposed to do. Conduct rigorous Manual tests as per defined test plans, keeping the end user in mind Conduct Exploratory tests based on existing or new test cases Conduct Browser compatibility testing to check performance of the application on different web browsers Conduct Regression testing on every release, minor upgrade, an integration or data migration. Automate Functional and Regression tests Conduct tests in target environment whether it is your data center or the Cloud. Conduct reliability testing to find the total defects of the application and thus reduce the number of failures, during real time deployment.

Multi-platform support/Compatibility Testing:

We can use combination different browsers versions and operating systems to perform cross-platform testing. We have to use browsers and operating systems majorly used by the end users to find issues that end user may encounter with his/her browser/operating system combinations.

Load over different clouds:

Application / system stability is a major factor as the user count is expected to be in multiples of hundreds. SaaS based application needs to handle large amounts of users and we dont have the luxury of re-booting or going down once in a while. Conduct load testing under normal as well as peak load conditions in multiple environments i.e. to determine the limits

Stress over different clouds:

Due to the cloud characteristics, it is imperative to identify issues as system is tested to breaking points maximum expected capacity or often beyond to 2x, 3x,nx expected usage. Pushing systems to maximum load capacity and beyond i.e. Exceed Break points

Capacity Testing:
Being hosted in a cloud environment it is prudent to determine maximum capacity for current or future hardware, bandwidth or other needs or to validate that installed hardware and network will support expected usage scenarios i.e. Plan for the Future Conduct scalability tests to determine the capacity of the application to scale up or down as per requirements

Availability Testing:
Conduct availability testing for a planned period of time and 24/7

Volume Testing:
Conduct volume testing for your data

Performance/Latency over different clouds:

Measure response times and isolate issues related to specific steps or actions while system is subjected to increasing load from different locations and multi user operations. Measuring response time variance over load and time

Reliability/Soak Testing:
Measuring performance degradation over longer periods at varying load levels i.e. Reliability over time

Remote Access and Usage:

We make sure that all users regardless if they come from the US, Holland, India, Argentina, or Australia can work with the system with good response times. There are many emulators that can help test this in your lab

Failover Testing/Disaster recovery & rollback procedures:

Another testing task coming to us from the IT side of the house. Here we run 2 main scenarios: o System down that needs to be brought up quickly (with the same machines or with new ones that require installation and configuration) o Rollbacks to the last known stable version, including data.

Verify Redundancy

Application/Infrastructure security testing:

The goal of this is to test the underlying infrastructure and security of the app: Test the security of the SaaS application for typical web application security issues such as HTTP header injection, Cross Site scripting (XSS), SQL Injection etc. Test security of the network where SaaS application is being deployed Test possible scenarios of security attacks/threats Test the application with respect to access privileges with the corresponding job roles (especially in a multi-tenant environment) Test the security, integrity & accessibility of test data (especially in a multi-tenant environment ) Determine situations that could make the SaaS application vulnerable Test compliance with Payment Card Industry Data Security Standard (PCI Compliance) Maintain logs of security warnings, errors and requests from unreliable sources

Security Management: Dedicated or shared firewall, firewall management; VPNs; intrusion detection &
IDS management; systems software hardening; security audit / vulnerability scanning and notification .

Vulnerability AssessmentsWhat are your weaknesses? Our vulnerability scans are based on a variety of compliance regulations such as:
Payment Card Industry (PCI) Data Security Standard Health Insurance Portability and Accountability Act (HIPAA) Sarbanes-Oxley (SOX) Gramm-Leach-Bliley Act (GLBA) Federal Information Security Management Act (FISMA) Statement on Auditing Standards Number 70 (SAS70)

Managed Firewalls: Test with different types of firewall devices such as From Cisco to Checkpoint to
Sonic wall and implement the best practices

Patch Management: Each customer environment is different and we stay aware of how change will affect
it. We work with end users to establish a patching strategy that meets end user needs. Every patch is analyzed. A risk assessment is made to be sure that your environment is not only safer by applying the patch but won't be adversely affected.

Intrusion Detection Systems (IDS): We use both Network Intrusion Detection Systems (NIDS) and
Host-based Intrusion Detection Systems (HIDS) to ensure that the bad guys stay out. With systems powered by Cisco, Checkpoint and OSSEC we perform log analysis, file integrity checking, policy monitoring, root kit detection, real-time alerting and active response.

Security Policies:We know that security is a top priority. Whether it's user admin or system builds, our
documented procedures are built on years of experience and industry best practices for security and compliance.

Incident Response:If you're learning the hard way at your own or a third-party location and experience an
attack, the vendor team can leap into action to control and repair the damage. We understand how to contain the breach, develop an action plan to systematically verify integrity of your network and all your devices, then recommend and help implement solutions to protect from future attacks.

GRC (Governance Risk Compliance) testing: Devise a unique comprehensive testing strategy for
compliance with standards like PCI and government regulations

Scalability over different clouds: This assures the quality of cloud elasticity to support SaaS and
cloud services inside a cloud.

Operational Testing:
This area is intended for the operations team whose objective is to make sure the apps are working fine, and take care of customer service & billing. Usually, there are tools that are built as part of the product which help the operational team members to monitor, track and analyze for issues. The areas to look for: Application, Services, App Server, Platforms (OS), Databases and Data Center Level Logs/Alerts/Warnings/Errors for functionality and performance. Billing and Customer Support Tools, especially for integration

Integration and API Testing:

Success of SaaS apps lie in how well you have thought of scenarios where third party developers can build their own apps using your APIs, and add value to your product. So testing all the APIs for functionality, security, usability, performance and completeness of documentation is critical to make them successful: SaaS based application interactions and cloud connections with different client interfaces, database servers SaaS based integration in a Cloud SaaS integration between Clouds

 Application oriented end to end integration over clouds

Enterprise oriented application integration between Saas/Cloud and with Legacy systems

Usability over different clouds: Test the Responsiveness, efficiency, Performance and
Personalizable

Cloud based Unit testing: Unit testing on different clouds Cloud-based application integration: Application integration testing on different clouds End-to-end system function testing: System testing on different Clouds Cloud based System Integration testing: SaaS usually provide certain service APIs and
connectivity interfaces to their customers, it is required task for engineers to validate these APIs and connectivity in a cloud environment. Saas based application interactions and cloud connections with different API interfaces and connectivity protocols(HTTPS, REST, SOAP and RMI).

Live updates and deployments:

Here is something we think about in regular applications:

How do you deploy the system while it is still running, and if needed how do you minimize down-time for the users? This is something that is developed and handled by our IT department, but since the delivery and update of the product is part of the overall user experience we test it constantly.

Internationalization/I18N:

Since our platform is used by people around the world we make sure

that we support the use International Characters.

Use Case Scenarios:

Ethernet Fabric by validating QoS of high density 10/40/100Gb Ethernet top of rack, end of row and WAN optimization switches Storage Networks by ensuring QoS of Fibre Channel and Fiber Channel over Ethernet storage

networking devices and storage systems Application Networking by testing QoS of Firewalls, IPS, WAN accelerators, Proxy Servers, SSL

VPNs, etc Cloud Virtualization by benchmarking QoS of virtual switching and virtual appliances from blade

servers to any point in the Ethernet and Storage fabric Exchange documents, such as purchase orders, with any business partner over the Web with

B2B integration technologies, while eliminating the costs of proprietary EDI solutions. Automate any mission-critical process such as Order-to-Cash, giving you more visibility into your

business, whether you represent marketing, sales, IT or support.

Interoperability Between Local and Global ADC Functions:

Cloud balancing is based on making routing decisions based on a combination of local and global variables. This requires interoperability between local and global ADC functions. Standards-based APIs may eventually emerge that will facilitate the cross-vendor exchange of cloud balancing variables. In the mean time, in those situations in which multiple ADC vendors are involved, IT organizations will need to take advantage of the APIs supported by each vendor in order to achieve an integrated set of variables to use to make routing decisions. Another option that IT organizations have is to adopt a single vendor strategy for both local and global ADC functions. The feasibility of implementing a single vendor strategy across the enterprise and one or more IaaS providers is enhanced if the ADC is available in a virtual appliance form factor. Focuses on different client interfaces and connecting to legacy systems

Synchronizing Data between Cloud Sites:

In order for an application to be executed at the data center that is selected by the cloud balancing system, the target server instance must have access to the relevant data. In some cases, the data can be accessed from a single central repository. In other cases, the data needs to co-located with the application. The co-location of data can be achieved by migrating the data to the appropriate data center, a task that typically requires highly effective optimization techniques. In addition, if the data is replicated for simultaneous use at multiple cloud locations, the data needs to be synchronized via active-active storage replication, which is highly sensitive to WAN latency.

Challenges:
Most organizations report impediments to SaaS testing like short notice periods for QA notification, frequent testing of live upgrades, short validation cycle times, impact on multiple subscriber organizations, privacy violations, errors due to rapid addition of new features, time taken for data migration, concerns over data security & integrity etc. cloud the obvious benefits of SaaS testing.

1.

Handling Changes through Frequent Releases: Every time the Application is upgraded,
the users have to understand the impact of the change, validate it against the existing system & ensure that the impact on the existing features of the application is minimal. Managing and executing all these activities within a short time span (1-2 weeks) is challenging. When SaaS upgrades involve interface upgrading, compatibility and integration issues across old and new interfaces crop up for the subscribers. Live upgrades being simulated or tested on the SaaS application impedes the activity of the existing users.

2.

Security Testing: Maintaining data security, accessibility & integrity on a single SaaS application
across multiple tenants. To understand individual privacy requirements, privilege levels, behavioral patterns and provide adequate privacy to the data can be a daunting task. Cloud computing security challenges fall into three broad categories: Data Protection: Securing your data both at rest and in transit User Authentication: Limiting access to data and monitoring who accesses the data Disaster and Data Breach Contingency Planning

3.

Integration Challenges: When subscribers integrate their internal enterprise applications with
SaaS, inbound and outbound data integration validations from client networks to the SaaS providers is needed. In such cases it is very difficult to conduct thorough validation simultaneously ensuring

100% data security and privacy

4.

Data Migration Issues: Data migration across different SaaS applications or from other
applications to SaaS can be challenging in terms of time taken for understanding the requirements and the exhaustive integration validation processes

5.

Licensing: The SaaS app licensing may vary by functionality, usage (such as volume of
transactions or amount of specific data) or # of named/concurrent users. All this needs to be tested across every release.

6.

Performance testing: Successfully modeling the most-used business transactions, application

usage and user mix may require greater diligence than an on-premise application.

Risks:
Accountability and Data Risk User Identity Federation Regularity Compliance Business continuity and Resiliency User Privacy & Secondary Usage of Data Service & Data Integration Multi-tenancy & Physical Security Incidence Analysis & Forensics Infrastructure Security Non-production Environment Exposure

1. Accountability: In traditional data center, the owning organization(End user) is accountable for
security at all layers i.e. Application/ Database/Computing/Network/Storage layers. You can outsource hosted services but you cannot outsource accountability. In a cloud, who is accountable for security at these layers? Data can be stored anywhere at different geographical locations: How sensitive is the data? (Informal blogs, public network sharing posts, public news, New group messages, Health Records, Criminal Records, Credit History and Payroll) Who owns the data? Is data encrypted single Vs multiple keys Data Mitigation:

Logical isolation of the data of multiple consumers Provider fully destroys deleted data Multiple encryption keys

2. User Identity Federation:

Security Risks Managing Identities across multiple providers Less control over user lifecycle (off-boarding) User experience Mitigations Federated Identity Auth for backend integrations Tighter user provisioning controls

3. Regulatory Compliance:
Data that is perceived to be secure in one country may not be perceived secure in another country/region. European Union (EU) has very strict privacy laws and hence data stored in US may not comply with those EU laws (US Patriot Act allows federal agencies limitless powers to access any corporate data etc) Lack of transparency in the underlying implementations makes it difficult for data owners to demonstrate compliance (SOX/HIPAA etc.) Lack of consistent standards and requirements for global regulatory compliance data governance can no longer be viewed from a point-to-point data flow perspective but rather a multi-point to multipoint. Mitigations Apply risk management framework, case-by-case basis Define data protection requirements and SLAs Provider / Consumer agreement to a pre-defined RACI model

4. Business Continuity and Resiliency:

Lack of know-how and capabilities need Cloud provider may be acquired by a consumers competitor Monetary losses due to an outage Mitigations Contract defines Recovery Time Objectives and monetary penalty for downtime

Cloud providers Business Continuity program certified to standard such as BS 25999

5. User Privacy & Secondary Usage of Data: Users Privacy of my data:

Address, Email, (Personally Identifiable Information) Health, personal financial info Personal Details (email, IMs,.)

Providers Keep Revenue Up/ Cost Down:

Push out the liabilities to user via Privacy and Acceptable Use Policy Build Additional Services on users behavior (targeted advertisements) e.g. Google Email, banner adv. Do minimal to achieve compliance Keep their social applications more open (increased adoption)

User personal data mined or used (sold) without consent-Targeted Advertisements, third parties

User Privacy data transferred across jurisdictional borders No opt out features for user (user can not delete data) Lack of individual control on ensuring appropriate usage, sharing and protection of their personal information.

Law Obligation for providers Key escrows to law agencies Subpoena

Mitigations Policy Enactment o o o Privacy and Acceptable Usage Consent (Opt In / Opt Out) Policy on Secondary Usage

De-identification of personal Information Encrypted storage Terms of Service with providers o o Responsibility on compliance Geographical affinity

6. Service and Data Integration:

Data traverses through the internet between end users and cloud data centers. How secure the integrations are? Mitigations Encryption keys single Vs multiple Secured protocols

7. Multi-tenancy and Physical Security:

Security Risks Inadequate Logical Separations Co-mingled Tenant Data Malicious or Ignorant Tenants Cross-Tenant Attacks Side channel Attacks Scanning other tenants DoS

Shared Service-single point of failuresWordpressOutage June 2010 Uncoordinated Change Controls and Misconfigs 100sof tenants (CNN,..) down in multitenant environment. Uncoordinated Change in database

Mitigations

Performance Risks

Architecting for Multi-Tenancy Data Encryption (per tenant key management) Controlled and coordinated Change Management Transparency/Audit-ability of Administrative Access Regular Third Party Assessments Virtual Private Cloud (VPC)

8. Incidence Analysis & Forensic Support:

Complex integration and dynamics in cloud computing present significant challenges to timely diagnosis and resolution of incidents such as: Malware detection and Immediate intrusion response to mitigate the impact.

Implications to Traditional Forensics? (Seizing equipment and analysis on media/data recovered) International differences in relevant regulations Mitigations Comprehensive logging Without compromising Performance Dedicated Forensic VMImages

 Infrastructure Security:
Malicious parties are actively scanning the internet for Vulnerable Applications or Services such as: Active Unused Ports Default Passwords Default Configurations Data

Mitigations Segregation of duties and role based administrative privileges Third party audits and app vulnerability assessments Tiered architecture with appropriate security controls between them Hardening(Networks, OS, Apps)

9. Non-production Environment Exposure:

Non-Production Environments are for design, development, and test activities internally within an organization: Typical non-prod environment use generic authentication credentials Security flaws Data copied to non-prod from its production equivalent High risk of an unauthorized user getting access to the non production environment

Mitigations Use multi layers of authentication Non-prod data is not identical to production Dont use cloud for developing a highly sensitive app in the cloud

Over Coming Challenges of Saas Testing: Challenges

Testing frequent SaaS upgrades Short notice period (1-2 weeks) for a QA notification to validate the application Business knowledge for effective testing of configurable and non-configurable components Gain comprehensive and competent knowledge on the configurable and non-configurable components of SaaS

Mitigation Plan
The use of automation tools for building regression suites brings in business value and helps quickly validate the impact of upgrades

applications Any non-configurable upgrade/change to the application will need to be assessed thoroughly since this will have an impact on all SaaS subscribers Though the configurable upgrade/change would not impact every client, it is advised to validate the impact of these changes as well Validating interface compatibility The backward compatibility of a SaaS interface needs to be validated to ensure that the organizations do not have to make any changes at their end, and can continue using SaaS applications as before Compliance with government regulations and other standards Data security and privacy Devise a unique comprehensive testing strategy for compliance with standards like PCI and government regulations Validation of strong encryptions is needed to ensure data security Data security and privacy would need to be thoroughly validated amongst multiple tenant scenarios to ensure that there are no loop holes Testing access controls, multi-privileges for security Perform access control and multi-privilege tests with users that have varied roles, different privileges and are executing unique activities (simulating real life usage scenarios) Data integration - inbound & outbound Test data transfers between an organizations network and SaaS applications. - Also, measure, compare and validate the performance of data migrations between SaaS applications and an organizations network Simulating live upgrade testing Live upgrade tests should be carried out in cloud based preproduction environments Use automation tools to simulate the scenario of multiple concurrent users logged on to a current SaaS version. Conduct live upgrades in cloud based environments Use automation tools to validate the accuracy of the upgrade Optimization of testing that is common to the impacted core and non-core areas of SaaS when getting customized Create a test strategy to test the core product of SaaS Create a standard suite of automated test cases to validate the core SaaS product Create a map/grid of the core and the non-core areas of the SaaS application that are most likely to be impacted during

customization Run a regression suite selecting the tests associated with the impacted areas Data migration from the existing system to SaaS application Identify the different data sources in the existing system that need to be migrated to the SaaS application. Select tools that will help in the data migration and in the post migration validation Frequent releases of feature rich SaaS applications increases the time taken for testing, owing to the significant number of pages to be covered Rapid addition of new features to the core SaaS product to meet new customer demands and to stay competitive. However, every change is a potential security bug/ performance issue Create an automated test library for SaaS applications that help reduce the associated testing effort that comes with each frequent release Formulate a comprehensive strategy for testing the SaaS applications with test tools that cover functional, performance and security requirements Maintain a test repository of results, performance benchmarks and access privilege grids, which would facilitate faster validation Execute comprehensive tests with automated tools that cover the functional and nonfunctional requirements. Conduct a continual impact analysis of requirements and regularly update the test library to help minimize risks.

Implementation of Saas Testing:

Now, lets take a look at the SaaS testing process itself. SaaS testing begins with assessing the functional and nonfunctional requirements for the SaaS application, including business, operational and non-functional needs. Once this is done, the focus then moves into understanding the usage pattern of the application. This particular set factors in the variations due to geographies, peak periods and network latencies across regions. A test plan would need to be developed to include all components of the SaaS application. The plan would also have details on how these components would be tested and the resources needed to carry out the same. Once the test plan is approved, the QA team would prepare test cases, test suites and eventually get the test data ready. The QA environment is then validated for its preparedness for SaaS Testing. After the assessments confirm the preparedness, test data is populated in the QA environment through data migration from the existing system. Then the test team focuses on the automated test suite generation for functional and non-functional validations. This would be followed by test execution, reporting, publishing and finally culminate with the issuance of the SaaS readiness certification. See figure 3 (The SaaS Testing Process) to get details on all steps and processes required for ensuring a systematic and successful SaaS Testing.

Assess the functional & non-functional test requirements

Understand the usage patterns Test strategy & plan Prepare test case & suite Prepare test environment Populate test data Generate automated test suite for functional & non-functional test requirements Execute SaaS testing, report & publish SaaS Certification

Benefits of Saas Testing

There are multiple benefits that SaaS testing delivers to organizations:

Reduces effort required and go-to-market time associated in procurement, upgrades, renewals, contracts, maintenance and deployment Lowers costs associated with test tools, test environments, maintenance and upgrades. Helps focus on the SaaS application configuration rather than on provisioning for the application and associated infrastructure requirements Significantly reduces CAPEX associated with setting up of environment for SaaS application, helping convert the same into OPEX Reduces shelf ware risk of SaaS application and testing tools associated with the validation of the application Testing costs are reduced by almost one third as the need to test client server installations, multi-platform backend support, multiple versions of upgrades and backward compatibility is completely eliminated Using SaaS testing tools are not system or machine dependent. For example, any local machine connected to a cloud network can be used for performance testing of the SaaS application This helps save effort and overhead expenses associated with the installation, configuration and maintenance of additional machines for enabling SaaS testing tools

Conclusion:
SaaS testing focuses on ensuring high quality across the application, its cloud characteristics and SaaS attributes. It also includes testing for security, privacy, accessibility and standards compliance as well. A thorough understanding of the SaaS application, the customer specific implementation, components that are configurable and nonconfigurable and how any change or upgrade would impact the application is absolutely needed to ensure a successful SaaS application testing. The automated validation of the functional and non-functional requirements of the SaaS application helps shorten the release cycle of frequent SaaS application upgrades and releases. The data integration/ migration pertaining to SaaS applications would also need thorough validation. The key to successful

SaaS testing is putting together the right test strategy, automating the tests for functional and non-functional requirements and leveraging best practices that would help maximize the investments in SaaS and in turn help the organization achieve the intended business outcome.