Processor Analysis: Generational Performance of Intel Processors in an Expedient Cloud

Benchmarking different Intel Processors in Expedient Generation 2, Generation 3, and Generation 4 Offerings

By Cloud Spectator July 2013

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

Table of Contents
Introduction Executive Summary Floating Point Math Integer Math Compression Encryption Physics Appendix A: Methodology Appendix B: Terms & Definitions Appendix C: Test Descriptions About Cloud Spectator 2 3 4 5 6 7 8 9 9 10 11

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

Introduction
Performance Testing Cloud Spectator monitors the processor, RAM, storage, and internal network performance of over 20 of the worlds most well-known IaaS services to understand important aspects of virtual server performance. Tests are run at least three times per day, 365 days per year to capture variability in addition to performance level. Tests are chosen based on reliability and practicality. The goal is to provide an indication of where certain providers perform well relative to others. This can give consumers an indication of which services would be best for their application(s) by understanding the performance of provider resources most critical to that application. Singular benchmarks alone should not be the only deciding factor in the provider selection process. Feature sets, configuration matches, pricing and ancillary services such as security, compliance, and disaster recovery should always factor into any vendor selection process. However, performance is a very important piece to the puzzle. The Comparison For the purpose of generating this document, Cloud Spectator measured the processor performance of Expedients virtual machines, across 2nd, 3rd and 4th Generation offerings. Each offering uses different Intel processor models. The goal was to understand how Intel CPU performance has increased over time, using Expedients offerings as a practical cloud environment to test in. Over a period of fifteen (15) days, Cloud Spectator ran the same benchmark tests across Expedients 2nd, 3rd and 4th Generation offerings. Tests were run from July 15th through July 29th, 2013. Cloud Spectator accounted for performance capability and stability for each environment to understand the value each one delivers to its users. Tests were run on 8GB servers with 4 vCPUs (for more details, see Methodology section in Appendix A). Statistical measures used to illustrate improvement included: average, standard deviation, coefficient of variation (CV), 15-day high and 15-day low. Please see Appendix B for definitions of each statistical measure.

The goal was to understand how Intel processor performance has evolved over time, using Expedients offerings as a practical cloud environment to test in.
Performance testing and benchmarking of cloud computing platforms is a complex task, compounded by the differences between providers and the use cases of cloud infrastructure users. IaaS services are utilized by a large variety of industries, and performance metrics cannot be completely understood by simply representing cloud performance with a single value. When selecting a cloud computing provider, IT professionals consider many factors: feature-sets, cost, security, location and more. However, performance is a key issue that drives many others including cost. Processors The processors below are used in each given offering: Expedient Generation 2: Intel Xeon Processor E5520 2.27GHz Expedient Generation 3: Intel Xeon Processor X5650 2.67GHz Expedient Generation 4: Intel Xeon Processor E5-2670 2.60GHz

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

Executive Summary
Findings
Newer Processors Deliver Significantly More Performance Generation 4 offerings performed better than Generation 2 offerings by an average of 46%. The largest margin of improvement was seen in the Floating Point Math Test, representing a value of 67%. Other notable improvements were seen in the Physics and Encryption Tests, representing values of 47% and 54%, respectively. Generation 3 offerings performed better than Generation 2 offerings by an average of 26%. Generation 3 performance improved most notably over Generation 2 in the Encryption Test, by 52%. All other tests represented a 15-21% improvement from Generation 2 to Generation 3. Generation 4 offerings performed better than Generation 3 offerings by an average of 17%. The most significant performance improvement margin, 38%, came in the Floating Point Math Test, while the Physics Test was second, with 27%. Other tests improved by an average of 7%.

Newer Processors Have Improved Stability Coefficients of variation (CVs), representing variability, decreased from Generation 2 to Generation 4 by an average of 34%. Most notable stability improvements were seen in the Floating Point Math, Compression and Encryption Tests with improvements of 48%, 52% and 65%, respectively. Averages in each scenario are much closer to the highs than lows, meaning that performance levels are more often higher than the mean. This means that the lows are responsible for most of the variability. Customers should expect a high level of performance most of the time. Highs are an average of 1.3 standard deviations from the mean, while lows are an average of 4.5 standard deviations from the mean.

Processor Performance
Why Processor Performance Matters
In a public cloud environment, resources are shared across virtual machines within the same physical server. Hardware components such as the processor are virtualized and usually shared across VMs. While modern-day physical processors may accomplish tasks at a desirable rate, virtualized processors, which may or may not receive a dedicated cores performance, may be less powerful. As the cloud industry progresses, though, with the introduction of newer cloud offerings with the latest processor technology, users experience a significant performance benefit while paying a comparable price for the IaaS service. This improvement in performance is highlighted within the tests below, which are run on congruent machines that only differ in processor generation.

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

Floating Point Math

Floating Point Math and Integer Math tests are synthetic benchmarks commonly used in gauging and comparing CPU performance because the operations used make up the basic operations in all computer software. The Floating Point Math test uses the same calculations as the Integer Math test, except using numbers with decimals, which are commonly used in modern-day applications. The floating point math performance level and variability showed a significant improvement in the more modern generations. Generation 3 average performance improved by 21% relative to Generation 2. Generation 4 average performance improved on Generation 3 by 38%. Overall, this represents a 67% performance increase from Generations 2 to 4. Variability also decreased by a significant amount, 48%, from Generation 2 to Generation 4. Generation 3 had a variability measure that was 25% lower than Generation 4, however, the Generation 4 15-day low was still 20% higher than the Generation 3 15-day high. These results were in line with what was expected after Intels introduction of Advanced Vector Extensions (AVX) technology in Generation 4 (Intel Xeon Processor E5-2670 2.60GHz). Intel Advanced Vector Extensions (Intel AVX) is an instruction set extension designed for applications that are floating point intensive. Intel AVX results in better management of data, improving performance through wider vectors, new extensible syntax, and rich functionality.

Floating Point Math

6000

Millions of Operations per Second

5000

4000

3000

2nd Generation 3rd Generation

2000

4th Generation

1000

0 July 16 July 17 July 18 July 19 July 20 July 21 July 22 July 23 July 24 July 26 July 27 July 28 July 15 July 25 July 29

The graph above compares Intel CPU floating point math performance on Windows servers between Expedients 2nd, 3rd and 4th Generation offerings. The graph shows the scores from the PassMark floating point math benchmark test for each provider over a period of 15 days, with two (high and low) data points shown for each day. For more information on the PassMark floating point math test, please see Appendix C.

PROVIDER 2nd Generation 3rd Generation 4th Generation

AVERAGE 3,220 3,902 5,367

STANDARD DEVIATION 149 71 127

CV 4.6% 1.8% 2.4%

15-DAY HIGH 3,348 4,004 5,493

15-DAY LOW 2,461 3,670 4,786

Note: numbers expressed in millions of operations per second. Copyright Cloud Spectator, LLC 2013. All rights reserved 4

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

Integer Math
Floating Point Math and Integer Math tests are synthetic benchmarks commonly used in gauging and comparing CPU performance because the operations used make up the basic operations in all computer software. The Integer Math test runs mathematical operations of addition, subtraction, multiplication, and division of whole numbers to give an indication of raw CPU throughput. Integer math performance improved by a large margin from Generation 2 to Generation 3 and by a much smaller margin from Generation 3 to Generation 4. Generation 3 saw a significant performance improvement of 20% over Generation 2. Generation 4 average performance improved slightly, by 6%, over Generation 3. The total performance improvement from Generation 2 to Generation 4 was 27%. Performance variability for all three generations was very low with the highest CV being 0.9%.

Integer Math
9000

Millions of Operations per Second

8000 7000 6000 5000 4000 3000 2000 1000 0 July 16 July 17 July 18 July 19 July 20 July 21 July 22 July 23 July 24 July 26 July 27 July 28 July 15 July 25 July 29 2nd Generation 3rd Generation 4th Generation

The graph above compares Intel CPU integer math performance on Windows servers between Expedients 2nd, 3rd and 4th Generation offerings. The graph shows the scores from the PassMark integer math benchmark test for each provider over a period of 15 days, with two (high and low) data points shown for each day. For more information on the PassMark integer math test, please see Appendix C.

PROVIDER 2nd Generation 3rd Generation 4th Generation

AVERAGE 5,945 7,139 7,563

STANDARD DEVIATION 49 40 67

CV 0.8% 0.6% 0.9%

15-DAY HIGH 5,996 7,235 7,628

15-DAY LOW 5,667 7,019 7,134

Note: numbers expressed in millions of operations per second.

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

Compression
Compressing files into smaller blocks of data is a common task used in software applications like Email and file backups. Compressing the same file in the same manner across different machines reveals a 36% improvement for this CPU-intensive task from the Expedient Generation 2 offering with the Intel Xeon Processor E5520 to the Expedient Generation 4 offering with the Intel Xeon Processor E5-2670. The compression results reveal a clear improvement from Generation 2 to 3, and from 3 to 4. The Expedient Generation 3 offering saw a 19% improvement in compression performance from Generation 2, with decreased performance variability. From Generation 2 to 3, the CV decreased by 48%, indicating more predictable and steady performance levels. Generation 4 displays further CPU compression performance over Generation 3. Performance increased by 14% from Generation 3 to 4, and 36% from Generation 2 to 4. Performance variability continued to decrease with newer models as Generation 4 saw a 6% decrease from Generation 3, and a 52% decrease from Generation 2.

Compression
8000 7000

KB Processed per Second

6000 5000 4000 3000 2000 1000 0 July 16 July 17 July 18 July 19 July 20 July 21 July 22 July 23 July 24 July 26 July 27 July 28 July 15 July 25 July 29 2nd Generation 3rd Generation 4th Generation

SAMPLE

The graph above compares Intel CPU compression performance on Windows servers between Expedients 2nd, 3rd and 4th Generation offerings. The graph shows the scores from the PassMark compression benchmark test for each provider over a period of 15 days, with two (high and low) data points shown for each day. For more information on the PassMark compression test, please see Appendix C.

PROVIDER 2nd Generation 3rd Generation 4th Generation

AVERAGE 5,271 6,275 7,158

STANDARD DEVIATION 165 103 111

CV 3.1% 1.6% 1.5%

15-DAY HIGH 5,433 6,464 7,277

15-DAY LOW 4,517 5,729 6,728

Note: numbers expressed in kilobytes processed per second.

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

Encryption
This encryption test encrypts random blocks of data using different encryption techniques. The machine will also create a hash of the data, which ensures that the encrypted data is not compromised. From a performance view, encryption uses many of the algorithms used in the Integer and Floating Point math test, as well as more complex ones such as to the power of functions. Encryption is used in many Internet-based applications to ensure privacy, including Internet browsers and communications tools. Generations 3 and 4 show a significant improvement in encryption performance over Generation 2. Generation 3 displays a 52% improvement in encryption performance over Generation 2. Generation 4 slightly improved in average encryption performance, increasing by 1% compared to Generation 3, but still representing a 54% improvement over Generation 2. Generation 3 variability decreased significantly, by 59%, from Generation 2. The Generation 4 CV decreased 13% compared to Generation 3, which was part of a 65% decrease in CV from Generation 2 to Generation 4. These results were in line with what was expected after Intels introduction of Intel AES-NI technology present in Generations 3 and 4 (Intel Xeon Processor X5650 2.67GHz and Intel Xeon Processor E5-2670 2.60GHz, respectively). Intel Advanced Encryption Standard New Instructions (Intel AES-NI) introduces new instructions for enhanced security and speed while implanting some intensive sub-steps of the AES algorithm into the hardware for faster execution.

Encryption
1200

MB Transferred per Second

1000

800

600

2nd Generation 3rd Generation

400

4th Generation

200

0 July 16 July 17 July 18 July 19 July 20 July 21 July 22 July 23 July 24 July 26 July 27 July 28 July 15 July 25 July 29

The graph above compares Intel CPU encryption performance on Windows servers between Expedients 2nd, 3rd and 4th Generation offerings. The graph SAMPLE shows the scores from the PassMark encryption benchmark test for each provider over a period of 15 days, with two (high and low) data points shown for each day. For more information on the PassMark encryption test, please see Appendix C.

PROVIDER 2nd Generation 3rd Generation 4th Generation

AVERAGE 624 951 961

STANDARD DEVIATION 23 14 13

CV 3.7% 1.5% 1.3%

15-DAY HIGH 649 974 975

15-DAY LOW 498 909 891

Note: numbers expressed in megabytes transferred per second. Copyright Cloud Spectator, LLC 2013. All rights reserved 7

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

Physics
Using the Tokamak Physics Engine, the Physics benchmark performs collision calculations and visual rendering simultaneously. The Tokamak Physics Engine is a high performance, real-time physics library designed specifically for games and has a built-in collision functionality, which is used for the test. Average performance level in the Physics Test improved steadily with more modern generations. Generation 3 average performance improved by 15% over Generation 2 while Generation 4 improved by 27% over Generation 3. This brings the total physics performance improvement from Generation 2 to Generation 4 to 47%. Variability decreased by 20% from Generation 2 to Generation 4.

Physics
450 400 350

Frames per Second

300 250 200 150 100 50 0 July 16 July 17 July 18 July 19 July 20 July 21 July 22 July 23 July 24 July 26 July 27 July 28 July 15 July 25 July 29 2nd Generation 3rd Generation 4th Generation

The graph above compares Intel CPU physics calculation performance on Windows servers between Expedients 2nd, 3rd and 4th Generation offerings. The graph shows the scores from the PassMark physics benchmark test for each provider over a period of 15 days, with two (high and low) data points shown for each day. For more information on the PassMark physics test, please see Appendix C.

PROVIDER 2nd Generation 3rd Generation 4th Generation

AVERAGE 272 314 399

STANDARD DEVIATION 12 9 14

CV 4.4% 3.0% 3.5%

15-DAY HIGH 288 327 417

15-DAY LOW 212 277 356

Note: numbers expressed in frames per second.

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

Appendix
Appendix A Methodology Server Setup:
The following server configurations were set up on each environment. The Intel CPU model that each environment uses is also detailed. Expedient Generation 2
http://www.expedient.com/products/cloudcomputing/public.php CPU: Intel Xeon Processor E5520 2.27GHz OS: Windows Server 2008 R2 vCPUs: 4 RAM: 8GB Disk: 50GB

Expedient Generation 3
http://www.expedient.com/products/cloudcomputing/public.php CPU: Intel Xeon Processor X5650 2.67GHz OS: Windows Server 2008 R2 vCPUs: 4 RAM: 8GB Disk: 50GB

Expedient Generation 4
http://www.expedient.com/products/cloudcomputing/public.php CPU: Intel Xeon Processor E5-2670 2.60GHz OS: Windows Server 2008 R2 vCPUs: 4 RAM: 8GB Disk: 50GB

Tests Used:
The benchmarks used to test the environments listed in this report are derived from PassMark Software. The following Passmark Tests were used to measure CPU performance: Compression, Encryption, Floating Point Math, Integer Math, and Physics. For the descriptions and details of each test, please see Appendix C.

Timeframe:
The test period ranged from 7/15/2013 though 7/29/2013.

Data Collection:
Performance data for each test was collected 10 times per day, every day throughout the above testing periods. Data shown in the graphs only display the highest and lowest points of each day. Statistical measures in the charts were calculated using all data points to obtain more accurate estimations of variability. Cloud Spectator obtains cloud servers by purchasing the server space directly from the providers as any user would. For certain providers, the client may reimburse Cloud Spectator for the server space needed for data collection relevant to that active project. Cloud Spectator collects and compiles the data into the CloudSpecs database and translates it into a visual display.

Appendix B Terms and Definitions

For the purpose of understanding the relational values of the data, all numbers within the tables below each graph are expressed as whole numbers except the percentages, which are expressed to the tenth decimal point. Percentages are expressed in that manner to account for instances when the coefficient of variation (CV), which is expressed as a percentage, falls below 1%, indicating a high degree of performance predictability.

Average:
When describing averages, Cloud Spectator refers to the average numerical value over a period of 15 days from July 15th until July 29th, 2013, taking into account each and every data point. Average scores can be found inside the tables underneath each graph within this document. The average is used to summarize the data in a simplified overview.

Standard Deviation:
The standard deviation is calculated over a period of 15 days from July 15th until July 29th 2013. The standard deviation can be found inside the tables underneath each graph within this document. The standard deviation is used to understand the amount of variation from the average Copyright Cloud Spectator, LLC 2013. All rights reserved 9

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013 benchmark score; i.e., how predictable a servers performance is for that test. The standard deviation can only be used to understand the amount of variation within a certain environment, and cannot be used to compare different environments because of performance differences.

Coefficient of Variation (CV):

The coefficient of variation is expressed as a percentage. The CV can be found inside the tables underneath each graph within this document. The CV is a measure of precision. It normalizes the standard deviation as a percentage of the average, which can be compared across providers. A lower CV means more stable performance.
[(Standard Deviation) / (Average)] * 100

15-Day Highs and Lows:

From the tested period between July 15th until July 29th 2013, Cloud Spectator extracts and presents the highest and lowest achieved scores by each provider in the tables underneath the graphs within this document.

Appendix C
Test Descriptions The following tests were taken from PassMark Software to measure the CPU performance in different Expedient environments: Compression Test: The Compression Test measures the speed that the CPU can compress blocks of data into smaller blocks of data without losing any of the original data. The result is reported in Kilobytes per Second. This test uses complex data structures and complex data manipulation techniques to perform a function that is very common in software applications, ranging from backup software to Email software. The compression test uses an Adaptive encoding algorithm based on a method described by from Ian H. Witten, Radford M. Neal, and John G. Cleary in an article called Arithmetic Coding for Data Compression. The system uses a model which maintains the probability of each symbol being the next encoded. It reports a compression rate of 363% for English text, which is slightly better than the classic Huffman method. This tests uses memory buffers totaling about 16kb per core. Encryption Test: The Encryption Test encrypts blocks of random data using several different encryption techniques, such that the resulting data can only be accessed by someone with the encryption key. It also tests the computers ability to create a hash of the data, which is also a common cryptographic technique that can be used to ensure the contents of data are not tampered with. The methods used are TwoFish, AES, Salsa20 and SHA256. This test uses many of the techniques in the math tests, but also uses a large amount of binary data manipulation and CPU mathematical functions like 'to the power of'. Encryption is a very useful benchmark, as it is now very widely used in software applications, ranging from Internet browsers, communications software and many different business applications. This tests uses memory buffers totaling about 1MB per core. Where available, the test will make use of specialized CPU instruction sets to accelerate performance, such as Intel AES-NI for the AES test. Floating Point Math Test: The Floating Point Math Test performs the same operations as the Integer Math Test, however, with floating point numbers. A floating point number is a number with a fractional part (ie. 12.568). These kinds of numbers are handled quite differently in the CPU compared to Integer numbers as well as being quite commonly used, therefore they are tested separately. This tests uses memory buffers totaling about 240kb per core. Integer Math Test: The Integer Math Test aims to measure how fast the CPU can perform mathematical integer operations. An integer is a whole number with no fractional part. This is a basic operation in all computer software and provides a good indication of 'raw' CPU throughput. The test uses large sets of random 32-bit and 64-bit integers and adds, subtracts, multiplies and divides these numbers. This tests uses integer buffers totaling about 240kb per core. Physics Test: The Physics Test uses the Tokamak Physics Engine to perform a benchmark of how fast the CPU can calculate the physics interactions of several hundred objects colliding. This tests uses memory buffers totaling about 30MB per core.
Source: cpubenchmark.net

Copyright Cloud Spectator, LLC 2013. All rights reserved

10

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, and 4 - July 2013

About Cloud Spectator

Cloud Spectator is the premier, international cloud analyst group focused on infrastructure pricing and server performance. Since 2011, Cloud Spectator has monitored the cloud infrastructure (IaaS) industry on a global scale and continues to produce research reports for businesses to make informed purchase decisions by leveraging its CloudSpecs utility, an application that automates live server performance tests 3 times a day, 365 days a year with use of open source benchmark tests. Currently, the CloudSpecs system actively tracks 20 of the top IaaS providers around the world. Cloud Spectator 485 Massachusetts Avenue Suite 300 Cambridge, MA 02139 Website: www.cloudspectator.com Phone: (USA) 1-617-300-0711

Intel, the Intel logo, and Xeon are trademarks of Intel Corporation in the U.S. and other countries.

Copyright Cloud Spectator, LLC 2013. All rights reserved

11