Professional Documents
Culture Documents
Contents
1. Introduction
2
Why Performance Matters
4
Findings
3. M ethodology
6
Process
Tests Used
10
10
Key Considerations
11
12
12
14
Internal Network
17
19
19
21
Internal Network
24
6. Conclusion
26
7. About
26
1
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Introduction
In an effort to simulate an end-user experience regarding performance of virtual machines across various cloud providers, Cloud Spectator
ran its iterative benchmark suite for 72 hours on each of the following providers: 1&1, AWS, Aruba Cloud, Microsoft Azure, and CloudSigma.
SolidFire sponsored this study on behalf of its client, 1&1.
In most cases, much of the work was straightforward regarding provisioning and the setup process. Occasionally, vendor-side issues
occurred during the provisioning process, and the Cloud Spectator team contacted the corresponding vendors support team in order to resolve
issues such as VM provisioning errors. Three primary VMs of each size were tested on all providers for 24 hours each (72 hours total). This was
done sequentially; once one VM had run the test suite for 24 hours, that VM was terminated and a new VM was created.
This study not only examined the performance of each vendor, but also tracked performance variability for each of the three 24-hour
periods. The methodology allowed Cloud Spectator to capture performance variability both over time on the same VM as well as across different
VMs on multiple physical hosts. Some providers, such as 1&1, show strong processor and memory bandwidth performance stability for all of its VMs
throughout the course of the study. Other providers, such as AWS, exhibited controlled periods of burst followed by throttled performance on network
storage depending on the size of the storage volume. Others, such as CloudSigma, exhibited unstable performance across all resources throughout
the study, possibly due to server-side issues at the time of the study, which contributed to provisioning problems as well.
Taking performance and stability a step further, price-performance analyses are conducted to help readers understand the value ratio
between the cost of the VM and the performance output. While the performance output is limited to the data points collected in the study, by
comparing the price-performance ratio, readers can gain better insight into the overall user experience seen on these providers.
Why Performance Matters
Performance and pricing are both key considerations in the public cloud industry, together having a substantial impact on a companys annual
operating costs. Cloud users may need fewer resources on better performing services, which can lower costs. Since many users only consider price
and not price-performance, these users may be paying more because they require additional resources to achieve a desired level of performance.
While some providers try to differentiate their offerings by cutting prices, others try to differentiate by focusing on improved performance and user
experience.
Differences in performance outputs of VMs across IaaS providers
can greatly impact quality of service as well as annual operating
costs. The graph on the right illustrates an example of the
average processor performance from a sample of six Cloud
Service Providers (CSPs) as studied by Cloud Spectator. CSP 1
has a processor performance three times as high as CSP 6
(names removed), which gives CSP 1 a notable advantage in
many processor-intensive workloads. CSPs 2-5 exhibit a closer
resemblance in processor performance, but do not offer nearly as
much processing power as CSP 1 does.
2
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
The table below lists the 3 hardware components studied in this project, and each purpose as a function in the server.
STORAGE PERFORMANCE
NETWORK PERFORMANCE
highly
key
The
performance
of
all
applications
is
consideration
for
best
application
3
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Executive Summary
On behalf of 1&1, SolidFire commissioned Cloud Spectator to gauge the performance of VMs on five different cloud providers European data
centers: 1&1, Amazon AWS, Aruba Cloud, CloudSigma, and Microsoft Azure. Both performance and price-performance were examined to evaluate
the value of each providers VMs. The purpose of the study was to understand, from an end-user perspective, the disparity of performance and value
(defined as price-performance) among cloud providers with similarly sized VMs. Overall, 1&1 exceled in performance and price-performance for all
component resources of the VMs tested. Its high performance rankings, combined with hourly pricing, introduces powerful, scalable cloud
infrastructure at low cost to its users.
Findings
vCPU & M em ory Perform ance Findings
For this study, Cloud Spectator evaluated vCPU and memory
4
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
1&1 VMs offered the highest disk performance among all included
storage. In this study an 8KB block size was used. Testing occurred
in this study.
between VMs within the same data center of the cloud provider
(measured using iperf and ping respectively) over the course of a 72-
performance tests.
5
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Methodology
Cloud Spectator strives to create a transparent and detailed methodology to allow readers to understand the testing process and recreate any
studies. If any information is unclear or if you have any questions, please email the team at contact@cloudspectator.com or call +1 (617) 300-0711.
Process
1.
Three iterations of 24-hour test cycles were run for each VM on each provider for a total of 72 hours of testing per VM size. After each 24hour block, VMs were terminated before beginning another cycle of tests on newly provisioned machines.
2.
Each VM was provisioned with a Linux Ubuntu 14.04 OS by default, available from all providers. For AWS, the HVM image was used.
3.
Before each 24-hour test period, and after provisioning the VMs, system updates and upgrades were conducted via apt-get.
4.
Git. Git was used to clone the test repository on the VM.
b.
MySQL. For automation purposes, mysql-server was installed to automate data uploads.
c.
Pip. Used to download the appropriate libraries for Python in order to run the testing. SQL Alchemy was downloaded to interact
with MySQL and upload data.
d.
5.
Libmysqlclient-dev. MySQL databases development files, which are necessary for the SQL Alchemy and MySQL interaction.
Each test cycled through in the following sequence: Geekbench 3 (process & memory), fio sequential operations, fio random operations,
Iperf internal network throughput (for more information on testing, see Tests Used).
a.
For fio testing (to measure disk IOPS), sequential operations ran first. Files from the sequential tests were deleted, and fio
recreated files before running random operations. Once random operations completed, the files were also deleted. Thus, before
each disk IOPS test, the files associated with the tests were deleted and recreated.
6.
On AWS and Azure, where VMs demonstrated varying internal network throughput depending on size and/or instance type, a
clone of that VM was created in the same region/availability zone. The cloned server listened for a TCP connection via Iperf.
E.g., two c4.large instances were created in Amazon AWSs EU West 1 region to test throughput. The cloned VM was
terminated alongside the tested VMs at the conclusion of each 24-hour test cycle.
b.
On 1&1, Aruba Cloud, and CloudSigma, where VMs did not demonstrate varying internal network throughput depending on size
and/or instance type, a screen session was created on each VM to listen for a TCP connection. Each category of VMs, which
contains 2 VMs, connected with each other to perform throughput testing; i.e., 1&1s 2 vCPU 4GB RAM and 2 vCPU 8GB RAM
virtual machines conducted network throughput tests across one another.
7.
A total of approximately 1.4 million data points were collected throughout the period of the study.
8.
At the end of each test iteration, results were uploaded into Cloud Spectators database through use of SQL Alchemy (Pythonsee 4c in
Process).
6
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Tests Used
Processor & Memory Bandwidth: Geekbench 3
Geekbench 3, a licensable product created by Primate Labs, is a cross-platform processor benchmark that can measure single-core and multi-core
performance by simulating real-world workloads. The Geekbench 3 test suite is comprised of 27 individual tasks/workloads: 13 integer workloads, 10
floating point workloads, and 4 memory-bandwidth tasks. While processor and memory bandwidth are both performance factors that contribute to the
final score provided by Geekbench 3, the test suite weighs processing performance much more heavily than memory bandwidth. Also, memory
bandwidth is not necessarily affected by the amount of memory available for the VM, so VMs with larger amounts of memory may not exhibit larger
bandwidth. For more information on Geekbench 3 and to see its individual workloads, please see http://www.primatelabs.com/geekbench/.
Geekbench 3 Tasks (Figure 3.1)
TEST
TOOL
TASK
DESCRIPTION
Integer
Geekbench 3
score.
SubtestsPNG
include:
Integer Math,
Point
PNG
Compression,
Decompression,
Sobel,Floating
Lua, Dijkstra
Math
Floating Point
Geekbench 3
Memory
Geekbench 3
7
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Instance
vCPU
RAM (GB)
Storage (GB)
Monthly ()
1&1
4GB
100
29.99
1&1
8GB
100
79.20
AWS
C4.large
3.75
95.04
AWS
M3.large
7.5
108.72
Aruba Cloud
4GB
100
64.80
Aruba Cloud
8GB
100
79.20
A2
3.5
Blob Storage
68.10
Azure
Azure
D2
Blob Storage
104.60
CloudSigma
4GB
100
49.33
CloudSigma
8GB
100
73.73
8
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Instance
vCPU
RAM (GB)
Storage (GB)
Monthly ()
1&1
8GB
400
93.60
1&1
15GB
15
400
144.00
AWS
C4.xlarge
7.5
208.08
AWS
M3.xlarge
15
236.16
Aruba Cloud
8GB
400
158.40
Aruba Cloud
15GB
15
400
183.60
Azure
A3
Blob Storage
143.65
Azure
D3
14
Blob Storage
216.59
CloudSigma
8GB
400
135.27
CloudSigma
15GB
15
400
177.97
Instance
vCPU
RAM (GB)
Storage (GB)
Monthly ()
1&1
15GB
15
800 (2 x 400)
172.80
1&1
30GB
30
800 (2 x 400)
280.80
AWS
C4.2xlarge
15
416.88
AWS
M3.2xlarge
30
473.76
Aruba Cloud
15GB
15
800
313.20
Aruba Cloud
30GB
30
800
367.20
Azure
A4
14
Blob Storage
287.24
Azure
D4
28
Blob Storage
433.11
CloudSigma
15GB
15
800
276.63
CloudSigma
30GB
30
800
368.15
Conversion Rates:
1.00
=
1.40
$1.00
=
0.89
9
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
4000
testing.
Mpixels/sec
3500
3000
2500
2000
95th percentile, then more high scores were observed than low scores; vice
1500
versa.
1000
500
0
AWS
Azure
Rackspace
testing.
Softlayer
CloudSpecs ScoreTM
100
100
80
60
44
41
41
40
25
20
The graph on the left is an example of how Cloud Spectators priceperformance analysis is visualized. The closer the score is to 100, the
higher price-performance value it indicates. The score 100 represents the
0
CSP1
CSP2
CSP3
CSP4
CSP5
best-value VM among all in the comparison. The value is scaled; e.g., the
VM from Cloud Service Provider 1 (CSP1) with a score of 100 gives 4x the value of the VM from CSP5 with a score of 25.
The CloudSpecs ScoresTM of any VM can change depending on the participants in the comparison. For example, if the highest score in a
comparison changes, the price-performance value represented by score 100 will change accordingly, and so will the other CloudSpecs ScoreTM
values.
If you have questions regarding Cloud Spectators price-performance calculation, please contact us at contact@cloudspectator.com.
Key Considerations
Pricing used in this study (for price-performance comparisons) is up to date effective June 10, 2015. Pricing may change for the specified
VMs after the release of this report.
Testing was conducted on specific VM types for each provider. Different VM configurations may yield different comparative results
between the providers. AWS and Azure offered fixed VM configurations, while 1&1, Aruba Cloud, and CloudSigma offered independently
customizable resource configurations.
Users may experience different performance across different physical hosts. Factors such as user contention or malfunctions of the
physical hardware can cause suboptimal performance. Cloud Spectator terminated and created new VMs for each test iteration to increase
the likelihood of testing on different physical hosts.
VMs selected were the base offerings across providers; greater performance may be obtained on certain providers by paying for additional
features/services.
11
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Provider
1&1
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
AWS
2 vCPUs
Indexed Score
Azure
4GB
8GB
1&1
c4.large
m3.large
AWS
4GB
8GB
A2
ArubaCloud
D2
Azure
4GB
CloudSigma
8GB
1&1
AWS
4 vCPUs
Indexed Score
Provider
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
Aruba
Cloud
Azure
8GB
15GB
1&1
c4.xlarge m3.xlarge
AWS
8GB
15GB
A3
ArubaCloud
D3
Azure
8GB
15GB
CloudSigma
CloudSigma
Provider
1&1
18000
16000
14000
12000
10000
8000
6000
4000
2000
0
8 vCPUs
AWS
Aruba
Cloud
Azure
15GB
30GB
1&1
c4.2xlarge m3.2xlarge
AWS
15GB
30GB
ArubaCloud
A4
D4
Azure
Min
4575
4059
3901
3125
3731
2973
1781
2176
2457
1150
5TH
4619
4663
3934
3139
4017
3883
2132
3557
2599
1374
Median
4711
4708
3952
3155
4085
4084
2166
3621
2896
2826
95TH
4731
4732
3973
3173
4165
4167
2185
3693
2965
2981
Max
4746
4748
4002
3192
4203
4197
2304
3746
2986
3000
VM
8GB
15GB
c4.xlarge
m3.xlarge
8GB
15GB
A3
D3
8GB
15GB
Min
7993
8304
6932
5699
6872
5660
3648
4978
1149
1192
5TH
8493
8688
7672
6105
7459
7388
4096
6650
1454
1474
Median
8764
8852
7736
6261
7693
7663
4118
6677
3698
4538
95TH
9088
9101
7775
6286
7842
7840
4136
6700
5782
4927
Max
9132
9144
7808
6308
7868
7881
4161
6716
5888
5387
CloudSigma
Indexed Score
Aruba
Cloud
VM
4GB
8GB
c4.large
m3.large
4GB
8GB
A2
D2
4GB
8GB
15GB
30GB
CloudSigma
CloudSigma
VM
15GB
30GB
c4.2xlarge
m3.2xlarge
15GB
30GB
A4
D4
15GB
30GB
Min
14218
14858
12718
10580
10739
10765
5541
8687
4019
1336
5TH
15648
15768
13257
11228
11848
11692
7418
12646
5076
1550
Median
16138
16097
13705
11519
13775
13874
7839
12739
7281
5900
95TH
16281
16767
14056
11588
14119
14206
7917
12793
7848
6803
Max
16349
16910
14140
11659
14337
14400
7963
12866
8056
7640
12
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
1&1s VMs achieved the highest performance across processor and memory performance in the study.
The Indexed Score, which weighs heavily on processor performance, is similar between VMs of the same vCPU count for 1&1, Aruba
Cloud, and CloudSigma. All three providers offer independently customizable VMs, which are backed with the same hardware. AWS and
Azure exhibit differences in performance results. AWSs C4 Family and M3 Family are provisioned with different processors; Azures A
Series and D Series are provisioned with different processors as well.
Although AWS and Azure VMs processor and memory performance scored lower than 1&1 VMs, both providers also achieved similar
levels of performance stability in the 72-hour test period, with CVs lower than 3% on all VMs. Aruba Cloud VMs CVs ranged up to 9.3%.
CloudSigma VMs large variation in performance, seen in the processor and memory performance results, are present in disk IOPS and
network performance as well.
CloudSigma and Azure VMs continue to display the lowest processor and memory bandwidth performance in each category (see tables in
Figure 4.1 A through C).
13
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
AWS
2 vCPUs
IOPS
Azure
4GB
8GB
1&1
c4.large
m3.large
AWS
4GB
8GB
A2
ArubaCloud
D2
Azure
4GB
8GB
CloudSigma
CloudSigma
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
AWS
4 vCPUs
IOPS
Aruba
Cloud
Azure
8GB
15GB
1&1
c4.xlarge m3.xlarge
AWS
8GB
15GB
A3
ArubaCloud
D3
Azure
8GB
CloudSigma
15GB
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
8 vCPUs
AWS
Aruba
Cloud
Azure
15GB
30GB
1&1
c4.2xlarge m3.2xlarge
AWS
15GB
30GB
ArubaCloud
A4
D4
Azure
Min
4477
4622
235
227
379
267
586
480
214
208
VM
8GB
15GB
c4.xlarge
m3.xlarge
8GB
15GB
A3
D3
8GB
15GB
Min
3052
2845
2093
2452
241
278
580
427
219
177
5TH
6138
6082
299
299
713
597
1056
1220
349
328
5TH
5437
6570
2099
3013
618
655
1069
1166
311
306
Median
7143
6996
299
299
1493
1372
1380
1382
753
749
Median
7675
7933
3015
3015
1186
1200
1380
1381
603
633
95TH
7947
7972
3064
3064
2494
4414
1384
1412
2170
2011
Max
8341
8358
3064
3064
4060
6204
1399
1436
4538
3621
95TH
9133
8804
3017
3015
2483
2228
1384
1385
1768
2155
Max
9437
9138
3017
3015
4496
3892
1421
1404
4191
3909
CloudSigma
IOPS
Aruba
Cloud
VM
4GB
8GB
c4.large
m3.large
4GB
8GB
A2
D2
4GB
8GB
15GB
30GB
CloudSigma
CloudSigma
VM
15GB
30GB
c4.2xlarge
m3.2xlarge
15GB
30GB
A4
D4
15GB
30GB
Min
3198
3265
2541
2907
342
367
396
531
202
195
5TH
4821
5709
3003
3013
617
599
620
1195
316
306
Median
6755
7650
3016
3015
1221
1300
663
1385
646
596
95TH
8703
9214
3017
3015
2352
3000
687
1414
1852
2074
Max
9280
9577
3017
3015
4345
5843
712
1435
4724
4012
14
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
AWS
2 vCPUs
IOPS
Azure
4GB
8GB
1&1
c4.large
m3.large
AWS
4GB
8GB
A2
ArubaCloud
D2
Azure
4GB
8GB
CloudSigma
CloudSigma
Provider
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
1&1
AWS
4 vCPUs
IOPS
Aruba
Cloud
Azure
8GB
15GB
1&1
c4.xlarge m3.xlarge
AWS
8GB
15GB
A3
ArubaCloud
D3
Azure
8GB
15GB
CloudSigma
CloudSigma
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
AWS
8 vCPUs
IOPS
Aruba
Cloud
Aruba
Cloud
Azure
15GB
30GB
1&1
c4.2xlarge m3.2xlarge
AWS
15GB
30GB
ArubaCloud
A4
D4
Azure
15GB
30GB
CloudSigma
CloudSigma
VM
4GB
8GB
c4.large
m3.large
4GB
8GB
A2
D2
4GB
8GB
Min
4710
4772
576
630
165
115
163
149
243
210
VM
8GB
15GB
c4.xlarge
m3.xlarge
8GB
15GB
A3
D3
8GB
15GB
Min
2830
3868
2759
2834
117
150
167
113
232
115
VM
15GB
30GB
c4.2xlarge
m3.2xlarge
15GB
30GB
A4
D4
15GB
30GB
Min
2814
3087
2816
2797
0
155
164
83
114
168
5TH
6002
5943
757
819
291
307
1021
1127
325
351
5TH
5110
6232
2985
2984
296
305
963
1116
308
302
5TH
4557
5490
2975
2984
295
296
341
1085
314
319
Median
6620
6621
760
822
612
552
1383
1386
661
652
Median
7272
7505
2987
2985
532
537
1376
1376
561
558
Median
6463
7364
2987
2985
534
518
369
1385
651
611
95TH
7346
7314
3064
3064
952
1064
1430
1428
1741
1435
95TH
8605
8359
2988
2985
983
951
1424
1380
1674
1557
95TH
8042
8751
2988
2985
992
919
382
1401
2156
2151
Max
8234
7695
3065
3064
1181
1709
1438
1436
3224
2997
Max
8896
8667
2988
2986
1167
1173
1436
1436
3049
2778
Max
8911
9109
2988
2986
1629
1175
392
1432
3897
3474
15
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Providers with SSD offerings exhibit little performance difference between random and sequential IOPS, which is expected for SSDs.
These providers include 1&1, AWS and CloudSigma. One exception is on the small VMs for AWS, which exhibited different IOPS
performance results due to throttling.
Although 1&1 and CloudSigma both offer SSDs, 1&1s volumes achieved 4.4 4.7x more IOPS than CloudSigmas SSDs when examining
95TH percentile figures.
1&1, Aruba Cloud, and CloudSigma express large performance variability across the 24 hours of testing. Despite the variability, the low
points of 1&1s disk IOPS in both sequential and random operations still surpass other providers for the majority of tests. Aruba Cloud and
CloudSigmas variability results in some of the lowest performance observed during the study.
Although AWS offers SSD technology on its block storage offering, EBS, the performance of that offering is more dependent on the size of
the block storage volume provisioned. Although the actual IOPS performance varies between sequential and random operations, the
pattern of performance remains similar based on the size of the provisioned storage (in this test scenario, 100GB, 400GB, and 800GB
sizes were provisioned on 2 vCPU, 4 vCPU, and 8 vCPU machines, respectively). For its General Purpose SSD volumes, AWS offers 3
IOPS per GB with burst up to 3000 IOPS. Figures 4.2 A and 4.3 A illustrate AWSs burst for a 100GB volume. While burst for the 100GB
volume never exceeded 3065 IOPS (see Max in the corresponding tables), the non-burst range, expressed by the median, stays at 299
IOPS for sequential operations, but is higher for random operations. The minimum guaranteed IOPS, 300 (100GB * 3 IOPS per GB), is
sustained for the most part, although minimum values showed lower IOPS, with dips to 227 IOPS. At 800 GB, AWS sustains a fairly stable
rate of approximately 2750 3000 IOPS. Burst credits are assigned to volumes, and as long as the volume still has credits, it can burst to
3000 IOPS. In the period of the study, for 4 vCPU and 8 vCPU machines on AWS, the 400GB and 800GB volumes sustained enough
credits for a continuous 24-hours of high performance; interestingly, the c4.xlarge VMs dropped in performance for IOPS on sequential
operations in all three 24-hour iterations, which suggests that credits for IOPS are being used up on the c4.xlarge VMs much faster than on
others.
Azure and Aruba Cloud do not offer SSD technology-backed storage. Therefore, both providers offer higher IOPS performance on
sequential operations when compared with random operations, which is common behavior for magnetic drives. Figures 4.2 and 4.3
illustrate this behavior for both providers. In one of the sequences for the 8 vCPU with 15GB RAM VM, Aruba Cloud did not successfully
complete a cycle of random read/write operations, and therefore achieved a minimum of 0 in the total 72-hour study.
1&1 achieved much higher IOPS than all other providers tested, with maximums exceeding 9500 IOPS for sequential operations. Even in
the median, 1&1s SolidFire-backed SSD volumes obtained a minimum of 6463 IOPS, which is 2.15x more IOPS than can be achieved per
800GB volume on AWS, unless a user decides to purchase provisioned IOPS at an additional charge.
For random IOPS, Azure and Aruba Cloud displayed the lowest results in most scenarios. Azure and Aruba Cloud, as mentioned before,
are the only providers in this study that use magnetic storage for their storage volumes.
16
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Provider
1&1
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
AWS
2 vCPUs
Throughput (Mbit/s)
Aruba
Cloud
Azure
4GB
8GB
1&1
c4.large
m3.large
AWS
4GB
8GB
A2
ArubaCloud
D2
Azure
4GB
8GB
CloudSigma
CloudSigma
Provider
1&1
18000
14000
AWS
12000
4 vCPUs
Throughput (Mbit/s)
16000
10000
8000
6000
Aruba
Cloud
Azure
4000
2000
0
8GB
15GB
1&1
c4.xlarge m3.xlarge
AWS
8GB
15GB
A3
ArubaCloud
D3
Azure
8GB
CloudSigma
15GB
Provider
1&1
5000
AWS
4000
8 vCPUs
Throughput (Mbit/s)
6000
3000
2000
1000
Aruba
Cloud
Azure
0
30GB
1&1
c4.2xlarge m3.2xlarge
AWS
15GB
30GB
ArubaCloud
A4
D4
Azure
Min
700
774
473
583
792
794
193
640
914
968
VM
8GB
15GB
c4.xlarge
m3.xlarge
8GB
15GB
A3
D3
8GB
15GB
Min
619
661
723
684
779
297
570
1451
1123
1423
5TH
929
928
473
628
834
838
302
863
1466
1979
5TH
885
896
725
899
826
743
863
1566
1942
2517
Median
955
955
473
628
875
894
708
880
2760
2900
Median
956
956
725
906
857
853
888
1624
6526
6970
95TH
955
955
492
663
895
895
734
895
4359
3660
Max
956
956
492
664
895
895
746
909
4832
4361
95TH
956
956
754
957
882
881
910
1683
13265
11396
Max
956
956
754
957
895
895
930
1735
16152
12821
95TH
956
956
962
957
895
893
925
1649
4657
5105
Max
956
956
962
957
895
895
936
1684
5983
5792
CloudSigma
15GB
VM
4GB
8GB
c4.large
m3.large
4GB
8GB
A2
D2
4GB
8GB
15GB
30GB
CloudSigma
CloudSigma
VM
15GB
30GB
c4.2xlarge
m3.2xlarge
15GB
30GB
A4
D4
15GB
30GB
Min
696
709
902
886
724
658
123
1432
1344
1005
5TH
841
891
921
903
801
820
890
1529
2001
2191
Median
953
956
925
906
852
861
910
1590
3208
3493
17
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
AWSs VMs deliver a specific network throughput depending on the size and family of VM. As the size of the VMs increase, the network
throughput increases as well. The difference in network performance between the C4 Family and M3 Family are comparable. By contrast,
Azures A Series provides less throughput than its D Series.
Azures VMs, which scale depending on size and Series, exceeds 1GB/s network throughput for D3 and D4s (see Figures 4.4B and C). All
other providers, with the exception of CloudSigma, never exceed 1GB/s, although 1&1 VMs and AWSs larger VMs come close.
1&1 provides a continuous and fairly steady throughput of slightly less than 1GB/s (956 GB/s max) regardless of VM size. Similarly, Aruba
Cloud provides a lower, continuous throughput at a little less than 900 MB/s, regardless of VM size as well.
While AWS displayed some of the lowest internal network throughput numbers for 2 vCPU and 4 vCPU VMs, Aruba Cloud, which does not
scale internal network throughput with increased VM size, displayed the lowest throughput numbers for 8 vCPU VMs.
18
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
90
90
80
80
70
60
50
40
40
37
26
30
20
CloudSpecs Score
100
70
49
40
30
A2
4GB
8GB
m3.large
8GB
D2
8GB
0
4GB
10
c4.large
20
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
100
100
100
90
90
80
80
70
52
60
50
40
31
40
29
30
CloudSpecs Score
100
68
70
60
43
50
50
41
40
30
8GB
15GB
m3.xlarge
15GB
D3
15GB
0
A3
10
0
8GB
20
10
c4.xlarge
20
8GB
CloudSpecs Score
64
50
10
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
47
51
42
A4
15GB
30GB
m3.2xlarge
30GB
D4
30GB
28
15GB
28
66
c4.2xlarge
29
100
90
80
70
60
50
40
30
20
10
0
15GB
35
CloudSpecs Score
100
CloudSpecs Score
58
60
20
100
90
80
70
60
50
40
30
20
10
0
87
4GB
CloudSpecs Score
100
100
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
19
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
For VMs with 2 vCPUs and approximately 8GB RAM (see Figure 5.1 B), all values are most evenly matched, compared with other VM
sizes.
As VM sizes scale up in processors and RAM, CloudSigmas CloudSpecs ScoreTM drops, and its rank falls in relativity to other tested
providers VMs, due largely to performance (see Figure 4.1).
Relatively, Azures D Series, which has an average CloudSpecs ScoreTM of 53 across all VM sizes, provides more relative priceperformance value than its A Series, which has an average CloudSpecs ScoreTM of 27. Because charts cannot be relatively compared,
though, the difference between the CloudSpecs ScoresTM does not equate to an almost 2x value on Azures D Series.
20
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
90
90
CloudSpecs
Score
CloudSpecs Score
100
100
80
70
60
50
40
70
60
50
40
30
30
20
2
10
4GB
A2
4GB
8GB
m3.large
8GB
D2
8GB
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
90
90
CloudSpecs
Score
CloudSpecs Score
100
100
80
70
60
50
40
23
30
20
80
70
60
50
40
12
20
12
10
c4.xlarge
8GB
A3
8GB
15GB
m3.xlarge
15GB
D3
15GB
8GB
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
CloudSpecs Score
100
25
m3.2xlarge
30GB
D4
30GB
30GB
12
15GB
10
A4
15GB
30
c4.2xlarge
100
90
80
70
60
50
40
30
20
10
0
15GB
CloudSpecs Score
30
30
7
10
100
90
80
70
60
50
40
30
20
10
0
11
c4.large
15
10
14
20
4GB
80
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
21
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
90
90
CloudSpecs
Score
CloudSpecs Score
100
100
80
70
60
50
40
70
60
50
40
30
30
20
10
10
4GB
A2
4GB
8GB
m3.large
8GB
D2
8GB
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
90
90
CloudSpecs
Score
CloudSpecs Score
100
80
70
60
50
40
24
30
10
80
70
60
50
31
40
30
12
20
20
10
8GB
A3
8GB
15GB
m3.xlarge
15GB
D3
15GB
c4.xlarge
12
8GB
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
CloudSpecs Score
100
25
m3.2xlarge
30GB
D4
30GB
30GB
12
15GB
A4
15GB
31
c4.2xlarge
100
90
80
70
60
50
40
30
20
10
0
15GB
CloudSpecs Score
11
1&1
100
100
90
80
70
60
50
40
30
20
10
0
16
6
c4.large
12
20
4GB
80
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
22
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
1&1 achieved the highest CloudSpecs ScoresTM as well as most IOPS for each VM tested.
Aruba Cloud offered the lowest value for random IOPS on SAN disk on all tested VMs, due to its magnetic drives and cost of the VMs.
AWSs c4.large and m4.large (see Figure 5.2 A and B) offered lower price-performance value due to throttled IOPS after burst limits were
exceeded for the 100GB storage volume.
Although CloudSigma used SSD technology for its storage volume, the price-performance value of those storage volumes tested are
similar to Aruba Cloud and Azure volumes, which use magnetic disks.
23
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
90
90
CloudSpecs
Score
CloudSpecs Score
100
100
80
70
60
57
50
40
30
13
20
16
70
60
50
40
31
30
19
21
20
21
20
10
10
c4.large
4GB
A2
4GB
8GB
m3.large
8GB
D2
8GB
0
4GB
80
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
90
90
CloudSpecs
Score
CloudSpecs Score
100
100
80
70
60
50
40
30
13
60
50
40
20
17
19
13
10
10
c4.xlarge
8GB
A3
8GB
15GB
m3.xlarge
15GB
D3
15GB
0
8GB
70
30
21
20
80
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
CloudSpecs Score
48
27
26
A4
15GB
30GB
m3.2xlarge
30GB
D4
30GB
19
15GB
18
39
36
c4.2xlarge
25
100
100
90
80
70
60
50
40
30
20
10
0
15GB
CloudSpecs Score
100
100
90
80
70
60
50
40
30
20
10
0
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
Azure also achieved a better CloudSpecs ScoreTM for its D Series in regards to internal network throughput. The D series offered approximately 1.7x
more network throughput on VMs tested in the study.
Although AWSs virtual machines increased throughput based on size and family, that increase did not surpass the price-performance ratio
of 1&1 VMs throughput and cost; therefore, 1&1 achieved a higher CloudSpecs ScoreTM on network throughput than AWS.
Aruba Cloud, which offered higher throughput than AWS on smaller VMs (see Figure 4.4 A), displayed higher internal network throughput
price-performance than AWS until VM sizes scale up. As machine sizes increased, AWSs throughput scaled while Aruba Clouds did not, which is
the reason AWS has higher CloudSpecs ScoresTM for the larger VMs. For these VMs, (see Figure 5.4 B and C), Aruba Cloud fell short of AWS, as
well as all other competitors in the study.
Additional Observations
Azures D3 and D4 VMs exceeded a throughput of 1 GB/s (see Figures 4.4 B and C), giving it better price-performance value in internal
network than all other providers for those VM sizes with the exception of CloudSigma (see Figures 5.4 D and F).
In all other VM sets, 1&1 achieves the highest CloudSpecs ScoreTM with the exception of CloudSigma for internal network.
Aruba Cloud, which does not scale internal network throughput with the size of the VM (see Figure 4.4), drops in value ranking as VMs are
scaled up in size; other providers, such as AWS and Azure, which scale internal network throughput with the corresponding VM size and
family/series, surpass Aruba Cloud in value for internal network as the VMs are scaled up.
25
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Conclusion
Selecting the right provider and virtual machines requires a thorough and accurate performance comparison. The study conducted for this
report offers a general understanding of performance and price-performance strengths and weaknesses across each included vendors VMs, and
should be considered a source of information to help guide readers in their own testing and analyses. The processor, memory bandwidth, network
storage, and internal network were all examined and results clearly show that no single provider can offer everything to fit everyones needs.
Therefore, in order to accurately select the right provider for a business or an application, performance and price-performance analysis is crucial.
Results from this study show that, 1&1s VMs displayed strong performance and price-performance. 1&1s VMs demonstrated high
performance and stability for processing and memory bandwidth both on the same VM, and also across VMs of the same size. Its network-attached
storage produced the largest amount of IOPS seen in the study as well, regardless of VM size and without the need for purchasing additional
volumes, sizing up, or provisioning IOPS. These high-performance results, combined with the low cost of the VMs, reflect on the value that 1&1s
VMs can deliver to potential users, as seen with the price-performance results. 1&1s performance and price-performance offers an excellent
alternative for high-performance environments such as distributed file systems and data analytics processing.
Performance in the industry cannot be assumed to be equal or even similar, as illustrated in this report. When it comes to processor and
memory bandwidth performance, tiered providers such as AWS and Azure may offer varying performance depending on the family/series of the VM,
despite having equivalent amounts of vCPUs and similar amounts of memory. For disk IOPS, although both 1&1 and CloudSigma advertise SSD
volumes, 1&1s SSDs achieved 4.4 4.7x more IOPS than their CloudSigma counterparts. Internal network performance on CloudSigma, though,
exceeds all providers examined in the study.
While this study was conducted in the manner of understanding a typical end user experience, it should not be assumed to be accurate for
all use cases. Stress testing was conducted to better understand fluctuation and theoretically sustained performance, and should be seen as a
general indication of provider performance. For more detailed analysis on any specific use case, please contact Cloud Spectator at
contact@cloudspectator.com or by phone at +1 (617) 300 0711.
About
About Cloud Spectator
Cloud Spectator is a cloud analyst agency focused on cloud Infrastructure-as-a-Service (IaaS) performance. The company actively monitors several
of the largest IaaS providers in the world, comparing VM performance (i.e., CPU, RAM, disk, internal network, and workloads) and pricing to achieve
transparency in the cloud market. The company helps cloud providers understand their market position and helps business make intelligent
decisions in selecting cloud providers and lowering total cost of ownership. The firm was founded in early 2011 and is located in Boston, MA.
For questions about this report, to request a custom report, or if you have general inquiries about our products and services, please contact Cloud
Spectator at +1 (617) 300-0711 or contact@cloudspectator.com.
For press/media related inquiries, please contact:
Ken Balazs
VP Sales & Marketing
kbalazs@cloudspectator.com
26
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.