Amazon Web Services:

Building Highly Scalable Web Applications

Institutional Web Management Workshop

July 2007

Jeff Barr
Senior Web Services Evangelist
jbarr@amazon.com
Today

Introduction
Amazon Web Services overview
Amazon EC2
Amazon S3
Amazon SQS
Application Architecture
Questions
Who am I?

Software development background

5 Years with Amazon:
Senior Developer
Web Services Evangelist
Senior Web Services Evangelist
Veteran of several startups
Microsoft .Net Team
Visix Software co-founder
Amazon?
Consumers (Buy)

Associates, Sellers & Merchants (Sell)

Developers (Innovate)
Issues Facing Developers
70% of Web Development Effort is “Muck”:
Data Centers
Bandwidth / Power / Cooling
Operations
Staffing

Scaling is Difficult and Expensive:

Large Up-Front Investment
Invest Ahead of Demand
Load is Unpredictable
Dream or Nightmare?

Slashdot/Digg/TechCrunch Effect
Rapid, unexpected customer demand/growth
Seasonal Spikes
More Seasonal Spikes

“Every year, we take the busiest minute of the

busiest hour of the busiest day and build
capacity on that, we built our systems to
(handle that load) and we went above and
beyond that.” *

-- Scott Gulbransen
Intuit Spokesman

* http://news.com.com/2100-1038_3-6177341.html
Solution: Web-Scale Computing

Scale capacity on demand

Turn fixed costs into variable costs
Always available
Rock-solid reliability
Simple APIs and conceptual models
Cost-effective
Reduced time to market
Focus on product & core competencies
Amazon Web Services Are...

A set of APIs and business models which give

developer-level access to Amazon’s infrastructure
and content:

Data As A Service Search As A Service

Amazon E-Commerce Service
Amazon Historical Pricing
Infrastructure As A Service
Amazon Simple Queue Service
Amazon Simple Storage Service
Amazon Elastic Compute Cloud
Alexa Web Information Service
Alexa Top Sites
Alexa Site Thumbnail
Alexa Web Search Platform
People As A Service
Amazon Mechanical Turk

10
The Commercial Side
Standard licensing terms
Commercially usable
Aggressive pricing
Monthly credit card billing
Self-serve model:
Sign up as developer
Choose services
Agree to service licenses
Enter payment info
Start coding
Web Services Billing
Infrastructure Services
Elastic Compute
Cloud

Compute

Simple Storage Simple Queue

Service Service

Store Message
Amazon Simple Storage Service

S3
Amazon Simple Storage Service
• Object-Based
Storage
• 1 B – 5 GB / object $.15 per
• Fast, Reliable, GB per
Scalable month
• Redundant,
Dispersed storage
$.01 for 1000
• 99.99% Availability to 10000
Goal requests
• Private or Public $.10 - $.18
• Per-object URLs & per GB data
ACLs transfer
Amazon S3 Concepts

Objects:
Opaque data to be stored (1 byte … 5 Gigabytes)
Authentication and access controls

Buckets:
Object container – any number of objects
100 buckets per account / buckets are “owned”

Keys:
Unique object identifier within bucket
Up to 1024 bytes long
Flat object storage model

Standards-Based Interfaces:
REST and SOAP
URL-Addressability – every object has a URL
S3 SOAP/Query API
Service:
ListAllMyBuckets

Buckets:
CreateBucket
DeleteBucket
ListBucket
GetBucketAccessControlPolicy
SetBucketAccessControlPolicy
GetBucketLoggingStatus
SetBucketLoggingStatus

Objects:
PutObject
PutObjectInline
GetObject
GetObjectExtended
DeleteObject
GetObjectAccessControlPolicy
SetObjectAccessControlPolicy
Amazon Simple Queue Service

SQS
Amazon Simple Queue Service

• Scalable Queuing
$.10 per
• Elastic Capacity
• Reliable, Simple, 1000
Secure messages
Inter-process messaging,
$.10 - $.18
data buffering, architecture per GB data
component transfer
Amazon SQS Concepts

Queues:
Named message container
Persistent

Messages:
Up to 256KB of data per message
Peek / Lock access model

Scalable:
Unlimited number of queues per account
Unlimited number of messages per queue
Amazon SQS At Work
SQS SOAP/Query API
Queues:
ListQueues
DeleteQueue
SetVisibilityTimeout
GetVisibilityTimeout

Messages:
SendMessage
ReceiveMessage
DeleteMessage
PeekMessage

Security:
AddGrant
ListGrants
RemoveGrant
Amazon Elastic Compute Cloud

EC2
Amazon Elastic Compute Cloud
• Virtual Compute
Cloud
• Elastic Capacity $.10 per
• 1.7 GHz x86
• 1.7 GB RAM
server
• 160 GB Disk hour
• 250 MB/Second
Network
• Network Security
Model $.10 - $.18
Time or Traffic-based Scaling, per GB data
Load testing, Simulation and transfer
Analysis, Rendering, Software as
Amazon EC2 Concepts

Amazon Machine Image (AMI):

Bootable root disk
Pre-defined or user-built
Catalog of user-built AMIs
OS: Fedora, Centos, Gentoo, Debian,
Ubuntu, Windows Server
App Stack: LAMP, mpiBLAST, Hadoop

Instance:
Running copy of an AMI
Launch in less than 2 minutes
Start/stop programmatically

Network Security Model:

Explicit access control
Security groups

Inter-service bandwidth is free

Root-level access
Amazon EC2 At Work

Startups
Cruxy – Media transcoding
GigaVox Media – Podcast Management

Fortune 500 clients:

High-Impact, S hort-Term Projects
Development Host

Science / Research:
Hadoop / MapReduce
mpiBLAST

Load-Management and Load Balancing Tools:

Pound
Weogeo
Rightscale
EC2 SOAP/Query API
Images: Image Attributes:
RegisterImage ModifyImageAttribute
DescribeImages
DeregisterImage DescribeImageAttribute
ResetImageAttribute
Instances:
RunInstances Security Groups:
DescribeInstances CreateSecurityGroup
TerminateInstances
GetConsoleOutput DescribeSecurityGroups
RebootInstances DeleteSecurityGroup
AuthorizeSecurityGroupIngress
Keypairs: RevokeSecurityGroupIngress
CreateKeyPair
DescribeKeyPairs
DeleteKeyPair
Sample Web-Scale Architecture

GigaVox Media
Web-Scale Architecture
GigaVox Economics

Implemented Amazon S3, Amazon EC2 and

Amazon SQS in November 2006
Created an infinitely scalable
infrastructure for less than $100 - building
the same infrastructure themselves would
have cost thousands of dollars
Reduced staffing requirements - far less
responsibility for 24x7 operations
Q&A
 Thank You!
Jeff Barr Pointers
Portal
Senior Web Services Evangelist http://aws.amazon.com

jbarr@amazon.com Blog
http://aws.typepad.com

EC2
http://aws.amazon.com/ec2

S3
http://aws.amazon.com/s3

Resource Center
http://aws.amazon.com/resources

Forums
http://aws.amazon.com/forums