Cloud Computing Setup and Probing Examples

Phil Pratt-Szeliga 3/22/2011

Setup Make your own cloud

OpenStack Compute
Docs
http://docs.openstack.org/openstack-compute/admin/oscompute-admin-book.pdf

Download
http://www.openstack.org/projects/compute/latestrelease/

OpenStack Storage
Docs
http://docs.openstack.org/openstack-objectstorage/admin/os-objectstorage-admin-book.pdf

Download
http://www.openstack.org/projects/storage/latest-release/

Setup Use Rackspace Cloud

Register an account
https://signup.rackspacecloud.com/signup

Call Rackspace on the phone to approve your account

1-877-934-0407

Setup your account to use the REST API

Login to https://manage.rackspacecloud.com/ Go to Hosting->Cloud Files and Click Activate. (Once this is done you can get your API KEY) Go to Your Account->API Access and Click Show Key Copy the text of your API key and save it to a file to be used with the Language Bindings (next) Dont share your API Key with anyone or transfer it over the internet in plaintext

Demo Rackspace User Interface

Java CloudServers Bindings Example: Finding a Flavor ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 private int findSmallestFlavor(CloudServersService service){
int smallest_mb = Integer.MAX_VALUE; int smallest_id = -1; FlavorManager fm = service.createFlavorManager(); EntityList<Flavor> flavors = fm.createList(true); while(flavors.hasNext()){ Flavor curr = flavors.next(); if(curr.getRam() < smallest_mb){ smallest_mb = curr.getRam(); smallest_id = curr.getId(); } } System.out.println("Smallest MB found: "+smallest_mb); return smallest_id; }

Java CloudServers Bindings Example: Finding an Image ID

1 2 3 4 5 6 7 8 9 10 11 12 13 private int findImage(CloudServersService service, String image_name){ ImageManager im = service.createImageManager(); EntityList<Image> images = im.createList(true); while(images.hasNext()){ Image curr = images.next(); if(curr.getName().equals(image_name)){ return curr.getId(); } } throw new RuntimeException(Cannot find image); }

Java CloudServers Bindings Example: Starting a Server

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 private int startServer(String username, String apikey, String image_name){
CloudServersService service = new CloudServersService(username, api_key, null);

int image = findImage(service, image_name); int flavor = findSmallestFlavor(service);

Server s = Server.create("test-server", image, flavor); ServerManager sm = service.createServerManager();

sm.create(s); sm.wait(s);
String ip = s.getPublicAddresses().get(0); System.out.println("New server created at: +ip); return s.getId();

CloudFiles Example: Creating a Container & Listing Contents

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 private FilesClient getFilesClient(){ FilesClient client = new FilesClient(m_Username, m_ApiKey); String auth_url = https://auth.api.rackspacecloud.com/v1.0; client.setAuthenticationURL(auth_url); client.setConnectionTimeOut(10000); client.login(); return client; } private void createContainerAndList(String container){ FilesClient client = getFilesClient(); client.createContainer(container); List<FilesObject> f_objects = client.listObjects(container); for(FilesObject file_object : f_objects){ System.out.println(File: +file_object.getName()); } }

Java CloudFiles Example: Uploading a file

1 2 3 4 5 6 7 8 9 10 11 12 13 private void uploadWithByteArray(byte[] bytes){ FilesClient client = getFilesClient(); Map<String, String> metadata = new HashMap<String, String>(); client.storeObject(container, bytes, "text/plain", filename, metadata); } private void uploadWithFile(File file){ FilesClient client = getFilesClient(); Map<String, String> metadata = new HashMap<String, String>(); client.storeObject(container, bytes, "text/plain", filename, metadata); }

Java CloudFiles Example Downloading a file

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 private byte[] downloadByteArray(){ FilesClient client = getFilesClient(); return client.getObject(container, filename); } private InputStream downloadStream(){ FilesClient client = getFilesClient(); return client.getObjectAsStream(container, filename); } private void printInputStream(InputStream is){ BufferedReader reader = new BufferedReader(new InputStreamReader(is)); while(true){ String line = reader.readLine(); if(line == null) break; System.out.println(line); } }

Building Java CloudServers: Apache REST and GSON

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 public void createServer(Server s){ try { Gson gson = new Gson(); String body = gson.toJson(s, Server.class); HttpClient httpclient = new DefaultHttpClient(); HttpPost method = new HttpPost(https://rackspace.com/servers"); method.setHeader(X-Auth-Token, getAuthToken()); method.setHeader("Content-Type", "application/json"); method.setEntity(new StringEntity(body); HttpResponse response = httpclient.execute(method); Header[] headers = response.getAllHeaders(); String respbody = HttpEntityToString.go(response.getEntity()); Server s2 = gson.fromJson(respbody, Server.class); s.copyIn(s2); } catch(IOException ex){ throw new ServiceUnavailableAPIFault(); } }

Demo Programmatic Server Launch

Final Project Statement

Create the basic java-cloudservers bindings (Done) Create a Rackspace Command and Control app
Launches N Servers (Done) Uploads computational application via scp (Done) Starts the computational application via ssh Automatically Destroys N Servers when done Configurable watchdog timer on each server
Terminates instance if no userspace activity

Accelerate a real world computational biology problem

Interactive Clump detection
Parallel Clump detection in Java (Done) Movie Compression that has a file format that is simple to read (raw bmp stacks are 7.7Gig, I can compress down to 385Meg) (Done)

Accelerated Interactive Clump Detection