WHITEPAPER

HarnessingtheCloudfor PerformanceTesting

Abstract
Inthiswhitepaper,Impetustalksabouthowthe infinitecomputingpowerofaCloudinfrastructurecan beleveragedforflexible,scalableandaffordableload testing. Thepaperprovidesinsightsonthevariousbenefitsof usingtheCloudforPerformanceTestingaswellashow toaddressthevariouschallengesassociatedwiththis approach.Itdiscussesthedifferentmodelswhere performancetestingtoolscanbedeployedonthe Cloud,inadditiontosomebestpracticesandreal worldexperiences.

ImpetusTechnologies,Inc. www.impetus.com July2011

Harnessin ngtheCloudfo orPerformance eTesting

Tableo ofConte ents

Introduction..... ...................... ............................................................... .......2 echallengesa andconseque encesofthegrowthindataandtheus ser The bas .................. se ...................... ............................................................... .......4 HarnessingthepoweroftheCloud.................................................... .......5 eflipsideofC Cloudandperformancete esting................................. .......6 The The etooldeploymentmodels sforCloudbasedperform mancetesting ......7 The eImpetussol lution............ ............................................................... .....10 Sum mmary.......... ...................... ............................................................... .....11

In ntroduct tion
Cloudbase edperformancetestinghassuddenlybe ecomeamajo orbuzzword today.From mstartups,to olargeenterp prises,everyo oneistryingto omakethem most oftheoppo ortunitiesoffe eredbytheC Cloud. Thefactist thatthatther reareseveraldistinctbenefitsthatCloudinfrastruct ture offersorganizations.Ite enablesfaster rtimetotest t,ishighlysca alableandan non demandso olution.TheCloudalsoreducestheoverallperforma ancetestingcosts. Itisimportanthowever, ,tounderstan ndthatwhyt theCloudise emergingasan paradigminp performancetesting. importantp TrendsshowthattheInternetisgrow wingatanunprecedentedrate.Globall ly billioninternetusers,withFacebookalo onecrossingt the thereareclosetotwob nusermarkan ndTwitterreceivingmillio onsoftweetseverymonth h.The 500million useoftheW Webandmob bileapplicatio onshasalsoincreasedphe enomenally,w with theavailabilityofdevice eslikesmartp phones,netbo ooksandiPad ds. Thisscenar rio,whileben neficialforend dusers,iscre eatingseveralchallengesf for applicationarchitectsan nddevelopers,whohavet toensurethatapplications sare gefficientlyan ndeffectively yunderthish huge,increasingload.The performing

Harnessin ngtheCloudfo orPerformance eTesting

concernisn notonlythet twobillionus sersontheInternet,butal lsotheincrea asein thesizeoft thedata. Today,data asizeswithinorganization nshavegoneupfromhund dredsofmillio ons ofrows,tobillionsofrowsperday.

Inter rnetGrowth

Challengesand uences Consequ Movin ngtoCloud

BigData B

Harnessin ngtheCloudfo orPerformance eTesting

Thech hallengesandco onsequencesofthe growth hindata aandthe euserbase

Businessesacknowledge ethatapplica ationperform mancedirectly yaffectstheir r bottomline e.Takethein nstanceofac companyope eratingwithin ntheadvertising domain.Th heamountite earnsisdirectlyrelatedto ohowfastitsadsaregettin ng displayedo ontheWebsit te.
It thasbeenobse erved th hatthedownti imeofa Websiteandthe W la atencyofeven millisecondscanresult m in nrevenuelosse esand cu ustomer dissatisfaction.Thus ap pplication performancedo oes matter,and m organizationsd do co onsideritason neof th hekeyattribut tesof th heirproduct st trategies.

Inthisenvironment,whencompanie esarenotsureaboutthep performance parameters sandapplicat tionscalabilit ty,movingtotheClouden nsureseffectiv ve manageme entofperform mancerequire ements. Exploitingt thepoweroftheCloudfor rPerformancetestingisga ainingimmen nse Amazonprov popularity. visioned18m millionEC2serverslastyea ar.Also,the ibingtoother rCloudvendo orssuchasM Microsofthas numberofuserssubscri significantly. increaseds Beforedisc cussingthebe enefitsofClou udbasedper rformancetes sting,itis importantt toexamineth heperforman ncetestingmodel,current tlyinuse. Theperform mancetesting gmodelinvolvesvariousc componentss suchasthe performanc cetestingtoo ols,thesystem mundertest, ,etc.thatare edeployedon n premise,w withinalocalenvironment t. Oneofthefirststepsinconductinga aperformance etestingassig gnmentwithina localenviro onmentisthe esettingupofatestlab.T Thelabrequir resinfrastruct ture, servers,hardwareands softwaretoho ostthesystem mundertest, ,andthe performanc cetestingtoo ols.Settingup psuchalabre equiresconsi iderable investment tandeffort,a andthemana agementofov verheadssuchasmaintenance costs.Anorganizationse ettingupsuchalabwillne eedtohiread dministrators sto maintainan ndsupportth helab.Thiswillfurtherinv volvetrainingcostsforthe resources,whowillperf formthesupp portrole. Also,witht thesubstantia alincreaseinInternettraf ffic,bothinte ermsofthe numberofusersandthe esizeofthed data,thecriticalchallenge eforthe performanc cetestingtea amistoscaleandtestimm mediatelytoh higher concurrenc cylevels. Thistypical llycallsforad dditionalhard dware,toollic censeextensi ions,andsett ting upthetoolsandothers softwareonth henewhardw ware,resultin nginadditional enditurewhic chmayleadt toadelaying gettingtheex xpected capitalexpe performanc cenumbers.

Harnessin ngtheCloudfo orPerformance eTesting

Anothersig gnificantaspe ectoftheonpremisemod delisthatitdoesnotmeas sure uptothegeographicald distribution.A Anapplication nwillgethits sfromallcorn ners oftheworldandinorde ertosatisfyusers,theperf formanceoft theapplicatio on emeasuredin npeaktraffic c.Thesimulat tedloadshouldmimicdiffe erent needstobe geographie esforamoreaccuratesimulationofrea alusage.Furt thermore,loa ad generatorsareanimpor rtantcompon nentofperfor rmancetestin ngandincase eof umes,needto obesetupinlargernumbers. highervolu Thegoodn newsisthatth heseissuesca anbeaddress sedbytheClo oud infrastructu ure.

Harnessi H ingthep powero oftheClo oud

TheCloudc canhelporga anizationssca aleforhighervolume,high hertransactio ons andhigherconcurrency, ,quicklyandaffordably.It thelpsinaddressingthe limitationsindicatedabo oveandoffer rsenterprises sdistinctadva antages,inclu uding thefollowin ng: Infinite ecomputingp power:TheC Cloudhasthecapabilitytosimulatemillions andmillionsofusersfromthousa andsofinstan nces.Thishel lpsintestingthe applica ationswithpr roductiondatavolumesan ndconcurrent tusers. Afforda ability:TheC Cloudhelpsinsavingtheupfrontcostsinsettingupt the perform mancetesting genvironmen ntastheCont trollersandlo oadgenerators canbeinstantiated,asandwhen nrequired. Thepayasyougob benefit:InthecaseoftheCloud,organ nizationsneed d ayforthedur rationthemachinesarebe eingused.Thu us,byplannin ng onlypa thetests,companiescansaveco ostsforusingthehardware eforalonger r periodoftime.Theinstancescan nbecreateda andconfigure edatthetime eof dterminatedafterrunning gthetests.W WiththeCloud d,companiesdo useand notnee edtoworrya aboutthelicensingcostsof fthesoftware,OS, perform mancetesting gtools,etc. Easyse etup:TheClo oudisquickandeasytose etupandthetestenvironm ment canbeupandrunni inginminutes.Manyvend dorsprovidea automatic provisio oningofthel loadgenerato orsthathelps sinquicklyse ettingupthe environ nmentandstartingthetes stexecutions.TheCloudalsooffersare eady touseenvironment twherenose eparateinstallationandconfigurationo of perform mancetesting gtoolsisrequ uired.Theins stancesarepr reconfigured das loadge eneratorsorc controllers. Abilitytochoosege eographies:T TheCloudmakesitpossible efor zationstocho oosegeographies,sothatinstancesare ecreatedandthe organiz

Harnessin ngtheCloudfo orPerformance eTesting

loadsim mulatedfrom mdifferentloc cations.Ther resultisthatt thereisamuc ch morea accuratesimu ulationofprod ductionusage eoftheapplicationundertest. Clearly,Clo oudbasedperformancetestingcanbebeneficialfor rInternetand d mobileapp plicationsthat thavealarge ecustomerba aseandhugetransaction volumes. Thoughthe eCloudoffers slotsofadvan ntages,there eareafewchallengesthat t needtobeaddressed,b beforeperform mancetestingcanbecome eviable.

Thef flipsideofCloud dandpe erformancetest ting

Despiteitsadvantages,t theCloudhas sitsshareofconcernarea as.DataSecu urity performancetestingactivi itythatisper rformedonth he isamajorissueforanyp eallthetestscripts,testd data,serverre esponseandtestresultsh have Cloud,since beenmove edtotheClou ud.Today,how wever,organ nizationscanh haveaprivate e Cloud,andsetupasepa aratetunnelf forsecurity. Anotherproblemareais stestingintra anetapplicationsontheCloud;wheret there workwiththe eclientsITde epartmentinordertorece eivepermissionto isneedtow accesstheapplicationfr romtheCloud d.Networkla atencyisano othercriticalf factor thatmaysk kewtheperfo ormancenum mbers.Itposesachallengeforintranet applications,butcanberesolvedtoa anextentbys statisticallyadjustingthe orintranetapplications,inc clusionoflate encyinthete est responsetimes.Also,fo resultsprov videsarealist ticviewofap pplicationspe erformance. Duringtheperformance etests,resour rcemonitorin nganddiagno osticactivityis sive.Incaseo oftheCloud,d datatransferthathappens sINandOUT Tof dataintens theCloudis shugeandm mayposeacha allenge. SincetheC Cloudinstance esmaynotbe eavailableall lthetime,itisimportantt to haveapert tinenttestda atamanagementstrategy,andaproper rbackup mechanism mfortestscrip pts,testdataandtestresu ults. TestscriptcreationontheCloudisatoughtaskas sitinvolvesp parameterizat tion, validationa anddebugruns.Ideally,th hescriptcreat tionshouldb bedoneinalo ocal environment,withthes scriptsbeingu uploadedontheCloudfor rfullloadtest ts.

Harnessin ngtheCloudfo orPerformance eTesting

Theto ooldeplo oymentmodelsfor Cloudb basedpe erformancetest ting

TheCloudmodel Thereares somestandar rdcomponent tsoftheperf formancetest tingtool architectur relikecontrol llers,loadgen nerators,reco ordersandan nalyzers.The variationsinthedeploym mentmodels sdependonw whereallthes secomponen nts rformancetestsarebeingconducted.O Organizations s areplacedwhiletheper havetocho oosetheappr ropriatedeploymentmodelbasedontheirneeds. InthecaseoftheCloudmodel,thep performancet testingtoolis smovedtoth he therefore,the econtroller,l loadgenerato orsandanaly yzerarealso Cloudandt deployedo ontheCloud.TheRecorderhowever,isprovidedon premisesot that userscanre ecordthetes stscript,debu ugandvalidat tethemandt thenfurtherpush totheCloudforloadtes sting.

Harnessin ngtheCloudfo orPerformance eTesting

Thereisan optionofhavingthereco ordingcomponentontheC Cloud;howev ver Cloudenviron nmentduring gscriptcreation.It thatmightresultinextrahitstotheC isrecomme endedthatth hescriptsarerecordedlocallyandthen nmovedtoth he Cloudforsc caling. Thismodelissuitablefo orperformanc cetestingofC Cloudbasedapplicationso or pplications,sin ncetheseapp plicationsget thitsfromallpartsofthe Internetap world.Thescenarioscan nbedesigned dinawaytosimulatereal listicload. Forinstanc ce,incaseanInternetappl licationisdep ployedonthe edatacenter, , where50p percentofthe etrafficiscom mingfromthe eeastcoast,2 20percentfrom thewestco oastandrema ainingfromtheAsiaPacifi icregion,itsh houldbeposs sible tospawnvirtualusersfr romtheCloud dbasedloadgeneratorsin nstantiatedfr rom ons. theseregio Thisrepresentsrealisticsimulationascomparedt totheonpremiseperform mance testingmod del.Itcanalsoprovideasmanyloadge eneratorsasr required,quic ckly andondem mand,withou uttheneedfo orhardwarep procurement,andtherefor reno massiveexpenditureonsettingupth heinfrastructure. Anotherbe enefitisthatt thereisnone eedtoinstallandconfigure etheperform mance testingtoolandotherso oftware.Ther rearenotool llicensecosts sinvolved.Th he comessimplifiedandtheo organizationisonlycharge edfortheperiod processbec thatituses stheCloudinfrastructure. TheHybrid dmodel Inthiscase e,thecompon nentsofthep performancet testingtoola aredeployed eitheronth heCloudoro onpremise.Thereareman nycombinatio onspossibleh here, butitisfea asibletotakeintoaccountthemostlike elyscenario. Here,thelo oadgenerato orsareontheCloudandal lltheotherco omponents,s such asthecont troller,record derandanalyzerareonpr remise. Thismodelismoresuita ablefororgan nizationswhic chhavealreadyinvestedin cetestlabsandhaveahardwaresetup p.Thesecomp paniescanus sethe performanc hybridmod delwhenthey yneedtoscalequickly. Oftenregularperforman ncetestingca anbedoneon npremise,no otonlytoget tthe cenumbers,butalsotode eterminetheperformance ebottlenecksin performanc theapplication.Howeve er,incasethe ereisarequir rementforte estingan applicationwithpeakvo olumes,highconcurrencie esandmoretransactions,t the Hybridmod delcanbeuse ed.

Harnessin ngtheCloudfo orPerformance eTesting

IntheHybr ridmodel,the econtrolleris sdeployedon npremiseand denablesthe euse oftheinteg gratedresour rcemonitorin nganddiagno osticfeaturesofthe performanc cetestingtoo ol.Thisalsoa addressesany ysecuritycon ncernsovercr rucial data,asthe econtrollerre emainsinhouse.Sincethe edataisavailablelocally,itcan archiveand dbereferredtoatanysuit tabletime. Incaseofr resourcemon nitoringandd diagnosticson ntheCloud,t therewillbea a hugedatat transferfrom mandintothe eCloudasthe ereispoolingofparameters likememor ry,CPU,netw workbyteseve ery5or10se econds.

Harnessin ngtheCloudfo orPerformance eTesting

TheImpet tussolut tion

ImpetusTe echnologieshasdeveloped dSandStorm,anenterprise eperformanc ce testingtool,whichcanb bedeployedo ontheCloud. .Usingthisso olution, onscanreducetheirperfor rmancetestin ngcostsby60 0percent. organizatio SandStormispreconfig guredontheC Cloudinstanc cesandoffers saReadyto use eover,thetestscriptscana alsobecreate edlocallyusin ng testenvironment.More theRecorderandautom maticallyuploadedtotheC Cloudbeforerunningthet tests. SandStormsupportsmu ultipleprotoco olsrangingfr romtheWeb, ,mobileande e maillikeHt ttp,Https,FLE EX,Silverlight t,Javaapplets s,OMAIMPS,WAP,WV,a and SMTPetc.I Itoffersinteg gratedresourcemonitoring gformostofthepopular applications,Databases serversandO OperatingSyst tems.Itprov videstheUser r wizardsforscr riptcreationa andconfigura ation. Interfacew Theanalysi isdashboards sprovideasu ummarizedviewoftestres sultswithdrill downcapabilitiestoidentifyperform mancebottlen necks(visit .impetus.com m). sandstorm.

CaseStudy:Perform mancetestingusingtheCloud
Thecustom mer:Aleading gproviderofworkforcemanagements solutionsrang ging frompayro oll,leave,sche edule,among gothers.Thecompanyoffe ersperformance benchmark kingservicest toitscustome erswhereint theperformancetesting activityisconductedatt thefacilitieso ofitscustome ers. Theneed:T ThecustomerrequestedImpetustoco omeupwitha amoreafford dable andflexible esolutionforitsclients. Thesolutio on:Impetussuggestedusin ngSandStorm montheCloud.TheRecord der component twaslocallyd deployedinthetestlabst tocreateandverifytestsc cripts. Thesescrip ptswereautomaticallyuploadedtotheCloudfortes stexecutions.The clientexposedtheapplicationtotheCloudbyset ttingappropri iatepermissio ons inthefirew wall.Theperfo ormancetest tingcouldbeeasilycomple etedintimef forall thebusines ssscenarios.T Thenetworklatencywasa adjustedbase edonthenet twork correctionf factoridentif fiedwiththehelpoftracer rtutility. Thebenefit ts:Thesolutionhelpedthecustomerto oincreasecustomer benchmark kingactivities.Thisinturnhelpedtheco ompanytoac chievehigher r revenuesandimprovec customersatisfaction.

10 0

Harnessin ngtheCloudfo orPerformance eTesting

Our rRecommendat tion

Summ mary
Inconclusio on,itcanbes saidthatmov vingtoaCloudbasedperf formancetest ting modelofferssignificantbenefitstoo organizations. .Atthesametime,when thismodel,itisimportantthatcompan niespayatten ntiontosome eof movingtot thefollowin ngrecommen ndations: Understandthe eperformanc cegoalsando objectives.Inthescenario os, herethereisn nocertaintyo onthepeakusage,itispru udenttooptf fora wh Clo oudstrategy. Che ecktoseewh hethertheor rganizationhasuncertaindemand.The e elasticityfromtheCloudinfr rastructurepr rovidesorgan nizationswith ha me echanismtod dealwithunce ertainty.Mov vingtotheClo oudforinstan nce, sav vescompanie esfrommakin ngcapitalexp penditureand dprocuring add ditionalhardw wareiftheex xistingsetupdoesmatchpeakusage,o orin cas sethepeakde emanddoesnotreachthe eexpectedca apacity. Thi inkfromtheenduserper rspective.Clo oudbasedapp plicationsand d Inte ernetapplica ationshaveto obeaccessedgloballyandcompaniesn need tod defineproperworkloadpa atternstosim mulatetherea alisticload.The

11 1

Harnessin ngtheCloudfo orPerformance eTesting

app plicationmus stbetestedatbothpeaka andnormaltr rafficlevelsfrom allthetargetedgeographical llocations. Loo okatthetype eoftheappli icationthatc canmovetot theCloud.Th his app proachisusef fulforapplica ationsthatha avehighconc currentusers, ,big dat taproblemsa andmobileap pplicationsac ccessiblefrom mtheInternet t. Cho ooseappropr riatetoolsan ndmodelsfor rtheCloudb based per rformancete esting.Them modelsandtoolsshouldbe echosenbase edon the etypeofappl lication,user base,nonfunctionalrequ uirements, scriptingrequire ements,andt testdataman nagement. mance.TheC Cloudenvironmentmightb beshared. Analyzeperform fferentvendo orshavediffer rentmechani ismofprovidingthevirtua al Dif instancesonthe eCloud.Incase,theunder rlyinghardwa areisshared, alyzetheperf formanceand dresourceco onsumption.A Anyunexplain ned ana cha angesinperfo ormancemay ybeattribute edtoeithersh haredresourc cesor the eCloudprovis sioningtool. Hav vecontinuou usperforman ncemanagem mentinplace.Performance e tes stingcanbeapartofconti inuousintegrationandcan nautomatically exe ecutecriticaltestscenarioswithinterm mediatebuilds s.Theprocess smay use eautomaticp provisioningtogenerateControllerand dloadgenerat tor instancesbased donthepeakusageoftheapplication.T Thesecan eentireproce ess. increasetheefficiencyofthe

AboutImpet tus ImpetusTech hnologiesoffersProductEng gineeringandT TechnologyR& &Dservicesforsoftwareprod ductdevelopment. Withongoing ginvestmentsinresearchan ndapplicationo ofemergingte echnologyarea as,innovativeb businessmode els,and anagileappr roach,wepartnerwithourclientbasecom mprisinglarges scaleISVsandt technologyinn novatorstodeliver cuttingedge esoftwareprod ducts.Ourexpertisespansth hedomainsofBigData,SaaS,CloudCompu uting,Mobility Solutions,Te estEngineering g,Performance eEngineering,andSocialMediaamongoth hers.

ImpetusTechnologies,Inc. vard,Suite450 0,SanJose,CA95129,USA 5300StevensCreekBoulev @impetus.com Tel:408.252.7111|Email:inquiry@ velopmentCentersINDIA:NewDelhiBangaloreIn ndoreHydera abad RegionalDev Toknowmo orevisit:http:/ //www.impetus.com

Di isclaimers
Theinformationcon ntainedinthisdocumentistheprop prietaryandexclus sivepropertyofIm mpetusTechnologi iesInc.exceptaso otherwiseindicate ed.Nopartof isdocument,inwh holeorinpart,ma aybereproduced, ,stored,transmitted,orusedforde esignpurposeswithoutthepriorwri ittenpermissiono ofImpetus thi 12 2 TechnologiesInc.