Distributed R For Big-Data Analitics

BIG DATA DEFINITION
Parallelization principles
Tools
Summary
Big Data Analytics Using R
Eddie Aronovich
October 23, 2014
Eddie Aronovich Big Data Analytics Using R

BIG DATA DEFINITION
Tools
Summary
Table of contents
1 BIG DATA DEFINITION
Definition
Characteristics
Scaling
Challange
2 Parallelization principles
Divide and Conquer
Amdahls and Gustafsons Law
Life experience
Where to parallelize ?
Storage issues
3 Tools
Explicit Parallelism
Hadoop
Hadoop - HDFS
BIG DATA DEFINITION Definition
Parallelization principles Characteristics
Tools Scaling
Summary Challange
Gartners term (probably)
Simple (personal) Definition

Too much data to be easily processed.
Used mainly for marketing and FUD

Tools Scaling
Summary Challange
Characteristics

Tools Scaling
Summary Challange
Characteristics
Volume

Tools Scaling
Summary Challange
Characteristics
Volume
Velocity

Tools Scaling
Summary Challange
Characteristics
Volume
Velocity
Verity

Tools Scaling
Summary Challange
Characteristics
Volume
Velocity
Verity
Veracity (integrity)

Tools Scaling
Summary Challange
ScaleUp vs. ScaleOut

Tools Scaling
Summary Challange
Scale Up - more ... in one box

Tools Scaling
Summary Challange

expensive (growing exp)
closed box approach
technical complexity

Tools Scaling
Summary Challange

closed box approach
Scale Out - more boxes

Tools Scaling
Summary Challange

closed box approach
cheap (avg.)
best of breed
usage is not trivial

Tools Scaling
Summary Challange

closed box approach
cheap (avg.)
best of breed
usage is not trivial
You want Scale Out? Well, Scale Out costs. And right here is
where you start paying ... in sweat....

Tools Scaling
Summary Challange
How long it takes to read / write data ?

Tools Scaling
Summary Challange
Generating a 1e7 var using runif() and writing it - 10-16 sec.

Tools Scaling
Summary Challange

Reading the same (96MB) file takes 1.5-6 sec.

Tools Scaling
Summary Challange

Generating 1e9 var was not possible on the host I used

Tools Scaling
Summary Challange

Calculating mean() and var() on all data was not possible

Tools Scaling
Summary Challange

Calculating mean() and var() on all data was not possible

Tools Scaling
Summary Challange
And in parallel ?

Tools Scaling
Summary Challange
And in parallel ?
Create 100 files, each size 1e7 and write them on the disk

Tools Scaling
Summary Challange
And in parallel ?
For each file: read it, compute mean() and var() for each file

Tools Scaling
Summary Challange
And in parallel ?
BUT - reading 100 files takes 100 times reading of one file :-(

Tools Scaling
Summary Challange
And in parallel ?
Compute mean and var of all files and ....

Tools Scaling
Summary Challange
And in parallel ?
Compute mean and var of all files and ....compute total VAR

Tools Scaling
Summary Challange
Now you know why you are here at this time !

Tools Scaling
Summary Challange
One is not enough

Data
Data can not be gathered on one node

Tools Scaling
Summary Challange
One is not enough

Data
One computer can not process all data

Tools Scaling
Summary Challange
One is not enough

Data
Data transfer is not feasible

Tools Scaling
Summary Challange
One is not enough

Data
Processors
One process, One thread (old and simple)

Tools Scaling
Summary Challange
One is not enough

Data
Processors
Multiple processes, One host

Tools Scaling
Summary Challange
One is not enough

Data
Processors
Multiple hosts

Tools Scaling
Summary Challange
One is not enough

Data
Processors
Multiple hosts
Reference
High-Performance and Parallel Computing with R
by Derik Eddelbuettel
Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Splitting tasks - How to ?
Functional

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Functional - requires sync

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

Data

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

Data - requires pre/post processing

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

Combination

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

Combination Hadoop made it easy !

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Can we parallelize forever ?
Gustafsons Law
S(P) = P (P 1)
S - scale-up, P - # of processors,
- proportion that can not be parallelized
Amdahls Law
T (P) = T (1)( + P1 (1 ))
T - processing time

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Thumb rules
Programs are small, data is big move program to data!

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Thumb rules

Parallel processing requires conversion of results Do it wise!

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Thumb rules

Queuing systems overhead is proportional to # of jobs & hosts

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Thumb rules

Multiple hosts might require authentication process.

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Thumb rules

Multiple hosts might require authentication process.
Checklist: Pre-processing, Synchronization and
Post-processing

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Little supercomputing center
Local cluster

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Local cluster
Hosting services

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Local cluster
Hosting services
Cloud

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Local cluster
Hosting services
Cloud
Storage (beware of outgoing traffic costs)

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Local cluster
Hosting services
Cloud
CPUs

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
Local cluster
Hosting services
Cloud
CPUs
Nice tool: segue - Allows running R jobs on AWS

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues
The data wont feet in ...
Loading the data is the first stage, but ...

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

Mapping instead of loading

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

The data is large and the memory is small ...

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

I/O is a performance killer

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

NAS / SAN is usually better than simple disk

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

Mounting remote file systems e.g. sshfs (degrades
performance)

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

performance)
Distributed file systems (w/o parallel processing) usually
cheap, maintenance

Divide and Conquer
BIG DATA DEFINITION
Life experience
Tools
Summary
Storage issues

performance)
Distributed file systems (w/o parallel processing) usually
cheap, maintenance
Move the program, not the data !

BIG DATA DEFINITION Explicit Parallelism
Parallelization principles Hadoop
Tools Hadoop - HDFS
Summary Hadoop - MR
Snow and Snowfall
MPI tools (beyond the scope of this talk)
Snow and Snowfall

https://www.stat.berkeley.edu/classes/s244/snowfall.pdf
biopara

Tools Hadoop - HDFS
Summary Hadoop - MR
Clusters and Grids
Queuing systems:
Can run almost any program
Very simple to use (for command liners)
Grids:
Requires authentication issues
Huge amount of resources
Ubiquitous mostly in the academic environment

Tools Hadoop - HDFS
Summary Hadoop - MR
Example: condor queuing system
Executable = / usr / local / bin / R

arguments = -- vanilla -- file = path / file . r
w h e n _ t o_ t ra n sf e r _ o u t p u t = ON_EXIT_OR_EVICT
Log = eddiea - ed2 . log
Error = del . eddiea - ed2 . err . $ ( Process )
Output = del . eddiea - ed2 . out . $ ( Process )
Requirements = ARCH ==" X86_64 " && \
( target . FreeMemoryMB > 6000 )
notification = never
Universe = VANILLA
Queue 3000

Tools Hadoop - HDFS
Summary Hadoop - MR
Clouds
Use resources as service (Ben-Yehuda et al.)

Very cheap for peak usage or low frequency
Wrong usage might be very expensive
Look for the service you need (it would usually be better and
cheaper)
In case of problem - Google it.

Tools Hadoop - HDFS
Summary Hadoop - MR
Hadoop - overview
An open source reliable, scalable, distributed computing

system
Designed to scale up from single servers to thousands of
machines
Each offering local computation and storage
Designed to detect and handle failures at the application layer
or hardware

Tools Hadoop - HDFS
Summary Hadoop - MR
Hadoop - components
Hadoop Common
HDFS
YARN, MR
Many other tools (incl. R package)

Tools Hadoop - HDFS
Summary Hadoop - MR
Using hadoop file system (HDFS)
library ( rhdfs )
hdfs . init ()
hdfs . ls ( / )

Tools Hadoop - HDFS
Summary Hadoop - MR
Using hadoop file system (HDFS)
library ( rJava )
. jinit ( parameters =" - Xmx8g ")

Tools Hadoop - HDFS
Summary Hadoop - MR
Writing an R object (1e8) to hdfs
> f = hdfs . file ("/ a . hdfs " ," w " , buffersize =10485760
> system . time ( hdfs . write (a ,f , hsync = FALSE ))
user system elapsed
7.392 2.192 15.072
> hdfs . close ( f )
[1] TRUE

Tools Hadoop - HDFS
Summary Hadoop - MR
Writing an R object (1e8) to tmpfs
> a < - runif (1 e8 )

> system . time ( save (a , file = a1 ))
user system elapsed
99.648 0.316 99.893

Tools Hadoop - HDFS
Summary Hadoop - MR
Hadoop - map reduce
Splits each process into Map and Reduce

map - Classifies the data that would be process together
reduce - process the data
Hadoop takes care of all the behind the scenes

Tools Hadoop - HDFS
Summary Hadoop - MR
MR parameters
library ( rmr2 )
Sys . setenv ( HADOOP_STREAMING =".../ hadoop - streaming
Sys . setenv ( HADOOP_HOME =".../ hadoop ")
Sys . setenv ( HADOOP_CMD =".../ bin / hadoop ")

Tools Hadoop - HDFS
Summary Hadoop - MR
Simple mapper
pi . map . fn < - function ( n ){

X < - runif ( n )
Y < - runif ( n )
S = sum (( X ^2+ Y ^2) <1)
return ( keyval (1 , n / S ))
}

Tools Hadoop - HDFS
Summary Hadoop - MR
MR parameters
pi . reduce . fn < - function ( p ){
keyval (4* mean ( p ))

}

Tools Hadoop - HDFS
Summary Hadoop - MR
The MR process
PI = function (n , output = NULL ){

mapreduce ( input =n , output = output ,
input . format =" text " ,
map = pi . map . fn , verbose = TRUE )
}

Tools Hadoop - HDFS
Summary Hadoop - MR
From command line
hadoop jar { PATH }/ hadoop - streaming -2.5.1. jar

- file map . R - mapper map . R
- file reduce . R - reducer reduce . R
- input input_file - output out }
map.R and reduce.R in this case have to be Rscript

Files should be located in HDFS

BIG DATA DEFINITION
Tools
Summary
The bottom line
Moving to cloud increases agility

BIG DATA DEFINITION
Tools
Summary
The bottom line

Moving to cloud reduces costs

BIG DATA DEFINITION
Tools
Summary
The bottom line

Moving to cloud improve operations

BIG DATA DEFINITION
Tools
Summary
The bottom line

Moving to cloud offers new business

BIG DATA DEFINITION
Tools
Summary
The bottom line

Moving to cloud offers new business
You cant refuse !

BIG DATA DEFINITION
Tools
Summary
Thank you

Distributed R For Big-Data Analitics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Distributed R For Big-Data Analitics

Uploaded by

Copyright:

Available Formats

BIG DATA DEFINITION

Big Data Analytics Using R

October 23, 2014

Eddie Aronovich Big Data Analytics Using R

Gartners term (probably)

Simple (personal) Definition

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

ScaleUp vs. ScaleOut

Eddie Aronovich Big Data Analytics Using R

ScaleUp vs. ScaleOut

Scale Up - more ... in one box

Eddie Aronovich Big Data Analytics Using R

ScaleUp vs. ScaleOut

Scale Up - more ... in one box

Eddie Aronovich Big Data Analytics Using R

ScaleUp vs. ScaleOut

Scale Up - more ... in one box

Eddie Aronovich Big Data Analytics Using R

ScaleUp vs. ScaleOut

Scale Up - more ... in one box

Eddie Aronovich Big Data Analytics Using R

ScaleUp vs. ScaleOut

Scale Up - more ... in one box

Eddie Aronovich Big Data Analytics Using R

How long it takes to read / write data ?

Eddie Aronovich Big Data Analytics Using R

How long it takes to read / write data ?

Generating a 1e7 var using runif() and writing it - 10-16 sec.

Eddie Aronovich Big Data Analytics Using R

How long it takes to read / write data ?

Generating a 1e7 var using runif() and writing it - 10-16 sec.

Eddie Aronovich Big Data Analytics Using R

How long it takes to read / write data ?

Generating a 1e7 var using runif() and writing it - 10-16 sec.

Eddie Aronovich Big Data Analytics Using R

How long it takes to read / write data ?

Generating a 1e7 var using runif() and writing it - 10-16 sec.

Eddie Aronovich Big Data Analytics Using R

How long it takes to read / write data ?

Generating a 1e7 var using runif() and writing it - 10-16 sec.

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Eddie Aronovich Big Data Analytics Using R

Now you know why you are here at this time !

Eddie Aronovich Big Data Analytics Using R

One is not enough

Eddie Aronovich Big Data Analytics Using R

One is not enough

Eddie Aronovich Big Data Analytics Using R

One is not enough

Eddie Aronovich Big Data Analytics Using R

One is not enough

Eddie Aronovich Big Data Analytics Using R

One is not enough