Professional Documents
Culture Documents
GEOINT Training
June 2015
1 What is Deep Learning?
2 GPUs and Deep Learning
3 cuDNN and DiGiTS
AGENDA 4 Neural Network Motivation
5 Working with Deep Neural Networks
6 Using Caffe for Deep Learning
7 Summary DL For GEOINT
2
What is Deep Learning?
3
DATA SCIENCE LANDSCAPE
Data Analytics
4
DEEP LEARNING & AI
CUDA for
Deep Learning
Machine Learning is in some sense a
rebranding of AI.
5
INDUSTRIAL USE CASES
machine learning is pervasive
6
TRADITIONAL ML HAND TUNED FEATURES
Images/video
Audio
Key components
Task
Features
Model
Learning Algorithm
8
THE PROMISE OF MACHINE LEARNING
ML Systems Extract Value From Big Data
9
WHAT MAKES DEEP LEARNING DEEP?
Todays Largest
Networks
~10 layers
1B parameters
10M images
~30 Exaflops
~30 GPU days
Input Result
10
IMAGE CLASSIFICATION WITH DNNS
Training Inference
truck
11
IMAGE CLASSIFICATION WITH DNNS
Training Typical training run
Pick a DNN design
cars buses trucks motorcycles
Input 100 million training images
spanning 1,000 categories
One week of computation
Test accuracy
If bad: modify DNN, fix training
set or update training parameters
12
DEEP LEARNING ADVANTAGES
Deep Learning
Dont have to figure out the features ahead of time.
Use same neural net approach for many different problems.
Fault tolerant.
Scales well.
14
CONVOLUTIONAL NETWORKS BREAKTHROUGH
16
Slide credit: Yann Lecun, Facebook & NYU
RECURRENT NEURAL NETWORK - RNN
AKA: LSTM
17
SENSOR/PLATFORM CONTROL
Reinforcement learning
(predicted future reward, actual reward)
Data sequence
Control
policy
Applications
Sensor tasking
Autonomous vehicle
navigation
[11] Google DeepMind in Nature
18
WHY IS DEEP LEARNING HOT NOW?
Three Driving Factors
350 millions
images uploaded
per day
19
GEOINT ANALYSIS WORKFLOW
TODAY
Big Data BOTTLENECK
NUMBERS
Metadata Human Mission focused
filters perception analysis
IMAGES VIDEOS
Near perfect
SOUNDS TEXT Noisy content
perception
VISION
Big Data
NUMBERS DL based Mission focused
Content
machine analysis
based filters
perception
IMAGES VIDEOS Near
SOUNDS TEXT Mission relevant
human level
content
perception
20
GPUs and Deep Learning
21
GPUs THE PLATFORM FOR DEEP LEARNING
GPU Entries
Image Recognition Challenge 120
100 110
1.2M training images 1000 object categories 80
60
Hosted by 60
40
20
4
0
2010 2011 2012 2013 2014
person
car bird
Classification Error Rates
helmet frog
30%
motorcycle 28%
25% 26%
20%
person 15% 16%
person 12%
hammer 10%
dog
flower pot 7%
5%
chair
power drill 0%
2010 2011 2012 2013 2014
22
GPUS MAKE DEEP LEARNING ACCESSIBLE
GOOGLE DATACENTER STANFORD AI LAB
Deep learning with COTS HPC
systems
A. Coates, B. Huval, T. Wang, D. Wu,
A. Ng, B. Catanzaro
ICML 2013
23
Deep Image: Scaling up Image Recognition
IMAGENET
CHALLENGE Baidu: 5.98%, Jan. 13, 2015
Accuracy %
DNN
Delving Deep into Rectifiers: Surpassing Human-Level
Performance on ImageNet Classification
84%
Microsoft: 4.94%, Feb. 6, 2015
CV
74%
72% Batch Normalization: Accelerating Deep Network
Training by Reducing Internal Covariant Shift
2010 2011 2012 2013 2014
Google: 4.82%, Feb. 11, 2015
24
GOOGLE KEYNOTE AT GTC 2015
25
GOOGLE USES DEEP LEARNING FOR UNDERSTANDING
What are all these numbers? What are all these words?
Large-Scale Deep Learning For Building Intelligent Computer Systems, Jeff Dean (Google), http://www.ustream.tv/recorded/60071572
26
WHY ARE GPUs GOOD FOR DEEP LEARNING?
ImageNet Challenge Accuracy
Neural 93%
GPUs
Networks
Inherently
Parallel 88%
Matrix
Operations
84%
FLOPS
GPUs deliver --
- same or better prediction accuracy
- faster results
- smaller footprint 74%
- lower power 72%
28
DEEP LEARNING EXAMPLES
29
GPU-ACCELERATED
DEEP LEARNING FRAMEWORKS
CUDA-
CAFFE TORCH THEANO KALDI
CONVNET2
Deep Learning Scientific Computing Math Expression Deep Learning Speech Recognition
Domain
Framework Framework Compiler Application Toolkit
Multi-CPU (nnet2)
Text-based
Python, Lua,
Interface(s) definition files, Python C++ C++, Shell scripts
MATLAB
Python, MATLAB
Embedded (TK1)
http://developer.nvidia.com/deeplearning
30
cuDNN
31
HOW GPU ACCELERATION WORKS
Application Code
Compute-Intensive Functions
Rest of Sequential
5% of Code CPU Code
GPU ~ 80% of run-time CPU
+ 32
WHAT IS CUDNN?
cuDNN is a library of primitives for deep learning
Applications
33
ANALOGY TO HPC
cuDNN is a library of primitives for deep learning
Application
Fluid Dynamics
Computational Physics
Tesla Titan
Intel CPUs IBM Power
TK1 TX1
34
DEEP LEARNING WITH CUDNN
cuDNN is a library of primitives for deep learning
Applications
Frameworks
cuDNN
35
CUDNN ROUTINES
36
CONVOLUTIONS THE MAIN WORKLOAD
Very compute intensive, but with a large parameter space
37
EXAMPLE OVERFEAT LAYER 1
/* Allocate memory for Filter and ImageBatch, fill with data */
cudaMalloc( &ImageInBatch , ... );
cudaMalloc( &Filter , ... );
...
/* Set descriptors */
cudnnSetTensor4dDescriptor( InputDesc, CUDNN_TENSOR_NCHW, 128, 96, 221, 221);
cudnnSetFilterDescriptor( FilterDesc, 256, 96, 7, 7 );
cudnnSetConvolutionDescriptor( convDesc, InputDesc, FilterDesc,
pad_x, pad_y, 2, 2, 1, 1, CUDNN_CONVOLUTION);
CPU is 16 core Haswell E5-2698 at 2.3 GHz, with 3.6 GHz Turbo
GPU is NVIDIA Titan X
39
CUDNN EASY TO ENABLE
41
DIGITS
Interactive Deep Learning GPU Training System
developer.nvidia.com/digits
42
DIGITS
Deep Learning GPU Training System
Available at
developer.nvidia.com/digits
Free to use
v1.0 supports classification on images
Future versions: More problem types
and data formats (video, speech)
Two options
44
Main Console
DIGITS Workflow
Configure your Network
Create your database
Start training
OR
Image parameter
options
OR add it here
Start training
47
DIGITS
Visualize DNN performance in real time
Compare networks
Download
network files
Accuracy and
loss values
during training
Learning rate
Classification on
the with the
network
snapshots
48
Neural Network Motivation
49
NEURAL NETWORK MOTIVATION
One learning algorithm hypothesis
We can connect any sensor to any part of the brain, and the brain figures it out.
50
NEURAL NETS SCALE EASIER
Why use neural nets? Consider computer vision
When the decision space is non-linear, and the number of features is very large.
X
X
Pixel 1 Pixel 2
x1 z = x1 w1 + x2 w2 + x3 w3
w1
w2 1
x2 Activation =
1 + e-z
w3
x3 1
1
x2 x3 Activation
-30
0 0 0
20
x2 1 0 0
20 0 1 0
x3 1
1 1
53
DEEP VERSUS TWO-LAYER NETWORKS
Theory says two fully-connected layers can solve any problem.
G. Cybenko - Approximation by Superpositions of a Sigmoidal Function, Mathematics of Control, Signals and Systems, 1989
CUDA for
Machine Learning
54
Working with Deep Neural Networks
55
OVERFITTING & UNDERFITTING
Important terminology
High Bias
High Variance
Underfitting Overfitting
Just right
56
LEARNING CURVE
Underfitting example
High Bias
validation
training
Actions
Increase size of neural network.
training
Actions
Get more data / examples Augmentation
Mirror images
Distorted / blurred
Rotations
Color changes
59
NEURAL NETWORK GUIDANCE
60
NEURAL NETWORK STRENGTH
Good
Neural Networks + Big Data =
Stuff
61
Using Caffe for Deep Learning
62
LEARNING A BIT MORE WITH CAFFE
Lets learn a bit more about DNNs by learning a bit about Caffe.
Well learn about layer types, and how to think about neural network architecture.
Though well use Caffe as our working example, these concepts are useful in general.
63
NETWORKS, LAYERS & BLOBS
Network Layer2
blob
Neural Layer2
Blob describes data
Layer1 - batch of images
blob - model parameters
Data
blob
Data Layer
64
OVERALL NETWORK STRUCTURE
Ignoring blobs here
Loss Layer
Neural Layer
Neural Layer
Data Layer
65
CAFFE MODELS DEFINED IN PLAINTEXT
66
CAFFE NEURAL LAYERS
Convolution
Neural Layer Inner Product = Fully Connected
1 or more Neural Layer
Pooling
Neural Layer
Local Response Normalization
67
CAFFE LOSS LAYERS
Softmax (Logistic)
Accuracy
Neural Layer
Neural Layer
Data Layer
68
Summary Deep Learning for GEOINT
69
DEEP LEARNING AS GEOINT FORCE MULTIPLIER
Managing Big Data
Real-time near-human level perception at web-scale
Data exploration and discovery
Semantic and similarity based search
Dimensionality reduction
Transfer learning
Model sharing
Compact model representations
Models can be fine-tuned based on multiple analysts feedback
70
SUMMARY - DL FOR GEOINT
Deep Learning
Adaptable to many varied GEOINT workflows and
deployments scenarios
Available to apply in production and R&D today
Approachable using open-source tools and libraries
71
Machine Learning and Data Analytics
72
TRADITIONAL MACHINE LEARNING
For your many non-DL applications
73
GPU ACCELERATION FOR GRAPH ANALYTICS
Comms & Social networks 1 GPU vs 60 Nodes
Cyber pattern recognition
280x vs optimized Spark
Shortest path finding
1440x vs Spark
3420x vs Hadoop
PageRank : 19x Speedup
1.2
Time per iteration (s)
1 0.967 3
2
1
1
0.8 2
Lower 2
0.6 is 2
0
Better 1 1
0.4 2
2 2
0.2 1
0.051 2 3
0 2
3
Intel Xeon E5-2690 v2
74
Thank you!