Course organization

Chapter 1: Introduction
Chapter 2: Bayesian Decision Theory

Chapter 3: Maximum likelihood and Bayesian

estimation.
Chapter 4: Nonparametric techniques
Chapter 5: Linear Discriminant Functions
Chapter 6: Multilayer Neural Networks
Chapter 8: NON-METRIC METHODS
Pattern Classification, Chapter 1

Lab.
Research Activity
Divide Students into groups.
Each group has a specific job.
Programming Tools

Matlab or other programming lang.

Start with basic image processing operations.
Choose a PR model to implement

Pattern Classification, Chapter 1

Chapter 1: Introduction to Pattern

Recognition
Machine Perception
An Example
Pattern Recognition Systems

The Design Cycle

Learning and Adaptation
Conclusion

Pattern Classification, Chapter 1

Human Perception
Humans have developed highly sophisticated Skills for

sensing their environment and taking

according to what they observe, e.g.,

actions

Recognizing a face,
Understanding spoken words,
Reading handwriting,
Distinguishing fresh food from its smell.

We would like to give similar capabilities to machines.

Machine Perception
Build a machine that can recognize patterns:
Speech recognition
Fingerprint identification
OCR (Optical Character Recognition)
DNA sequence identification

Pattern Classification, Chapter 1

Pattern Samples

Pattern Classification, Chapter 1

What is pattern recognition?

Definitions from the literature

The assignment of a physical object or event to one of

several pre-specified categories Duda and Hart

A problem of estimating density functions in a highdimensional space and dividing the space into the regions of
categories or classes Fukunaga
Given some examples of complex signals and the correct
decisions for them, make decisions automatically for a stream
of future examples Ripley
The science that concerns the description or classification
(recognition) of measurements Schalkoff
The process of giving names to observations x,
Schrmann
Pattern Recognition is concerned with answering the
question What is this? Morse

Examples of applications
Handwritten: sorting letters by postal code,
input device for PDAs.

Optical Character

Printed texts: reading machines for blind

people, digitalization of text documents.

Recognition (OCR)

Biometrics

Face recognition, verification, retrieval.

Finger prints recognition.
Speech recognition.

Diagnostic systems

Medical diagnosis: X-Ray, EKG analysis.

Machine diagnostics, waster detection.

Military applications

Automated Target Recognition (ATR).

Image segmentation and analysis (recognition
from aerial or satelite photographs).

Why we need pattern recognition?

Human beings can easily recognize things or objects

based on past learning experiences! Then how about

computers?

Features
Feature is any distinctive aspect, quality or characteristic.
Features may be symbolic (i.e., color) or numeric (i.e.,

height)
Definitions
The combination of d features is represented as a ddimensional column vector called a feature vector
The d-dimensional space defined by the feature vector is
called the feature space
Objects are represented as points in feature space. This
representation is called a scatter plot

Features (cont.)

Basic Structure
Two basic factors: Feature & Classifier
Feature:

Car
Boundary
Classifier: Mechanisms and methods to define what
the pattern is.

System structure
The feature should be well-chosen to describe the

pattern!!
Knowledge: experience, analysis, trial&error
The classifier should contain the knowledge of each
pattern category and also the criterion or metric to
discriminate among patterns classes.
Knowledge: direct defined or training

Pattern
Pattern is a composite of features characteristic of

an individual
In classification tasks, a pattern is a pair of
variables {x,} where

x is a collection of observations or features (feature

vector)

is the concept behind the observation (label)

Feature extraction
Task: to extract features which are good for classification.
Good features: Objects from the same class have similar feature values.
Objects from different classes have different values.

Good features

Bad features

More feature properties

Classifiers
The task of a classifier is to partition feature space

into class-labeled decision regions

Borders between decision regions are called decision boundaries
The classification of feature vector x consists of determining which

decision region it belongs to, and assign x to this class.

t

classifier can be represented

discriminated function i
The classifier assigns a feature vector x to class

If

g i ( x) g j ( x)

i j

as

set

of

Pattern Recognition Models

The four best known approaches
Template matching: measure similarity an image and a

template.
Structural (or Syntactic) PR: pattern classes

represented by means of formal structures as

grammars, automata, strings, etc.
Statistical PR: based on underlying statistical model of

patterns and pattern classes.

Neural Network: classifier is represented as a network

of cells modeling neurons of the human brain

(connectionist approach).

A comparison

Pattern Classification, Chapter 1

20

An Example
Sorting incoming Fish on a conveyor according to

species using optical sensing

Sea bass
Species
Salmon

Pattern Classification, Chapter 1

21

 Problem Analysis
Set up a camera and take some sample images to extract

features

Length
Lightness
Width
Number and shape of fins
Position of the mouth, etc
This is the set of all suggested features to explore for use in our
classifier!
Pattern Classification, Chapter 1

22

 Preprocessing
Use a segmentation operation to isolate fishes from one

another and from the background

Information from a single fish is sent to a feature extractor

whose purpose is to reduce the data by measuring certain

features
The features are passed to a classifier

Pattern Classification, Chapter 1

23

Pattern Classification, Chapter 1

24

 Classification
Select the length of the fish as a possible feature for

discrimination

Pattern Classification, Chapter 1

25

Pattern Classification, Chapter 1

26

The length is a poor feature alone!

Select the lightness as a possible feature.

Pattern Classification, Chapter 1

27

Pattern Classification, Chapter 1

28

 Threshold decision boundary and cost relationship

Move our decision boundary toward smaller values of

lightness in order to minimize the cost (reduce the number of

sea bass that are classified salmon!)

Task of decision theory

Pattern Classification, Chapter 1

29

 Adopt the lightness and add the width of the fish

Fish

xT = [x1, x2]

Lightness

Pattern Classification, Chapter 1

Width

30

Pattern Classification, Chapter 1

31

 We might add other features that are not correlated with the

ones we already have. A precaution should be taken not to

reduce the performance by adding such noisy features
Ideally, the best decision boundary should be the one which

provides an optimal performance such as in the following

figure:

Pattern Classification, Chapter 1

32

Overfitting and underfitting

underfitting

good fit

overfitting

Figure: Overly complex models for the fish will lead to decision boundaries that are
complicated. While such a decision may lead to perfect classification of our training
samples, it would lead to poor performance on future patterns. The novel test point
marked ? is evidently most likely a salmon, whereas the complex decision boundary
shown leads it to be misclassified as a sea bass.

Pattern Classification, Chapter 1

34

 However, our satisfaction is premature because the

central aim of designing a classifier is to correctly

classify novel input

Issue of generalization!

Pattern Classification, Chapter 1

35

Figure 1.6: The decision boundary shown might represent the optimal tradeoff
between performance on the training set and simplicity of classifier.
Pattern Classification, Chapter 1

36

Pattern Recognition System

Pattern Classification, Chapter 1

37

Pattern Recognition Systems

Sensing
Use of a transducer (camera or microphone)
PR system depends of the bandwidth, the resolution

sensitivity distortion of the transducer

Segmentation and grouping

Patterns should be well separated and should not

overlap
Pattern Classification, Chapter 1

38

 Feature extraction
Discriminative features
Invariant features with respect to translation, rotation and
scale.
Classification
Use a feature vector provided by a feature extractor to assign
the object to a category
Post Processing
Exploit context input dependent information other than
from the target pattern itself to improve performance
Pattern Classification, Chapter 1

39

The Design Cycle

Data collection

Feature Choice
Model Choice
Training

Evaluation
Computational Complexity

Pattern Classification, Chapter 1

40

Design Cycle

Pattern Classification, Chapter 1

41

 Data Collection
How do we know when we have collected an adequately
large and representative set of examples for training and
testing the system?
Feature Choice
Depends on the characteristics of the problem domain.
Simple to extract, invariant to irrelevant transformation
insensitive to noise.
Model Choice
Unsatisfied with the performance of our fish classifier
and want to jump to another class of model
Pattern Classification, Chapter 1

42

 Evaluation
Measure the error rate (or performance and switch from
one set of features to another one.
Computational Complexity
What is the trade-off between computational ease and
performance?
(How an algorithm scales as a function of the number of
features, patterns or categories?)
Training
Use data to determine the classifier. Many different
procedures for training classifiers and choosing models
Pattern Classification, Chapter 1

43

Learning and Adaptation

Supervised learning
A teacher provides a category label or cost for each

pattern in the training set

Unsupervised learning
The system forms clusters or natural groupings of

the input patterns

Pattern Classification, Chapter 1

44

Classification and Clustering

Classification (Supervised learning)
Clustering (Unsupervised learning)

Pattern Classification, Chapter 1

45