Computer Science 212

 Title: Data Structures and Algorithms

 Instructor: Harry Plantinga
 Text: Algorithm Design: Foundations,
Analysis, and Internet Examples by Goodrich
and Tamassia
 Computer Science 212
Data Structures and Algorithms
 The Heart of Computer Science
 Data structures
 Study of important algorithms
 Algorithm analysis
 Algorithm design techniques
Plus
 Intelligent systems

 Course Information:
http://cs.calvin.edu/curriculum/cs/212/
Grades, turning in programs on moodle
Why study DS & Algorithms?
 Some problems are difficult to solve and
good solutions are known
 Some “solutions” don’t always work
 Some simple algorithms don’t scale well
 Data structures and algorithms make good
tools for addressing new problems
 Interviews
 Fun! Beauty! Joy!
Place in the curriculum
 The study of interesting areas of computer
science: upper-level electives
 108, 112: basic tools (like algebra to
mathematics)
 212: More advanced tools. Introduction to
the science of computing. (A little more
mathematical than other core CS courses, but not as
rigorous as a real math course.)
 Example: Search and Replace
 Goal: replace one string with another
 Which version is better?

#!/usr/bin/perl #!/usr/bin/perl
my $a = shift; my $a = shift;
my $b = shift; my $b = shift;
my $input = ""; my $input = "";

while (<STDIN>) { while (<STDIN>) {

$input .= $_; $input .= $_;
} }

while ($input =~ m/$a/) { $input =~ s/$a/$b/g;

$input =~ s/$a/$b/;
}
Empirical Analysis
 Implement
 Gather runtime data for various inputs
 Analyze runtime vs. size

 Example: suppose we test swap1

 Input sizes: 1, 2, 4, 8, 16, 32, 64, 128, 256
 Runtimes: 70, 110, 250, 560, 810, 1750, 6510,
25680, 102050
 swap1 runtime analysis
Swap1 sw ap1 runtime
size (ms)
1000000
1 70
100000 y = 1.555x 2 - 0.3518x + 232.72
2 110
Runtime (ms)

10000
4 250
1000
8 560
100
16 810
10
32 1750 1
64 6510 1 10 100 1000
Problem size (1000s)
128 25680
256 102050
 swap2 runtime analysis
size swap2
4 70 swap2 runtimes

8 50
7000
16 70 6000
32 80 5000
Runtime (ms)

64 100 4000
128 120 3000

256 230 2000

512 380 1000

0
1024 580
0 5000 10000 15000 20000
2048 910
Problem size (1000s)
4096 1680
8192 3130
16384 6160
 Runtime vs. Problem Size

120000

100000

80000

swap1
60000
swap2

40000

20000

0
0 2000 4000 6000 8000 10000 12000 14000 16000 18000
 Runtime vs. Size, lg-lg plot

18
y = 1.9577x + 0.957
16

14

y = 0.9175x - 0.2822
12
Lg (runtime)

10

0
0 2 4 6 8 10 12 14 16
Lg(problem size)

Note that slope gives growth rate of runtime

Note that first few points are inflated by startup costs
Empirical runtime analysis
 What can we say about the runtimes of these
two algorithms?
 Could give runtime as a function of input size, for
a particular implementation
 But would that apply on different hardware?
 What about a different implementation in C++
instead of Perl?
 Is there anything we can say that is true in
general?
 Is version 2 better than version 1 in general?
Why?
Empirical analysis: conclusions
 Empirical analyses are implementation-
dependent
 You have to use representative sample data
 You can learn something about the growth
rate from the slope or curve of the runtime
graph
 Bad algorithms can’t be fixed with faster
computers
 Big companies (e.g. Microsoft) are not
immune to bad algorithms
What’s the runtime?

for i  0 to n  1 do
for j  0 to n  1 do
if A[i]  A[j] then
A[j]  A[i]
 Theoretical Analysis
 Uses a high-level description of the algorithm
instead of an implementation
 Characterizes running time as a function of the input
size, n.
 Takes into account all possible inputs, often analyzing
the worst case
 Allows us to evaluate the speed of an algorithm
independent of the hardware/software environment
 Pseudocode (§1.1)
 High-level description Example: find max
of an algorithm element of an array
 More structured than Algorithm arrayMax(A, n)
English prose
Input array A of n integers
 Less detailed than a Output maximum element of A
program
 Preferred notation for currentMax  A[0]
describing algorithms for i  1 to n  1 do
 Hides program design if A[i]  currentMax then
issues currentMax  A[i]
return currentMax
Pseudocode Details
 Control flow  Method call
 if … then … [else …] var.method (arg [, arg…])
 while … do …  Return value
 repeat … until … return expression
 for … do …  Expressions
 Indentation replaces braces  Assignment
(like  in Java)
 Method declaration  Equality testing
Algorithm method (arg [, arg…]) (like  in Java)
Input … n2 Superscripts and other
Output … mathematical
formatting allowed
 Questions…
 Can a program be asymptotically faster on
one type of CPU vs another?

 Do all CPU instructions take equally long?

The Random Access Machine
(RAM) Model
 A CPU

 An potentially unbounded bank 2

of memory cells, each of 1
which can hold an arbitrary 0
number or character

 Memory cells are numbered and accessing any

cell in memory takes unit time.
 Primitive Operations
 Basic computations  Examples:
performed by an algorithm  Evaluating an
expression
 Identifiable in pseudocode
 Assigning a value
 Largely independent from to a variable
any programming language  Indexing into an
 Exact definition not important array
(constant number of machine  Calling a method
cycles per statement)  Returning from a
method
 Assumed to take a constant
amount of time in the RAM
model
 Counting Primitive
Operations (§1.1)
 By inspecting the pseudocode, we can determine the maximum
number of primitive operations executed by an algorithm, as a
function of the input size

Algorithm arrayMax(A, n) # operations

currentMax  A[0] 2
for i  1 to n  1 do 2+n
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 2(n  1)
{ increment counter i } 2(n  1)
return currentMax 1
Total 7n  1
 Estimating Running Time
 Algorithm arrayMax executes 7n  1 primitive operations
in the worst case. Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
 Let T(n) be worst-case time of arrayMax. Then
a (7n  1)  T(n)  b (7n  1)
 Hence, the running time T(n) is bounded by two linear
functions
Theoretical analysis: conclusion
 We can count the number of RAM-equivalent
statements executed as a function of input
size
 All remaining implementation dependencies
amount to multiplicative constants, which we
will ignore
 (But watch out for statements that take more
than constant time, such as $input =~ s/$a/$b/g)
 Linear, quadratic, etc. runtime is an intrinsic
property of an algorithm
What’s the runtime?
int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++) {
cout << "Hello, ";
cout << "greetings, ";
cout << "bonjour, ";
cout << "guten tag, ";
cout << "Здравствуйте, ";
cout << "你好 ";
cout << " world!";
}
What’s the runtime?
int n;
cin >> n;
if (n<1000)
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << "Hello\n";
else
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << "world!\n";
 Function Growth Rates
1E+30
1E+28 Cubic
 Growth rates of 1E+26
functions: 1E+24 Quadratic
1E+22
 Linear  n Linear
1E+20
 Quadratic  n2 T (n ) 1E+18
 Cubic  n3 1E+16
1E+14
1E+12
 In a log-log chart, the 1E+10
slope of the line 1E+8
corresponds to the 1E+6
growth rate of the 1E+4
1E+2
function 1E+0
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
n
 Constant factors
1E+26
 The growth rate is not 1E+24 Quadratic
affected by 1E+22 Quadratic
 constant factors or 1E+20 Linear
1E+18 Linear
 lower-order terms
1E+16
 Examples
T (n )

1E+14
 102n + 105 is a linear 1E+12
function 1E+10
 105n2 + 108n is a 1E+8
quadratic function 1E+6
1E+4
1E+2
1E+0
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
n
 Asymptotic (big-O) Notation (§1.2)
10,000
 Given functions f(n) and g(n), 3n
we say that f(n) is O(g(n)) if
2n+10
there are positive constants 1,000
c and n0 such that n
f(n)  cg(n) for n  n0 100
 Example: 2n + 10 is O(n)
 2n + 10  cn
 (c  2) n  10 10
 n  10/(c  2)
 Pick c  3 and n0  10
1
1 10 100 1,000
n
 Example
1,000,000
n^2
 Example: the function 100n
n2 is not O(n) 100,000
10n
 n2  cn 10,000 n
 nc
 The above inequality 1,000
cannot be satisfied
since c must be a 100
constant
10

1
1 10 100 1,000
n
 More Big-O Examples
7n-2
7n-2 is O(n)
need c > 0 and n0  1 such that 7n-2  c•n for n  n0
this is true for c = 7 and n0 = 1

 3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3n3 + 20n2 + 5  c•n3 for n  n0
this is true for c = 4 and n0 = 21
 3 log n + log log n
3 log n + log log n is O(log n)
need c > 0 and n0  1 such that 3 log n + log log n  c•log n for n  n0
this is true for c = 4 and n0 = 2
Asymptotic analysis of functions
 Asymptotic analysis is equivalent to
 ignoring multiplicative constants
 ignoring lower-order terms
 “for large enough inputs”
 Big-O and growth rate
 Big-O gives an upper bound on the growth rate of
a function
 Think of it as <= [asymptotically speaking]
 Big-O Rules
 If is f(n) a polynomial of degree d, then f(n) is O(nd),
i.e.,
1. Drop lower-order terms
2. Drop constant factors
 Use the smallest possible class of functions, if possible
 Say “2n is O(n)” instead of “2n is O(n2)”
 (The former is a stronger statement)
 Use the simplest expression of the class
 Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”
 Asymptotic Algorithm Analysis
 Asymptotic analysis: determine the runtime in big-O
notation
 To perform the asymptotic analysis
 Find the worst-case number of primitive operations
executed as a function of the input size
 Express this function with big-O notation
 Example:
 We determine that algorithm arrayMax executes at most
7n  1 primitive operations
 We say that algorithm arrayMax “runs in O(n) time”
 Since constant factors and lower-order terms are
eventually dropped anyway, we can disregard them
when counting primitive operations
 Computing Prefix Averages
 Runtime analysis example: 35
X
Two algorithms for prefix 30 A
averages 25
 The i-th prefix average of
20
an array X is average of the
15
first (i + 1) elements of X:
10
A[i]  (X[0] + X[1] + … + X[i])/(i+1)
5
 Computing the array A of
0
prefix averages of another
1 2 3 4 5 6 7
array X has applications to
financial analysis
 Prefix Averages (Quadratic)
 The following algorithm computes prefix averages in quadratic
time by applying the definition

Algorithm prefixAverages1(X, n)
Input array X of n integers
Output array A of prefix averages of X #operations
A  new array of n integers n
for i  0 to n  1 do n
s  X[0] n
for j  1 to i do 1 + 2 + …+ (n  1)
s  s + X[j] 1 + 2 + …+ (n  1)
A[i]  s / (i + 1) n
return A 1
 Prefix Averages (Linear)
 The following algorithm computes prefix averages in linear time
by keeping a running sum

Algorithm prefixAverages2(X, n)
Input array X of n integers
Output array A of prefix averages of X #operations
A  new array of n integers n
s0 1
for i  0 to n  1 do n
s  s + X[i] n
A[i]  s / (i + 1) n
return A 1
 Algorithm prefixAverages2 runs in O(n) time
 Arithmetic Progression
 The running time of 7
prefixAverages1 is 6
O(1 + 2 + …+ n)
5
 The sum of the first n
integers is n(n + 1) / 2 4
 There is a simple visual 3
proof of this fact
2
 Thus, algorithm
prefixAverages1 runs in 1
O(n2) time 0
1 2 3 4 5 6
What’s the runtime?
int n;
2n3+n2+n+2?
cin >> n;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++) O(n3) runtime
for (int k=0; k<n; k++)
cout << “Hello world!\n”;

What if the last line is replaced by:

string *s=new string(“Hello world!\n”);

O(n3) time and space

What’s the runtime?
int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << “Hello world!\n”;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << “Hello world!\n”;

O(n3) + O(n3) = O(n3)

Statements or blocks in sequence: add
What’s the runtime?
int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=n; j>1; j/=2)
cout << “Hello world!\n”;

Loops: add up cost of each iteration

(multiply loop cost by number of iterations
if they all take the same time)

log n iterations of n steps  O(n log n)

What’s the runtime?
int n;
cin >> n;
for (int i=1; i<=n; i++)
for (int j=1; j<=i; j++)
cout << “Hello world!\n”;

Loops: add up cost of each iteration

1 + 2 + 3 + … + n = n(n+1)/2 = Q(n2)
What’s the runtime?
template <class Item>
void insert(Item a[], int l, int r)
{ int i;
for (i=r; i>l; i--) compexch(a[i-1],a[i]);
for (i=l+2; i<=r; i++)
{ int j=i; Item v=a[i];
while (v<a[j-1])
{ a[j] = a[j-1]; j--; }
a[j] = v;
}
}
 Math you need to Review
 Summations (Sec. 1.3.1)
 properties of logarithms:
 Logarithms and Exponents (Sec. 1.3.2)
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
Logb xa = a logb x
logba = logxa/logxb
 properties of exponentials:
a(b+c) = aba c
abc = (ab)c
ab /ac = a(b-c)
 Proof techniques (Sec. 1.3.3) b = a logab
 Basic probability (Sec. 1.3.4) bc = a c*logab
 Relatives of Big-Oh
 big-Omega
 f(n) is (g(n)) if there is a constant c > 0

and an integer constant n0  1 such that

f(n)  c•g(n) for n  n0
 big-Theta
 f(n) is Q(g(n)) if there are constants c’ > 0 and c’’ > 0 and an
integer constant n0  1 such that c’•g(n)  f(n)  c’’•g(n) for n 
n0
 little-o
 f(n) is o(g(n)) if, for any constant c > 0, there is an integer
constant n0  0 such that f(n)  c•g(n) for n  n0
 little-omega
 f(n) is (g(n)) if, for any constant c > 0, there is an integer
constant n0  0 such that f(n)  c•g(n) for n  n0
 Intuition for Asymptotic Notation

Big-Oh
 f(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n)

big-Omega
 f(n) is (g(n)) if f(n) is asymptotically greater than or equal to g(n)

big-Theta
 f(n) is Q(g(n)) if f(n) is asymptotically equal to g(n)

little-oh
 f(n) is o(g(n)) if f(n) is asymptotically strictly less than g(n)

little-omega
 f(n) is (g(n)) if is asymptotically strictly greater than g(n)
 Example Uses of the
Relatives of Big-Oh
 5n2 is (n2)
f(n) is (g(n)) if there is a constant c > 0 and an integer
constant n0  1 such that f(n)  c•g(n) for n  n0
let c = 5 and n0 = 1
 5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer
constant n0  1 such that f(n)  c•g(n) for n  n0
let c = 1 and n0 = 1
 5n2 is (n)
f(n) is (g(n)) if, for any constant c > 0, there is an integer
constant n0  0 such that f(n)  c•g(n) for n  n0
need 5n02  c•n0  given c, the n0 that satisfies this is n0 
c/5  0
Asymptotic Analysis: Review
 What does it mean to say that an algorithm
has runtime O(n log n)?
 n: Problem size
 Big-O: upper bound over all inputs of size n
 “Ignore constant factor” (why?)
 “as n grows large”

O: like <= for functions (asymptotically speaking)

: like >=
Q: like =
Asymptotic notation: examples
 Asymptotic runtime, in terms of O, , Q?
 Suppose the runtime for a function is
 n2 + 2n log n + 40
 0.0000001 n2+ 1000000n1.999
 n3 + n2 log n
 n2.0001 + n2 log n
 2n+ 100 n2
 1.00001n+ 100 n97
Asymptotic comparisons
 0.0000001 n2 = O(1000000n1.999 )?
No – a polynomial with a higher power dominates one with a lower power

 n1.000001 = O(n log n)?

No – all polynomials (n.000001) dominate any polylog (log n)

 1.0001n = O(n943)?
No – all exponentials dominate any polynomial

 lg n = Q(ln n)?
Yes – different bases are just a constant factor difference

(Evaluate the limit of the quotient of the functions)

Estimate the runtime
 Suppose an algorithm has runtime Q(n3)
 suppose solving a problem of size 1000 takes 10 seconds.
How long to solve a problem of size 10000?
runtime 10-8 n3; if n=10000, runtime 10000s = 2.7hr

 Suppose an algorithm has runtime Q(n log n)

 suppose solving a problem of size 1000 takes 10 seconds.
How long to solve a problem of size 10000?

runtime 10-3 n lg n; if n=10000, runtime 133 secs

Worst vs. average case
 You might be interested in worst, best, or average
case analysis of an algorithm
 You can have upper, lower, or tight bounds on each
of those functions.
 Eg. For each n, some problem instances of size n
have runtime n and some have runtime n2.

 Worst case: Q(n2), (n), (log n), O(n2), O(n3)

 Best case: (n), (log n), O(n2), Q(n)
 Average case: (n), (log n), O(n2), O(n3)
Average case: need to know distribution of inputs
Problem: Extensible Arrays
 A data structure for a stack that is…
 As fast and easy as an array (usually)
 Not limited in size
 How?

 Extensible Array:
 Usually works like a normal array
 When it fills, copy the entire contents into a new
array of twice the size.

 Runtime?
Analysis of Extensible Arrays
 Worst-case runtime for a push() operation?
 Is that the most useful way of describing the result?

 Amortized analysis: averaging out the cost over

many operations.
 Amortized runtime of a push() operation?
 Total runtime of m push operations:
 m steps when array doesn’t grow
 Worst-case cost of grow operations:
 1+2+4+8+…+m < 2m
 Total: O(m) time for m operations.
 O(1) amortized runtime per operation
Theoretical and empirical analysis
 Why do we ignore constants in analyzing
algorithms theoretically?
 Wouldn’t it be better if we knew the actual
constants?
 Theoretical and empirical analysis gives us
 Theoretical knowledge of asymptotic runtime
 Empirical knowledge of actual constants for some
implementation

 Best of both worlds!

Analyze this…
function fun (int n)
if (n<1000)
for (i=1; i<=n; i++)
for (j=1; j<=n; j++)
for (k=1; k<=n; k++)
cout << “Hello world!\n”;
else {
for (i=1; i<=n; i++)
j=i;
while (j >= 1)
j = j / 2;
cout << “Hello world!\n”;
if (n%2)
for (i=1; i<=n; i++)
for (j=1; j<=n; j++)
cout << “Hello world!\n”;
}
cout << “Have a nice day.\n”;