Source Coding

Modulator
· Digital Source Channel
Source Symbols Encoder Binary Encoder
Source sequence
Entropy
Source coding reduces redundancy and hence reduces
bandwidth requirement.

For the source encoder to be efficient, we require the

knowledge of the statistics of the source.

Example: If some source symbols are known to be more

probable than others, shorter codewords should be assigned
to these symbols D.1

Efficient Source Encoding

N eed to know statistics of source

For exam ple :

1) Frequent sym bols Sym bols are equally probable
assign short code w ord
2) Rare sym bols to happen and/or statistically
assign long code word independent

V ariable-length Code
(M orse Code)

Sym bols occur with different

probability
(eg. A lphabet in the English
Language)

U se C ode word w ith

U se V ariable length fixed block length
C ode ie. 4 sym bols A B C D
A Æ 00
B Æ01
C Æ10
Code according to the probability of D Æ11
occurrence of the source sym bols
D.2
EN TR O PY C O D IN G

1
 Shannon’s source coding theorem
Given a discrete memoryless source of entropy H(X), the
average codeword length Lav for any distortionless source
coding scheme is bounded as
Lav ≥ H ( X )

According to this theorem, the source entropy H ( X )

represents a fundamental limit on the average number of bits
per source symbol necessary to represent a discrete
memoryless source in that it can be made as small as, but no
smaller than the source entropy.

D.3

Source Efficiency

Source efficiency η = H ( X ) / Lav

η ≤1

At the receiver,
Channel
Binary Decoder
Symbols
sequence

Note: Symbol to Sequence or Sequence to Symbol is a one

to one mapping
D.4

2
 Example 1
Source symbols
Discrete P ( A) = 1 /2
Memory
Source A, B, C, D P( B) = 1/ 4
(DMS) P (C ) = 1 / 8
P( D) = 1 / 8

H(X) = 1.71 bits/symbol

The symbols are coded in 3 different ways using variable-
length code as below.

D.5

Example 1

Source Probability Code I Code II Code III

Symbol of Occurence
A 1/2 1 0 0

B 1/4 00 10 01

C 1/8 01 110 011

D 1/8 10 111 111

Lav H(X)=1.71 1.5 1.75 1.75

D.6

3
 Code I
There is a problem in this decoding process.

Suppose we have 001001……..

Is this B,A,B,A,…… or B,D,C,………?

∴ Code I is not uniquely decodable.

Sometimes this ambiguity can be resolved by waiting for

additional bits. But then the decoding is not instantaneous
and not desirable in practice.

D.7

Code II
Uniquely decodable and instantaneous.

A B C

0 0 0 Code tree for

D code II
Initial state
1 1 1

D.8

4
 Code III
Uniquely decodable but not instantaneously decodable.

A 1 B 1 C Code tree for code III

0
D
1 1 1

D.9

Instantaneously decodable
Question: How can we have codes that are instantaneously
decodable?
Answer: No code word in the code is a prefix of another code
word.

For a given code word C k of length k, C k = [b1 b2 b3 ⋅ ⋅ ⋅ bk ]

C l = [b1 b2 b3 ⋅ ⋅ ⋅ bl ] is a prefix of C k l < k

Prefix condition requires that for any code word C k , there

is no other code C l having the same l elements
[b1 b2 b3 ⋅ ⋅ ⋅ bl ] .
D.10

5
 Kraft inequality
A prefix code always satisfy the Kraft inequality
L

∑2
k =1
− nk
≤1

where n1 ≤ n 2 ≤ n3 ≤ ... ≤ n L are the lengths of the

codewords.

Note: Kraft inequality does not tell us that a source code is a

prefix code. Rather, it is merely a condition on the codeword
lengths of the code and not on the code words themselves.

D.11

Example
Code I violates the Kraft inequality; it cannot be a prefix
code.

Kraft inequality is satisfied by both codes II and III; but only

code II is a prefix code.

D.12

6
 Average code word length

H ( X ) ≤ Lav ≤ H ( X ) + 1
⇒ H ( X n ) ≤ Lnav ≤ H ( X n ) + 1
⇒ nH ( X ) ≤ Lnav ≤ nH ( X ) + 1
⇒ H ( X ) ≤ Lnav ≤ H ( X ) + 1 / n

In the limit, as n approaches infinity, we have

1
lim Lnav = H ( X )
n →∞ n

Therefore, the average codeword length of an extended prefix

code can be make as small as the entropy of the source.
However, the price we have to pay for decreasing the
average code-word length is incrased decoding complexity.
D.13

Shannon-Fano Encoding
Variable-length encoding scheme proposed by Shannon and
Fano and the algorithm is simply stated as below.

For M possible messages with probability P2 , P2 ,..., PM

and N symbols per message, a unique binary codeword C i of
length li for message i can be generated such that the
average number of bits per symbol LNav is closer to G N where

1
GN = −
N
∑ P(m ) log
i
i 2 P ( mi ) as before

D.14

7
 M
1
In other words , Lav =
N
∑l P → G
i =1
i i N

1 X X X….X C1 (length l1)

2 X X X….X C2 (length l2)
M possible
. Encoding .
message of
. .
N symbols long
. .
M X X X….X CM (length lM)

where l1 , l 2 ,...l M might or might not be the same.

High probability messages will have short codewords. Each

message has a unique codeword and can be decoded
uniquely.
D.15

The average number of bits per symbol (entropy) is bounded

by
G N ≤ Lav < G N + 1 / N

Encoder rate efficiency e is

H
η= H: Source Entropy
Lav

D.16

8
 Coding: Step 1
Arrange the N messages in order of decreasing probability
and let
i −1
Fi = ∑ Pk with F1 = 0
k =1

P1 F1
P2 F
i.e. → 2
: :
PN FN
D.17

Step 2
Calculate the number of bits li required for message i by the
following equation
− log 2 Pi ≤ li < 1 − log 2 Pi

For example, Pi = 1 / 5 ,

− log 2 (1 / 5) = 2.32 and 1 − log 2 (1 / 5) = 3.32

∴ li = 3

D.18

9
 Step 3-4
Convert Fi into a binary fraction. The binary fraction
satisfies the following equality
b1 b2 b
b1b2 b3 ⋅ ⋅ ⋅ bk = + 2 + ... + kk
2 2 2
For example, 27 0 0 1 1 0 1 1
=
+ + + + + +
128 2 4 8 16 32 64 128
0 0 1 1 0 1 1
= + 2 + 3+ 4 + 5 + 6 + 7
2 2 2 2 2 2 2
→ 0011011
Step 4
The code word for message i is the truncated version of the
fraction Fi and has length li .
D.19

Example 2

i Message Pi Fi li Binary Fi Codeword

1 AAA 27
128 0 3 0000000 000
2 BBB 27 27
128 128 3 0011011 001

3 CAA 9 54
128 128 4 0110110 0110

D.20

10
 Example 3
Consider a Markov source encoder with N =2 symbols per
message per message. The encoding operation is
i Symbols Pi li Codeword Fi Binary Fi
1 AA 9
2 00
32
2 BB 9
2 01
32
3 AC 9
4 1001
32
4 CB 3
4 1010
32
5 BC 3
4 1100
32
6 CA 3
4 1101
32
7 CC 2
4 1111
32 D.21

Example 3
H N = 1.44bit / symbol
CA,AA,BB,BC 1101,00,01,1100
Source Encoder

The decoding is accomplished by For example,

matching the incoming data with 1101 00 10 1100
the codewords having the least
number of bits until there is no
AA
match. The incoming data is the
matched with the codewords No Match
having the second smallest
number o bits, then so on.
CA D.22

11
 Huffman encoding
Variable-length scheme for the encoding of symbols.
– Arrange the source symbols in descending order of
probability
– Create a new source with one less symbol by combining
(adding) the two symbols having the lowest probability.
– Repeat step 1 and 2 until a single -symbol source is
achieved.
– Associate '1' and a '0' with each pair of probabilities so
combined.
– Encode each original source symbol into binary sequence
generated by the various combinations, with the first
combination as the least significant digit in thee sequence.
D.23

Example 4
Source symbols A,B,C,D,E and probabilities are P(A)=0.4,
P(B) =0.2, P(C) =0.2, P(D)=0.1, P(E)=0.1
A 0.4 Æ 0.4 0.4
B 0.2 Æ 0.2
C 0.2 Æ 0.2 0
0.4 0
D 0.1 0 0.2 1 0.6 0
Æ
E 0.1 1 0.2 1 0.4 1Æ 1.0

A encodes into 1 B encodes into 01

C encodes into 000 D encodes into 0010
E encodes into 0011
D.24

12
 Example 4

Note that the Huffman's encoding rule is uniquely

decodable; i.e. a stream of encoded symbols can decoded
without the need for synchronization or forming pulses
between the binary sequences representing each symbol. The
tree diagram confirms this property, since no encoded
sequence is a prefix to another/any other valid sequence.

Example:
The encoded sequence for symbol C is not a prefix of any
other valid sequence.

D.25

Example 4

Tree Diagram E
A B 1
D
1 1 1 0

Initial state C
0 0 0
m
Source Entropy H ( X ) = −∑ Pi log 2 ( Pi )
i =1
= −0.4 log 2 0.4 − 2[0.2 log 2 0.2] − 2[0.1log 2 0.1]

= 2.12 bits/symbol

D.26

13
 Example 4
If P(A) = P(B) = P(C) = P(D) = P(E), H ( X ) = H ( X ) max

∴ H ( X ) max = −5[0.2 log 2 0.2]

= 2.322 bits/symbol

H(X )
Source efficiency before encoding = = 0.913
H ( X ) MAX

D.27

Example 4
The average number Lav of digits used to encode each source
symbol after encoding is
0.4(1) + 0.2(2) + 0.2(3) +0.1(4)+0.1(4) = 2.2 bit/symbol
Source efficiency η after encoding is η=H(X)/Lav=0.964

1- η: defined as the source redundancy.

M: number of binary digits required to represent the source
without any source coding
Lav/M: compression ratio CR.
In our example, M=3 and CR = 2.2/3 = 0.73
CR indicates the proportion by which the bandwidth
required to transmit the source over a binary channel can be
reduced. D.28

14
 Run-length Coding
Run length coding technique is particularly suitable for
binary sources where one of the symbols (the '1' say) occurs
very much less often then others, so that there are long runs
of successive '0's (e.g. a scanned and digitized line drawing).
In this case, it is more efficient to encode by counting the
number of the consecutive '0's between '1's.

D.29

Example 5
1 000
01 001
001 010
0001 Source 011
00001 Encoder 100
000001 101
0000001 110
0000000 111

If a binary source generates a sequence of 20 digits,

001,0000000,1,1,0000001
This sequence is encoded into a 15-digit sequence
010,111,000,000,110.
D.30

15
 Example 5
The compression in this particular case is 15/20 = 0.75

Input block length varies, Output block length fixed.

There are schemes where both the input block length and
output block length are variable length. (FAX)

D.31

Lempel-Ziv Coding
In practice, source statistics are not always known a priori.

To overcome this limitation, Lemp-Ziv Coding could be

used.

Encoding in the Lempel-Ziv algorithm is accomplished by

parsing the source data stream into segments that are the
shortest subsequences not encountered previously.

D.32

16
 Example 6
Consider 000101 110010 100101…
Assumed that the binary symbols 0 and 1 are already stored
in that order in the code book, We can write
Subsequences stored: 0, 1
Data to be parsed: 000101 110010 100101…

The encoding process begins at the left. With symbols 0 and

1 already stored, the shortest subsequence of the data stream
encountered for the first time and not seem before is 00, so
we write
Subsequences stored: 0, 1, 00
Data to be parsed: 0101 110010 100101… D.33

Example 6
Similarly, we have
Subsequences stored: 0, 1, 00, 01
Data to be parsed: 01 110010 100101…

Subsequences stored: 0, 1, 00, 01, 011

Data to be parsed: 10010 100101…

The process is continued until the given data stream has

been completely parsed. Thus, we get the code book as
follows:

D.34

17
 Example 6
Numerical Subsequences Numerical Binary
Position Representation encoded
blocks
1 0
2 1
3 00 11 0010
4 01 12 0011
5 011 42 1001
6 10 21 0100
7 010 41 1000
8 100 61 1100
9 101 62 1101 D.35

Lempel-Ziv coding
Lempel-Ziv algorithm uses fixed-length codes to represent a
variable number of source symbol; this feature makes the
Lempel-Ziv code suitable for synchronous transmission.

In practice, fixed block of 12 bits long are used, which

implies a code book of 4096 entries.

D.36

18