Error-Resilient Coding and Decoding

Strategies for Video Communication

Thomas Stockhammer and Waqar Zia

Presented by Li Ma

Background & Motivation

Video becoming more popular

Advances in bandwidth, capacity enhancements
Requirements:

Limitation:

data transmission rate

Real-time delivery of multimedia data
QoS available is not sufficient to guarantee error-free
delivery for all receivers

Motivation:

Provide means of dealing with various transmission

impairments

Content

Focus on MCP-coded video

Concentrate on tools and features integrated
in standard H.264/AVC
Focus on specific tools for improved error
resilience
Other advanced error-resilience features not
covered:

Multiple description coding

Distributed video coding
Combinations with network prioritization and FEC

Outline

Video Communication Systems

Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary

Video Communication Systems

End-to-End Video Transmission

Video Applications

wide variety of applications

Different bit rate ranges:

Different tolerable end-to-end delay

HDTV: 20 Mbit/s
MMS on cell: 20 Kbit/s
Conversational applications constraints: 200250 ms

Transmission Impairments

Differences of errors

Wireless networks:

IP network:

Congestion results in packets lost

Methods

Fading and interference cause burst errors: multiple

lost bits

Detect presence of errors

Intermediate protocol layers (UDP) could drop
erroneous pkts
Video data pkts treated as lost if delayed more
than threshold

Data Losses in MCP-Coded Video

Transmitted over error-prone channels

Error concealment
Error propagation / Spatiotemporal error
propagation

Example of Error Propagation

Pt @ t=0 is lost.
Error propagation till t=8
Intra-coded image transmitted @ t=9

Therefore, when data units might get lost, a

video coding system should provide:

10

Means that allow completely avoiding

transmission errors
Features that allow minimizing the visual effects of
errors
Features to limit spatial and spatiotemporal error
propagation

Outline

Video Communication Systems

Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary

11

Error-resilient Video Transmission

System Overview

12

Features

@Sender

MBs are grouped in data units and entropy coding used

Error Control before transmission over lossy channel

Forward Error Correction(FEC)

Backward Error Correction (BEC)
Prioritization Methods
Combinations of above

@Receiver

13

Erroneous and missing data detected and localized

Decoder gets correct data units or error indication
Error concealment applied at positions where no data
received
Report loss of data units to encoder

Design Principles

error-resilience tools decrease compression

efficiency
Main goal:

Shannons separation principle: compression separated

with transport
In low delay situations, error-free transport is impossible

System Design Principles

14

1.
2.
3.
4.
5.

Loss correction below codec layer

Error detection
Prioritization methods
Error recovery and concealment
Encoder-decoder mismatch avoidance

Video Compression Tools Related to Error

Resilience

Slice Structured Coding

Flexible MB Ordering
Scalable Coding
Data Partitioning
Flexible Reference Frame
Intra Information Coding
Pictures Switching

15

Slice Structured Coding

Slices provide spatially distinct resynchronization

points within the data for a single frame
Several MBs grouped together: a slice header
Variable sized data units
Encoder can select the location of sync. Points
Motion vector prediction not allowed over
boundaries
Encoder decides either:

16

Allocate fixed number of MB to one slice

Or fixed bits to one slice (matched to pkt size in
network)

Flexible MB Ordering

Flexible Macroblock Ordering (FMO)

17

Allows mapping of MBs to Slice groups

A slice group may contain several slices
MBs can be transmitted in flexible and efficient way
Spatially collocated images areas can be
interleaved in different slices greater probability
of concealing lost MB
Protection: Can map ROI (region of interest) into a
separated slice group

Data Partitioning

Loss of some syntax elements of a bit stream

results in larger degradation of quality compared to
others

E.g. Loss of motion vector

Data partitioning results in Graceful Degradation

Categorize syntax elements

18

Header information
Motion information
Texture information

Layout of compressed data

Without Data
Partitioning:

With Data
Partitioning:

2 additional sync. Points

available
19

Data Partitioning (Cont.)

Unequal Error Protection (UEP)

20

Protect partitions of different importance

More important data offered more protection

Flexible Reference Frame

H.263 v.1 & MPEG-2 allow only a single

reference frame for predicting P frame and
mostly 2 for B frame.
Possible to have significant statistical
dependencies between other pictures too
Use more frames than just the recent one as
reference

Advantages:

21

Increased compression efficiency

Improved error resilience

Example of Flexible Reference Frame

22

Example of Flexible Reference Frame

Enable Subsequences

Use a subsequence of anchor frames at lower frame rate

Other frames inserted in between to achieve overall frame

rate

E.g P

Error propagation:

23

E.g P

Only till next P received

Outline

Video Communication Systems

Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary

24

Resynchronization and Error Concealment

Video Packetization Modes

Without FMO(flexible macroblock ordering)

25

1. a constant number of MBs within one slice (arbitrary

size)
2. the slice size bounded to some max bytes (arbitrary
# of MBs)

Video Packetization Modes (Cont.)

With FMO (more flexible)

26

Slice interleaving
Dispersed MB allocation using checkerboard
patterns
Subpictures within a picture
etc.

Error Concealment

Basic Idea

Decoder should generate a representation for lost

area
Match as close as possible to the lost info
Within manageable complexity

Techniques

27

Spatial Error Concealment

Temporal Error Concealment
Hybrid Concealment
Other Techniques

Spatial Error Concealment

Based on assumption of continuity of natural

scene content in space
Use pixel values of surrounding available MBs
Estimate of lost pixel:

are weighing factors

Determine relative impact of vertical,

Horizontal, upper, lower

Disadvantage

28

Blurred reconstruction

Temporal Error Concealment

Rely on the continuity of a video sequence in

time
Use temporally neighboring areas to conceal
lost regions
Previous Frame Concealment (PFC)

29

Use previous corresponding data to copy to

current frame
Only good when little motion
Widely used due to simplicity

Hybrid Concealment

When only apply spatial concealment

When only use temporal error concealment

Concealed regions are significantly blurred

Significantly discontinuities in the concealed
regions

Hybrid temporal-spatial technique applied

30

MB mode info of reliable and concealed neighbors

decide which concealment method to use

Hybrid (cont.)

For intra-coded images

For inter-coded images

Only use spatial concealment

Use temporal concealment when more than half of
the available neighbor MBs are inter-coded
Otherwise, use spatial concealment

Referred to as Adaptive temporal and spatial

Error Concealment (AEC)

31

Selected Results

Performance of different error concealment

strategies

32

Selected Performance Results for Wireless

Low-delay and low-complexity requirements

Max allowed buffering at encoder limited to
250ms

33

Using a smaller slice size of 150 bytes is lower

in PSNR when error free

Because:

increased packetization overhead

prediction limitations on slice boundaries

Performs good when in lossy channel

34

Because the loss affects only a small area of the

image for fixed slice size

Outline

Video Communication Systems

Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary

35

Motivation

Error propagation is major problem over lossy

channels
Encoder can change encoding behavior when
he finds its likely to be lossy or knows
decoder suffering losses

36

Operational Encoder Control

Encoder appropriately select parameters

37

Motion vectors
MB modes
Quantization parameters
Reference frames
Spatial and temporal resolution

38

Conclusion

Bad decisions at the encoder can lead to

If no feedback is available

Poor results in coding efficiency

Poor in error resilience
Or both
an increased percentage of intra MBs performs
best

If feedback available

39

Interactive Error Control is best

Outline

Video Communication Systems

Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary

40

Summary

Important to understand video can benefit

significantly when data delivered reliably
Introduced error-resilience tools and impact
For good overall performance, should take into
account:

41

the selection of error-resilience tools

rate-distortion-optimized mode selection
the channel characteristics

42