Introduction

Rawesak Tanawongsuwan
ccrtw@mahidol.ac.th
 1

Prologue
A common-sense description of multimedia might be: the ability
to combine the creative possibilities of radio and television
programs, newspapers, books, magazines, comic books, animated
films and music disks into one set of computer files accessed by the
same piece of software to provide and integrated seamless
experience, where user input to some extent determines the manner
in which the material is accessed. It is therefore interactive. The
computer’s ability to have rapid access to the files which constitutes
this material makes the linear model of the radio or television
program seem old-fashioned and limited. Interactivity, where the
user to some extent determines the text or, more accurately, the
order in which the text unfolds, offers great creative potential. It also
offers a great creative challenge, with the possibility of an interactive
movie very likely to give the traditional script writer something
approaching a migraine headache to the power of 5.

Digital Multimedia, 2nd edition Nigel Chapman & Jenny Chapman

Prologue “The Creative Challenge” by Brent MacGregor – Edinburgh College of Art
 1

Prologue
A common-sense description of multimedia might be: the ability
to combine the creative possibilities of radio and television
programs, newspapers, books, magazines, comic books, animated
films and music disks into one set of computer files accessed by the
same piece of software to provide and integrated seamless
experience, where user input to some extent determines the manner
in which the material is accessed. It is therefore interactive. The
computer’s ability to have rapid access to the files which constitutes
this material makes the linear model of the radio or television
program seem old-fashioned and limited. Interactivity, where the
user to some extent determines the text or, more accurately, the
order in which the text unfolds, offers great creative potential. It also
offers a great creative challenge, with the possibility of an interactive
movie very likely to give the traditional script writer something
approaching a migraine headache to the power of 5.
 1

Prologue
A common-sense description of multimedia might be: the ability
to combine the creative possibilities of radio and television
programs, newspapers, books, magazines, comic books, animated
films and music disks into one set of computer files accessed by the
same piece of software to provide and integrated seamless
experience, where user input to some extent determines the manner
in which the material is accessed. It is therefore interactive. The
computer’s ability to have rapid access to the files which constitutes
this material makes the linear model of the radio or television
program seem old-fashioned and limited. Interactivity, where the
user to some extent determines the text or, more accurately, the
order in which the text unfolds, offers great creative potential. It also
offers a great creative challenge, with the possibility of an interactive
movie very likely to give the traditional script writer something
approaching a migraine headache to the power of 5.
 1

Prologue
A common-sense description of multimedia might be: the ability
to combine the creative possibilities of radio and television
programs, newspapers, books, magazines, comic books, animated
films and music disks into one set of computer files accessed by the
same piece of software to provide and integrated seamless
experience, where user input to some extent determines the manner
in which the material is accessed. It is therefore interactive. The
computer’s ability to have rapid access to the files which constitutes
this material makes the linear model of the radio or television
program seem old-fashioned and limited. Interactivity, where the
user to some extent determines the text or, more accurately, the
order in which the text unfolds, offers great creative potential. It also
offers a great creative challenge, with the possibility of an interactive
movie very likely to give the traditional script writer something
approaching a migraine headache to the power of 5.
 1

Prologue
A common-sense description of multimedia might be: the ability
to combine the creative possibilities of radio and television
programs, newspapers, books, magazines, comic books, animated
films and music disks into one set of computer files accessed by the
same piece of software to provide and integrated seamless
experience, where user input to some extent determines the manner
in which the material is accessed. It is therefore interactive. The
computer’s ability to have rapid access to the files which constitutes
this material makes the linear model of the radio or television
program seem old-fashioned and limited. Interactivity, where the
user to some extent determines the text or, more accurately, the
order in which the text unfolds, offers great creative potential. It also
offers a great creative challenge, with the possibility of an interactive
movie very likely to give the traditional script writer something
approaching a migraine headache to the power of 5.
 1 2

Digital Multimedia
• The same story, information, etc can be
represented in different media
•Text, images, sound, moving pictures

• All media can be represented digitally as a

structured collection of bits

• Manipulated by programs, stored,

transmitted over networks

• Digital media can be combined into multimedia

 1 3–4

Interactive Multimedia
• Combination of media is actually commonplace
(e.g. TV news) and natural – we perceive the
world through all our senses at once

• Novelty of digital multimedia is that all media

can be treated as data

• Programs can manipulate data in response to

user input, so digital multimedia can be
interactive
 1 4

Historical Context
• Some dates:
• CD-ROM specification published 1985
• CD-ROM drives on desktop machines from ~1989

• WWW publicly available at start of 1992

•Handful of servers; line-based browser
• HTML 3.2 adopted as W3C Recommendation in
January 1997
• Audio and video proprietary extensions
 1 5–6

Cultural Development

• Takes time for conventions about content and

consumption to become established – cf film:

• 1895 footage of train arriving at station

• Early animations and trick films shown as part

of vaudeville acts at the same time as
narrative films were being shown in cinema

• Established forms translated into new medium

(e.g. newsreels based on newspapers)
 1 6–7

Terminology
• Multimedia production – display and
presentation is the sole purpose

• Multimedia application – display is driven by

computation

• e.g. Web application presenting data stored

in a database

• Multiple media – user must switch between

modalities (read, watch, listen,…) instead of
combining them
 1 7

Definition
• Digital multimedia: any combination of
two or more media (text, audio, images,
drawings, animation, video,…),
represented in a digital form, sufficiently
well integrated to be presented via a
single interface, or manipulated by a
single computer program
 1

Example usages
• Video teleconferencing.

• Distributed lectures for higher education.

• Tele-medicine.

• Co-operative work environments.

• Searching in (very) large video and image

databases

• Augmented reality by placing real-appearing

computer graphics and video objects into scenes
 1

Example usages
• Including audio cues for where video-conference
participants are located.

• Building searchable features into new video, and

enabling very high- to very low-bit-rate use of
new, scalable multi- media products.

• Making multimedia components editable.

• Building “inverse-Hollywood” applications that can

re-create the process by which a video was made.

• Using voice-recognition to build an interactive

environment
 1 8–9

Delivery
• Online

• Uses a network (usually the Internet) to

send information from one computer to
another
• World Wide Web is commonest form of
online delivery of multimedia

• Offline

• Removable storage medium is used to carry

the data
• CD-ROM, DVD
 1 10

Page-Based Multimedia
• Text, images laid out in 2-D arrangement as in
book or magazine

• Time-based elements embedded as if they

were images

• Playback controls may be provided

• Pages combined using links (hypermedia)

• Essentially static
 1 11

Time-Based Multimedia
• Elements arranged in time

• Presented in sequence on a timeline

• Elements may be frames or discrete pages

(slideshow)

• Often incorporates parallelism

• Parallel elements may be synchronized

 1 10

Linearity
• Film: fixed order of
frames defines a
single playback
sequence

• Book: physical
arrangement of text
and pages implies a
linear reading order
 1 11

Non-Linearity
• Flash: jumps between
frames, controlled by
interactivity, permit
branching and loops

• Hypermedia: links
between pages permit
multiple arbitrary
reading orders
 1 13–14

Interactivity

• User input may control a multimedia

production, but only within limits set by the
multimedia producer

• Only choices that are coded into the

program are possible

• Can allow the user to control events at many

points, leading to combinatorial growth in
number of possibilities

• e.g. 4 choices at each of 5 points implies 20

branches but 1024 possible sequences
 1 15–16

User Interfaces
• Means of presenting choices and accepting
user input can vary enormously from

• Stylized interface elements used by

mainstream OSs and applications

• to

• Free-form, dynamically changing interaction

of games

• Conventions are predictable, but limited to

(static) context in which they were developed
 1 17–20

Access
• If access to multimedia is the norm, those denied
access become marginalized ('digital divide')

• Access may be limited by lack of:

• Access to equipment and skills
• Network infrastructure
• Literacy and education
• Physical and cognitive abilities

• These factors may depend on wealth, geographical

location,…
 1 21–23

Traditional Media Production

• Access to production of traditional media highly

restricted
• Books: distributed through publishers, subject
to editorial scrutiny; barriers to newcomers
• Film: very high cost; studios prefer safe bets
• Music: mostly distributed by few labels
controlled by small number of multinationals;
hard to break in to the business
• TV: video production relatively low cost, but
access to broadcast rigidly controlled
 1 21

Web Site Production

• Potentially anyone with Internet access can
have their own Web site

• ISPs provide free Web space

• Free and inexpensive tools are adequate

• WWW has potential for revolution in access

to the means of production and distribution
of digital material
 1 24–25

Control of Content
• All sufficiently complex societies seek to
control what people may see or hear, either by
explicit policing, economic or other means

• Rapid growth of the Internet and its potential

for disseminating unacceptable content has
given new impetus to debates about
censorship

• Complicated ethical issues with no enduring

conclusion or consensus despite thousands of
years of debate
 1 25–26

Diversity
• WWW is global network, hence material
reaches many different societies and cultural
and religious groups within those societies

• Many different models of censorship – none,

rigid centralized control, self-regulation, …

• Unrealistic to expect a single model of

censorship to be acceptable everywhere

• Difficult to assign responsibility for

disseminaton of content on Internet
 1 27–29

PICS
• Platform for Internet Content Selection
• Attempt to provide a mechanism that supports
a diversity of attitudes towards content and
censorship
• Labels attached to each page, providing a
rating of its contents
• PICS only defines standard label format
• Screening software rejects material deemed
unsuitable according to user's criteria
• Defers the difficult decisions
 1

Let’s take a detour a bit…

• I wonder what multimedia means to technical
people

• Mean the same thing as we just talk about

today

• And more technical

• ACM Multimedia conference

 1

ACM, IEEE
• ACM, the Association for Computing Machinery, is an
international scientific and educational organization
dedicated to advancing the arts, sciences, and
applications of information technology. With a world-wide
membership ACM is a leading resource for computing
professionals and students working in the various fields of
Information Technology, and for interpreting the impact
of information technology on society. (www.acm.org)
• The initials I-E-E-E represent the legal name of the IEEE,
the Institute of Electrical and Electronics Engineers. IEEE
is pronounced "EYE triple E." The IEEE is a global technical
professional society serving the public interest and
members in electrical, electronics, computer, information
& other technologies. (www.ieee.org)

ACM MM’01 vs. MM’04
• Video applications
• Video processing
• Streaming
• Image retrieval
• Video retrieval and browsing
• Audio processing
• Network games
• Authoring support
• Video storage
• Coding and encryption
• Voice over IP
1
• Content-based image retrieval
• Networked multimedia
applications
• Audio processing
• Multimedia streaming
• Learning in multi-modal data
• Compression, streaming &
retrieval of 3D objects
• Still & moving images
• Watermarking & multi-media
processing
• Video processing
• Managing image
• Multimedia over wireless
 1
Examples of Multimedia Topics
• Multimedia processing and coding: multimedia content
analysis, content-based multimedia retrieval, multimedia
security, audio/image/video processing, compression, etc.

• Multimedia system support and networking: network

protocols, Internet, operating systems, servers and clients,
quality of service (QoS), and databases.

• Multimedia tools, end-systems and applications:

hypermedia systems, user interfaces, authoring systems.

• Multi-modal interaction and integration: “ubiquity”
web-

everywhere devices, multimedia education including Computer
Supported Collaborative Learning, and design and applications
of virtual environments.
 1
Examples of Multimedia Projects
• Camera-based object tracking technology: tracking of the
control objects provides user control of the process.

• 3D motion capture: used for multiple actor capture so that

multiple real actors in a virtual studio can be used to
automatically produce realistic animated models with natural
movement.

• Multiple views: allowing photo-realistic (video-quality)

synthesis of virtual actors from several cameras or from a
single camera under diering lighting.

• 3D capture technology: allow synthesis of highly realistic

facial animation from speech.
 1
Examples of Multimedia Projects

• Specific multimedia applications: aimed at handicapped

persons with low vision capability and the elderly

• Digital fashion: aims to develop smart clothing that can

communicate with other such enhanced clothing using wireless
communication, so as to articially enhance human interaction
in a social setting.

• Electronic Housecall system: an initiative for providing

interactive health monitoring services to patients in their
homes

• Augmented Interaction applications: used to develop

interfaces between real and virtual humans for tasks such as
augmented storytelling.
 1
Multimedia Software Tools
• Music Sequencing and Notation

• Digital Audio

• Graphics and Image Editing

• Video Editing

• Animation

• Multimedia Authoring
 1
Music Sequencing and Notation

• Cakewalk: now called Pro Audio.

• The term sequencer comes from older devices that

stored sequences of notes

• It is also possible to insert WAV les and Windows MCI

commands (for animation and video) into music
tracks (MCI is a ubiquitous component of the
Windows API.)

• Cubase: another sequencing/editing program, with

capabilities similar to those of Cakewalk. It includes
some digital audio editing tools.

• Macromedia Soundedit: mature program for creating

audio for multimedia projects and the web that
integrates wellwith other Macromedia products such as
Flash and Director
 1

Digital Audio

• Cool Edit: a very powerful and popular digital audio

toolkit; emulates a professional audio studio

multitrack productions and sound file editing including
digital signal processing effects.

• Sound Forge: a sophisticated PC-based program for

editing audio WAV files.

• Pro Tools: a high-end integrated audio production and

editing environment
MIDI creation and manipulation;
powerful audio mixing, recording, and editing software.
 1
Graphics and Image Editing
• Adobe Illustrator: a powerful publishing tool from Adobe.
Uses vector graphics; graphics can be exported to Web.

• Adobe Photoshop: the standard in a graphics, image

processing and manipulation tool.

• Allows layers of images, graphics, and text that can be

separately manipulated for maximum flexibility.

• Filter factory permits creation of sophisticated

lighting-effects filters

• Macromedia Fireworks: software for making graphics

specifically for the web.

• Macromedia Freehand: a text and web graphics editing

tool that supports many bitmap formats such as GIF, PNG,
and JPEG.
 1
Video Editing

• Adobe Premiere: an intuitive, simple video editing tool for

nonlinear editing, i.e., putting video clips into any order:

• Video and audio are arranged in “tracks”

• Provides a large number of video and audio tracks,

superimpositions and virtual clips

• A large library of built-in transitions, lters and motions for

clips
effective multimedia productions with little effort

• Adobe After Effects: a powerful video editing tool that

enables users to add and change existing movies. Can add
many eects: lighting, shadows, motion blurring; layers.

• Final Cut Pro: a video editing tool by Apple; Macintosh

only.
 1
Animation

Multimedia APIs:

• Java3D: API used by Java to construct and render 3D

graphics, similar to the way in which the Java Media
Framework is used for handling media les.

• Provides a basic set of object primitives (cube, splines,

etc.) for building scenes.

• It is an abstraction layer built on top of OpenGL or

DirectX (the user can select which).

• DirectX : Windows API that supports video, images,

audio and 3-D animation

• OpenGL: the highly portable, most popular 3-D API.

 1
Animation
• Rendering Tools:

• 3D Studio Max: rendering tool that includes a number

of very high-end professional tools for character
animation, game development, and visual effects
production.

• Softimage XSI: a powerful modeling, animation, and

rendering package used for animation and special
effects in films and games.

• Maya: competing product to Softimage; as well, it is a

complete modeling package.

• RenderMan: rendering package created by Pixar.

• GIF Animation Packages: a simpler approach to

animation, allows very quick development of eective small
animations for the web.
 1
Multimedia Authoring

• Macromedia Flash: allows users to create interactive

movies by using the score metaphor, i.e., a timeline
arranged in parallel event sequences.

• Macromedia Director: uses a movie metaphor to create

interactive presentations | very powerful and includes a
built-in scripting language, Lingo, that allows creation of
complex interactive movies.

• Authorware: a mature, well-supported authoring product

based on the Iconic/Flow-control metaphor.

• Quest: similar to Authorware in many ways, uses a type of

flowcharting metaphor. However, the flowchart nodes can
encapsulate information in a more abstract way (called
frames) than simply subroutine levels.