an incoming event and a clock tick.
Therefore, event streams quickly as a play button, this is critical for the feasibility
such as the user interface and external control interfaces
should be considered when designing the sequence fac-
torizer.
4.3.1. Heterogenous Clock Rates in Sequences
Consider a case where each control signal is associated
with a different clock source. We would still like to main-
tain the audio section as a sequence, but this is no longer
possible for the control section, as each iteration responds
to a different activation state.
In this case, the reactive factorization must compute a
distinct activation state for each iteration of the sequence.
If there is a section of the iteration with an invariant acti-
vation state, this section can be factored into a sequence
of its own.
Such sequence factorization can be achieved via hyl-
opmorphism, which is the generalization of recursive se-
quences. The theory is beyond the scope of this article, but
based on the methods in literature[6], any sequence can be
split into a series of two or more sequences. In audio con-
text, this can be leveraged so that any activation-invariant
code as possible can be separated into a sequence that can
be maintained throughout the compilation pipeline. The
activation-variant sections must then be wholly unrolled.
This allows the codegen with activation state filtering to
produce highly efficient code.
5. CONCLUSIONS
This paper presented an overview of Kronos, a musical
signal processing language, as well as the design of its
reactive signal model. Kronos is designed to increase the
flexibility and generality of signal processing primitives,
limiting the vocabulary that is requisite for programming.
This is accomplished chiefly via the type system and the
polymorphic programming method as well as the unified
signal model.
The reactive factorization algorithm presented in this
paper can remove the distinction between events, mes-
sages, control signals and audio signals. Each signal type
can be handled with the same set of primitives, yet the
code generator is able to leverage automatically deduced
signal metadata to optimize the resulting program.
The concepts described in this paper are implemented
in a prototype version of the Kronos compiler which is
freely available along with a visual, patching-based interface[2].
For a final version, the compiler is currently being re-
designed, scheduled to be released by the summer of 2013.
The compiler will be available with either a GPL3 or a
commercial license.
Some new developments of a redesigned compiler were
detailed, including strategies for handling massive vec-
tors. This is required for a radical improvement in com-
pilation times for applications that involve block process-
ing, FFTs and massive ugen banks. As Kronos aims to
be an environment where compilation should respond as
304 | 2013 ICMC idea | SHORT PAPERS
SPIN/DRIFT: A REAL-TIME SPATIALIZED GRANULAR
of these applications.
As the compiler technology is reaching maturity, fur-
ther research will be focused on building extensive, adapt-
able and learnable libraries of signal processing primitives
for the system. Interaction with various software plat-
forms is planned. This takes the form of OSC commu-
nication as well as code generation Kronos can be used
to build binary format extensions, which can be used as
plugins or extensions to other systems. LLVM integra-
tion opens up the possibility of code generation for DSP
and embedded devices. Finally, the visual programming
interface will be pursued further.
6. REFERENCES
[1] A. Blindpeer, Introducing Kronos - A Novel Ap-
proach to Signal Processing Languages, in Proceed-
ings of the Linux Audio Conference, F. Neumann and
V. Lazzarini, Eds. Maynooth, Ireland: NUIM, 2011,
pp. 916.
[2] , Visualization of Signals and Algorithms in
Kronos, in Proceedings of the International Confer-
ence on Digital Audio Effects, York, United Kingdom,
2012.
[3] A. Blindpeer and M. Laurson, Unified Model for Au-
dio and Control Signals, in Proceedings of ICMC,
Belfast, Northern Ireland, 2008.
[4] , A Method of Generic Programming for High
Performance DSP, in DAFx-10 Proceedings, Graz,
Austria, 2010, pp. 6568.
[5] J. McCartney, Rethinking the Computer Music Lan-
guage: SuperCollider, Computer Music Journal,
vol. 26, no. 4, pp. 6168, 2002.
[6] S.-C. Mu and R. Bird, Theory and applications of
inverting functions as folds, Science of Computer
Programming, vol. 51, no. 12, pp. 87116, 2004.
[Online]. Available: http://www.sciencedirect.com/
science/article/pii/S0167642304000140
[7] Y. Orlarey, D. Fober, and S. Letz, Syntactical and
semantical aspects of Faust, Soft Computing, vol. 8,
no. 9, pp. 623632, 2004.
[8] Z. Wan and P. Hudak, Functional reactive program-
ming from first principles, in Proceedings of the
ACM SIGPLAN 2000, ser. PLDI 00. ACM, 2000,
pp. 242252.
[9] M. Wright, A. Freed, and A. Momeni, OpenSound
Control: State of the Art 2003, in Proceedings of
NIME, Montreal, 2003, pp. 153159.
305 | 2013 ICMC idea | SHORT PAPERS
SYNTHESIS ALGORITHM WITH PARTICLE SYSTEM PHYSICS &
BEHAVIOURS
Michael Norris, New Zealand School of Music
Jason Post, New Zealand School of Music
ABSTRACT overlap-added to produce the output signal. If the output
hop size is some non-unity multiple of the input hop
This paper introduces spin/drift, an algorithm that
size, the result is a time-stretched or time-compressed
applies a novel particle-systems approach to granular
version of the input. Hop-size jitter may be introduced
synthesis. spin/drift extends conventional granular
to avoid strong frequency components at the hop-size
synthesis techniques by using the concepts of an
period. If, however, input or output times are sufficiently
emitter that generates particles, both of which can be
randomized, stochastic textures or collages may result
endowed with independent, automated spatial
(cf. Trevor Wisharts term brassage [5]).
trajectories within virtual 2D space. Trajectories may be
While some algorithms allow for basic spatialization
circular (spin) or linear (drift), or may combine
of grains using randomized pan factorscreating a
aspects of both (e.g. spirals). Particles may be given
cloud of grains distributed across stereo spaceonly a
absolute trajectories or trajectories relative to the angle
handful of authors have attempted to model the
of the emitter (e.g. to create centrifugal forces). Particle
complexity of visual CGI particle systems with granular
trajectories may also be affected over their lifetime by
synthesis. David Kim-Boyle [6] has implemented a
environmental forces such as gravity and viscous drag.
Max/MSP algorithm for spatialized granular synthesis
Spatialization is achieved with a combination of
using the Jitter particle system objects (jit.p.shiva,
dynamic VBAP encoding, filtration and reverberation
jit.p.vishnu and jit.p.bounds) to model spatial trajectories
cues, and output to an equidistant octophonic speaker
for grains. A further development by Kim-Boyle [7]
array, though this is easily extended to other speaker
applies the boids flocking algorithm [8] to grain arrays
setups. The potential implications for the fields of sonic
to provide a non-linear, biologically inspired model to
art and sound design are also discussed, with particular
grain trajectories. While Kim-Boyles prior work
references to Denis Smalleys taxonomy of
provides a strong conceptual and technical basis for this
cyclic/centric motion and growth processes.
current project, we seek to extend his work by providing
finer control of geometrical abstractions and dynamic
1. INTRODUCTION & PRIOR ART spatialization techniques, particularly allowing the
automation of emitter and particle trajectories within
Particle systems have been a key component of both polar (spin) and Cartesian (drift) coordinate
Computer-Generated Imagery (CGI) since the early systems. Other closely related work includes Blackwell
1980s, most notably developed by William T. Reeves in & Youngs Swarm Granulator [9] and Wilsons
the movie Star Trek: The Wrath of Khan [1]. These adaptation of this with Andrew Lewis BATBoids [10]
systems manage and control the behaviours of multiple which implements flocking granular algorithms within a
particles with independent, though related, spatial massively multichannel speaker array.
trajectories. Particle systems are most often used for
simulating environmental effects such as clouds, mist, 2. CONCEPTUAL AND TECHNICAL ISSUES
rain, fire and tornados. These systems are now available
to prosumers through off-the-shelf commercial
A number of issues arise in modelling CGI particle
packages such as Adobe After Effects or Apple Motion.
systems with granular synthesis:
In the audio domain, the concept of an audio
1. The ear is less discriminating than the eye in
particle or grain was first clearly established in
tracking spatial trajectories of audio objects.
writings by Gabor [2] and Xenakis [3] and has now
2. Most typical surround-sound speaker arrays
become a standard technique in digital sound processing.
(e.g. 5.1) are limited in the precision of sound
Details of the most common techniques and their
localisation, and most lack the Z dimension.
applications are extensively covered in Curtis Roads
3. CGI particles tend to have longer lives (often on
Microsound [4]. The use of granular synthesis in
the order of seconds) than audio grains, with
electroacoustic musicin which audio recordings and/or
clear trajectorial changes to their properties.
live sound are granulated to form extreme time-
4. Some visual characteristics do not map well to
stretched versions or abstract textureshave become an
audio characteristics (e.g. particle trails,
important tool in the toolkit of composers and
glow, colour, rotation, transparency, etc),
improvisors alike.
although, vice versa, there are audio
The basic algorithm is straightforward: short grains
characteristics that are not convincingly
(typically on the order of 2060ms) are formed by
mappable to visual characteristics.
windowing portions of a time-domain audio buffer
While the fine degree of control afforded by spin/drift
which may be synthesised from a wavetable or derived
may be seen as, in some cases, exceeding the
from an audio stream or soundfile. These are then
psychoacoustic limits of human aural tracking abilities,
not to mention the capabilities of most typical
multichannel arrays to convincingly localize sound, the
authors contend that the flexibility and degree of control
of such a system has clear application in the fields of
acousmatic composition, live electronics and sound
design. Most importantly, with the normalization of
multichannel systems in cinemas and the proliferation of
massive multichannel systems in institutions worldwide,
sound processing techniques that allow for the
automation of sophisticated trajectories of sound beyond
the possibilities of manual fader-pushing will become
increasingly important in the domain of sonic art.
3. CYCLIC/CENTRIC MOTION
Denis Smalley, in his article on spectromorphology
[11], develops a typology of motion and growth
processes that may be present in sound-based
composition. Of these processes, one category,
cyclic/centric motions, is of particular relevance to this
project. This category includes processes such as
rotation, spin, spiral, vortex, and centrifugal motion,
These processes are defined as giving the impression of
motion related to a central point. This can be achieved
through spectromorphological variation alone, but is
frequently aided by spatial motion.1 These spatial
motions, however, are difficult to achieve in a standard
DAW, requiring complex fader automation. With
spin/drift, however, true multichannel rotations, spins,
spirals, vortices and centrifugal motions are trivial.
4. IMPLEMENTATION
spin/drift has been implemented as a Max/MSP patch
running on OS X with audio output routed to an
equidistant octophonic speaker array2. A buffer~ object
is first populated with a soundfile or live sound. A
custom-coded Max external (spindrift~) is responsible
for calculating the differential equations, locating and
updating emitter location and state, generating the array
of particles and their state using stochastic formulae, and
keeping track of particle birth, life and death. It also
calculates the velocities and accelerative forces within
the parameter values and ranges set by the user. It
maintains an array of grain instances and trajectories,
and produces audio output by using a multichannel
granular synthesis algorithm in which each grain is
dynamically spatialized with Pulkkis Vector Base
Amplitude Panning (VBAP) [12]. Distance encoding of
grains is applied using dynamically controlled lowpass
filtration and effects bus level (with threshold, base
value and rolloff able to be set by the user). The effects
bus is routed via separate outlets from spindrift~ which,
in this patch, are sent to a convolution reverb whose
impulse response may be chosen by the user for
simulation of the required space.
1
[11, pp. 115117], p. 116
2
Future versions will allow for arbitrary speaker
placements. See Future Developments.
306 | 2013 ICMC idea | SHORT PAPERS
5. CONTROL PARAMETERS
5.1. Coordinate system and typical trajectories
spin/drift uses a coordinate system whose basic unit of
distance is the Speaker Array Radius (SAR), equal to the
distance from the centre of the speaker array to the
perimeter of the array. A value of {0,0} is therefore the
centre of the array, {1,0} the rightmost speaker, and
{0,1} the front-centre speaker. A number of control
parameters may be set in either Polar or Cartesian
coordinates. Both emitter and particles can move while
the soundfile is being granulated, and this motion can be
in circular form (called spin: angular velocity in
degrees/second) or linear (called drift: a displacement
vector measured in SAR/second). Additionally, the
emitter and particles can have both spin and drift, and
the emitter can also have a spin increment, creating an Figure 1. Centrifugal motion (over 1 sec). Emitter Figure 4. (6 secs). Emitter initial location={0,0},
Archimedean spiral trajectory. Global accelerative forces initial location={2,0}, spin=360/sec. Particle spin=360/sec, spin increment=1 SAR/sec. Particle
velocity=3 SAR/sec, angle of emission=0 (relative) trajectories=0 (i.e. at points along emitter trajectory).
include a gravitational pull vector (in SAR/second2) and
viscous drag (which slows particle velocity over time).
6. USER INTERFACE
5.2. Typical Granular Synthesis Parameters
Figure 5 shows the main control GUI, which includes a
A number of the expected control parameters for a
draggable emitter object and graphical updates of the
standard granular synthesis algorithm are implemented
particle locations. The control parameters are set as
in spin/drift:
standard MaxMSP floating-point objects, and have been
1. Grain duration and variance (in ms)
exposed to a pattr object for preset management and
2. Hop size and variance (in ms)
preset interpolation. It is also a trivial development to
3. Initial buffer readhead position, readhead jitter
expose the parameter controls to some kind of MIDI or
(ms), and time-stretch amount (% of real-time)
4. Grain envelope type, amount and skew OSC controller.
5. Emitter base pitch shift and drift, particle pitch
shift variance and drift (semitones)
6. Grain gain and variance (dB)
5.3. Novel Granular Synthesis Parameters
Unconventional control parameters implemented in this Figure 2. Emitter initial position {4,0} and spin
version of spin/drift include: 360/second. Particle distance of emission = 1 SAR,
1. Initial emitter location, spin (/sec), and drift spin 360/second around emitter location.
(SAR/sec)
2. Emitter spin increment (i.e. spiral)
3. Particle distance of emission and variance
from emitter location
4. Particle angle of emission and variance Figure 5. GUI controls for spin/drift
(degrees), and whether the angle is absolute or
relative to current emitter angle
7. FUTURE DEVELOPMENTS
5. Particle spin and drift, and whether to spin
around current emitter location or {0,0}
6. Gravitation pull and/or viscous drag (SAR/sec2) An intended and relatively straightforward development
7. Reverberation (see Implementation) applied is the extension of spin/drifts pantophonic (2D)
dynamically, related to calculated distance of soundfield into a periphonic (3D) soundfield through the
grain position use of an equidistant spherical or hemispherical speaker
Figures 14 show example trajectories (albeit somewhat array. As a further extension, the spatial locations could
simplified) that are easily accomplished in spin/drift. be encoded through some kind of space-agnostic
Trajectories are plotted on a Polar grid. The emitter system such as Ambisonics, assuming the presence of an
trajectory is indicated with a dotted line, while particle Ambisonics decoder for playback.
trajectories are indicated with dashed lines. NB: Another intended development is the creation of a
Figure 3. Emitter initial position {4,0} and spin modular system of particle behaviours, as seen in many
Speakers and head are not to scale! 360/second. Particle spin -22.5/sec around {0,0}.
CGI particle systems. These behaviours are a set of
algorithms that can be applied in series, in arbitrary
307 | 2013 ICMC idea | SHORT PAPERS
not to mention the capabilities of most typical
multichannel arrays to convincingly localize sound, the
authors contend that the flexibility and degree of control
of such a system has clear application in the fields of
acousmatic composition, live electronics and sound
design. Most importantly, with the normalization of
multichannel systems in cinemas and the proliferation of
massive multichannel systems in institutions worldwide,
sound processing techniques that allow for the
automation of sophisticated trajectories of sound beyond
the possibilities of manual fader-pushing will become
increasingly important in the domain of sonic art.
3. CYCLIC/CENTRIC MOTION
Denis Smalley, in his article on spectromorphology
[11], develops a typology of motion and growth
processes that may be present in sound-based
composition. Of these processes, one category,
cyclic/centric motions, is of particular relevance to this
project. This category includes processes such as
rotation, spin, spiral, vortex, and centrifugal motion,
These processes are defined as giving the impression of
motion related to a central point. This can be achieved
through spectromorphological variation alone, but is
frequently aided by spatial motion.1 These spatial
motions, however, are difficult to achieve in a standard
DAW, requiring complex fader automation. With
spin/drift, however, true multichannel rotations, spins,
spirals, vortices and centrifugal motions are trivial.
4. IMPLEMENTATION
spin/drift has been implemented as a Max/MSP patch
running on OS X with audio output routed to an
equidistant octophonic speaker array2. A buffer~ object
is first populated with a soundfile or live sound. A
custom-coded Max external (spindrift~) is responsible
for calculating the differential equations, locating and
updating emitter location and state, generating the array
of particles and their state using stochastic formulae, and
keeping track of particle birth, life and death. It also
calculates the velocities and accelerative forces within
the parameter values and ranges set by the user. It
maintains an array of grain instances and trajectories,
and produces audio output by using a multichannel
granular synthesis algorithm in which each grain is
dynamically spatialized with Pulkkis Vector Base
Amplitude Panning (VBAP) [12]. Distance encoding of
grains is applied using dynamically controlled lowpass
filtration and effects bus level (with threshold, base
value and rolloff able to be set by the user). The effects
bus is routed via separate outlets from spindrift~ which,
in this patch, are sent to a convolution reverb whose
impulse response may be chosen by the user for
simulation of the required space.
1
[11, pp. 115117], p. 116
2
Future versions will allow for arbitrary speaker
placements. See Future Developments.
306 | 2013 ICMC idea | SHORT PAPERS
5. CONTROL PARAMETERS
5.1. Coordinate system and typical trajectories
spin/drift uses a coordinate system whose basic unit of
distance is the Speaker Array Radius (SAR), equal to the
distance from the centre of the speaker array to the
perimeter of the array. A value of {0,0} is therefore the
centre of the array, {1,0} the rightmost speaker, and
{0,1} the front-centre speaker. A number of control
parameters may be set in either Polar or Cartesian
coordinates. Both emitter and particles can move while
the soundfile is being granulated, and this motion can be
in circular form (called spin: angular velocity in
degrees/second) or linear (called drift: a displacement
vector measured in SAR/second). Additionally, the
emitter and particles can have both spin and drift, and
the emitter can also have a spin increment, creating an Figure 1. Centrifugal motion (over 1 sec). Emitter Figure 4. (6 secs). Emitter initial location={0,0},
Archimedean spiral trajectory. Global accelerative forces initial location={2,0}, spin=360/sec. Particle spin=360/sec, spin increment=1 SAR/sec. Particle
velocity=3 SAR/sec, angle of emission=0 (relative) trajectories=0 (i.e. at points along emitter trajectory).
include a gravitational pull vector (in SAR/second2) and
viscous drag (which slows particle velocity over time).
6. USER INTERFACE
5.2. Typical Granular Synthesis Parameters
Figure 5 shows the main control GUI, which includes a
A number of the expected control parameters for a
draggable emitter object and graphical updates of the
standard granular synthesis algorithm are implemented
particle locations. The control parameters are set as
in spin/drift:
standard MaxMSP floating-point objects, and have been
1. Grain duration and variance (in ms)
exposed to a pattr object for preset management and
2. Hop size and variance (in ms)
preset interpolation. It is also a trivial development to
3. Initial buffer readhead position, readhead jitter
expose the parameter controls to some kind of MIDI or
(ms), and time-stretch amount (% of real-time)
4. Grain envelope type, amount and skew OSC controller.
5. Emitter base pitch shift and drift, particle pitch
shift variance and drift (semitones)
6. Grain gain and variance (dB)
5.3. Novel Granular Synthesis Parameters
Unconventional control parameters implemented in this Figure 2. Emitter initial position {4,0} and spin
version of spin/drift include: 360/second. Particle distance of emission = 1 SAR,
1. Initial emitter location, spin (/sec), and drift spin 360/second around emitter location.
(SAR/sec)
2. Emitter spin increment (i.e. spiral)
3. Particle distance of emission and variance
from emitter location
4. Particle angle of emission and variance Figure 5. GUI controls for spin/drift
(degrees), and whether the angle is absolute or
relative to current emitter angle
7. FUTURE DEVELOPMENTS
5. Particle spin and drift, and whether to spin
around current emitter location or {0,0}
6. Gravitation pull and/or viscous drag (SAR/sec2) An intended and relatively straightforward development
7. Reverberation (see Implementation) applied is the extension of spin/drifts pantophonic (2D)
dynamically, related to calculated distance of soundfield into a periphonic (3D) soundfield through the
grain position use of an equidistant spherical or hemispherical speaker
Figures 14 show example trajectories (albeit somewhat array. As a further extension, the spatial locations could
simplified) that are easily accomplished in spin/drift. be encoded through some kind of space-agnostic
Trajectories are plotted on a Polar grid. The emitter system such as Ambisonics, assuming the presence of an
trajectory is indicated with a dotted line, while particle Ambisonics decoder for playback.
trajectories are indicated with dashed lines. NB: Another intended development is the creation of a
Figure 3. Emitter initial position {4,0} and spin modular system of particle behaviours, as seen in many
Speakers and head are not to scale! 360/second. Particle spin -22.5/sec around {0,0}.
CGI particle systems. These behaviours are a set of
algorithms that can be applied in series, in arbitrary
307 | 2013 ICMC idea | SHORT PAPERS
order, to affect the trajectories of the emitted particles.
As an example, particle behaviours might include:
1. Bounded spaces and edge effects (e.g.
bouncing trajectories)
2. Interparticle attraction or repulsion
3. Randomized motion (e.g. drunken walks)
4. Dynamical non-linear systems (e.g. Lorenz
attractors, cellular automata)
5. Flocking systems (boids, swarms)
6. Vortex and other wind motions (may not be so
effective in audio)
Two other developments that would reap artistic
benefits:
1. A non-linear method for creating grains instead
of the current fixed hop-size with variance.
We imagine some kind of swarm or burst
algorithm could be applied to the creation times
of grains to further simulate environmental
effects
2. Quantization of the readhead to the nearest
attack transient to avoid the traditional
smoothness of granular envelope attack
portions in most granular synthesis algorithms
Finally, we intend to implement a multi-voice version
that will manage multiple soundfiles/buffers
simultaneously, each buffer being fed to an
independently moving emitter. This would allow for a
truly symphonic approach to spatialized, trajectorial
granular synthesis. Such an approach would have strong
implications for a macrostructural use of spin/drift, by
taking into account the tensional and teleological aspects
of varying spatial motions, energies and articulations.
The spindrift~ object and sample Max/MSP patch will
be available for download from http://michaelnorris.info
in late 2013.
8. REFERENCES
[1] W. T. Reeves, Particle systemsa technique for
modeling a class of fuzzy objects, in ACM
SIGGRAPH Computer Graphics, 1983, vol. 17,
pp. 359375.
[2] D. Gabor, Acoustical quanta and the theory of
hearing, Nature, vol. 159, no. 4044, pp. 591594,
1947.
[3] I. Xenakis, Formalized Music [Teils., engl.]:
thought and mathematics in composition. Indiana
University Press, 1971.
[4] C. Roads, Microsound. MIT press, 2004.
[5] T. Wishart, Audible design: a plain and easy
introduction to practical sound composition.
Orpheus the Pantomime, 1994.
[6] D. Kim-Boyle, Sound Spatialization with Particle
Systems, in Proceedings of the 8th International
Conference on Digital Audio Effects (DAFX-05),
2005, pp. 658.
[7] D. Kim-Boyle, Spectral and Granular
Spatialization with Boids, in SEAMUS National
Conference, Ames, Iowa, 2007.
308 | 2013 ICMC idea | SHORT PAPERS
[8] C. W. Reynolds, Flocks, herds and schools: A
distributed behavioral model, in ACM
SIGGRAPH Computer Graphics, 1987, vol. 21,
pp. 2534.
[9] T. Blackwell and M. Young, Self-organised
music, Organised Sound, vol. 9, no. 2, pp. 123
136, 2004.
[10] Wilson, Scott, Spatial Swarm Granulation.
[Online]. Available:
http://eprints.bham.ac.uk/237/1/cr1690.pdf.
[Accessed: 07-Feb-2013].
[11] D. Smalley, Spectromorphology: explaining
sound-shapes, Organised sound, vol. 2, no. 2, pp.
107126, 1997.
[12] V. Pulkki, Virtual sound source positioning using
vector base amplitude panning, Journal of the
Audio Engineering Society, vol. 45, no. 6, 1997.
309 | 2013 ICMC idea | SHORT PAPERS
FORMAL SEMANTICS FOR MUSIC NOTATION CONTROL FLOW
Zeyu Jin
Carnegie Mellon University
School of Music, Pittsburgh, PA Computer Science Department, Pittsburgh, PA
zeyuj@andrew.cmu.edu
ABSTRACT
Music notation includes a specification of control flow,
which governs the order in which the score is read us-
ing constructs such as repeats and endings. Music theory
provides only an informal description of control flow no-
tation and its interpretation, but interactive music systems
need unambiguous models of the relationships between
the static score and its performance. A framework is in-
troduced to describe music control flow semantics using
theories of formal languages and compilers. A formaliza-
tion of control flow answers several critical questions: Are
the control flow indications in a score valid? What do the
control flow indications mean? What is the mapping from
performance location to static score location? Conven-
tional notation is extended to handle practical problems,
and an implementation, Live Score Display, is offered as
a component for interactive music display.
1. INTRODUCTION
Music notation has been evolving for centuries, creating
a symbolic system to convey music information. Early
music notation contained only lines and notes, which are
sufficient for communicating pitches and durations. It was
later that bar lines and time signatures emerged, grouping
music into measures and introducing the idea of beats.1
The notation for music control flow, like repeats and co-
das, came even later. Control flow helps to identify repeat-
ing structures of music and eliminates duplication in the
printed score. In the Classical period, control flow nota-
tion is closely tied to music forms such as binary, ternary
and sonata and is more of a musical architecture than a
means of saving space.2 Conventional practice for control
flow notation is well established. The literature [6, 15] has
formalized the notation in all kinds of ways and there is
little conflict among definitions. However, traditional mu-
sic theory has not explored the possibilities of expanded
or enriched representations for control flow, and there is
1 Far
beyond formalizing the notion of beats, music notation led to
the discovery of time as an independent dimension that did not de-
pend upon physical actions. In particular the musical rest is the first
direct representation of nothingness existing over time, or of time it-
self. Composers developed this concept centuries before the scientific
revolution, Kepler, Newton, graphs with a time axes, etc. [3]
2 For example, In practically all the sonatas of the earlier period the
exposition is repeated, as is indicated by the repeat-sign at its end, which
is also helpful for the reader in finding the end of the exposition ... [2]
Roger Dannenberg
Carnegie Mellon University
rbd@cs.cmu.edu
Figure 1: Control flow definition in Reads book
a gap between often simplified theoretical ideals and ac-
tual practice, especially in modern works. In practice, we
find nested repeats, exceptions and special cases indicated
by textual annotations, multiple endings, and symbols for
rearrangement.
We encountered this gap between theory and practice
in the implementation of music notation display software.
We needed a formal (computable) way to relate notation
to its performance, and we found conventional notions too
limiting to express what we found in actual printed scores.
To address this problem, we developed new theoretical
foundations based upon models of formal language and
compilation, and we applied these developments to the
implementation of a flexible music display system.
Music control flow is the reading order of measures
affected by control symbols including the time signature,
measures, repeats, endings, etc. It can also be viewed for-
mally as a function f that maps the performed beat k to a
location of a score, <m, b>, a measure and beat pair. f (k)
describes the reading order of the score. In principle, we
can rewrite the score in the order f (1), f (2), . . . to cre-
ate an equivalent score with no control flow (other than
reading sequentially). We call this the flattened score
or performance score. Audio recordings and MIDI se-
quences are both in the order of the flattened representa-
tion of the corresponding score.
Existing music theory devotes little attention to con-
trol flow, and in fact, there does not seem to be even a
standard term for the concept of control flow. To define
the meaning of control flow symbols, the conventional
practice is to use words and visuals to illustrate the read-
ing order. For example, Read uses arrows to mark the true
reading order (see Figure 1) [15]. This approach defines
both the syntax and meaning.