Interactive Live Instrumental Music 21-06-2016

"INTERACTIVE LIVE INSTRUMENTAL MUSIC"
THESIS
SUBMITTED TO THE DEPARTMENT OF MUSIC AND
TECHNOLOGY
AND THE COMMITTEE ON GRADUATE STUDIES
OF HKU UTRECHT UNIVERSITY OF THE ARTS
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF MASTER OF MUSIC
Than van Nispen tot Pannerden - than.vannispen@hku.nl - student id nr. 2051299

June 2016
ii
Abstract
This thesis contextualises the portfolio of relevant works that I created during the period of
my Master of Music studies from 2014 to 2016.
The projects described involve interactive live instrumental music, which is the subject of
this thesis, and the interactive music is mainly based on video-game music methods.
In these projects the music is performed live by several musicians, and an interactive system
defines which music should by played by whom and when, based on certain events. The
projects that do not have an interactive system for the music at least have a set of rules for
play and interaction for, or with the music.
A focus of this Master of Music studies is the exploration of solutions for interactive music,
based on game music methods, with the use of traditional music notation.
Keywords: interactive music, audience participation, game music, live music, videogames, audience games, ensembles, orchestral music
iii
Acknowledgement
I would like to express my enormous gratitude to everyone involved in the realisation of
the projects described in this thesis, especially the ensembles, conductors and musicians
that made all the interactive live instrumental music possible:
Metropole Orkest, Residentie Orkest, Rotterdam Philharmonic Orchestra, Royal Concertgebouw Orchestra, Gelders Orkest, Space Invaders Ensemble, Hartkamer-orkest (Nacht
van de Kunst & Wetenschap), Kaiser String Quartet, Ludwig Live and specifically: Jan
Stulen, Jules Buckley, Roland Kieft, Katinka Reinders, Ernst van Tiel, Bob Zimmerman,
Loek Dikker, Ivan Meylemans, Bas Wiegers, Michael Klier, Hans-Joachim Esser-Mamat,
Wilmar de Visser, Peppie Wiersma, Derk van der Kamp.
I would also like to thank the HKU for the opportunity to enroll in this Master of Music
and my supervisor Gerard van Wolferen in particular. I am most grateful of my colleagues
at Utrecht School of the Arts for providing a fruitful discourse on many subjects relating
to my Master of Music in particular Rens Machielse, Jan IJzermans, Hans Timmermans,
Marcel Wierckx, Sander Huiberts, Richard van Tol, Kees Went, Ciska Vriezenga, Bart
Delissen, Joris Weijdom, Machiel Veldkamp, Martijn van Gessel, Marinka Copier, Iris
Douma and, again, Gerard van Wolferen.
Finally I would also like to thank Ivo van Dijk, Sebastiaan Donders, Iris Oltheten, Bertus
van Dalen, Jegor van Opdorp, Thijs Koerselman, Stan Koch, Nick van Kleef, Ruben
Bergshoeff, Tony Roe, Pat Cleaver, Bobby Petrov, Friederike Darius, Mendel Bouman,
Cesco Homminga, Kirsten Dabekaussen and Maxime Goulet for their involvement in one
or more of the projects and my family and friends for visiting the concerts I worked on and
showing a sincere interest in my music projects.
iv
Contents
Abstract
iii
Acknowledgement
iv
introduction
1.1
Why interactive, live and instrumental? . . . . . . . . . . . . . . . . . . .
1.2
Definitions? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Interactive Music . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Interactive music strategies (in video games) . . . . . . . . . . . . . . . . .
Instrumental Music . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Live instrumental music
. . . . . . . . . . . . . . . . . . . . . . . . . . .
Interactive live instrumental music . . . . . . . . . . . . . . . . . . . . . .
interactive live instrumental music experiences

2.1
15
Horizontal re-sequencing . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Karmaflow in Concert 2015 - democratic sequence . . . . . . . . . . . . . 16
NLN-Live - interactive live instrumental music with iPads . . . . . . . . . 19
v
Space Invaders performed live at the 2015 Indie Games Concert . . 22

Space Invaders performed live at the 2015 Night of Arts & Sciences
(Nacht van Kunst & Wetenschap) . . . . . . . . . . . . . 23
Other Space Invaders compositions performed live at the Media
Sound Hamburg 2015 . . . . . . . . . . . . . . . . . . . 24
Experiences from the use of NLN-Live . . . . . . . . . . . . . . . 25
Musikalisches Wrfelspiele live at 2015 Media Sound Hamburg . . . . . . 26
2.2
Vertical Re-orchestration . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Johan Sebastian Joust performed live at the 2015 Indie Games Concert . . . 29
Parallel staffs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Johan Sebastian Joust performed live at the 2016 Night of Art and Science
32
King of Games - live interaction with children (2015 adaptations) . . . . . 34

2.3
Combinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Interactive music experiment with the audience as an instrument at 2015
Media Sound Hamburg (not performed) . . . . . . . . . . . . . . . 36
Hart Joust performed live at the 2016 Night of Arts & Science 2016 . . . . 38
2.4
Other
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
RCO - live music interaction with children
. . . . . . . . . . . . . . . . . 40
Pepernoten voor groot orkest . . . . . . . . . . . . . . . . . . . . 41

Dansen met Bach (Dance with Bach) . . . . . . . . . . . . . . . 42
Other relevant repertoire during the period of my Master of Music studies . 44
Hat Trick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Project-week Future Sound of Live with Tin Men and the Telephone . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
vi
Discussion and Conclusions

3.1
48
Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Vertical re-orchestration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Horizontal re-sequencing . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Combinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.2
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
A Karmaflow Interactive Music Approach
59
B First eight pages of the Space Invaders score for the Indie Games Concert 2015 61
C First two pages of Kirnberger Trio with fragments in order of table
70
D First two pages of the J.S. Joust score with parallel staffs
73
E Part for first Violin for Hart Joust
76
F Score for Hart Joust
78
Bibliography
82
vii
viii
Chapter 1
introduction
This Master of Musics thesis describes the projects, linked to a big part of my design
practice and the subject of my main focus:
Interactive Live Instrumental Music.
The final presentation of my Master of Music studies includes both a creative practice
component in the form of a portfolio of relevant works, created during period of 2014 to
2016, and this thesis that contextualises my practice.
After the introduction in this Chapter 1, I describe, in Chapter 2, the most important
terminology relevant for the contextualisation of my work and how I use these terms. I
relate the context of the subject of this thesis to other repertoire and explain my motivations
behind the projects I worked on.
In Chapter 3 I describe these projects and their contexts, as well as some other relevant
experiences during this period, and in Chapter 4 I discuss my findings, recommendations,
future plans and ideas.
1.1
Why interactive, live and instrumental?
As a composer with a background in composition for media, games and interaction I happened to be invited to participate in the realisation of several live instrumental music performances, often with some element of video game music, or interaction. I noticed that
1
CHAPTER 1. INTRODUCTION
most concerts with music from video games did not have any live interactive music in the
performance1 , even though sometimes audience members were invited to play a game live
on stage (as was the case in the concerts of VideoGamesLive2 and the Level Up-concert of
St. Jan Marke3 ).
Possibly this is due to the nature of the use and implementation of pre-recorded music in
video games and the transformation of this digital music implementation to live instrumental performance proves to be challenging.
One other situation that led to my interest in this topic was the release in 2013 of a new
version of a composition application for children, called http://ikcomponeer.nl. In this application one can place musical fragments of orchestral material in a sequence and/or on
parallel, thus creating a new composition. Bertus van Dalen developed the web-application
and I composed and arranged most of the musical content.
To celebrate the release of this new version a concert was being organised and the organisation of the concert was exploring the idea to perform compositions that would be made by
children at the release event and asked me whether this idea would be possible. Besides this
connection with this composition-tool for children I was invited to write a new interactive
composition.
With my experiences in composition for games and interaction I started to experiment with
the transformation from digital use of (interactive) music to a physical (live and instrumental) realisation of interactive music.
These elements led to one of the central research questions in my projects described in
this thesis, namely whether it is possible to transform the interactive music strategies that
are most common in video game applications to be used in the context of interactive live
instrumental music.
Before, during (and after) my studies for my Master of Music (2014-2016) I have been
enrolled in a wide range of projects and with several ensembles of different sizes, which
made these experiments possible in practical contexts.
1 Games
in Concert with Metropole Orkest (2006, 2007, 2008, 2014 http://www.npo.nl/games-inconcert/04-04-2010/NCRV_1367415), Indie Games Concert (2013), Distant World-Final Fantasy concerts
http://www.ffdistantworlds.com/, etc.
2 http://www.videogameslive.com/
3 http://levelup.sjmarke.be/
1.2. DEFINITIONS?
One other goal (that will be explained in Interactive live instrumental music on page 9)
of my studies is to explore if these music strategies could possibly expand the field of live
instrumental music. A field with a long history and rich tradition, but also with its restrictions and challenges when it comes to the realisation of non-linearity and interactivity.
Some of these restrictions and challenges concern the musical background of musicians4 ,
the size of the ensemble5 and the role of the audience6 .
The exploration of solutions for interactive music by means of notation, instead of leaving
many aspects of the music to the interpretation of (and improvisation by) the musicians, is
part of my studies and this thesis.
1.2
Definitions?
The goal of this Master of Musics thesis is to describe as good as possible the context and
content of the musical projects that cover interactive live instrumental music. It is not my
intention to discuss too much in depth the different definitions that are covered in this topic.
It is important however to define what the most important terms I use mean in the specific
context of my Master of Music studies and this thesis.
Many other authors (Went, 2014; McAlpine et al., 2009; Collins, 2008) have made
overviews of different definitions and will be cited where relevant. I will first try to describe what interactive live instrumental music means, by explaining the meaning of the
different components in the context of this thesis .
Interactive Music
Both the terms Interactive and Music can lead to long debates on what this actually
could (or should) be. For instance what can be considered Music is open to interpretation
4 whether
the musicians are trained in improvisation, or score-reading

requires for some notation solutions for, for instance the string section of a large orchestra to play in
unison when responding to interaction
6 the audience is traditionally rarely invited to participate in a concert, or interact with the music performance, other than listening and being silent when the music is being played and applaud when the music has
ended
5 it
What it is about
What it is not about
(members of)
audience
Interactive
Music system
Music
Sheet music
vs
Improvised
Musicians
Instrumental
vs
Synthesized
Performance
Live
vs
Recorded
Figure 1.1: Figure to describe what this Master of Music is about and what it is not about.
Interaction with the music is always executed via a system, and in the context of this thesis, via
(members of) an audience.
and differs culturally, historically and even biologically between different persons (Hijleh,
2012; Levitin, 2008; Sacks, 2008). A definition that is often found is that music is humanly
organised sound7 .
In the context of the music projects described in this thesis Im personally satisfied with the
definition from the Websters Collegiate Dictionary "the science or art of ordering tones or
sounds in succession, in combination, and in temporal relationships to produce a composition having unity and continuity".
In this context Im actually even more pleased with the definition by Apples 10.11s Dictionary App (that uses the British Oxford Dictionary of English (Oxford-Dictionaries, 2010)),
which defines Music as: 1. vocal or instrumental sounds (or both) combined in such a
way as to produce beauty of form, harmony, and expression of emotion. 2. the written or
printed signs representing vocal or instrumental sound..
7 John
Blacking & Edgar Varse
1.2. DEFINITIONS?
Although this is a very narrow and limited definition of music, it describes best my intentions for the music projects discussed in this thesis.
Interaction according to the Apple dictionary App means reciprocal action or influence (Oxford-Dictionaries, 2010). This definition would suggest that in interactive music a reciprocal action between something and the music would occur. One may question
whether a musical response to an action would always necessarily lead to a reciprocal response, or that we can consider the musical response being (inter)active enough.
Kees Went describes in his 2014 thesis Sound Design for Nonlinear Artefacts the troublesome terminology regarding interactive audio (page 36 and 37), and states that sound
by itself can not react to anything. It is the [..] system that responds to gameplay, and uses
sound as a medium to communicate this [..]. Likewise, the music itself can never react,
or be interactive, it is the performance of the music (the system) that reacts, or interacts.
This would mean that interactive music could be better understood as a reciprocal action
between one system and another, using music as a medium. A system can be a musician,
but can also be a computer system presenting the appropriate new musical material to the
musician in response to an action, or event.
In this thesis interactive music is considered to be music where aspects of the music
performance, such as tempo, dynamics and the musical material, can respond dynamically
to specific events, such as events caused by a person playing a video-game, or (members
of) an audience playing an audience game8 .
Interactive Music in video-games

Interactive music is well known in video games. In some games specific events can for
instance be the pick up of a power-up (the music in PacMan and/or Donkey Kong change
when the user picks up such a power-up), or the quality of correctly playing along with
existing music (in, for instance, Guitar Hero).
Aspects of the music performance can for instance be the volume of the music, or the
(playback)speed, but also the use of effects and filters on the music and, of course, the
variation of the musical material itself (Collins, 2008).
8 See
Hat Trick on page 44 for an example of an audience game
This broad definition of interactive music includes the interaction with pre-recorded (linear)
music, such as the music in the 2008 video game Braid9 , in which the playback-speed and
-direction are related to the user playing the game.
One could also consider games such as AudioSurf, Vib Ribbon, Russian Dancing Men,
Dance Dance Revolution etc. as being interactive music-games, but since the music is not
responding to events in the game, these games could better be considered as music-active
games, for the user acts on the music, without the music responding / adapting to the
actions of the user.
Instead of the music responding to the action, these applications respond to the music.
This approach to music (inter)action can sometimes also be considered, or experienced as
interactive music, as one is acting on the music.
Historically this kind of music (inter)action include for example dancing and moving on
the music.
A video game like Guitar Hero would also be considered as a music-active game if
there would be no response in the music to the users playing accuracy. This would also be
the same for SingStar-like games if one wouldnt need to sing along, as the music doesnt
change when one doesnt sing.
Playing along with a controller (Guitar Hero, RockBand), an instrument (RockSmith),
voice and/or (body)percussion is however considered as changing the music itself with
an additional music layer and therefore always interactive.
Games like J.S. Joust10 , Musical Chairs (stoelendans in Dutch)11 require the musical
output to change in order for the game to take effect. There are events that relate to both
the gameplay and the music. These events are either realised by a person (in Musical Chairs
someone presses pause for the music to stop), or a(nother) system like the game itself.
These are also considered as interactive music games.
Most of the interaction that will be discussed in this thesis involves events, occurring
by the interaction with a single audience member (Space Invaders), several members (J.S.
Joust), or an entire audience (King of Games, Pepernoten voor Groot Orkest and Hat Trick).
9 http://braid-game.com/
10 http://jsjoust.com/
11 https://en.wikipedia.org/wiki/Musical_chairs
1.2. DEFINITIONS?
Interactive music strategies (in video games)

In video games most interactive music is realised by either one, or a combination, of the
three most used interactive music strategies:
vertical
re-orchestration
(see
figure
1.2
re-
orchestration) is a music method in which the mix
game-play variable(s)
of separate tracks (stems) of music is changed based on

changes in gameplay variables (McAlpine et al., 2009;
Collins, 2008), is also described as a disjointed linear
music system in Went (2014) and vertical mixing in
Sweet (2015). An example of a video-game using this
time
re-orchestration
approach is the 2001 Red Dead Redemption.

Fading (see figure 1.2 fades) from one composition to
a variation of the composition can sometimes also be
considered as vertical re-orchestration, depending on the
music. In the 2012 video-game Faster than Light12 for
time
fades / switching
instance a "battle" version is usually reasonably similar

to the "explore" version with the addition of drums and
similar dark timbres, making transitions between the two
versions smooth. This is in contrast to the invincibility
theme in for instance the Super Mario series which aims
time
re-sequencing
to stick out, making the change sudden and obvious13 .
state 1
state 2
state 3
state 4
This last variation to vertical re-orchestration can be considered what is called the Switching of soundtracks:
depending on the game variable another (sound)track is
time
played.
This switching (sometimes via a crossfade) of sound- Figure 1.2: Most common intertracks is a very important and much used method in active music strategies (in video
video games.
12 http://www.ftlgame.com/
13 https://en.wikipedia.org/wiki/Dynamic_music
games)
horizontal re-sequencing (see figure 1.2 re-sequencing) is a music method in which precomposed segments of music can be re-shuffled based on changes in game variables. This
music strategy is also known as the variable open form (Collins, 2008), has been described
as a disjointed linear music system Went, 2014 and is more thoroughly explained and discussed in my earlier articles on the NLN-Player and NLN-Live (van Nispen tot Pannerden,
2011, 2014).
There are also writings in which the earlier mentioned switching of soundtracks is considered as being part of horizontal re-sequencing (Sweet, 2015).
An early example of a video-game using horizontal re-sequencing is the 1984 Lazy Jones.
algorithmic music is music in which the musical content is generated and performed
within the video-game via synthesizers and/or music sampler instruments.
Although this method is not used very frequently a notorious example of a video-game
using this approach is the 2008 video game Spore in which melodies and rhythms are
generated.
A category other could consist of previously mentioned examples (in Interactive Music
on page 3), such as playback speed, the use of effects and filters, etc (Collins, 2008).
Instrumental Music
In this thesis only projects with instrumental music are discussed in which the instrumental part states that the music is primarily or exclusively produced by musical instruments,
played by musicians. A field which is for instance excluded is interactive synthesized, or
(other) computer music, where some technology realises the sound without a human being
performing an instrument. This aspect also excludes robots playing musical instruments.
One goal of my studies is to explore the possible expansion of the field of live instrumental music. A field with a long history and rich tradition, but also with its restrictions
and challenges when it comes to the realisation of certain aspects of non-linearity and interactivity.
In this thesis I approach the field of instrumental music from a traditional perspective as it
is and exclude experiments with, or the introduction of, new musical instruments.
1.2. DEFINITIONS?
Live instrumental music

In this thesis projects with a focus on live instrumental music are described, meaning
the music is to be performed live and does not cover the field of recorded instrumental
music as we know from soundtracks, both interactive and not.
Interactive live instrumental music

With the previous definitions in mind we would define interactive live instrumental music
as music that responds (via a system) to certain events and is being performed live by one,
or more musicians.
Or in a variation we could define interactive live instrumental music as music, being performed live by one, or more musicians, in which a system defines which music should by
played by whom and when, based on certain events.
This last definition could be the better of the two in the context of this thesis, since it emphasizes the interactive music system more. As the music systems described in this thesis
are based on interactive music methods in video game music, this aspect is quite noteworthy.
One can assume that certain aspects of these interactive live instrumental music performances are non-linear and unique for every occasion of the performance, as the events
leading to different musical output would probably differ at different occasions.
When one thinks about this definition, interactive live instrumental music performances
may not seem new at all, as the history of (world) music for example demonstrates (Sweet,
2015). In other cultures, for instance in the historic "topeng" and "kecak" dances in Bali,
the music is also realised in a reciprocal reactive way between the dancer and the gamelan,
based on certain rules (Terry E. Miller, 2012; Tenzer, 2011).
When playing improvised music, some of the music is generated, invented, or at least
modified live in a reciprocal reaction to the other musicians and/or audience and/or other
variables (such as the actual length of the current improvisation).
Composer and musician John Zorn formalised certain aspects of improvisation in his 1984
composition Cobra. Musicians get their cues from descriptions on cards, and need to
10
learn how to interpret these cues, by learning rules corresponding to these cues. Some of
these cues include: POOL: Those playing stop, or radically change. Those not playing
may enter, or V(olume change): Crescendo, diminuendo, or faders as determined by
caller.
The number of players, instrumentation, and length of the piece is indeterminate. Because
there is no traditional musical notation and the players improvise, every piece most probably sounds different in every performance.
The interaction lies within the reciprocal influence of musicians and/or caller (who can
be considered as a sort of improvisation conductor) and a performance usually doesnt
comprise any audience interaction.
The Dutch multimedia musical ensemble
Tin Men and the Telephone is a jazz trio that
plays interactive live instrumental music during
their concerts. The audience can install an App,
called Tinmendo, with which they can control certain aspects of some of the compositions.
These interactions consist of changing the speed
and volume of the music, but also who should
be playing a solo, whether the trio should conFigure 1.3: Tin Men and the Telephone: on tinue playing, have a break, or should stop etc.
the screen the result of the audience poll is
The audience sees what they can vote for and
visible
the outcomes of the voting on the stage projec-
tion screen as well as on their smart device.

In one notorious piece frontman and pianist Tony Roe, uses his piano (by means of a Moog
Pianobar to transform his music to the midi-domain) as an interface to type chat-messages
to the audience, communicating with the audience with both music and text.
The works by composer and computer music researcher Jason Freeman are also very
noteworthy when it comes to interactive live instrumental music. For instance in his 2004
composition Glimmer the members from the concert audience are invited as musical collaborators who do not just listen to the performance but actively shape it. Each member of
the audience is given a light stick which can be waved back and forth over the course of the
piece. Musicians receive instructions, via multi-colored lights, based on video analysis of
1.2. DEFINITIONS?
11
the audience (Freeman, 2004).

In his 2013 work Sketching, a chamber group of musicians improvises, guided by a
graphic score. Audience members use their mobile devices to draw the score collaboratively, in real time, during the concert (Freeman, 2013a).
And in Freemans 2013 Teamwork the audience use their mobile phones to changes the
tempo and dynamics of the performance of an athletic music band playing their teams fight
song (Freeman, 2013b).
In the rock music-scene there are of course also interesting examples of audience participation in relation to the music, where the audience is for instance invited to sing along
(while the vocalist doesnt sing at that moment and/or points the microphone towards the
audience), make noise, do the wave, etc. Bas Frishert (Frishert, 2015), in his 2015 Master
of Arts thesis, gives a nice overview of a history of rock music, as well as certain examples
regarding audience participation and rock music.
Oliver Hdl and Geraldine Fitzpatrick experimented with audience interaction in their Experimence by means of a big red balloon the audience could toss around, controlling effect parameters, while the rock band was playing a precomposed elements of a rock song
(Oliver Hdl, 2014).
As one might notice, many of these examples are based on improvisation, avant-garde
and/or highly experimental music with certain aesthetics that are quite different from
most traditional and multimedia music, such as (interactive) game music. Another element
which might have been noticed is that in many of these examples either there are no scores,
nor parts (improvisation agreements aside), or non-traditional scores and parts, such as
graphical scores.
Very little examples of interactive live instrumental music exist that can be considered as
an extension to a more traditional form of music notation and performance, known in for
example classical music ensembles. These classical ensembles are not known for their
quality in improvisation, but for their skills in interpreting and performing notated music14 .
As I explained, in the Section Why interactive, live and instrumental? on page 1 of the in14 The
willingness of classical ensembles to experiment and think about audience participation in their
traditional music program can be seen in for instance the Philharmonic Jukeboxes on http://www.npo.nl/ntrpodium/30-11-2014/VPWON_1221432 during the Edison Klassiek Gala 2014. An example of audience
choices can be seen around 2m50s and the voting via a decibel measuring system around 4m00s.
The generation of as much sound as possible to vote is quite a limited method of interaction and although
this might work well in the context of a rock concert, in the context of a classical concert this noise-
12
troduction, the transformation from music which has been notated and recorded for video
games and/or interactive music applications to live performances is one of the reasons of
my early experiments in interactive live instrumental music. The need for solutions for
interactive music by means of notation, instead of leaving many aspects of the music to the
interpretation (and improvisation) by the musicians will be elaborated on.
Furthermore, the interaction described in this thesis is often based on video-games and
comprises interaction with (members from) an audience and not the interaction between
musicians of an ensemble per s15 .
This means that in this thesis (members from) an audience drives these certain events an
interactive system responds to, leading to the musicians playing music corresponding to
the (game-play-)variables.
Some of the musical elements one can think of an audience can interact with are summarised in figure 1.4. These elements are largely based on video-games and other ludic
applications.
When considering all the conditions and restrictions that illustrate my ambitions and
work for my Master of Music studies (summarised in picture 1.1 on page 4), examples of
repertoire are very limited. I only know of relating repertoire examples by Canadian composer Maxime Goulet.
One of the first interactive live instrumental music from a video game soundtrack that has
been performed was in 2010, when the music of Maxime Goulets Dungeon Hunter 2 was
performed live at the Montreal International Game Summit 201016 . A different version of
the score is being played when the conductor gives certain cues to the musicians. These
cues are based on the gameplay of Dungeon Hunter 2 being played at that moment. This
interactive live instrumental music strategy is explained in subsection 2.2 as Parallel Staffs.
Maxime Goulet also was one of the first composers to implement the horizontal resequencing technique in his 2010 composition Bungalopolis17 .
generating approach was considered annoying by certain members from the audience (including myself)
and some left before the end of the concert.
15 The interaction between musicians is another field of studies ranging from a matter of course, or regular
aspect of any live performance, to a more specific aspect of a performance, or composition.
16 A video registration of live performance of Maxime Goulets Dungeon Hunter 2 soundtrack can be found
on https://youtu.be/9bVq3jG4Fs4
17 Youtube extract of the Bungalopolis performance with an example of interaction around 3m50s
https://youtu.be/78sMKVEIRn4?t=3m51s
1.2. DEFINITIONS?
13
More recently a 2016 crowdfunding campaign18 to bring the music from the 2012 video
game Journey to a live concert seemed very promising. In this concert the music by
composer Austin Wintory is being performed live while the video-game is being played.
As can be seen in the video registrations, the concert seems to be played from paper scores
and there seems to be very limited interaction. The game is quite linear by itself, the only
variation is the amount of time a player takes to end the levels and the game. The only
non-linear aspects are the interaction with other online players, the timing and duration of
the silences and cues for the music for next stages and/or levels (these are depending on
how fast the game is being played).
When conducting his own music, Austin Wintory cleary gives some kind of
cue around the famous slide in for instance this amateur video-registration on
https://youtu.be/72zSjzUAmrg around 37m18s.
Although the wikipedia page19 describes that The ensemble will react to the players
actions, using a specially-scored version of the soundtrack,..., that breaks the music into
small pieces to enable this reaction I didnt receive any additional information, or insight
in some of the scores and parts to acknowledge any non-linearity during the concert.
A final, very recent, example is the Metromorphose concert with Metropole Orkest and
Tin Men and the Telephone (also see 1.2 on page 10) of June 2016, where music from the
jazz trio was adapted so that the orchestra could also play the interactive music.
Noteworthy is the audience composition participation in which members from the audience
design rhythms, harmonies and a melody for the trio and orchestra to play and have control
over which sections, or musicians should play nd in which style, tempo and dynamics.
An amazing realisation of a very complex challenge, leading to an almost magical musical
experience.
18 The 2016 Kickstarter campaign for Journey Live:
live
https://www.kickstarter.com/projects/40058920/journey-
19 https://en.wikipedia.org/wiki/Journey_(2012_video_game)#Music
14
Audience interaction & live instrumental music

(digital effects, mixing etc. have been
left out of this equation)
audience as controller(s)
(video, or sound analysis /
balloon / laser pointers /
smartdevices)
gaming
(video and/or sound
analysis: movement,
volume, pitch, timing)
democratic choice
(ex. blue vs. red in
Karmaflow /
Philharmonic Jukebox)
memory games
Simon Says / Loom

(remember the
sequence of notes)
GAMES *
music games
time (Duck, duck, goose

(zakdoekje leggen) /
Musical chairs (stoelendans) /
Hot Potato
puzzle games
timing (Guitar Hero /
DanceDance R. / audiosurf)
Body Percussion
"guess the instrument"

(multiple choice?)
Single person interaction & live instrumental music
play / move
along (dance)
direct interaction
dynamics
tempo / pulse
note material
instruments /
groups
pitch (SingStar / Plink)
* other games and game-genres than the

ones described can also be used by
means of mapping gameplay, or narrative
Figure 1.4: live instrumental music audience interaction based on video-games
Chapter 2
interactive live instrumental music
experiences
In this chapter I describe the projects in which I designed interactive live instrumental
music based on music strategies used in video-games. These projects are grouped in three
sections: horizontal re-sequencing, vertical re-orchestration and combinations.
These sections are then followed by a section other in which I describe projects I worked
on that have interesting interactive live instrumental music and a section other relevant
repertoire in which I describe relevant projects I was not participating in the Music design
process, but as a lecturer at the Utrecht University of the Arts was coordinating students
projects.
2.1
Horizontal re-sequencing
In horizontal re-sequencing pre-composed segments of music can be re-shuffled according to gameplay variables. The composition process in both game music and music which
will be performed live by musicians require the composer to think of methods to make
sure these segments combine in a (musically) convincing way. Generally, it is not considered desirable for the transitions between segments to be audible; the music should both
continue smoothly (as if the intention of the composer), and follow the gameplay variable
15
16
CHAPTER 2. INTERACTIVE LIVE INSTRUMENTAL MUSIC EXPERIENCES
and/or narrative.
Both the composition and/or data-structure for these musical segments is a challenge as
well as a skill composers using this approach soon discover once starting to work with it
(van Nispen tot Pannerden, 2011; Sweet, 2015).
Karmaflow in Concert 2015 - democratic sequence

Karmaflow is a single player Rock Opera
Video-game in which the protagonist, the Karmakeeper, travels through several worlds and
learns about conflicts and possible solutions in
these worlds. As Karmaflow is an interactive
rock opera video game, the narrative is sung
and the cast includes vocalists from bands such
as DragonForce, Cradle of Filth, Epica, Scarlet
Stories, Arch Enemy and more. Together with
4 other composers and music lead Ivo van Dijk
I was responsible for some of the compositions
and scores for the recording of the soundtrack
with the Dutch Metropole Orkest.
The video-game was released on January 19th
2015 and to celebrate the launch of this videoFigure 2.1: Poster for Karmaflow in Concert game an interactive rock opera concert has been
organised with the Dutch Metropole Orkest, an

international cast of rock vocalists and a rock band on January 18th and 19th in theatre
Theater aan de Parade in Den Bosch, the Netherlands1 .
Similar to the interaction in the video-game, the audience is invited to play the role of
the Karmakeeper and to decide at certain moments during the play (concert) whether
1 more
information on this concert can be found on: www.karmaflowinconcert.com
2.1. HORIZONTAL RE-SEQUENCING
17
the narrative should go in one, or the other direction. Both music and narrative adapt to the
choices the audience makes2 .
The interaction consisted of a voting mechanism (poll), where members from the audience that had the Karmaflow companion app installed on their smartphone, or tablet, could
vote for two possible outcomes of the plot. This companion app, developed by DBLR Media3 , worked both as an overall synopsis of the concert, as well as a means of interacting
at certain points during the performance. Depending on the outcome of all the votes both
the narrative and the music respond accordingly (as well as the stage-lighting, the backingand click-tracks, etc.).
My proposed solution for the musical response was with either a rehearsal mark I, or II in
scores and parts, signaled by the conductor holding either one, or two fingers in the air (and
communicating the outcome of the poll via a talkback microphone for the vocalists that
had their backs faced toward the conductor). See Appendix A on page 59 for the approved
proposal of the interactive music in the scores and parts.
Part 2 A
Music Part 1
(Optional) Part 3
T
Part 2 B
timed audience poll

T = time to inform
result to musicians
Figure 2.2: Musical structure and audience interaction of interactive pieces
One can discuss whether this limited musical variability fits the description of horizontal
re-sequencing as the music basically only varies in one musical segment, or movement.
As this live music is based on the music of the Karmaflow video-game, in which there
are more possibilities of interaction and changes in the sequential order of the musical
fragments during gameplay, I still consider the interactive parts of this concert as being
based on horizontal re-sequencing.
2 This
could be considered similar to certain performances (and adaptations) of "Der Ja-Sager / NeinSager" by Bertold Brecht and Kurt Weil (also known as "He Who Said Yes, He Who Said No") and Maxime
Goulets Bungalopolis as described in the introduction.
3 http://dblrmedia.com/
18
The added value of a real-time synopsis during the performance was appreciated by
audience members, but the feel of interaction with the music was considered very limited
at best. The democratic outcome of your vote could be considered somewhat similar to
voting for your party during political elections and in this sense one may wonder to what
extent one personally feels involved with the election process itself and with the ultimate
outsome. Does one feel personally involved during an election and how does one feel when
getting to know the results after a few hours, or day(s)? Probably not so much.
The latency between voting and actual performance of the vote would make the feel of
interaction more limited. Interaction might work better when members of the audience can
understand and feel somewhat of a faster, or potentially real-time, relation in the response.
The successful use of the companion app was dependent of the WiFi-network in the venue
and, although additional access points were installed for this concert, depending on where
members of the audience were seated their experiences were different ranging from no
problems to no connection.
There was only one rehearsal with the
Metropole Orkest and international cast
on January 17th, the day before the concert.
Before, and at, the rehearsals for
Karmaflow in Concert the conductor concluded that we would need extra time for
the amount of musical material and the musicians and conductor to gain experience
with the interactive parts.
Figure 2.3: Full cast and orchestra for Karmaflow

To gain some time we decided that some
in Concert
of the pieces (where a gauze screen would

be between the audience and the orchestra) would not be played live, but played from the
earlier recordings for the game soundtrack with the same Metropole Orkest.
Another solution was to play less interactive parts, as that would require less musical material to rehearse. Playing all the variations would require more rehearsal time, so some
of the interaction was actually staged during the concert. The audience of course didnt
know, nor experience this occasional dodgery and by pure coincidence even made their
choice for the parts we pre-selected as the outcome.
19
NLN-Live - interactive live instrumental music with iPads

Video-games using horizontal re-sequencing as
a method for the interactive music often have
some form of database structure for possible
musical sequences. A music system decides
which musical fragments should be played next,
based on gameplay variables, the current musical fragment and/or musical history, and this
database with possible successors.
One can
Figure 2.4: Ernst van Tiel conducts the imagine that transforming this video-game muGelders Orkest at the world-premiere of sic approach to a live instrumental context inSpace Invaders during the museumnight volves the need of specific technology to dis2013
tribute to the musicians and conductor the in-
formation about which musical fragment to play and/or the musical fragments themselves.
NLN-live is a dynamic score system for nonlinear and interactive instrumental music performances, developed for this purpose. The goal of NLN-live is to facilitate dynamically
controlled collaborative performances and make live interactive video game music possible. Musicians and conductor have their musical material presented on a tablet, or computer
screen. This musical material is being changed dynamically based on gameplay variables
and the musicians alternatingly play their top and bottom musical fragment, with the currently active musical fragment illustrated with a green border (see figure 2.5).
NLN-Live was developed for the 2013 Museumnacht by Jegor van Opdorp and myself and
further developed later in 2014 by Thijs Koerselman at the Utrecht University of the Arts,
HKU.
The NLN-Live software is thoroughly explained in my 2014 article as presented at the
2014 International Computer Music Conference (van Nispen tot Pannerden, 2014) and the
music that will be discussed concerns adaptations to my 2013 composition called Space
Invaders, which I composed in collaboration with colleague composer Stan Koch for the
Amsterdam Museumnacht in 2013.
The original 2013 music was based on a variation to the 4 note motif, known from the
original 1978 Space Invaders video-game. Space Invaders is considered the oldest videogame to have interactive video game music (Collins, 2008). Although limited to its 4 notes,
20
Figure 2.5: Example of a musicians interface on an iPad running the NLN-Live web application
with the music fragment with the green border as the fragment that is currently being played and
the other fragment as the next fragment to play
the music gradually accelerates when there are less aliens on the screen and the aliens move
faster.
The original 1978 Space Invaders music had a 4 note motif with a tone distance downward
of whole-whole-half, whereas in the new Space Invaders composition we used a wholehalf-half tone distance as our musical starting point (as can be seen in the cello and double bass parts, starting on a G in Appendix Chapter B on page 61). As a composer for
games and interaction I have some experience in composing in the context of horizontal
re-sequencing. One of the solutions for easy data-structuring and generating enough musical variation, while limiting the musical material, is to have musical fragments which can
easily connect (or follow in sequence).
In for instance level 3 of the Space Invaders music all musical fragments are written to
21
start in one of the three fundamental chords of Gm, Bm, Ebm. These chords all have a
chord-distance of a major third, making them third-related as the IIId, or VId degree4 .
The data-structure for the first Space Invaders concerts were realised on a node-basis
(meaning that every possible successor for any musical fragment was administrated),
whereas the new NLN-Live software worked on a group-basis (where nodes, or musical
fragments, were grouped and the possible successors for groups were administrated)5 .
The concerts described in this thesis used the new 2014 NLN-Live software with groupdata-structure.
New arrangements for the music with minor adaptations in both the musical material, orchestration and data-structure were made for the different ensembles.
A suite version of the music was created for every concert for rehearsal purposes and failsafe for when the NLN-Live technology, or tablets would not work for some reason. The
first 8 pages of the score of this suite version for the Indie Games Concert 2015 are included
in Appendix Chapter B on page 61.
In the next sub-sections I give short reports with my experiences involving NLN-Live
at the Indie Games in Concert 2015, Night of Arts & Science 2015 (Nacht van Kunst &
Wetenschap) and Media Sound Hamburg 2015 in combination with a mod of the videogame Space Invaders:
4 in
Dutch: tertsverwant
typical data-structure for the music for Space Invaders on a node-basis can be seen
http://arborjs.org/halfviz/#/NTUyOQ and on a group-basis on http://arborjs.org/halfviz/#/ODI3OQ
5A
22
Space Invaders performed live at the 2015 Indie Games Concert

The Indie Games Concert 2015 was a concert
program with music from Indie Games performed on May 16th 2015 by a medium large
ensemble consisting of professional musicians
from the Residence Orchestra (22 instrumentalists) and conductor Bas Wiegers.
Part of the program were interactive music
pieces with for instance J.S. Joust being played
Figure 2.6: Member from the audience plays live with live instrumental music (see Johan SeSpace Invaders, while Bas Wiegers conducts bastian Joust performed live at the 2015 Indie
the Residence Orchestra
Games Concert on page 29), the student-project
audience game Hat Trick ( on page 44) and the

video-game Space Invaders with interactive live instrumental music.
A mod of the video game Space Invaders was linked to variables in the NLN-Live data
and was connected via Nodejs and web-sockets.
Through a separate computer, where the score could be followed, a conductor assistant
would click next to have the next fragment presented to musicians and conductors
tablets, based on the state of the video game.
There were two rehearsals and a final rehearsal on the day of the concert for the music for
Space Invaders to be rehearsed. The rehearsal on May 15th, the day before the concert,
was also a rehearsal with the video-game being played, so that the interaction could be experienced and rehearsed. At the first rehearsal the musicians played the suite version of the
music and there were three moments that the musicians played with music presented via
NLN-Live on the tablets and with three different WiFi-experiences [in between brackets]:
1. 1st rehearsal in the Paard van Troje-venue6
[WiFi from the venue, with Cisco Router: good experience, but with a few delays in
some tablets]
6 An
impression of the first rehearsal with tablets can be seen on: https://youtu.be/cq9AZO6Ht1A
23
2. 2nd rehearsal (final rehearsal) in the Paard van Troje-venue

[HKU_WPA WiFi access point, sublime and fast experience]
3. the concert
[the HKU_WPA network was recognized by many of the smartphones etc. of the
audience and (probably) caused an overload of the access point. The result was
that some of the tablets lost the connection with the NodeJS server. This made it
impossible to perform with NLN-Live, and the suite version of music was played
with the only "interaction" on cue at the end of the game (win or lose).
The audience will probably not have noticed very much from this technical problem.]
A video of Space Invaders at the Indie Games Concert 2015 can be found on:
https://youtu.be/NWiZ27wJCMw
Space Invaders performed live at the 2015 Night of Arts & Sciences (Nacht van Kunst
& Wetenschap)
I was invited as part of the Night of Arts &
Science 2015 event to re-arrange my interactive
music for Space Invaders for a short game music program that would be played several times
during this event. The music was performed by
a small ensemble, named the Space Invaders
Ensemble, consisting of professional and nonprofessional musicians (nine instrumentalists),
most of them students from the Groningen Conservatory, and a conductor with a background as
Figure 2.7: The Space Invaders Ensemble
orchestrator and composer. A mod of the video

game Space Invaders was linked to variables in the NLN-Live data and was connected via
Nodejs and web-sockets.
24
The WiFi was provided by TalentWeb Groningen and consisted of a closed (invisible) network, so that only registered devices could use the network. The musicians, who were
generally around the age categories between 20 - 30, were explained how they themselves
could control the tablet for use of the NLN-Live application.
There were two moments that the ensemble played from tablets:
1. 2nd rehearsal and also 1st experience with the non-linear music for Space Invaders
(Music and playability proved the most difficult for the musicians, the technique
proved simplest as the musicians were very quickly used to it)
2. The final rehearsal before the continuous performance (over 20 games of Space Invaders have been played with live interactive music during the course of the event)
An example of a winning game can be found on: https://youtu.be/OnfG6oVUYvs
Other Space Invaders compositions performed live at the Media Sound Hamburg
2015
It is good to mention in advance that I did not write new interactive music for NLN-Live for
this concert and this section could be considered as part of section Other relevant repertoire
during the period of my Master of Music studies on page 44. I organised a composition competition, in collaboration with MediaSoundHamburg, to see what other composers
would do with the NLN-Live technology, or what other technologies they would use, or
develop.
The concert was realised by a professional String Quartet (the Kaiser Quartet) from Hamburg.
A mod of the video game Space Invaders was linked to variables in the NLN-Live data and
was connected via Nodejs and web-sockets.
The string quartet had never worked with the NLN-Live technology before, but was immediately enthusiastic and handled the new technology well. For these musicians, the tablets
25
were prepared and they did not need to select anything, or even touch the tablet.
One of the "problems" that arose was the outdated tablets with an outdated iOS, which did
not support the NLN-Live software. Updating the iOS solved this problem.
The WiFi was from the location itself, and one had to log in through a web form with
username and password before WiFi would work. This worked well, but is not ideal for a
concert situation.
A video of one of the performances with the Kaiser string quartet playing from iPad can be
found on:
https://youtu.be/ZiiyrsgkmtM (NLN-Live Simon Schmidt game 1)
I did write a short report on the different approaches by the composers using NLN-Live,
but since this is mainly focussing on the data-side of the compositions, I decided not to
include this information in this thesis. Suffice to say it was inspiring and confirming in
enthusiasm to see other composers work with the NLN-Live approach and technology.
Experiences from the use of NLN-Live
NLN-Live makes it possible to perform interactive live instrumental music, based on horizontal re-sequencing.
As can be seen in my 2014 article on NLN-Live (van Nispen tot Pannerden, 2014) there is
the challenge of latency and the problem of responsiveness due to the length of the musical
fragments that have to be finished before a new fragment, combined with the necessity of
reading ahead.
It is also interesting to see the use of NLN-Live by other composers, as writing music in
which the sequence can change is quite a challenge.
The current version of NLN-LIVE is actually not really user friendly enough for all musicians to set up their tablets themselves. This now means that all tablets are often prepared
at a central point7 , to then be distributed to the musicians by a number of stage-hands. This
distribution should of course be as fast, effective and, preferably, as invisible as possible
not to disturb the concert.
NLN-Live requires a reliable and fast (wireless) network to distribute the musical material
to all the musicians interfaces.
7 preparation means:
and instrument
set up the WiFi network, open the NLN-Live webpage, select the current performance
26
Musikalisches Wrfelspiele live at 2015 Media Sound Hamburg

Kaiser string quartet and antique horizontal re-sequencing
As an ode to non-linear music and as a research into horizontal re-sequencing methods in
the context of my Master of Music studies two music students of the Utrecht University
of the Arts (Ruben Bergshoeff and Nick van Kleef) and I have transcribed a score dating
from 1757 and realized a web-application that brings the non-linear music compositions of
Johann Philipp Kirnberger to life. For the first time his compositions can be played and
heard interactively with the press of a button.
Figure 2.8: Table for the composition of the Trio movement
German composer Johann Philipp Kirnberger (1721-1783) can be considered as the inventor of das Musikalisches Wrfelspiel, a musical dice game which became particularly
popular in the 18th century. In a Musikalisches Wrfelspiel one uses dice to randomly
generate music from precomposed fragments (see figure 2.8 for the table to interpret the
outcome of a dice throw). This game can be considered as one of the earliest music games.
27
In 1757 Kirnberger composed "Der allezeit fertige Menuetten- und Polonaisencomponist" (German for "The Ever-Ready Minuet and Polonaise Composer") and at the occasion
of a unique game-music program at the MediaSoundHamburg this piece was performed
live by the Hamburg Kaiser String Quartet on the 4th of July 2015.
The web-application has a simple user interface and with the press of a button one of the
three movements, or a full composition of this
musical dice game is randomly generated, using
the rules set out by Kirnberger. The application
offers the musicians to choose which part to see,
whether it is the entire score, the first, or second
violin, cello, or a piano reduction. Lastly, the
application can generate a MIDI-file, or play a
web-audio version to hear the generated piece
Figure 2.9: Music fragments for the genera-
before playing it, or to store it if one likes this tion of the Trio movement
unique randomly re-sequenced composition.
The structure:
1. the original music table (see figure 2.8) and music fragments (see figure 2.9) were
transcribed in music notation software such as Sibelius and Finale
2. the music was then exported as MusicXML and translated to musical ABC-notation
via xml2abc8
3. the ABC9 -syntax music fragments were placed in the order of the table via javascript.
These ordered versions make it possible to use fragments with other block-sizes than
one bar, for example 2, or 4 bars and could be used for other purposes such as NLNLive and ikcomponeer.nl.
In the application:
8 http://wim.vree.org/js/xml2abc-js.html
9 https://en.wikipedia.org/wiki/ABC_notation
28
1. the music fragments were selected by random chance, generating the data structure
in ABC-music data
2. this ABC-music data is distributed via nodejs socket.io to the different clients and
processed via abc2svg10 to the generated sheet music.
An online version of the web-application (but without the websocket functionality) can be
found on: http://silab.hku.nl/generative/dice-games/kirnberger/
Experiences
The Kaiser String Quartet had never before played non-linear music from iPads, but the
concept was immediately clear at the one and only rehearsal. The Kaiser quartet was already used to playing Polonaises, and familiar to the overall structure of the music. The
performance of the String Quartet was prima vista since the actual composition was generated during the introduction of the piece. The Kaiser Quartet had no problem in playing
the new composition at first sight. Of course the Kaiser Quartet is a professional string quartet with very experienced musicians. This approach of prima vista performances might not
be considered appropriate for other string quartets and ensembles.
The interactive part of the composition is limited to the generation of the composition.
This is very limited and doesnt provide for a meaningful relation of interactive music. This
performance should be considered as an experiment in prima vista music performances and
the realisation of a historic music piece. It should also be considered as an experiment in
the generation of interactive music scores and the experiences are very promising for future
use in interactive live instrumental music, for instance in the realisation of generative, and
algorithmic music, but also as an update to the NLN-Live software (which at this moment
uses pre-rendered images (van Nispen tot Pannerden, 2014)).
A video fragment of the concert can be found on: https://youtu.be/3SQYWsfL_Fo (and
with a full introduction on: https://youtu.be/MO9aikXYqjA )
10 http://moinejf.free.fr/js/
2.2. VERTICAL RE-ORCHESTRATION
2.2
29
Vertical Re-orchestration
In vertical re-orchestration the tracks, stems or layers of music are changed based on
changes in game variables (see figure 1.2).
Johan Sebastian Joust performed live at the 2015 Indie Games Concert
The Indie Games Concert 2015 was a concert program with music from Indie Games
Part performed on May 16th by a medium
large ensemble consisting of professional
musicians from the Residence Orchestra
(22 instrumentalists) and conductor Bas
Wiegers.
Part of the program were interactive music pieces with for instance Space Invaders
being played live with interactive live in- Figure 2.10: J.S. Joust being played during the
strumental music (see Space Invaders per- 2015 Indie Games Concert
formed live at the 2015 Indie Games Concert on page 22), the student-project audience game Hat Trick (see Hat Trick on page 44)
and the game J.S. Joust with live instrumental music.
There were two rehearsals and a final rehearsal on the day of the concert that the music for
J.S. Joust was rehearsed. The rehearsal on May 15th, the day before the concert, was also a
rehearsal with a small audience of play-testers, so that the actual games could be rehearsed
with interaction with members from the audience.
J.S. Joust is a no-graphics, contact sport video (or playground) game designed for motion
controllers, developed in 2012 11 .
In J.S. Joust when the music normally J.S. Bachs Brandenburg Concertos plays in
slow-motion, the controllers of the players are very sensitive to movement.
When the music speeds up, the threshold becomes less strict, giving the players a small
11 http://www.jsjoust.com/

#
&
34
30
Rall. until as slowly as possible

CHAPTER
MUSIC EXPERIENCES
2. INTERACTIVE LIVE INSTRUMENTAL
# # n b
&
b n b
Rall.
37
41
&
F Subito Allegro (q=90)
b b
Lento (q= ca. 40)
b gliss.
rit.
G
accel. Allegro (q=c. 90)
b
b b
& b b
45
gliss.
Lento (q= ca. 40)
Figure 2.11: J.S. Joust Part for Violin I with the transitions written out
b
# b b
J
&
window to dash at their opponents. If the controller is ever moved beyond the allowable
49
threshold, the player loses a love, or is out.

Players have to try to jostle their opponents controllers while protecting their own.
53
b
b
b b
&
This interactive music concept of speeding up and slowing down is of course very easy to
realise in the digital domain, where the playback-speed is directly related to the controller
57
b When
U as well, just as a tape, or record
the music slows down the pitch lowers
sensitiveness.
J
&
J
J
playing at a slower speed.
In the physical domain, where a live ensemble has to slow-down, and/or speed-up, with a
2
similar drop, or rise of pitch is less easy to realise, especially when this has to be interactive.
The tempo-change is something which can be done on cue from, for instance, the conductor, but the gradual transposition to a new pitch, while maintaining the musical contour
proves much more challenging to realise (or notate) live.
One simple reason why this is challenging is that the range of the instruments becomes
a limiting factor for the transposition. A violin for instance couldnt play lower than its
lowest G on its G-string. One way to solve this range-problem is by having other instruments take over the previous musical role. For instance the Viola and Cello could extend
the range downwards, where the violin cannot go any lower and take over the voice during
the transition.
As I did not see a (safe) solution for taking over voices during the transitions in a live
context I solved this challenge by creating a score and parts where the diminuendos, or
ritardandos, and accelerandos were completely notated including the glissandi which give
31
the feeling of a tape playing at a slower and faster speed (say pitch-shifts).
An example can be seen in figure 2.11 where there is a slowing down from bar 44 to bar 45
12 .
The people playing J.S. Joust did not know when to expect the changes in music
speed (making their controllers more, or less, sensitive to their movement) and therefore this gave these players an interactive experience to the changing music. A recording of the performance of J.S. Joust can be seen, with a short introduction (in Dutch) on:
https://youtu.be/wjAa-pJMEhA.
Parallel staffs
Another solution which might have worked at the Indie
Games Concert 2015 would be one of parallel staffs,
or parallel systems, where the musicians would have
a normal and a slow (and low) version of the score
and a short explanation of the transition between the two
staffs (see fig. 2.12).
In the case of J.S. Joust this transition would be a glissando combined with a tempo-change and a disclaimer
what to do when you would have nothing to play before,
or after the transition (possibly due to range problems).
The same fragment as in figure 2.11 would then have
been notated as in figure 2.13.
Figure 2.12: Illustration of Parallel staffs
12 rit.
actually stands for ritenuto, while actually ritard. or rall., which stand ritardando resp. rallentando,
would have been more appropriate.
32
F
42
Sol. Vln.
(normal)
&
Sol. Vln.
(slow)
&
b b
j b
b
b

b b
&

G
45
Sol. Vln.
(normal)
b
& b

b b
Sol. Vln.
(slow)
48
Sol. Vln.
(normal)
b #
b
&
#
&
b
Sol. Vln.
(slow)
Figure 2.13: J.S. Joust Part for Violin I with parallel staffs and the transitions not written out
51
Sol. Vln.
(normal)
&
b b
b
b

b
J
Johan Sebastian Joust performed live at the 2016 Night of Art and SciSol. Vln.
ence(slow) & b b
b b
J
Similar to the 2015 edition of the Night for Art and Science, the ensemble consisted again
55
b
U
b (fifteen
musicians
instrumentalists),
of them
of professional
and
non-professional
most
stuSol. Vln.
J
&
(normal)
dents from the Groningen Conservatory, and a conductor with a background as orchestrator
U
j played
Vln.
and Sol.
composer.
Similar to the 2015 Indie
game
Joust
b J.S.
j was
bGames
Concert,
the
j
&
(slow)
audience.
by members fromthe
The only difference was that the musicians and conductor
now had two systems in their sheet music, instead of one, and a textual explanation what
to do with the transitions (see figure 2.13 and First two pages of the J.S. Joust score with
parallel staffs on page 73).
On cue the
4 ensemble would transition (in the course of 2 measures) to the other staff (or
system in the case of the conductor), making the interaction between the members from the
audience playing the game and the live music less predictable in the context of this event,
33
where the game was to be played several times.

This was the approach from the parallel staffs, as mentioned in the previous section.
As the musicians, nor conductor were used to these parallel staffs, the first rehearsal took
some more time than usual for the explanation and getting familiar with the switching between these two systems.
The musicians didnt have any music notation for the transitions, just a read-me-file combined with their parts and my explanation at the first rehearsal.
It was interesting to notice that the explained transitions seemed not well remembered at
the next rehearsal and had to be explained (and rehearsed) again. It might have been a good
idea to have some transition suggestions notated for every musician, instead of leaving
certain aspects open to interpretation, or improvisation.
A video of J.S. Joust being performed live can be seen on: https://youtu.be/it_tuUCK0tg
and an impression in 360 of the same concert can be seen on https://youtu.be/TMu0QyVpN4k
34
King of Games - live interaction with children (2015 adaptations)

In King of Games, protagonist Jim is a gamer
that ends up in his game console, near his sound
card. The entire venue can be considered as
the inside of a (game)computer and Jim tries
to solve certain problems and challenges (with
help from the audience) to repair the console
and return back home. There are many elements
of gameplay with the audience during the con- Figure 2.14: King of Games, a concert with
cert, such as guess the instrument, find the in- many themes from video-games and interacstrument which has the highest, or lowest note,
tion for Philharmonic Orchestra
pieces where the audience plays along by means

of body percussion and eventually (when the battery of the console, or sound-card is depleted and the orchestra has fallen asleep) an energizing chant based on the main 4 note
motif wakes the members of the orchestra.
King of Games is a concert with video games as its theme, for children in the age of 6+.
The concept and music for the concert was realised before the start of my Master of Music
studies, but in May 2015 an adaptation to the concert was performed by the Rotterdam
Philharmonic Orchestra. I will briefly discuss the 2015 adaptations and experiences.
In the 2015 adaptation the venue of the Rotterdam
De Doelen hosted several hundreds of children that
had learned a short song called the King of Games
Lied13 , based on the 4 note motif that recurs during
the concert (see figure The original 4 note motif
based on G-Am-E ).
Figure 2.15: The original 4 note motif

When the children start singing this motif, together
based on G-Am-E
with the protagonist, Jim, an energy bar on the
screen is being filled and the children have the sensation of re-energizing the battery of the
orchestra. When the bar is filled, members of the orchestra act as if they wake-up and start
to play their parts. This way the music responds to the singing of the audience.
13 The
King of Games Lied for the 2015 adaptation can be heard on: https://youtu.be/z16m1a78aO4
live
registration
of
the
audience
35
singing
this
song
in
combination
with the King of Games composition Awakening can be listened to on

https://soundcloud.com/thanvannispen/king-of-games-lied-awakening-live-de-doelen2015
This combination of a song with orchestral parts being added could be considered as a
form of vertical re-orchestration, where instead of an audience playing (singing) along, the
orchestra starts to play along with the chanting audience.
The only question whether it could be considered as vertical re-orchestration is if there is
any (gameplay) variable which changes anything in the performance?
Actually there is none.
The piece might seem and feel very interactive to the audience, but it eventually is all following the same rules as a completely written out composition with very limited reciprocal
influence: the audience sings, the orchestra plays. The reason that the project is described
in this thesis and this section is that the music in the concert is based on video-game music
and video-game music methods and the specific piece Awakening is based on vertical
re-orchestration.
In the original 2014 concert the audience didnt learn a song in advance and another method
was used to have the audience singing a 4 note motif. In that performance the level of
motif-completion, or the level of success, could be considered as the (gameplay) variable.
Although the other music for King of Games had already been written before the period
of my Master of Music studies and was not adapted during that period it is good to mention
that most of the interactive music in King of Games was often on cue, for instance play
rehearsal number [II Wrong answer] when a wrong answer is given.
Musicians from the orchestra were asked to know these 2 bar cues by heart, so that the
music could respond immediately to the action, which the conductor would communicate
by holding up either 2 (wrong answer), or 3 (good answer) fingers.
36
2.3
Combinations
Interactive music experiment with the audience as an instrument at

2015 Media Sound Hamburg (not performed)
This composition was to be performed at the Abschlufest (goodbye party) of MediaSoundHamburg 2015 and consisted of musical elements summarising the week long event
and was also underlining the international character of the event with for example a musical
fragment consisting of singing 1 2 3 4 in ones native tongue. In this composition I was
exploring a combination of a horizontal re-sequencing and vertical re-orchestration.
On large, A0, sheets musical fragments were written, with which the audience was invited to act as "an interactive musical instrument while a mod of the video game Space
Invaders was being played.
The interns and volunteers acted as a live analog sequencer and everyone had their own
cue to react to events in Space Invaders. Their sequencer-sheets were by default muted
(figure 2.16) and in fact they would un-mute their loop at one point by flipping the sheet
(figure 2.17) while the presenter would also play and sing along, as well as conduct the
audience.
Because of a heat-wave and a very full program we decided to cancel this performance.
The rehearsal was very promising though and was one of the experiences which led to
some later ideas regarding new interactive live instrumental compositions (for instance for
the piece Hart Joust for the 2016 Night of Art & Science, explained in the next section).
2.3. COMBINATIONS
37
Figure 2.16: Human Sequencer Muted
Figure 2.17: Sequencer with 2 loops unmuted
Figure 2.18: Other (polyrhythmic) musical

fragment
Figure 2.19: An end-fragment to be shown

on a victory-cue
38
Hart Joust performed live at the 2016 Night of Arts & Science 2016
The theme of the 2016 edition of the Night of Arts
& Sciences (Nacht van Kunst & Wetenschap) was
Heart. Similar to the 2015 edition, the ensemble
consisted again of professional and non-professional
musicians (fifteen instrumentalists), most of them
students from the Groningen Conservatory, and a
conductor with a background as orchestrator and
composer. As I had already written an overture on
the Heart-theme, based on a heartbeat and was arranging Bachs Brandenburger Concerto Nr. 2 BWV
1047 for the video game J.S. Joust (see section Johan Sebastian Joust performed live at the 2016 Night
of Art and Science on page 32) I wondered how it
would work if we would use the game-play from J.S.
Joust and write new music for this game, with Par-
Figure 2.20: Poster for the Night of Art

allel Staffs in mind.
and Science 2016
I combined this idea with my ambition to experiment

with a minimal music-approach like Terry Rileys 1968 In C (de Ruiter, 1993).
One of the things the composition In C doesnt have is a relation between musical material and some form of narrative, or game-play variable. If for instance the first twenty
or so musical fragments would have been categorised as group 1 and the next as group
2, with some narrative meaning attached to these different groups, Terry Rileys approach
would probably have been very suitable for interactive live instrumental music. Some sort
of variation to both horizontal re-sequencing and vertical re-orchestration.
In my piece Hart Joust - nonlinear music the musical fragments are grouped in a top and
bottom half of every musicians part, divided by a transition up, or down (see figure 2.21).
When musicians choose any of the top-group musical fragments to play, the combination
will make up for danceable music, based on a heartbeat. When, after a transition downwards to F# and retarded to half speed, the bottom part-group called Eery, half time is
played, this results in eery, tense music, making the gameplay with higher sensibility of the
& .. .
. . .
F espress.
. .
ad lib
..
. .
2.3. COMBINATIONS
39
& .. .
#.
Move-controllers
very
intense.
F
#.
Transition
b
# b b
Transition
& #
rit.
accel.
Freely Lydic scale up to next note
Freely Octatonic scale down to F#
Eery, half-time qd = 40
Figure 2.21: Transition up from F# with speeding up and Lydian scale, transition down with slowing
Growl (ad lib)
down and octatonic scale
& ..
The music was written in the key of C in a lydian scale for the upper part (movement
joyfull, dansant) and in F# in an octatonic scale for the bottom part (movement eery.
& ..
alternate fingering
j
b
fl
.. # . b . # . b . # . b . # .b
j
j . ..
. b of the melodies to a Dorian scale). F
next 4 bars an Am chord (changing the context
fl
p
In the dansant movement the double bass and bass clarinet have musical fragments of 8
bars, with the first 4 bars giving the sensation of a C chord (with lydian melodies) and the
This harmonic and rhythmic grid, realised by double bass and bass clarinet is meant to give
Fluttertongue ad lib
.. .. # . . . b . . b . . . ..
F
A challenge
I. END occurs for the conductor as he can have no more clue of what will be played
II. END
Only when previously playing
and when. To have some insight in the musical material that the musicians can choose
2
.
.
.
from, and
with
& to be able to use the score in the live context, I made a reduction to 3 pages,
the transition fragments op page 2 (approximately in the middle of the score).
f
..
the music more stability and clarity, in relation to the other musical fragments that are of
different length (from one to 8 bars).
An impression of the sheet music for the musicians can be found with an example of the
part for violin I in Appendix Part for first Violin for Hart Joust on page 76, the score for the
conductor in Appendix Score for Hart Joust on page 78 and an impression of a relatively
long (over 2 minutes) game with the interactive live instrumental music can be found on
https://youtu.be/BXhXDHcKMYs and with a 360 view angle on https://youtu.be/S01mlvsU88.
40
2.4
Other
In the category other I describe some of my live interactive instrumental music projects
that are not directly related to the previously mentioned methods used in video game music,
but have interesting live instrumental music interaction.
RCO - live music interaction with children

During the period of the Master of Music studies three projects with the Royal Concertgebouw Orchestra (RCO) have been realised in a collaborative participation with HKU and
Ciska Creative Design and a fourth project project was under development.
Together with Ciska Vriezenga and Michael Klier we developed a formula for the family
concerts consisting of:
a preparation and anticipation phase (in which the children were invited to do something in advance to the concert)
a short introduction to the concert (where the young visitors and their (grand)parents
could for instance meet some of the musicians, and learn something more about the
music, the musicians and/or the concert)
the concert
an after-experience (which could consist of photos of the concert, but also applications to continue with the theme of the concert, such as a web application called
Muzikale Penselen (Musical brushes) in which you could draw with brushes that
change on the music of Brahms Symphony nr. 3).
Most of my activities consisted in the conceptualisation of the concert, and the realisation
of (web)applications with action on music (like painting and dancing), or were interactive
music applications with recorded instrumental material and will not be discussed in this
thesis.
One of the three projects also included playing along with the orchestra with percussive
2.4. OTHER
41
instruments the children created themselves before the concert and will briefly be discussed.
This was part of the concert Pepernoten voor groot orkest on November 22, 2015.
In the introduction to the concert Dansen met Bach (Dance with Bach), March 13, 2016
an interactive element which involved movement was present that I will also discuss.
Pepernoten voor groot orkest

Pepernoten are Dutch cookie-like brown confectioneries traditionally associated with the
early December Sinterklaas festivities. The title of the concert is both a link to the Sinterklaas event, which was part of the theme of the narrative of the concert and a word-play of
the word noten which can also be translated as music notes.
Children were invited in the preparation phase to build their own percussive instrument
(see figure 2.22).
A short instruction video to get to know the
rhythm was part of the preparation phase that
young visitors could participate in before going
to the concert. It consisted of a short fragment
of the percussion theme from Britten Young
Persons Guide to the Orchestra.
As mentioned before, every concert also had
an introduction to the concert. In this specific
Figure 2.22: Young musicians with percus- introduction children were divided into three
sive instruments that they built themselves groups and one group was learning about perbefore the Pepernoten voor groot orkest- cussion and learned the rhythm that was also
concert
to be played in the concert, while the two
other groups were learning about woodwinds,

or brass and were singing along with popular Dutch Sinterklaas tunes. It was inspiring
to see that many children brought their own percussive instruments, based on the Do-ItYourself (DIY) schemes on the RCO-website and that they were very enthusiastic to play
on these instruments.
This category of playing along, making you a participant (or co-creator) of the music is of
42
course a very ancient method of interacting with the music. It is in the context of orchestral
music that this is quite un-common.
When we consider the audience as (a group
of) musicians, playing along on cue we might
also be tempted to consider this approach as
similar to vertical re-orchestration. The only
reason I havent placed it in that chapter is
the fact that there is not a clear interaction to
the variation, other than that people are playFigure 2.23: Movie still from the short ing along. The mixing of stems is not related
video-fragment to learn a rhythm on a DIY- to any gameplay variable, when we would apinstrument, in preparation to the concert
proach it from a video-game perspective. In par-
allel to video games that use this playing along

as an aspect of game play (such as Guitar Hero, Rocksmith etc) the audience needs some
kind of guide to know what to play along and when. A music video-game approach, such as
in Guitar Hero, of seeing your notes coming to you was used in the short video-fragment
to learn the rhythm in advance14 . Instead of music notes you see pepernoten fly on a
music staff (as can be seen in figure 2.23). Of course this method could also have been
used during the concert. Instead of using video-material a percussion player was giving the
cues when and what to play.
Dansen met Bach (Dance with Bach)

In one of the concert-introductions an interactive element was present which I would like
to discuss, the introduction to Dansen met Bach (Dance with Bach).
In the briefing to the musicians (violin players) and dancers I tried to explain the concept
of from music to movement and from movement to music. From music to movement
was easy to grasp for both dancers and musicians, as were all quite used to moving on
music in for instance the context of dancing. The concept from movement to music was
somewhat more complex to explain, even with examples of a conductor directing the music with movements and examples of the musician moving the bow to insert energy to the
14 Link
to the Pepernoten video fragment: https://www.youtube.com/watch?v=oXxrj4tK9Eg
2.4. OTHER
43
strings of the violin, thus creating the sound and music.

After inviting the musicians to play specific parts from Bachs violin Concerto (BWV 1041)
and to play them again, but for instance with another tempo (slower and faster), or intensity
(softer and louder), the dancer was also invited to dance differently to the variation of the
music.
When this worked well I asked the dancer and musician to change their roles. Where the
music and the variations were leading the dancer at first, the dancer was now directing the
musician to respond to the variations in the dance. The concept was that this approach of
movement to music could also be communicated to some of the young audience members
visiting this introduction, inviting some of them to conduct the music with their movements.
44
Other relevant repertoire during the period of my Master of Music

studies
In this Section I briefly describe other relevant new repertoire of interactive live instrumental music in which I did not take part in the musical design process, but was participating
in a coordinating role as lecturer at the Utrecht University of the Arts, HKU.
Hat Trick
Hat-Trick is an audience game with interactive live
instrumental music.
The game and composition
were created by 11 students from the HKU and UU

under supervision of Richard van Tol and myself.
Hat-Trick was first performed at the Indie Games
Concert 2015 in the Hague Paard van Troje venue
by the Residentie Orkest (Residence Orchestra). The
game was the interactive finale of the concert.
Hat-Trick is about two rival magicians, "Hans
Croissant" and "David Pepperfield". The audience
must help Hans Croissant by letting his magical hat
crowd surf. The audience can see themselves from
Figure 2.24: Poster of Hat Trick
above projected on a large screen. Through augmented reality there are objects on the screen that
must be taken up with the hat, or have to be dodged.
The hat in a way acts as a mouse-cursor for the game to be played. The audience must play
three different levels to win the game. The first level is to collect objects, the second level
is to dodge / avoid walking peppers and the third level is about picking up cherry bombs as
ammunition to load and fire a cannon to defeat Pepperfield.
The music was performed by the Residence Orchestra with conductor Bas Wiegers using
a PlayStation Move controller as his baton. Controller signals were transmitted from and to
the game system so that the performance of the music could be been synchronized with the
2.4. OTHER
45
UTP
Flute Oboe Clarinet
Bassoon Horn
Trombone
AUDIO
VIDEO
Trumpet
CAMERA
Percussion 2
Harp
Violin II
Viola
Cello
Violin I
Laptop
CB
Monitor
Percussion 1
BLUETOOTH
Conductor
PS Move
Controller
SFX
10 m
27 m
beamer
Hat Trick
Game
computer
camera tracking
crowdsurfing
hat on audience
Figure 2.25: Hat Trick technological infrastructure
game and vice versa. The conductor, for example, gives the cue to the game for throwing
a bomb when he also directs the orchestra to play a specific musical cue. In addition, the
conductor receives specific cues via the light on the Move controller, for instance when the
game should move to the next level. The conductor can than direct the orchestra according
to these cues and, again, let the game system know that he is performing his cue at that
moment. This way the music is synchronized with whats happening on screen and vice
versa.
15 m
Read more on: http://unst.hku.nl/943?prev=1 (in Dutch)

Look at the Hat Trick performance during the 2015 Indie Games Concert on:
https://youtu.be/DZSpdKM8RMs
46
Project-week Future Sound of Live with Tin Men and the Telephone
The project-week, Future Sound of Live was a project of HKU Jazz and Pop Music together with Music & Technology, the Expertise Centre for Creative Technologies (ECT)
and the Jazz-trio Tin Men and the Telephone. Students realised a live interactive instrumental music concert at Tivoli Vredenburg Cloud Nine within the scope of the 5 day project
week from April 4 - 8, 2016.
Together with Tony Roe, Pat Cleaver, Bobby Petrov and Marcel Wierckx (Tin Men and
the Telephone) I was coordinating and coaching the student-groups. The end results of the
student group presentations can be found on: https://www.youtube.com/playlist?list=PLuSty7ZhL3XP29FhKDmQjFzEaqVPI40e
I only describe briefly and discuss the interactive musical components of every performance.
In Music Dolls, the ensemble consisted of 1 bass player, 5 singers and a music technology student. The singers could be controlled via an iPad with 3 faders. The faders control
the lights on the stage and the intensity of the light represents the overall volume of the
singer(s) vocals. When the lights start flashing (after a specified time-interval) a new musical sequence is started. People from the audience were invited to control the Music Dolls
with the iPad. The singers were responding very fast to changes in their lights, making this
interactive music piece very responsive. In a way this approach can be considered as a realtime live mixing approach, similar to the vertical re-orchestration method in video-game
music.
In Nr. 14, the ensemble consisted of 2 drummers and a music technology student. The
title Nr. 14 refers to Johan Cruijff, an historic Dutch soccer-player and -coach that was
also known for some remarkable sayings. In Nr. 14 the drummers first play with the
rhythm of one of Johan Cruijffs quotes, to gradually develop a more elaborate variation
on the rhythms and in the playback of the quote. After this introduction the performance is
presented more and more as a sports match, including an arbiter-whistle and members from
the audience are being invited (after the half-time) to vote via a yellow and/or red card
which of the drummers should play, or take a penalty. One of the drummers represented
red, while the other drummer represented yellow.
2.4. OTHER
47
In Motives the ensemble consisted of a saxophone, pianist, drummers and a music

technology student. Members from the audience were invited, after the introduction of the
motives, to use any of three large buttons on the stage to alter the music. The use of the buttons would randomly change the sequence to play (motives), the style to play, the dynamics,
which instrument(s) would play and the speed. The result led to a very interesting musical performance, where the musicians only had to know 5 different motives (sequences)
to play and of course had to practice a lot on the large amount of possible variations in
styles, dynamics and speed. For the audience the interaction and resulting variations were
very clear and entertaining (as one can hear in the response from the audience during the
concert). The interaction was of course limited to people that would walk up to the stage
and use one, or more of the three buttons, but as there was a constant queue for the buttons
this did not prove to be a problem.
In Light Fragments the ensemble consisted of piano, guitar, saxophone, drums, vocalist
and a music technology student that also realised an electronic music layer and effects. The
audience was invited to select a new musical sequence by making a choice using the light
on their mobile phones. The circle with the highest percentage of light would become the
new sequence that would be played.
In Tinderen voor Kinderen the ensemble consisted of a saxophone, flugelhorn, synthesizer, bass, guitar, drums, vocalist and a music technology student. The audience was
invited, after a musical introduction, to take part in a musical story in which a princess was
feeling lonely and decided to start dating via (a dating app) called Tinder. Members of the
audience were able to chat with the princess, to help her with her profile and to choose her
dates. The role of princess was performed by the vocalist that used elements from the chat
in her narrative and vocals.
Chapter 3
Discussion and Conclusions
3.1
Discussion
General
In general I think it is safe to consider that most of the experiments during this Master
of Music studies with interactive live instrumental music performances realised with technologies and methods based on interactive music strategies in video games were quite successful.
More generalised I would like to carefully conclude that interactive live instrumental music
is a field in which there is a lot more that can be explored, based on the findings of these
experiments, concerts and feedback from musicians and audiences. The goal to explore
whether it is possible to transform the most used, interactive (video game) music strategies
for use in interactive live instrumental music can at least be considered achieved.
One other pre-mentioned goal (in the section Why interactive, live and instrumental? on
page 1) of this Master of Music studies, to explore if these music strategies could possibly
expand the field of live instrumental music, can also be considered achieved, although
this expansion might be considered quite humble and only lightly explored. It requires
more experience with different musicians, ensembles and contexts to be able to draw more
conclusions in that regard.
48
3.1. DISCUSSION
49
Artisticity and games

One of the overlapping features one might notice when looking at the described interactive live instrumental music experiences is that almost all examples are somehow related
to gaming and ludic interaction. This poses a challenge to maintaining a balance of autonomic artisticity and designing effective music in the particular context of games and ludic
interaction.
One might discuss the artistic relation in most of the projects mentioned, like a form of
counterpoint, between the music and the interaction. The music can be considered quite
submissive to the narrative and/or interaction in many of the projects. With my background
as a composer for media, games and interaction, I didnt find this a problem for the projects
described, as the audiences in general were pleased with the surprising and immersive elements of interaction and I personally liked the fact that in most cases the audience was
smiling, laughing and, seemingly, having a good time.
When I transform the music by Johann Sebastian Bach to an extremely slow (and low)
version, as if a record is slowing down and accelerating, this is of course based on the original gameplay and interactive music of J.S. Joust (see for instance Johan Sebastian Joust
performed live at the 2015 Indie Games Concert on page 29). The artisticity lies in the
transformation of this digital medium to a live performed acoustic medium, again somewhat ludic.
When one would listen to the recordings of concerts1 of J.S. Joust it is hard to imagine a
listener would be enjoying this music (especially when we appreciated a definition of music as described in the introduction: ...instrumental sounds (or both) combined in such a
way as to produce beauty of form, harmony, and expression of emotion.).
Same goes for many of the described live interactive instrumental music performances. It is
often less interesting to listen to recordings of these interactive performances, without understanding or experiencing the context and one of the reasons why most of the interactive
music pieces are not included in online music albums (such as the Indie Games Concert
2015 album2 ). The new beauty of form,... and expression lies more in the context of the
interaction and play, than in the music itself.
When we compare this to other (non-interactive) concerts of game music, we see that the
1 Registration
of J.S. Joust with live performed music on: https://youtu.be/wjAa-pJMEhA

free Indie Games Concert 2015 album can be listened on:
https://dutchgamemusic.bandcamp.com/album/indie-games-concert-2015-live
2 The
50
CHAPTER 3. DISCUSSION AND CONCLUSIONS
linear renditions of these game soundtracks are quite popular (Video Games Live, The
Greatest Video Game Music, Games in Concert, etc).
On distracting
I would conclude that interactive live instrumental music can work really well in the context of live music performances and in that sense might add a value to the experiences of
going to concerts, other than listening to music one already might know. The added value
of being there and participating is also felt and understood by people that werent there.
What I consider a new problem to take into account, is the fact that with some of the interactive live instrumental music pieces, the audience was much less aware of the music
being played, as they were playing, or experiencing a game and/or concentrated in their
playing along.
The balance between listening and participating is something to take into account in the
overall program of a concert.
This is something the Tin Men and the Telephone would also concur, as in their programs
the beginning of the concert is also mostly music the audience just has to listen to and experience. The balance between listening and interacting with the musicians and the music
seems to be well managed in their concerts. From my conversations with band-members
from Tin Men and the Telephone I learned that this is something which has been balanced
over the course of many performances and based on their years-long experience.
Oliver Hdl describes a similar challenge in his article on Experimence when he mentions that their interactive effect became too intense for the whole song and became even
distracting and annoying somehow... (Oliver Hdl, 2014).
On play-testing with audiences
One question in parallel to the previous mentioned problem of balancing is how do you
play-test a concert with audience participation?. This question was part of the challenge
the students were facing that worked on the audience game Hat Trick (see Hat Trick on
page 44), but also the case for the J.S. Joust and King of Games performances. The actual
audience might behave and react different from the play-test group. In the case of Hat
Trick and J.S. Joust the play-testers were mostly students, while the concert was visited
by a mixed audience of game-affiliates and regular visitors of the Residence Orchestra
concerts.
3.1. DISCUSSION
51
Another remark on concerts with elements of game, play and interaction is that certain
(traditional) conventions are broken. The audience was apparently so much engaged in the
game-aspect of the performance that they would always, in all occasions cheer and applaud
when a game was finished and especially when a victory was achieved, although the music
hadnt finished yet3 .
This is something which is rare for most other concerts and for the musicians (but in a
way also for the audience) this is a somewhat awkward situation. For the Night of Art and
Science 2016 concert we used this assumption as a cue and the musicians were prepared
for this possibility.
On audiences
In regard to the different audiences that took part in the concerts that are described in this
thesis, I must say that in general we were very lucky with our audiences and the enthusiasm
and willingness of members of the audience to participate. An audience that for instance
Dutch composer Merlijn Twaalfhoven was facing in a more formal context during the 2013
European Culture Forum seemed much less willing to actively engage in his piece with
audience participation4 .
Attracting audiences to the different concerts also went well, especially when one takes
into consideration that interactive live instrumental music is quite a new phenomenon and
people might not be attracted to something they dont know, or understand yet. The fact
that most contexts had another, more familiar, component might have helped in getting the
audiences to visit the concerts (for instance game-music in general).
From the experience with the J.S. Joust games being played at different occasions I can
highly recommend to have a number of pre-informed audience members standby to participate. We noticed during the Night of Arts and Science 2016 that finding audience members
that want to play a game they dont know in front of an audience was less easy than expected, especially during the second show where there was a smaller audience than the first
show. Some audience members mentioned that they thought it was to warm to partake in a
game which involves moving, something we hadnt anticipated in advance.
Having some actors standby to, seemingly, voluntarily join in the game can show the
game mechanics and rules faster and heighten the overall enthusiasm for next rounds. It
3 This
effect can for instance be seen during the J.S. Joust performances on https://youtu.be/it_tuUCK0tg
around 3m24s, and on https://youtu.be/wjAa-pJMEhA around 3m30s and around the end
4 as can be seen on: https://youtu.be/9jfZyJAd1Vs?t=12m3s
52
was actually a risk to not have any actors standby during the Night of Arts and Science
concerts.
On improvisation
One of the aspects of live interactive instrumental music is whether the musicians are
skilled in improvisation, or they are more used to interpreting sheet music. In this thesis the
focus lays strongly on interactive live instrumental music with (digital) sheet music. This
is partly because of the background of the musicians I worked with. The experiences with
Tin Men and the Telephone, for instance during the Future Sound of Live project-week,
but also their unique concert Metromorphose with Metropole Orkest, were very interesting
and promising. In both examples often a combination of notation and improvisation was
used, a very interesting and promising area to gain further experience with.
The two different music pieces for the J.S. Joust game, played during the Night of Art and
Science 2016, both made use of a transition that was relatively free to interpret. This can
be considered as leaving aspects of the music to the interpretation (and improvisation) by
the musicians which I explicitly mentioned I would not do, in the introduction. Although
the musicians had a few questions about the transitions and they had to get used to this new
fact, the method seemed promising. A possible solution for more clarity would be to give
certain suggestions for the transitions on a separate page, and/or write out the transitions.
These written out suggestions are much appreciated, especially by musicians that are used
to interpret notated music.
During the rehearsals for the different J.S. Joust pieces for the Night of Arts and Science
2016 one of the musicians suggested to first rehearse the Hart Joust piece, since (a similar
kind of) transition was written out, so that the ensemble would get the feel for the J.S. Joust
piece (without a written transition).
On rehearsals
Although the techniques and strategies discussed make live interactive instrumental music possible, the need for extra rehearsal time makes it more time consuming and therefore
often more expensive. As one can imagine, the higher level of complexity requires more
time: introducing new methods, new technologies require explanation and time for the musicians to understand and practice with these new experiences.
3.1. DISCUSSION
53
Another time-consuming aspect is the fact that there is often more musical material to rehearse. In for instance the parallel staffs-method the amount of music is twice the amount
of music one would hear during a concert, with only ne variation.
In Karmaflow, the variations have to be rehearsed as well, but might not even be heard
during the concert. In video games this is sometimes also the case: there is more musical
material in the game than one will eventually hear, depending on the gameplay.
On playing along
Making the audience part of the music by playing, or singing along with (body) percussion, other instruments and vocals was considered a very strong and effective way of feeling
part of the music. Seeing and hearing the audience participate in the King of Games concert, but also during the Pepernoten voor Groot Orkest these elements of (body) percussion and vocals felt as the epiphany of my interactive live instrumental music experiences.
Although simple in concept and technology, it was very effective!
Vertical re-orchestration
A limitation in most video-game music using vertical re-orchestration projects is the fixed
tempo and key of the music (Sweet, 2015). The making of documentary of the Red Dead
Redemption video game mentions (on https://youtu.be/IJAIFmOdB00?t=3m30s around
3m30s) that the interactive music was limited to 130 bpm in A minor. When this method
is transformed into a live music equivalent, which I called Parallel staffs, this limitation is
less an issue, since the musicians can change their tempo and depending on the use of a
transition the key could also change. This was the case in the J.S. Joust experiences where
the slowing down and accelerating were part of the gameplay and the transition between
the variations made a shift in both key and tempo possible. A next question could be how
could these experiences from interactive live instrumental music inspire for possible solutions back in the video-games-domain?
A comparable transition method is known in video game music design as the stinger
(Sweet, 2015).
54
Horizontal re-sequencing
When using horizontal re-sequencing as a method sheet music will often not suffice, except
for an approach such as Terry Rileys in C and my 2016 Night of Art & Science Hart
Joust and will require additional technology. The use of NLN-Live and/or DFScore make
it possible to use this interactive music method. A problem of latency is something to take
into account when using this method, since the time required to finish the currently playing
musical fragment and the musical fragment that was presented for reading ahead introduce
delay (van Nispen tot Pannerden, 2014).
Conclusions of the use of NLN-Live in these concerts:
Although NLN-Live facilitates the realisation of interactive live instrumental music, the
current version of NLN-Live is actually not user-friendly enough for musicians to set up
their tablets themselves and the software needs further development and improvements.
Kirnberger experiences
The approach used for the web application for Kirnbergers Musical dice game seemed
very promising with regard to musical notation in a web environment. The amount of
(ABC music notation) data transmitted via the network is much smaller compared to the
.jpg-, and .png-image files in NLN-Live.
In the case of the Musical dice game the entire piece was generated with one button and
there was no narrative, or game-play variation associated with the different musical content. The music just had variations. Of course the connection between musical fragments
and a (game) variable would be a logical next step in this approach.
This technology could also work well with generative music and real time notation (Jason Freeman, 2014), an area which to my knowledge has been little explored in the context
of video-games and interactive live instrumental music.
Combinations
The combinational approach of horizontal re-sequencing and vertical re-orchestration is
quite common in interactive music for video-games (Sweet, 2015), but poses certain challenges for the realisation of live music performances.
3.2. CONCLUSIONS
55
One of these challenges is how to distribute the sequential information to the musicians
and how the sheet music could be adapted so that they can easily find the right musical
fragment to play. If all the musical fragments would for instance fit on 2, or 3 pages and all
fragments would be orderly labeled with a number, the conductor, or a screen5 could signal
which musical fragment should be played next.
Similarly the conductor, or a screen could signal when to play (for the vertical reorchestration part of the combination)6 .
In my composition Hart Joust, described on page 38 I experimented with this combinational approach and left most of the interpretation of the sequential order to the musicians.
Only the cues for the start, the transitions and the end are decided by the game and the
conductor. As the musicians can choose which fragment to play, and may decide when and
if they play, the music varies both in the (horizontal) sequence and (vertical) orchestration
and results in an infinite number of musical outcomes.
One limitation of this approach is that the music in the case of Hart Joust is limited to two
musical modes and very limited chord progressions. Introducing a system, like NLN-Live,
would make it possible to use the combinational approach in more elaborate ways.
3.2
Conclusions
Some of the foreseen challenges have successfully been solved and the original goals of
the Master of Music studies achieved.
The transformation from digital video game music technologies to physical representations
broaden the possibility-space of live instrumental music with more interaction.
Some of the noticeable advantages, disadvantages and challenges of the video game music
methods in the context of interactive live instrumental music are summarised in table 3.1.
Some of the new challenges that seem to arise are:
5 or
a paper sheet, as is described in Interactive music experiment with the audience as an instrument at
2015 Media Sound Hamburg (not performed) on page 36
6 an example can be seen on https://youtu.be/3OQl3Bed4GE where the Metropole Orkest with Tin Men
and the Telephone rehearse the interactive music cues for the Metromorphose concert. Chief conductor Jules
Buckley developed a few new signs to communicate his cues to the ensemble.
56
the challenge of artistry in combination of live interaction (can I think of other uses
than game and how do I meaningfully integrate live interaction with interactive
music practice in other than ludic concerts?)
the (limited) added value of interaction to the overall musical experience (users tend
to get distracted by interacting and although they might have an overall greater experience, they might have missed some of the music and the musical experience.
There might be a parallel with music for media, such as films and games, where the
music is often not consciously noticed, but experienced nonetheless (Prendergast,
1977).)
In general it is quite a different experience to play with music, or to just listen to it.
audience participation (how to experiment and play-test when writing interactive music that would respond to audience-interaction and/or -participation? How to design
audience participation that is easy and intuitively to understand and doesnt take
much time to explain during a concert? How to predict the different audience behaviours and how to prepare and respond adequately?)
rehearsal time (although the techniques and strategies discussed make live interactive instrumental music possible, the need for extra rehearsal time makes it more
time consuming and therefore often more expensive. This is something to take into
account, for there will most probably not be an easy solution to this problem. One
suggestion to look into would be the approach as in my 2016 Night of Art & Science
Hart Joust, with little musical material and a lot of possible variation in the musical
outcome.)
specific event technology is required to realise some of the new techniques and strategies. For instance NLN-Live requires a fast (wireless) network to distribute the musical material to all the musicians interfaces (tablets), while the Karmaflow app would
also require a wireless network for the audience to be able to make their votes.
The use and dependency of these kind of technologies also introduce new challenges
and inventiveness for how to test the technology for use with audiences, how to prevent the technology from failing and what to do when the technology fails (as mentioned in the many NLN-Live experiences).
In the case of Space Invaders for example a suite version of the music was avail-
3.2. CONCLUSIONS
57
able, as a backup in case the technology would let us down, but also for the rehearsal
of the musical material without the need for tablets.
In the case of Karmaflow in Concert the only fallback needed was that the conductor
would always decide what the outcome of the interaction would be, but for members
from the audience that would not be able to vote anymore the interactive component
would be completely gone.
In other contexts, such as Tin Men and the Telephone concerts I visited, there are
always a small number of mobile devices that are not supported by the TinMendo
software. This is something to take into account when composing for, and with,
interaction.
I am aware of the fact that the interactive live instrumental music projects I worked on are
(at this moment) quite a rarity in the concertante music context and I do not dare to make
any predictions about the future viability, nor popularity of these approaches in instrumental
music. Although both the audiences, ensembles and I were very enthusiastic about the
different experiences, I have no idea whether this musically very interesting period for me
was incidental, or possibly a new trend in live instrumental music. It will also depend on
this popularity whether I will be able to continue these, personally, very interesting and
challenging projects.
For now Im just very pleased that I was able to realise a humble part in these extraordinary
concerts with interactive live instrumental music, was able to share some of the experiences
in this thesis and that I have the fortunate perspective of some fantastic composition projects
with some of the best musicians and musical ensembles I could wish for!
58
Vertical
Re-orchestration
Horizontal
Re-sequencing
Combination
Example
J.S. Joust (as performed

at the Night of Arts and
Science 2016)
Hart Joust (as performed

at the Night of Arts and
Science 2015)
Advantage
Fast response possible.

Easy to predict what the
result will sound like
Similar to game music
using this approach, the
harmonic grid can be
quite limiting. The
rhythmic grid can be
less limiting than in
games, though.
Space Invaders (as

performed at the Night
of Arts and Science
2015)
Much variation possible
with relatively little
material.
Latency for reading
ahead and finishing the
current musical
fragment.
Most often would
require the use of some
special technology like
NLN-Live, or the
number of musical
segments will have to be
very limited.
The composition
process can be
challenging. In most
cases one cant predict,
nor pre-listen all
possible sequences.
Disadvantage
Challenges
As one doesnt know

when a transition can
occur, placement of
tempo marks, dynamic
symbols and techniques
(like pizzicato, or arco)
can be overlooked.
Much variation nd fast

response possible.
Limitation of the
musical mode and
harmonies, or rapidly
becomes complex for
musicians to read, or
requires use of some
special technology.
A promising
combination in which
the re-sequencing of
musical fragments could
be vertically
re-orchestrated based on
gameplay variables.
This would require new
technology, or an
adaptation to the
NLN-Live software.
Table 3.1: Summarizing some advantages and disadvantages for the discussed methods
Appendix A
Karmaflow Interactive Music Approach
59
60
APPENDIX A. KARMAFLOW INTERACTIVE MUSIC APPROACH
Score
Proposal interactive music

Than van Nispen
Intro
4
&4
Variation A
&
!
!
!
!
!
!
!
To Coda
16
20
&
II
Variation B
&
&
&
fi
&
End
&
To Coda
Appendix B
First eight pages of the Space Invaders
score for the Indie Games Concert 2015
61
Space Invaders
Score
Indie Games Concert 2015
Stan Koch / Than van Nispen tot Pannerden
Allegro {m q = c 120}
accel. poco a poco
Flute
4
&4
Oboe
& 44
Clarinet in B b
& 44
The music piece is performed by means of tablets such as Samsung

Galaxy, iPad etc. with the use of the nln-live web application.
The nln-live web application can be previewed on www.nln-live.nl
Bassoon
? 44
This paper / pdf score and parts are solely intended for preparation and
rehearsal purposes of the musical material.
The live performance will be interactive.
Horn in F
& 44
Trumpet in C
& 44
Trombone
? 44
Timpani
? 44
Cymbals
4
4
4
&4
? 44
& 44
? 44
Violin I
4
&4
Violin II
& 44
Viola
B 44
Cello
?4
4
Double Bass
? 44
Harp
Piano
Introduction to this non-linear & interactive music piece

by Stan Koch & Than van Nispen tot Pannerden
The piece starts at a tempo of ca. 120 bpm and gradually accelerates,
unless specifically mentioned otherwise. This acceleration is similar to the
pace of the 1978 videogame "Space Invaders".
Notated tempo changes can be overruled by the conductor.
The musicians will be presented small musical fragments of approx. 4
measures in length on their tablet. These musical fragments will
depend on the variables of the game Space Invaders.
Rehearsal marks represent these musical fragments.
There are two possible outcomes: I "Defeat" or II "Victory" which will
be performed on cue by the conductor and based on the outcome of the
videogame.
Due to the nature of the non-linear order each musical fragment has
it's dynamic-, div., etc.-markings. These can be overruled by the 'overall'
feel of the performance, as well as the interpretation by the conductor.
The videogame Space Invaders will be played during the performance.
Intermezzo 0
Fl.
&
Ob.
&
Cl. in B b
&
Hn.
&
Tp. in C
&
&
&
Vln. I
&
Vln. II
&
Vla.
Bsn.
Tb.
Timp.
Cym.
Hp.
Pno.
Vc.
D.B.
P
?
P
b -
b -
b -
b -
Intermezzo 05
Fl.
&
Ob.
&
Cl. in B b
&
Bsn.
Hn.
Tp. in C
Tb.
?
&
w
p
w
P
&
?
?
w
P
# # -
w
w
# -
# -
&
&
Vln. I
&
Vln. II
&
Vla.
Timp.
Cym.
Hp.
Pno.
Vc.
D.B.
w
"
P
?
P
b -
b -
b -
b -
Intermezzo 08
4
Fl.
&
Ob.
&
Cl. in B b
&
Bsn.
Hn.
Tp. in C
Tb.
Timp.
?
&
w
p
w
P
&
w
P
?
w
"
bw
bw
bw
&
&
Vln. I
&
Vln. II
&
Vla.
Cym.
Hp.
Pno.
Vc.
D.B.
P
?
P
b -
b -
b -
b -
Intermezzo 09
Fl.
&
Ob.
&
Cl. in B b
&
Hn.
&
Tp. in C
&
13
Bsn.
Tb.
Timp.
Cym.
Hp.
Pno.
&
?
pOoPOOOO
# ww
# ww
# ww
&
&
Vln. II
&
Vla.
D.B.
Vln. I
Vc.
# ww
# # Ow
# Ow
p
Ow
Ow
p
b Ow
p
p
?
w
p
# Ow
# Ow
Ow
b Ow
#
P
Intermezzo 10
6
Fl.
&
Ob.
&
Cl. in B b
&
Hn.
&
Tp. in C
&
&
&
17
Bsn.
Tb.
Timp.
Cym.
Hp.
Pno.
Vln. I
Vln. II
Vla.
Vc.
D.B.
&
# Ow
#
&
F
#
B
F
? #
F
?
w
OOoPoOOo
# Ow
w
P
w
# ww
bw
ggg b www
ggg
gg P
gg w
gg ww
ggg
g
.
.
p
p
..
i ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
w
21
&
F
p
Level 2A
Fl.
Ob.
&
Cl. in B b
&
Bsn.
i ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
w
!
!
b !
Hn.
&
Tp. in C
&
Tb.
Timp.
Cym.
Hp.
Pno.
Vln. I
Vln. II
Vla.
Vc.
D.B.
Wooden sticks
&
&
&
&
Fi
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
w
Fi
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
w
B
F
?
F
?
F
poco
b
b i
b
i ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
w
i
b
i ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
w
poco
j

3
b -
b
b i
b
poco
i
b
poco
j

3
b -
b -
b -
b -
Level 2B
Fl.
&
Ob.
&
Cl. in B b
&
25
Bsn.
i ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
w
!
F
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#w
w
!
-
b -
!
!
b.
Hn.
&
Tp. in C
&
Tb.
Timp.
Cym.
Hp.
Pno.
&
Vln. II
&
D.B.
&
Vc.
&
Vln. I
Vla.
b
b Fi
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ bi
w
w
w
B
F
?
F
?
F
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#w
#w
poco
.
.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#w
#w
poco
F
i~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ i
b
w
j

3
j

3
b -
b -
Appendix C
First two pages of Kirnberger Trio with
fragments in order of table
70
Vln. II
Vln. I
? 3
b 4
?b
J #
..
..
3

J
..
..
..
..
.
.
..
..
..
..
..
3
j
j j
J
J
j
j

3

J J
J

J
j

#.
. #
#.
.
J
#
J
#.
#
#
3

& b 4

& b J #
j
b
&

j
& b #
J
Clav.
Violin II
3
&b 4
Violin I
& b 43
Kirnberger - Trio (op volgorde)
j
# #.
& b .
? b .
Vln. II
. .
# n n

Vln. I
& b .
Vln. I
Vln. II
&b
3
J #
j
..
..
..
..
..
..
..
.
.
..
.
j 3

3

J
..
j 3

J
3

J
j
..
.
.
J .
j
j
# J
J
j
#
J
j
J
Kirnberger - Trio (op volgorde)
# n
#
J
3

j
b
&
? . .
b
J
J #
25
.
& b # n n # # .
17
&b
Appendix D
First two pages of the J.S. Joust score
with parallel staffs
73
Score
Brandenburger Concerto tbv J.S. Joust

Nacht van Kunst & Wetenschap 2016
Flute
Clarinet in B b
Trumpet in C
Horn in F

4
& 4 J b
f
4 J b
&4
f

4
& 4 j
f
4
&4
Allegro (q=c. 90)
b b

b b b
?4
4
Timpani
? 44
NORMAL
Trombone
1
Piano
Violin 1
Violin 2
Viola
Cello
Double Bass
Flute
slow

& 44

f
? 44 j b
& 44 J
f
4
&4 J
f
B 44 J
f
?4 j
4
f
t 44 j
4
&4
& 44
Horn in F
slow
& 44
Piano
slow
Violin1
slow
Violin2
slow
Viola
slow
Cello
slow
Double Bass
slow
Ultra Slow (at least half-time q=c. 45)
? 44
J
f
? 44
b b b
b b b
b b
b b b
b
f
B 44
J
f
? 44 j
b
f
t 44 j
b
f

V 44

f
t 44 j b
4 j
&4

b
f
& 44
b b
Trumpet in C
slow
Tuba
slow
Bass Clarinet
slow
Trombone
slow
4 j
&4
b
f
?4
Clarinet in B b
slow
SLOW
J.S. Bach
Than van Nispen t.P.
b b
b b b
b

b

b b

f

j

b b
Brandenburger Concerto tbv J.S. Joust
2
7
Fl.
&
B b Cl.
&
7
C Tpt.
Hn.
Tbn.
7
Timp.
7
1
Pno.
Vln.1
&
&

J

J J
&
Hn.
slow
&
Tbn.
slow
Tuba
slow
?
V
t
Vc.
slow
D.B.
slow
b

b
Solo
. b b
b

j
j
b
j
b
b b
b
b
b
b b
b
& b
Vla.
slow
& j
&
Vln.2
slow
C Tpt.
slow
Vln.1
slow
& b
slow
Pno.
D.B.
Vc.
B. Cl.
slow
B b Cl.
slow
B J
& J
Fl.
slow

7
b
&
Vla.
b
b
&
Vln.2
b
b
Solo
Solo
j
b
. b b
b b
b
Solo
j
b

b b

b
J
Appendix E
Part for first Violin for Hart Joust
76
Violin I
Hart Joust
non-linear music
Than van Nispen
Joyfull, dansant and with a heartbeat qd = 83

.. # # # J . ..
f nobile
..
.
.
. ..
.. .
ad lib
f nobile
# # ..
.. #
& 68
& ..
& ..
& #
& ..
..
~
b
J
..
..
.. # .
F
&
b.
.
.
~~~
o
nd
espress.
..
~~~
I. END
~~G~
& ..
i
i. b . .
#.
rit.
accel.
n .
Transition
Transition
& .. b . b .
sa
lis
.
o~
n~d~~
a
& lis~s ~
.~G~~
b .
J
b.
.
& ..
p
b.
b.
..
P
II. END
b. n.

..
. b . . b . . . ..

2
Appendix F
Score for Hart Joust
78
Score
Hart Joust
non-linear music
Joyfull, dansant and with a heartbeat qd = 83
Flute
6
&8
..
Clarinet in B b
6
&8
..
Bass Clarinet
? 68
..
Horn in F
& 68
..
Trumpet in B b 1
6
&8
Trumpet in B b 2
& 68
Trombone
? 68
Tuba
?6
8
Timpani
? 68
& 68
Solo (with D.B.)
& 68
..
Violin II
& 68
..
Viola
B 68
Cello
? 68
t6
8
Solo (with piano)

pizz.
& ..
B b Tpt. 1
& ..
?
& ..
j
. .
f nobile
espress.
17
Hn.
? ..
17
Timp.
j .
f nobile
.
f nobile
j
.
j
.
f nobile
.
f nobile
f nobile
j .

& ..
17
ad lib
j
.
j
.
.. .. .
..
..
j
.
. .
.. ..
.
f nobile
.. ..
. #.
.
f nobile
f nobile
j
.
.
.
f nobile
j .
f nobile
..
.
.
..
j
j
j
j
j ..

f nobile
.. ..
..
.. ..
j
( )

..
f nobile
.. . . . . . . . . . . . . ..
arco
f .
.
p .
p
.
..
..
..
. j .. ..
.. ..
f nobile
.. ..
j
,
f nobile
.. . . .
f. . .

..
f
. #.
..

poco
p
p
.
.
.
.
.. ..
..
f nobile
..
..
..
j
..
j
.
. #.
F
..
..
j
j
j

j
.
.
J
f nobile
..
f nobile
pizz.
j
j
. . j .. .. ( j) . .
f nobile
#.
# .
.. ..
7
7
f nobile / P
ad lib
.. ..
j
.
j .
..
. .
..
..
..
.
f nobile
j. .
. . .
f nobile
#.
j
.

.. # ..
..
.. ..
.
.. .. .
f nobile
.. ..
# # .
.
..
p
..
..
p
espress.
# # .. .. .
P
..
t
.. ..
D.B.
j
.
..
j
j
j
j .. ..

Pno.
Vln. I
.. ..
j
j

j
.
pizz.
..
j
.. ( j) .
.
f nobile
pizz.
j
j .. j .
.
( )
f nobile
# . #.
..
..
# # #
J .
f nobile
.. ..
..
.. ..
..
j .. ..
j
j

f nobile
. #.
ad lib
j
j

f nobile
17
Tbn.
espress.
Violin I
Double Bass
B. Cl.
j
..
f nobile
.
F
j j
j j
j j
j j ..
j
j
( j)
f nobile
f nobile
? 68
..
Piano
Fl.
..
J .
f nobile
Than van Nispen
arco
f nobile
pizz.
j. .
. . ( j) .
f nobile
.
.
..
. #.
..
j
.
j
.
j
..
. #.
..
.
j
.
..
& ..
29
Fl.
B. Cl.
ad lib
#
7
.
#
f nobile / P
? .. .
. .
.. ..
. . . .

f nobile
Hart Joust
. . .
Transition
accel.
..
..
rit.
b
# b b
b #
b b
Transition
#
B b Tpt. 2
& .. #
p
b
# b b
Freely Octatonic scale down to G#
b
# b b
.
# # # .
poco
p
.
~~~~
.~~~~
Freely Octatonic scale down to G#
~. ~G~~lissa
~~~n~do
~~ .

do
an
iss
Gl
? .. j
29
Timp.
#
& ..
29
Pno.
? ..
& .. #
Vc.
? .. .
f nobile
arco
arco
D.B.
t .. .
B b Cl.
& ..
B. Cl.
? ..
#
p
& ..
44
Hn.
B b Tpt. 1
B b Tpt. 2
Tbn.
Tuba
Timp.
Pno.
.
& .. b
& ..
? .. # .
p
? ..
#.
p
44
? .. j
b .
P
44
? .. # j
J
p
t .. # j .
# .
b Ped.
. b.
J
& ..
p
b.
.
.
&
44
Vln. I
Vln. II
Vla.
B ..
Vc.
? ..
D.B.
t ..
b.
b.
arco
. .
F
.
.
b.
Growl (ad lib)
#.
. b .
b. n.

#.
#.
.
b.
b.
b.
j .
b b
.. ..
..
..
~.~~G
~~li~ssa
~~n~do
~~~ .
~.~~G
~~li~ssa
~~n~do
~~~ .
~.~~Gli
~~s~san
~~d~o
~~ .
.
do~~
an~~
~
~
~
.~~
.
do~~
an~~
is~s ~
G~l ~
.~
.
o
n~d~~
ssa~
.G~~li~~
..
..
.
.
P
P
f
iss
Gl
~. ~G~~lissa
~~~n~do
~~ .
.
o
~n~d~~
.~~~~
a
iss
Gl
.. ..
.
i
.
.
i
. # . b

p
#.
p
.. .. # .
p
~. ~G~~lissa
~~~n~do
~~ .
.
o
~n~d~~
.~~~~
..
a
iss
Gl
..
j
.
.
..
b .
fl
f
.
j .
b
fl
..
b.
b.
b.
#.
#.
b ..
.. .
.
F
.. ..
#.
p
..
..
.. # .
p
# - .
j .. .. b .
b
b. b.
.. b .
b n .
b .
.. .. .
p
j
# .. ..
Ped.
poco
- . # .
-
b j b .
b b.
.. b ..
J
.
- . # .
-
b( )( ) j
( ) b( )
p
. #. #.
.. ..
j . .
b . . b.
#.
j
b n b
.
.
.
j
b b.
#. .

..
.. # .
p
.. # .
p
b.
#.
b.
..
#.
p
.. ..
.. ..
b.
#.
.. # #
. . . . .>. . . . . > . .>.>. . . . .
p>
b
J .
..
j
. . b b
j
b( )( )( ) b( ) . . b b b
F
..
b j ..
..
.. ..
b.
#.
palternate fingering
..
# - . # .
-
alternate fingering
. b.
F
Growl (ad lib)
..
..
j .. ..
b
fl
.. ..
. b .
b #
b b
.. ..
..
#. . #. .
fl
fl
. b j .. .. b .
. j
b
P
b. b.
.. b b ..
b
.
.
# & .. ..
J
J
p
j
j
j
# .. .. # .
. #
#
# .
#
. b.
.. .. b . b . n .
..

# #
J
J J
J
..
..
.. ..
ad lib
. b.
F
.
. b j b .
#.
..
.
p
. b.
#.
j
. #
#
. b b
J
#.
. .
poco
. .
F
j
b
i
b .
p
.
#.
p
.
.
(Slow trill)
& .. # # .
ad lib
p
44
Fl.
# # .
Vla.
# # ..

29
Vln. I
b.
b.
p
b.
..
Pi
.. .
p
Ped.
.. .
#
p
.. ..
.
b ..
#.
..
b.
b .
f
b ..
n.
i
.
espress.
..
..
Sul Pont
..
58
Fl.
&
#.
b.
#.
b.
.. ..
#.
b.
#.
b.
..
B b Cl.
&
B b Tpt. 1
&
..
> # . . . . . > . . . . . # > . . . > . > . . . . .
Tbn.
Tuba
Vln. II
Vc.
..
ex.
Non /
slow vib.
.. .. .
..
b.
b .
&
t
& ..
i
.
multo
vib.
Non /
slow vib.
#.
multo
vib.
.. .. # . .
F
.. .. # .
.
F
. b.
. b.
. .
..
. b.
. b.
. .
..
..
..
b .
#.
b ..
.. ..
i
.
..
b b ..
..
Ped.
..
b b ..
j
#
#
#.
#.
..
Sul Pont
ad lib
.. .
b( ) ( ) j b
()()
() ad lib
Growl (ad lib)
.. ..
espress.
() ad lib
..
b ..
& .. # .
p
?
..
58
Vln. I
ad lib
() ad lib
Hart Joust
in.
poco
58
Pno.
b b b b b n
#.
.. .. .
#.
Sul Pont
j
b n b
poco
b
b J
#.
variate with vibrato ad lib
..
#.
p
#.
.. ..
#.
p
.. # .
.
F
. b .
. b.

. .
..
. .
..
#.
..
j
# ..
#
.. # .
F
.. .. # .
F
.. .. .
b
F
. b .
. b.

Sul Pont ad lib
j
. b b - .
b.
b.
. .

..
j. .
. b . . b .
F
Sul Pont ad lib
j
. b b -.
j.
. b .
I. END
73
Fl.
B b Cl.
B. Cl.
&
& .
73
Hn.
B b Tpt. 1
B b Tpt. 2
Tbn.
Tuba
Timp.
& .
& .
Pno.
.
t .
.
73
Vln. I
Vln. II
Vla.
Vc.
D.B.
.
ffl
.
.
.
.
ffl
.
.
ffl
? .
73
?
.
?
.
ffl
.
f
& .
.
?
73
? .
&
.
ffl
.f
.
.
.
&
? .
pizz.
t
f
.
ffl
B .
.
ffl
II. END
.
.
.
.
.
Bibliography
Karen Collins. Game Sound. MIT Press Books, 2008.
Wim de Ruiter. Compositietechnieken in de twintigste eeuw. De Toorts, Haarlem, 1993.
Jason Freeman. Glimmer (2004) for chamber orchestra and audience, 2004. URL http:
//distributedmusic.gatech.edu/jason/music/glimmer/.
Jason Freeman. Sketching (2013), for improvising musicians with audience participation
via mobile phones, 2013a. URL http://distributedmusic.gatech.edu/jason/
music/sketching-2013-for-improvis/.
Jason Freeman.
Teamwork (2013), for athletic band with fan participation via mo-
bile phone, 2013b. URL http://distributedmusic.gatech.edu/jason/music/

teamwork-2013-for-athletic/.
Bas Frishert. The next level rock concert - master of arts in creative design for digital
cultures. Masters thesis, Utrecht School of the Art Department of Art, Media and Technology, 2015.
Mark Hijleh. Towards a Global Music Theory : practical concepts and methods for the
analysis of music across human cultures. Ashgate Publishing Limited, 2012.
Sang Won Lee Jason Freeman. Real-time music notation in mixed laptopacoustic ensembles. Computer Music Journal, 37:4, pp. 2436, Winter 2014, 2014.
Daniel J Levitin. This is Your Brain on Music. Atlantic Books, 2008.
82
BIBLIOGRAPHY
83
Kenneth B. McAlpine, Matthew Bett, and James Scanlan. Approaches to creating real-time
adaptive music in interactive entertainment: A musical perspective. In The Proceedings
of the AES 35th INTERNATIONAL CONFERENCE. AES Royal Academy of Engineering, 2009.
Simon Holland Oliver Hdl, Geraldine Fitzpatrick.
Experimence: Considerations
for composing a rock song for interactive audience participation.
In Proceedings
ICMC|SMC|2014. Proceedings ICMC|SMC|2014, 2014.

Oxford-Dictionaries. Oxford Dictionary of English. Oxford University Press, 2010.
Roy M. Prendergast. Film Music, a neglected art. W. W. Norton, 1977.
Oliver W. Sacks. Musicophilia : Tales of Music and the Brain. Picador, 2008.
Michael Sweet. Writing interactive music for video games: A Composers Guide. AddisonWesley, 2015.
Michael Tenzer. Balinese Gamelan Music. Tuttle, 2011.
Andrew Shahriari Terry E. Miller. World Music a Global Journey. Routledge, 2012.
N.J. van Nispen tot Pannerden. Nln-live, an application for live nonlinear and interactive instrumental music. Proceedings ICMC 2014 International Computer Music Conference,
2014.
Than van Nispen tot Pannerden. The nln-player: A system for nonlinear music in games.
Proceedings of the International Computer Music Conference 2011, University of Huddersfield, UK, 2011.
Kees Went. Sound Design for Nonlinear Artefacts - Models for Sound Design in the Transformation between Linear and Nonlinear Artefacts. PhD thesis, Bournemouth University, 2014.

Interactive Live Instrumental Music 21-06-2016

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Interactive Live Instrumental Music 21-06-2016

Uploaded by

Copyright:

Available Formats

"INTERACTIVE LIVE INSTRUMENTAL MUSIC"

Than van Nispen tot Pannerden - than.vannispen@hku.nl - student id nr. 2051299

Why interactive, live and instrumental? . . . . . . . . . . . . . . . . . . .

Interactive music strategies (in video games) . . . . . . . . . . . . . . . . .

Live instrumental music

Interactive live instrumental music . . . . . . . . . . . . . . . . . . . . . .

interactive live instrumental music experiences

Space Invaders performed live at the 2015 Indie Games Concert . . 22

King of Games - live interaction with children (2015 adaptations) . . . . . 34

RCO - live music interaction with children

Pepernoten voor groot orkest . . . . . . . . . . . . . . . . . . . . 41

Discussion and Conclusions

A Karmaflow Interactive Music Approach

E Part for first Violin for Hart Joust

F Score for Hart Joust

Why interactive, live and instrumental?

the musicians are trained in improvisation, or score-reading

What it is not about

Blacking & Edgar Varse

Interactive Music in video-games

Hat Trick on page 44 for an example of an audience game

Interactive music strategies (in video games)

orchestration) is a music method in which the mix

of separate tracks (stems) of music is changed based on

approach is the 2001 Red Dead Redemption.

instance a "battle" version is usually reasonably similar

to stick out, making the change sudden and obvious13 .

Live instrumental music

Interactive live instrumental music

the outcomes of the voting on the stage projec-

tion screen as well as on their smart device.

the audience (Freeman, 2004).

18 The 2016 Kickstarter campaign for Journey Live:

Audience interaction & live instrumental music

Simon Says / Loom

time (Duck, duck, goose

"guess the instrument"

Single person interaction & live instrumental music

pitch (SingStar / Plink)

* other games and game-genres than the

Figure 1.4: live instrumental music audience interaction based on video-games

CHAPTER 2. INTERACTIVE LIVE INSTRUMENTAL MUSIC EXPERIENCES

Karmaflow in Concert 2015 - democratic sequence

organised with the Dutch Metropole Orkest, an

information on this concert can be found on: www.karmaflowinconcert.com

2.1. HORIZONTAL RE-SEQUENCING

timed audience poll

Figure 2.2: Musical structure and audience interaction of interactive pieces

CHAPTER 2. INTERACTIVE LIVE INSTRUMENTAL MUSIC EXPERIENCES

Before, and at, the rehearsals for

Figure 2.3: Full cast and orchestra for Karmaflow

of the pieces (where a gauze screen would

2.1. HORIZONTAL RE-SEQUENCING

NLN-Live - interactive live instrumental music with iPads

tribute to the musicians and conductor the in-

CHAPTER 2. INTERACTIVE LIVE INSTRUMENTAL MUSIC EXPERIENCES

2.1. HORIZONTAL RE-SEQUENCING

CHAPTER 2. INTERACTIVE LIVE INSTRUMENTAL MUSIC EXPERIENCES

Space Invaders performed live at the 2015 Indie Games Concert

audience game Hat Trick ( on page 44) and the

2.1. HORIZONTAL RE-SEQUENCING

2. 2nd rehearsal (final rehearsal) in the Paard van Troje-venue

Figure 2.7: The Space Invaders Ensemble

orchestrator and composer. A mod of the video