Guitar Hero

Tod Machover believes that, through technology, everyone can experience the transformative power of music. But creating music is not just fun, it can also aid mental and physical health.

We all love music deeply. It is capable of entertaining, stimulating, moving and transforming us as few other activities are. In fact, there is increasing research on the how and why of music’s power, some of it – like the much-hyped ‘Mozart Effect’ – suggesting that merely listening to music on your iPod while driving, reading, sleeping or perhaps even playing music to your baby in the womb is enough to let music work its full magic.

Unfortunately, that isn’t quite true. Music exerts its power when we are actively engaged, not when we listen subliminally. For this reason, I have been working with my group at the MIT Media Lab to create musical tools – often with specially designed technologies – that enable everyone to participate directly in music-making regardless of background.

This field has undergone a revolution in the past several years through the huge public success of the Guitar Hero and Rock Band videogames. Alex Rigopulos and Eran Egozy, students of mine from the MIT Media Lab, developed them based on ideas that we were working on in the early 1990s. The good news about Guitar Hero and Rock Band is that they clearly demonstrate the public’s willingness to dive in and immerse themselves in music-making, given the right environment. The bad news is that neither platform is truly musical, nor do they encourage learning, expression or creativity.

What would happen if we could combine the excitement and ‘stickiness’ (bordering on addiction) of Guitar Hero with a more sustaining, personal and open-ended musical experience? How could we embed such a new type of activity in a more integrated musical ecology, where the current exaggerated distinctions between celebrities and amateurs would be diminished and the level of musical sophistication, excellence and, hence, enjoyment would be raised for all?

The research projects that I direct at the MIT Media Lab and many of the musical projects I undertake are attempting to nudge the field in these directions. We started by developing Hyperinstruments for some of the world’s greatest performers, including Yo-Yo Ma and Prince, as well as orchestras, chamber music ensembles and rock bands. All kinds of sensors are built into the Hyperinstrument so it knows how it is being played. By changing the interpretation and feeling during performance, a cello – for instance – can be morphed into a voice or a whole orchestra or something that nobody has heard before.

As these virtuosic Hyperinstruments developed, we started imagining that we could use similar technologies and interpretation strategies to produce instruments and interfaces for music-lovers who weren’t highly trained virtuosi. We have designed a series of such instruments and interfaces. One of the largest collections was the Brain Opera that launched at the first Lincoln Center Festival in New York in 1996, toured the world and is now permanently installed at the Haus der Musik in Vienna.

We created this rather large orchestra of specially designed fantasy instruments (including Rhythm Trees, Harmonic Driving, Gesture Walls and Melody Easels) so that anybody could play them using natural skill. You can play a video game, drive through a piece of music, use gestures to control huge masses of sound, touch a special surface to make melodies and use your voice to make a whole aura. We designed the Brain Opera for adult concert-goers, but found that everywhere we went it was most easily understood and most creatively manipulated by the youngest (under 8) and oldest (over 70) visitors. This was perhaps due to lack of inhibition and desire for social play and creativity among those ages. We therefore decided to concentrate on activities that might engage those groups more directly.

This led to our Toy Symphony project (2002–2005), which attempts to reconsider how to introduce children to music in the most immersive, creative and enjoyable way possible. The goal is to have kids fall in love with making music first and then demand to learn more because of that love. We designed a set of new Music Toys, including the soft, squeezable Music Shapers that manipulate intensity and tone colour; Beatbugs, which capture rhythms that can be manipulated and shared with friends; and a software-composing environment called Hyperscore that lets anyone compose original music by shaping lines and colours. Another goal of Toy Symphony was to develop a project model – learning musical skills, creating new music and then rehearsing and performing a concert – that would bring children and orchestras together.

Our results with Toy Symphony were encouraging enough that we decided to bring this model to other populations where there might be clear impediments to personal expression and creativity, where music, made accessible through new musical tools, might be an ideal medium.

In 2004, we began concentrating on providing musical experiences and tools – based on ongoing research by colleagues at the MIT Media Lab and increasingly around the world – to help improve health, diagnose illness and provide a medium of expression and communication that would otherwise be lacking. This new area of Music, Mind and Health has led to research in using music for early detection of Alzheimer’s disease, for social and emotional adaptation for autistics, for aiding physical and mental rehabilitation and for a growing number of other areas. With my student Adam Boulanger, I started this work at Tewksbury Hospital near Boston, where we were invited to work with a group of long-term residents with a wide range of severe physical and mental disabilities. We organised composing workshops with Hyperscore that resulted in a series of public concerts featuring music by patients. This process has become so successful that it has been replicated at many sites, resulting in marked and unexpected improvement in a wide range of conditions, and inspiring a number of patients to themselves mentor others in the uses of new tools and environments for creative musical expression.

A new performance system designed for and with Dan Ellsey (see panel) is an example of a new category of interfaces that we call Personal Instruments. Even an instrument as sophisticated as the Hypercello we designed for Yo-Yo Ma is a generalised instrument. In other words, anyone familiar with cello technique can play it, drawing more from it according to one’s mastery and understanding. But Dan’s instrument was designed for him and him only: it takes account of his particular style and detail of moving and the way that translates into musical expression, and compensates for his particular physical limitations. Future instruments and interfaces can and must be adaptable and tunable to each of our skills and limitations. For us, Dan’s performance system represents the first step in this direction.

If we extend this idea that music can be one of the most powerful interfaces we have for showing the outside world who we are, another current project – a ‘robotic’ opera called Death and the Powers – pushes it in an unusual direction. With an original libretto by former US Poet Laureate Robert Pinsky, it tells the story of Simon Powers, a rich, successful, powerful man who wants to live forever. He designs a ‘System’ to download himself into his environment, which includes a vast library and a set of objects and artefacts. The main singer disappears into the System at the beginning of the opera and the entire stage becomes the main character; it becomes Simon Powers’ legacy. The opera is about what we can share, what we can pass on to others and what we can’t. It features a large Musical Chandelier – a robotic music instrument with gigantic piano-like strings. These are remotely controlled mechanical and electroacoustic elements that tickle, vibrate, stroke and punch out unusual music. The opera also has a ‘chorus’ of autonomous robots that observe the action and intervene between Simon’s family and Simon-as-System. And there is the System itself, which is an abstraction of how Simon chooses to represent himself. Together, the thousands of moving, pulsating books create a new visual, gestural and sonic language for translating a human life into a compressed, expressive, dynamic and interactive form.

Another unusual aspect of Death and the Powers is that, unlike our previous Hyperinstruments, the interchange between human expressive behaviour and ‘system’ response is invisible to the public. By definition, the performer playing Simon Powers is offstage while his voice and gestures are being measured and translated into sounds and movements of the various set pieces, from System to Chandelier. This means that, rather than being an explicit control-and-result relationship, we must measure subtler and more fundamental aspects of the performance, such as breathing, muscle tension and heartbeat, in addition to more musical measurements. We must also translate these so that a powerful sense of ‘disembodied performance’ is established, with Simon Powers’ presence felt on stage, even though it doesn’t look, feel or sound like him.

The live performance of Death and the Powers premieres in Monaco in September 2009, but we are also developing a version that will allow anyone to pour their own memories, images and sounds into a similar System to create a Personal Opera of/for oneself, friends or family – a kind of next generation Hyperscore.

As with all the projects described, an important element has been to integrate a professionally composed and performed musical work with numerous activities that can engage the general public in substantial ways. My most recent opera project, Skellig, explores this new interrelationship in an unusual way.

Skellig premiered in November 2008 at the Sage Gateshead in Newcastle. I based the opera on David Almond’s bestselling novel, which was written for young adults but has a wide appeal. In an extension of the Toy Symphony model, we decided to make a youth chorus – recruited from motivated but not ‘specialist’ young people aged 12–18 – a central part of the work. This chorus is on stage for much of the two-hour-long opera and represents an extension of the mysterious central character Skellig, who is part-beast, part-human and part-angel. In doing so, the chorus must perform a wide range of music, from intricate multi-part counterpoint to singing the sound of animals, the wind and ‘the world’. In addition, the youth chorus is physically integrated into the production, with constantly varying choreography by Mark Bruce. For both music and choreography, we had to invent systems that would allow young people to both learn and memorise almost two hours of performance details, and to absorb all of this so they could enact the opera with ease, freedom and expressive power. Bruce accomplished this by using a language of natural gestures and movements drawn from ordinary life and then working extensively with the young people over a period of months to teach and shape the performance. I used a combination of carefully notated music, which had to be learned, mostly by ear, and an audio score that the chorus follows for cues, imitation and extrapolation. Under the training of choral director Sharon Durant at the Sage, the amateur chorus was able to absorb this complex and ever-varying soundscape and to provide a constant auditory counterpoint to the music sung by the world-class professional soloists playing the principal roles in Skellig.

All the projects mentioned suggest a new model for the interrelationship between experts and amateurs in musical listening, performance and creation. Some of the boundaries to active engagement in music have eroded, but there is still much to be done to create a truly vibrant musical culture.

In my view, a prime example of the kind of new musical ‘ecology’ that we should seek is found in our culture’s relationship with cuisine. We all enjoy eating at three-star restaurants and admire the achievements of the world’s greatest chefs. At the same time, we do not hesitate to dive in ourselves to prepare special meals of high quality on special occasions. We also put together daily meals for ourselves, improvising content that reflects our personal styles. We enjoy eating and even studying the most ‘expert’ cuisine we can find, but are not scared to make and invent our own. In turn, the fact that we constantly prepare food ourselves makes us better understand and appreciate other food that we encounter.

Music – and most of the arts – has come very far from such a ‘healthy’ ecology, and it is this that we need to reinvent. Technology can help, as it can act as a bridge to each of us depending on our background and experience, taking advantage of our skills and compensating for our limitations. Even more importantly, we need to establish a fundamentally new partnership between all of the potential participants in our musical culture, including individual artists, all parts of the music business, technology, lifestyle, health and social organisations, music presenting and broadcasting entities, research institutions, artists-as-mentors and – last but not least – the music-loving public. Only in this way can we establish a culture that will allow music to reach its full potential in shaping and transforming our experience. Doing so will allow music to exert its most powerful possible influence on society at large. Surely we can imagine a world where music is at least as nourishing as a three-star meal? 

Find out more about Tod Machover and his acclaimed new opera Skellig.

Tod Machover is Professor of Music and Media at MIT's Media Lab.

Music and health: the work of Dan Ellsey

Working with Dan Ellsey, a 32-year-old with severe cerebral palsy, led us to the next stage of our music and health research. Dan has great limitations in mobility and normally ‘speaks’ through an infrared-activated computer interface mounted on his wheelchair, a laborious process for an intelligent, animated and expressive young man. Dan took to our Tewksbury Hyperscore workshop with gusto and learned Hyperscore quickly. After Adam Boulanger designed a head-mounted interface so Dan could control it directly, Dan wrote a terrific piece that was performed at our final concert by the local community orchestra. After our initial workshops, Dan continued to compose prolifically in Hyperscore and eventually started training others. Adam and I invited Dan to compose a new piece using Hyperscore in the spring of 2007 and offered to work with him to design a system so Dan could perform his piece in public. We worked with Dan to track his head movements, to explore what motions he associated with which expressive musical parameters and to extract the meaning from the noise in his interpretative head gestures. Dan’s condition limits the pure predictability of timing and position but not his expressive intent or enthusiasm.

Dan premiered the new system by performing his new Hyperscore piece, 'My Eagle Song', at a major MIT conference in May 2007 and then reprised his performance at the TED Conference in Monterey, California, in March 2008. On both occasions, he was able to craft a powerful and communicative performance through his mastery of the new head-mounted interface. 

Significantly, everyone who has viewed Dan’s performances has been able to ‘see’ Dan directly, understanding his ideas and feelings and seeing past the usual barrier of his physical limitations. At least as significantly, Dan’s physical symptoms subside when he is rehearsing or performing with the new system and his movements become more regular, rhythmic and precise.