World Music Visualizations: Score-Driven Musical Storytelling

Within the animist belief system of Indigenous Australians, a songline, also called dreaming track, is one of the paths across the land (or sometimes the sky) which mark the route followed by localized "creator-beings" during the Dreamtime. The paths of the songlines are recorded in traditional songs, stories, dance, and painting. (From Wikipedia)

One of the more intriguing possibilities opened up by a single page music application such as foreseen here is that of scripted educational narratives, where various short score-driven but highly interactive components (synchronized film snippets, score phrase playback, text, instrument models and theory tools) can be brought together -possibly several times in different manifestations on one page- to illustrate some educational theme in one short article. Here, the one-score, one-instrument, one theory tool mould is broken.
Another narrative arena is that of animated musical stories: graphical (in the sense of artistic song- or storyline) animations whose narrative is driven directly from a musical score. Different score, different narrative. This has potentially profound impact on storytelling across a swathe of application areas, be they musical or artistic, or 2D, 3D, on-screen or even open-air.

Animate Musical Storytelling
Data, Goal, Story, Metaphor

Not only will this allow musicologists to bring information-rich technical visualization to a world-wide audience, but give storytellers the means to create score-driven (and hence music-dependent) storylines. It will also release the storyteller into the realms of networked thinking, in the sense that one animation set will give different results depending on the score. Effectively, the score is projected into -and drives- the story's graphical artifacts.

So what might these animated musical stories look like? The only limits I see are those of own imagination..

Big, brave, open-source, non-profit, community-provisioned, cross-cultural and off the reservation crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or simply register as a potential crowdfunder..


The Storyteller

Musical Storytelling

Through song, saga, ritual dance, or celebration of the great cycles of nature, music has always been intimately associated with storytelling. Music is one of the vehicles by which cultural continuity is assured.

Cultural diversity implies distribution of power. Cultural continuity implies a sense of identity, and so wellbeing. Cultural understanding implies experience and openness towards others. Cultural mobility implies acceptance.

No stories, no culture? No musical diversity, no democracy? No understanding, no tolerance? Neither museums nor archives, but people, their sense of identity and their values keep culture alive.

We devalue cultural identity at our peril. Culture seems, even in more retrenched forms, an abstraction central to wellbeing. Nevertheless, to understand someone else's cultural values is not to sacrifice your own.

Understanding -and especially in context- is the foundation of respect and tolerance, and ultimately of mutual advancement.
Culture is catalyst, portal and key. It awakens curiosity, motivates, fosters identification and enriches. Moreover, there is no freedom greater than acceptance where none was asked or expected.

Cultural Continuity

Potential Crowdfunder? First Name Email Address

We live in challenging times. No matter where we look, we encounter fault lines. Discontinuities. Conflict.

Cultural memory and values, last remnants of a refugee's life before speculation-fuelled wars, come under attack: scapegoats for the real aggressors - armaments industry, hedge funds, banks and politicians locked in last-gasp, pre-singularity economic overreach. Raptors, psychopaths, hoarders, narcissists and tax evaders united in one goal: to beat each other to the other ninety-nine percent's last, if symbolic, tooth filling.

Cultural diversity implies distribution of power. Cultural continuity implies a sense of identity, and so wellbeing. Cultural understanding implies experience and openness towards others. Cultural mobility implies acceptance.

So is it worthwhile keeping culture alive?

Narrative Threads

Culture is a narrative. Where our forebears were masters in sustaining it's continuity, we seem to be losing it.

Too many threads, too many interrupts, too many demands. If honest with ourselves, we are losing even the willingness and ability to relax in other's company - central to sustaining the narrative in the first place.

Music, dance and community are fertile ground. If we can adjust our ways, narrative and continuity stand a chance of taking root once more.

Storytelling Tool

Part of the challenge at present are the limitations of the tools at our disposal. It's all very well talking community if these only compound our troubled, hectic lifestyles. Even a 200-page book on music theory?

With this framework, however, we are equipped to drive our musical narrative -whatever their storytelling payload- directly from the musical score.

Elegy. Dance. Theatre. Animation. Parody. Edutainment. Theory. Instruments. Esoterics. History. Anything that can find accompaniment in music can be brought to life on-screen. The only limit is imagination.

Narrative Threads or Arcs

There are, however, many, many ways in which musical score playback can be used to underpin and highten narrative tension.

Story Threads or Arcs

Multiple threads are a widespread device. The aggregator platform focussed on in this site is no exception.

In bringing together the many different aspects of a musical narrative, we foster immersion, immediacy and discovery.

With SVG at our disposal, we have the means to take these ideas much further.

Music Animation Examples

Interference Music Visualizer

In another context, I've already mentioned Richard Merrick's Interference Music Visualizer, which -in content- bears comparison with ideas being developed in the music aggregator platform at the centre of focus here.

Interference Theory's Visualisation Dashboard

This is musical storytelling in the sense that the development of a musical theme or (here) phrase is brought to pictorial life in time.
Appealing use is made of saturated colour, pitch, position on a timeline, instrument model, and a variety of theory tools in support of his underlying interference theory.
It's not clear if an interactive version was ever made available, but the demo in itself remains a little mesmerising.

Notation Mimickry

Another approach effectively duplicates the notation, adding form and colour to hold the viewer's attention. Again, some of these (there are many) can be quite beautiful. In the absence of any additional storyline or data, interest can be quickly lost.

Musical Engines

Musical engines exploit our child-like curiosity for the intricate, if the signal to our brains is confused. Is the engine producing the sound, the sound driving the engine, or both externally held in synch?

Narrative Support

Sergei Prokofiev's "Peter and the Wolf" is famous for bringing greater depth to story characters by associating each with a specific instrument and musical theme.

This supports the narrator (storyteller) by providing additional, musical clues as to the temperament and movement of the creature concerned. Earlier film used static pictures, later ones animation synchronised with the music. Nevertheless, all the work setting this up was strictly manual. 

Learning Reinforcement

Much as in earlier playground counting rhymes, song and music can be used to reinforce memorisation, as in this short, surgical "Brain Parts Song" by Aaron Wolf.

Brain Parts Song by Aaron Wolf from Aaron Wolf on Vimeo.

Physics

As a child, I was fascinated by the standing waves produced in water or using filings on a metal plate, musing if every piece of music might not have it's own, unique signature. Nowadays, designers experiment with the detailed audio response of entire rooms.


You may feel that the only story these patterns tell is their own. I feel they hint at much more. Here, for example, Lissajous standing waves are taken to an entirely new level: drawing with sound.

Summary

For all their flexibility, SVG animations driven from musical sources, whatever these may be, remain few. Decoding a time-based source such as MusicXML or audio, and manipulating complex SVG animations can be time-consuming to code, and bears an initially steep learning curve.

Many early adopters of libraries such as D3.js have been snapped up by industry. Nevertheless, the quality and breadth of data visualisation work continues to grow.

The main challenge to notation-driven animations remains, however, the complexity of music exchange formats such as MusicXML. Midi represents a simpler access mechanism, but one that sacrifices much of MusicXML's musical expressiveness.

MusicXML is, as internet technologies go, long established and well proven.

Beyond this, coding highly interactive yet robust music animations also has it's challenges, but where these find widespread use by a large community, this is time well invested. With the range of entry points (notation, audio etc) to controlling SVG on the rise, opportunity calls.

---

Keywords

---

online music learning, online music lessons	distance music learning, distance music lessons	remote music lessons, remote music learning
p2p music lessons, p2p music learning	world music visualisation world music visualization	musical instrument models interactive music instrument models
music theory tools musical theory	p2p music interworking p2p musical interworking	comparative musicology ethnomusicology
world music international music	folk music traditional music	P2P musical interworking, Peer-to-peer musical interworking
WebGL, Web3D, WebVR, WebAR	Virtual Reality, Augmented or Mixed Reality	Artificial Intelligence, Machine Learning
Scalar Vector Graphics, SVG	3D Cascading Style Sheets, CSS3D	X3Dom, XML3D
Online Music Education, Remote Music Education	Equal Temperament, Just Intonation	score-driven storytelling notation-driven storytelling