World Music's DIVERSITY and Data Visualisation's EXPRESSIVE POWER collide. A galaxy of INTERACTIVE, SCORE-DRIVEN instrument model and theory tool animations is born. Entirely Graphical Toolset Supporting World Music Teaching & Learning Via Video Chat ◦ Paradigm Change ◦ Music Visualization Greenfield ◦ Crowd Funding In Ramp-Up ◦ Please Share

Monday, July 18, 2016


Music Visualization: Why A Toolset and Platform, And Not Just An App?

More websites are now loaded on smartphones and tablets than on desktop computers, a milestone that underlines the pace of digital disruption. But in the context of immersion, immediacy and expressiveness, does the mobile app always make sense?

Big, brave, open-source, non-profit, community-provisioned, cross-cultural and chicken'n'waffles crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or just register as a potential crowdfunder..

Why A Music Visualization Toolset And Platform - And Not Just An App?

Why a Platform, and Not Just an App? #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory

Chicken & Egg

If one thing characterizes mobile usage, it is pervasive 'distraction'. The mobile experience is conceived around being on the move and at the mercy of reduced bandwidth, capabilities and countless interrupts. Indeed, with exception of advertising, the entire experience is built around mitigating distraction's impact.

Learning, though, is something we should do completely distraction-free. At home. Broadband-tethered. On the real estate luxury of the desktop. With instrument. In peace.

Moreover, mobile first -no less than the earlier PC/Mac model- has the user tied to one provider and their framework. The only more or less platform-agnostic environment is that of open (W3C) browser standards, with the bonus -and tyrannies- of the browser DOM.

Within the browser-based ecosystem, we have two main (and diametrically opposed) schools of development thought.

Graceful Degradation vs. Progressive Enhancement

Responsive - Graceful Degradation #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory
Responsive: Graceful Degradation
o graceful degradation: associated with the view that the transition from fixed to mobile or VR device screen estate is progressive, and that this can and should be accommodated over time and without compromising functionality.
o progressive enhancement: mobile first, larger devices being accommodated through enhanced (and possibly dedicated) solutions (turn the arrows above around).

Graceful degradation in our context implies above all expressive power over convenience. It is a unique feature of this platform that it makes full use of d3.js, perhaps the most expressive of in-browser data visualisation libraries. The hope and expectation is that it's capabilities will, with time, be brought to all devices and form factors.

I also include ER / VR (enhanced/virtual reality) devices in this context in the expectation that web stack technologies can progressively be integrated in ways that feel natural. I see VR, incidentally, more in a mobile than a fixed scenario in the long run.

Whatever. None of us are clairvoyants. In the context of our aggregator platform, what do these two approaches mean in real terms? With some generalisations:

Fixed First Mobile First
Graceful Degradation Progressive Enhancement
Aggregation (Platform) Fragmentation (App)
W3C Standards Mobile Framework
Broadband Internet Mobile Internet
Resources Bottlenecks

For me the choice is clear, the web browser providing a possible future bridge between the two. Starting with fixed-first, the right mind- and technology set, we may -in the long term and given further technology advances- be able to cater to both.
Model / Tool Family Preselection Matrix. #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory
Model / Tool Family Preselection Matrix
Critically, we should not expect to be able to serve everything available on a workstation to a mobile user without adapting to it's fragmentation model.

This implies a filtering of content to ease the load on, and traffic to the mobile device, but under user control.

Approaches to this are to restrict the mobile environment to at most a user's preferred models and tools (implies preselection either on a workstation or directly on the mobile device), or impose a hard (app-like) limit of a short score excerpt with:
  • one instrument
  • one tool
  • one shared configuration
This equates to at most just one of the cubes on the preselection matrix above.

Ball-and-Chain Solutions

Potential Crowdfunder?

Mobile Apps

Own ecosystems and native apps allow owners of prominent mobile platforms to control the entire value chain.

Apps follow the age-old strategy of 'divide and conquer'. They fragment the market, then integrate the debris into a framework ensuring the host has control. The effect is a dumbing down of domain, device and end user.


By fencing-in social interactions, Facebook can 'tollgate' every interaction between business and customer. Access to a user's wall is restricted to those who pay, and to product advocates (the friends list). Similarly, maps exploit location, search engines discovery, and news portals the thirst for timely, topical information.

Some simple technical impracticalities associated with mobile also stand in our way. Technologies such as WebRTC, for example, (used for browser-to-browser interworking) are not yet robust enough for use in a mobile environment. Without these, we are forced towards proprietary solutions, which kinda sucks for your 'willfully open-source' type of initiative.

Free Flight

Assuming we can get past these hurdles and have a means of aggregating domain-relevant visualisations, there is the question of how much information is actually needed, what can be comfortably made available within limited screen real estate, and how well it supports person-to-person interaction (without some form of connectivity with others, nothing holds it's fascination for long).

Let's assume a parallel video chat session, then focus on the supporting tools..

Potential Information Overload. #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory
Information Overload
There comes a point on a small device where a large range of menu choices can get out of hand. Same for the number of models or tools ultimately displayed.

The watershed is a combination of needs, ergonomics, device size and common sense, perhaps a choice best left to the user.

Nevertheless, there are a couple of other issues.

Musical Immersion

With freedom of choice and enabling technologies comes the notion of multi-sensory and multi-model learning: the ability, for example, to forge leaps of musical understanding by networking concrete instrument models with abstract theoretical models, and seeing -very plainly- the mappings between the two.

The value of this should not be underestimated. With complete control over which facets of a musical culture (music, instruments, history, tools, genres and so on) should be shown, the user can establish an own, preferred ecosystem.

With visual control and focus in the user's hands, we have the basis for immersion.

Musical Immediacy

Deep musical understanding is often a process of assimilation: leaps of intuition or insight, spread over many years of practice and play.

The underlying information is widely available but static (cannot be directly related to the music at hand), spread over countless websites, videos and academic texts, and therefore tedious to access and apply.

Directly associating a musical score with instrument models and theory tools, theory, instrument and music are intimately linked. This represents immediacy.

Music Visualization

There is a user experience / GUI designer wisdom to the effect 'don't make the user think', which suggests that nothing short of mind-reading will lead to learning success.

Well, mind-reading may have to wait. Our next best shot? Direct, visual learning. Our visual processing is blisteringly fast, and appears to engage cognition far better than text.

Multi-Sensory Feedback

Chromatic Half Tone Displacements Lattice Tonnetz 12 Octave. #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory
Tonal Lattice (Tonnetz)
Visual modelling allows us to approach a problem from a variety of viewpoints.

Clearly, not all senses can be usefully engaged for music learning, but combined with tactile and audio inputs, visualisations quickly draw us into the realm of immersive learning.

As immersive learning is better understood and exploited, we may see dramatic leaps of learning effectiveness.

Multi-Model Music Visualization

Any specific piece of music may find visual expression in a variety of fully interactive contexts: for example, a linear interval map, circle of fifths, spiral, double helix, lattice or tonnetz, torus or even 3D modal space. Similarly, cross-cultural musical understanding may, depending on needs, best be served by any of instrument fingering roadmaps, frequency waveforms, 3D matrices (music-cultural landscapes) or any of many other models.

Depending on their characteristics, each of these models may support a specific kind of insight or understanding particularly well, or have particular subjective appeal. Two central issues in this context are:
  • freedom of choice in configuring an own, personalised environment.
  • sufficient resources to facilitate a good overview and easy navigation (compared to desktops, mobile devices still have rather skimpy processors, batteries and RAM).
Both seem to me arguments for a bold, platform-scale solution.

Musical Homogeneity

With multiple models arises the question of how to ensure colouring or naming conventions are consistently applied across multiple models. This is also closely connected to development overhead: homogeneity is much more difficult to control across a range of fragmented, standalone apps than across a single, integrated platform.


Where there is choice (personalisation), there is diversity, synergy and communication. A cloud- and browser-based solution is well placed both to leverage and offer these, through public web APIs.

Musical Sharing

Here, perhaps, the real crux of the matter, at least in the long run. A fully configured, personal instrument model and tool display can be passed to a student as a whole. The entire learning context is shared.

This has profound implications for the quality and ease of teaching. In the case of larger P2P classes, teacher and learners are on the same page.

We need to distinguish this form of 'on the same page' from that of hardcopy publishing 'on the same page'. The latter ensures that sometimes hundreds of thousands of learners learn from the same material - a sure route to monoculture. Under P2P learning, however, teacher and immediate students being on the same page can be seen as some guarantee of diversity.

Music Visualisation Aggregator Platform

Alone the idea of a multi-model, immersive environment implies something far beyond the dimensions of the current mobile app. Thoughts are inevitably drawn in the direction of aggregation, and hence 'platform'. With this arise wider issues of context, networking, ideas exchange and, inevitably, synergies.

The App is not Dead. Yet.

Clearly, however, the iOS or Android app is not going to go away overnight, and though -for all but the lucky few developers- a flawed model, it may provide an outlet for a limited range of applications derived from those of the 'aggregator platform', with size-limited window onto their driver scores. Central to this is an elegant, 'responsive' degradation across increasingly smaller devices, arriving at a point where only a tiny excerpt of score is visible, along with at most one or two models.

Mobile Hardware Limitations

Should mobile browser apps remain unable to leverage device-specific functions such as camera or hardware sensors, the arguments in favour of a browser-based aggregator platform approach would only be strengthened.

There are, however, initiatives afoot to provide just such access on mobile devices. Currently, the strongest arguments appear to be in favour of so-called hybrid apps, which attempt to combine the strengths of the browser stack with those of mobile devices.


online music learning,
online music lessons
distance music learning,
distance music lessons
remote music lessons,
remote music learning
p2p music lessons,
p2p music learning
music visualisation
music visualization
musical instrument models
interactive music instrument models
music theory tools
musical theory
p2p music interworking
p2p musical interworking
comparative musicology
world music
international music
folk music
traditional music
P2P musical interworking,
Peer-to-peer musical interworking
WebGL, Web3D,
WebVR, WebAR
Virtual Reality,
Augmented or Mixed Reality
Artificial Intelligence,
Machine Learning
Scalar Vector Graphics,
3D Cascading Style Sheets,

Comments, questions and (especially) critique welcome.