Music Visualization: Why A Toolset and Platform, And Not Just An App?

Anyone whose nose is regularly buried in news site comments will quickly come away with the impression that the internet is inhabited by nothing but trolls. Envy, aggression, self-profiling, spite, racism, polarization and goading seem the order of the day.

The internet has far to go before it realizes it's full potential. Central to that journey are the notions of social value, mutual assistance, of giving, of win-win. The first step is to replace the predatory and the exploitative with platforms true to the notion of 'digital commons' and the general good.

In contrast, there is little more heartening, life-affirming and human that the hand-to-hand or eye-to-eye contact of dance. Music and dance are the antithesis of conflict: a gateway to the inner peace of the crowd, of the release from self and self-consciousness: a catalyst for group or cultural identity - and a sense of belonging.

If we are to strengthen these forms of contact, why in particular a musical aggregator platform? Why, indeed on the internet?

Big, brave, open-source, non-profit, community-provisioned, cross-cultural and ouija board crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding #VisualFutureOfMusic. For the future live platform #WorldMusicInstrumentsAndTheory. Or just cut to the chase as a potential crowdfunder..

Goal: ANY Music Source, Instrument Configuration, Theory Tool, Mentor or Location

Why a Music Visualisation Aggregator Platform?

Cultural Identification

Eye-to-eye and hand contact, diversity of exchange and community are central to wellbeing.

Dance in particular powerfully reinforces these, in many cases breaching reserve or contact thresholds. In the crowd lies freedom, bringing people to a point where consciousness of time and self simply falls away.

In this sense, music and dance can be seen as a form of social currency. In participating, you contribute to the pool of social wellbeing.

Music Visualization

Potential Crowdfunder? First Name Email Address

We already encounter many types of music visualization in our daily lives: from the simple tone and volume level displays on audio equipment through sophisticated but largely cosmetic displays generated in real time from music audio, to score-synchronized video.

In parallel, we have seen the growth of an entire data visualization industry around the exposure of patterns and trends in governmental or regional datasets.

What has -until now- been missing is a broad-based, graphical social media site acting as a focus for audio- or score-driven interactive animations relating to music instruments and theory.

Moreover, imagine being able to send someone a link to such an animation and -regardless of platform- have it simply work?

Aggregator Platform

In directly and programmatically linking scores with animations, such an aggregator platform would unite a variety of entirely configurable instrument models and conceptual music theory tools, music exchange formats and scoring conventions to create a shared, highly interactive and customisable environment for teachers and learners.

It would also open the possibility of direct, P2P (here, person to person) teaching and learning worldwide over the internet - serving both musical diversity and community.

World Music Aggregator Platform Configuration Overview

There are dozens of more or less interactive music sites, and -all else failing- blow by blow 'learn to play..' videos on YouTube. Why, if so many other tools exist, create yet another?

Populated Main Menu

1. in building scores using the fundamental building blocks of data visualisation (the array), they are open to manipulation by sophisticated transformation techniques. Greenfield.
2. existing teaching offerings lie low in the learning value chain. Music visualisations, on the other hand, expose the complex relationships between instrument and abstract theoretical models. These are the associations on which virtuosity is built, and which normally take years to acquire. Greenfield.
3. in supporting the modeling of any instrument and configuration, visualization opens up the entire world music instrumentarium to learners. Learning freedom. Greenfield.
4. world music systems as a whole are poorly understood. Indeed, many 'western' music-lovers are barely aware of the existence of other than the 12-tone equal temperament system. The potential for comparative musicology and experimental music is huge. Greenfield.
5. existing music platforms are primarily concerned either with (largely passive) music consumerism, the profiling of own band or musical tastes, or the individual's pursuit of excellence. But social collaboration? Greenfield.
6. from a motivational point of view, music learning is (or should be) a shared experience. This platform's promise is person-to-person (P2P) musical collaboration across cultural and geographical boundaries. Greenfield.

The Singularity and Music

Advances in artificial intelligence, machine learning and automation are expected to bring major societal upheaval within the next decade (leaving as much as 40% of industrialised populations irrelevant to wealth creation).

Music, Community, Contentment, Wealth

Music has (along with religion and art) long been recognized as a last resort of the disenfranchised.

The benefits both of music and occupation for the hands are well documented. If the bulk of humanity is to be condemned to economic irrelevance, there are cultural and social alternatives to work - with their own reward.

The most important contribution of music I see, however, is in offering a continued channel for curiosity, creativity and social needs. In doing so, it more or less inevitably addresses the issues represented in the Ikigai diagram. In doing so, it pulls us towards a much healthier inner state.

More websites are now loaded on smartphones and tablets than on desktop computers, a milestone that underlines the pace of digital disruption. But in the context of immersion, immediacy and expressiveness, does the mobile app always make sense?

Big, brave, open-source, non-profit, community-provisioned, cross-cultural and chicken'n'waffles crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or just register as a potential crowdfunder..

Why A Music Visualization Toolset And Platform - And Not Just An App?

Chicken & Egg

If one thing characterizes mobile usage, it is pervasive 'distraction'. The mobile experience is conceived around being on the move and at the mercy of reduced bandwidth, capabilities and countless interrupts. Indeed, with exception of advertising, the entire experience is built around mitigating distraction's impact.

Learning, though, is something we should do completely distraction-free. At home. Broadband-tethered. On the real estate luxury of the desktop. With instrument. In peace.

Moreover, mobile first -no less than the earlier PC/Mac model- has the user tied to one provider and their framework. The only more or less platform-agnostic environment is that of open (W3C) browser standards, with the bonus -and tyrannies- of the browser DOM.

Within the browser-based ecosystem, we have two main (and diametrically opposed) schools of development thought.

Graceful Degradation vs. Progressive Enhancement

Responsive: Graceful Degradation

o graceful degradation: associated with the view that the transition from fixed to mobile or VR device screen estate is progressive, and that this can and should be accommodated over time and without compromising functionality.
o progressive enhancement: mobile first, larger devices being accommodated through enhanced (and possibly dedicated) solutions (turn the arrows above around).

Graceful degradation in our context implies above all expressive power over convenience. It is a unique feature of this platform that it makes full use of d3.js, perhaps the most expressive of in-browser data visualisation libraries. The hope and expectation is that it's capabilities will, with time, be brought to all devices and form factors.

I also include ER / VR (enhanced/virtual reality) devices in this context in the expectation that web stack technologies can progressively be integrated in ways that feel natural. I see VR, incidentally, more in a mobile than a fixed scenario in the long run.

Whatever. None of us are clairvoyants. In the context of our aggregator platform, what do these two approaches mean in real terms? With some generalisations:

Fixed First	Mobile First
Graceful Degradation	Progressive Enhancement
Aggregation (Platform)	Fragmentation (App)
W3C Standards	Mobile Framework
Broadband Internet	Mobile Internet
Resources	Bottlenecks

For me the choice is clear, the web browser providing a possible future bridge between the two. Starting with fixed-first, the right mind- and technology set, we may -in the long term and given further technology advances- be able to cater to both.

Model / Tool Family Preselection Matrix

Critically, we should not expect to be able to serve everything available on a workstation to a mobile user without adapting to it's fragmentation model.

This implies a filtering of content to ease the load on, and traffic to the mobile device, but under user control.

Approaches to this are to restrict the mobile environment to at most a user's preferred models and tools (implies preselection either on a workstation or directly on the mobile device), or impose a hard (app-like) limit of a short score excerpt with:

• one instrument
• one tool
• one shared configuration

This equates to at most just one of the cubes on the preselection matrix above.

Ball-and-Chain Solutions

Potential Crowdfunder? First Name Email Address

Mobile Apps

Own ecosystems and native apps allow owners of prominent mobile platforms to control the entire value chain.

Apps follow the age-old strategy of 'divide and conquer'. They fragment the market, then integrate the debris into a framework ensuring the host has control. The effect is a dumbing down of domain, device and end user.

Tollgating

By fencing-in social interactions, Facebook can 'tollgate' every interaction between business and customer. Access to a user's wall is restricted to those who pay, and to product advocates (the friends list). Similarly, maps exploit location, search engines discovery, and news portals the thirst for timely, topical information.

Some simple technical impracticalities associated with mobile also stand in our way. Technologies such as WebRTC, for example, (used for browser-to-browser interworking) are not yet robust enough for use in a mobile environment. Without these, we are forced towards proprietary solutions, which kinda sucks for your 'willfully open-source' type of initiative.

Free Flight

Assuming we can get past these hurdles and have a means of aggregating domain-relevant visualisations, there is the question of how much information is actually needed, what can be comfortably made available within limited screen real estate, and how well it supports person-to-person interaction (without some form of connectivity with others, nothing holds it's fascination for long).

Let's assume a parallel video chat session, then focus on the supporting tools..

Information Overload

There comes a point on a small device where a large range of menu choices can get out of hand. Same for the number of models or tools ultimately displayed.

The watershed is a combination of needs, ergonomics, device size and common sense, perhaps a choice best left to the user.

Nevertheless, there are a couple of other issues.

Musical Immersion

With freedom of choice and enabling technologies comes the notion of multi-sensory and multi-model learning: the ability, for example, to forge leaps of musical understanding by networking concrete instrument models with abstract theoretical models, and seeing -very plainly- the mappings between the two.

The value of this should not be underestimated. With complete control over which facets of a musical culture (music, instruments, history, tools, genres and so on) should be shown, the user can establish an own, preferred ecosystem.

With visual control and focus in the user's hands, we have the basis for immersion.

Musical Immediacy

Deep musical understanding is often a process of assimilation: leaps of intuition or insight, spread over many years of practice and play.

The underlying information is widely available but static (cannot be directly related to the music at hand), spread over countless websites, videos and academic texts, and therefore tedious to access and apply.

Directly associating a musical score with instrument models and theory tools, theory, instrument and music are intimately linked. This represents immediacy.

Music Visualization

There is a user experience / GUI designer wisdom to the effect 'don't make the user think', which suggests that nothing short of mind-reading will lead to learning success.

Well, mind-reading may have to wait. Our next best shot? Direct, visual learning. Our visual processing is blisteringly fast, and appears to engage cognition far better than text.

Multi-Sensory Feedback

Tonal Lattice (Tonnetz)

Visual modelling allows us to approach a problem from a variety of viewpoints.

Clearly, not all senses can be usefully engaged for music learning, but combined with tactile and audio inputs, visualisations quickly draw us into the realm of immersive learning.

As immersive learning is better understood and exploited, we may see dramatic leaps of learning effectiveness.

Multi-Model Music Visualization

Any specific piece of music may find visual expression in a variety of fully interactive contexts: for example, a linear interval map, circle of fifths, spiral, double helix, lattice or tonnetz, torus or even 3D modal space. Similarly, cross-cultural musical understanding may, depending on needs, best be served by any of instrument fingering roadmaps, frequency waveforms, 3D matrices (music-cultural landscapes) or any of many other models.

Depending on their characteristics, each of these models may support a specific kind of insight or understanding particularly well, or have particular subjective appeal. Two central issues in this context are:

• freedom of choice in configuring an own, personalised environment.
• sufficient resources to facilitate a good overview and easy navigation (compared to desktops, mobile devices still have rather skimpy processors, batteries and RAM).

Both seem to me arguments for a bold, platform-scale solution.

Musical Homogeneity

With multiple models arises the question of how to ensure colouring or naming conventions are consistently applied across multiple models. This is also closely connected to development overhead: homogeneity is much more difficult to control across a range of fragmented, standalone apps than across a single, integrated platform.

Connectivity

Where there is choice (personalisation), there is diversity, synergy and communication. A cloud- and browser-based solution is well placed both to leverage and offer these, through public web APIs.

Musical Sharing

Here, perhaps, the real crux of the matter, at least in the long run. A fully configured, personal instrument model and tool display can be passed to a student as a whole. The entire learning context is shared.

This has profound implications for the quality and ease of teaching. In the case of larger P2P classes, teacher and learners are on the same page.

We need to distinguish this form of 'on the same page' from that of hardcopy publishing 'on the same page'. The latter ensures that sometimes hundreds of thousands of learners learn from the same material - a sure route to monoculture. Under P2P learning, however, teacher and immediate students being on the same page can be seen as some guarantee of diversity.

Music Visualisation Aggregator Platform

Alone the idea of a multi-model, immersive environment implies something far beyond the dimensions of the current mobile app. Thoughts are inevitably drawn in the direction of aggregation, and hence 'platform'. With this arise wider issues of context, networking, ideas exchange and, inevitably, synergies.

The App is not Dead. Yet.

Clearly, however, the iOS or Android app is not going to go away overnight, and though -for all but the lucky few developers- a flawed model, it may provide an outlet for a limited range of applications derived from those of the 'aggregator platform', with size-limited window onto their driver scores. Central to this is an elegant, 'responsive' degradation across increasingly smaller devices, arriving at a point where only a tiny excerpt of score is visible, along with at most one or two models.

Mobile Hardware Limitations

Should mobile browser apps remain unable to leverage device-specific functions such as camera or hardware sensors, the arguments in favour of a browser-based aggregator platform approach would only be strengthened.

There are, however, initiatives afoot to provide just such access on mobile devices. Currently, the strongest arguments appear to be in favour of so-called hybrid apps, which attempt to combine the strengths of the browser stack with those of mobile devices.

---

Keywords

---

online music learning, online music lessons	distance music learning, distance music lessons	remote music lessons, remote music learning
p2p music lessons, p2p music learning	music visualisation music visualization	musical instrument models interactive music instrument models
music theory tools musical theory	p2p music interworking p2p musical interworking	comparative musicology ethnomusicology
world music international music	folk music traditional music	P2P musical interworking, Peer-to-peer musical interworking
WebGL, Web3D, WebVR, WebAR	Virtual Reality, Augmented or Mixed Reality	Artificial Intelligence, Machine Learning
Scalar Vector Graphics, SVG	3D Cascading Style Sheets, CSS3D	X3Dom, XML3D