Modeling a game with melody recognition and machine training using ClojureScript and real instruments

Being great at a first-person shooter game requires a lot of hand-eye coordination and skill with the mouse. Being great at an MMORPG requires a working knowledge of a vast variety of keyboard shortcuts. Being great at a fighting game requires speed and skill with the gamepad.

The underlying philosophy behind this experiment is to showcase that you can potentially take the input to any stringed instrument (with a pickup) lying around in your house, to use to design a game that requires as much skill (if not more) as playing any other game with any other kind of input device. By treating the instrument as a game controller, and moving the onus of deriving quality from the playing into the system, we can harness some of the basic qualities – speed and melody – of the instrument to make our game more interesting and a lot more unique and fun.

The idea is to go as far away from being a Guitar Hero derivative as possible, and closer to a system that treats it as just another input device. The game should not be about playing guitars or any kind of music, but should be about playing any other regular game. Unlike other Musical Instrument Games™, we want ours to work for anyone, anywhere. So we're going to use a simple semi-electric guitar plugged into the microphone line-in and demonstrate this on the browser using the Web Audio Interface.

We'll go through a few papers and look at implementing some of the algorithms to pitch / note detect in near real-time, process the input to make sense of it, run it through a few simple constraints and algorithms to recognise what a melody might be like and eventually explore how we can make the system intelligent enough to train itself based on the player and the styles so that we can generate smarter and harder levels for our game.

 
5 favorite thumb_down thumb_up 2 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

Before we deep dive into the implementation, we're going to discuss music in general. The game is rather a secondary goal for this exercise and the outcome will serve merely as a demonstration of possibility.

Music Theory

We'll cover the basic building blocks of music from ground-up. Starting from,

  • Note
  • Tuning
  • Intervals
  • Temperaments
  • Scales
  • Chords

and how these apply to various traditions of music (western, contemporary, carnatic etc.)

Waves

We'll examine what fundamental frequencies, harmonics and the frequency-ratio characteristics (twelfth-root of 2 etc.) of these different styles are like.

Detection

We'll go through these papers and discuss some of the pros and cons of the each of these algorithms. We'll explore how the accuracy of each of them vary based on the kind of signals we receive. Polyphonic sounds for any proper guitar playing holds a lot of frequency fluctuations that are hard to deal with, so the robustness for pitch detection is still more or less an unsolved problem.

ClojureScript Wrapper

I've been working on a fork of the ClojureScript library called Hum. The original is still small and lacks a few APIs that we require, for e.g. AnalyserNode

Improvements

Even the high-end hardware at this point fails to properly do chords. As you can design chord notes to hit directly on the harmonics of the fundamental frequency. We'll talk about some simple solutions that can potentially make our output better, for e.g.: putting six piezos on each string of the guitar that can send MIDI data.

Game

We'll devise a simple Fleuch-esque game where a space rover can shoot in a 360° arc around itself to destroy incoming droids. The speed and attack radius of this rover is controlled by how melodic the player plays. Here's a simple shot of what this would look like:



Training

We can back this up with a persistent connection to a server that takes all this frequency data to bucket our notes into trying to determine what a good melody might be. This is obviously a very non-deterministic problem, and an objectively correct solution is impossible.

We could start by bucketing our melody engine into good, reasonable and bad templates. Each of these buckets would have a set of constraints that we would run our notes through.

  • Pre-feed frequency data from popular music for comparisons.
  • Playing a series of notes in a certain window frame that usually sound good (based on some simple rules of musical scale)
  • Not let the player beat the game by merely playing the same nice-sounding notes / scales over and over again

Learning Outcome

ClojureScript, Music Theory, Web Audio API

Target Audience

Clojure / ClojureScript enthusiasts, music tweaking, DIY games

schedule Submitted 2 years ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Tejas Dinkar
    By Tejas Dinkar  ~  2 years ago
    reply Reply

    I spoke to the proposer. He's ok turning this into a workshop-cum-talk. Maybe people can bring their own instruments and play around as well

    • Akshay Gupta
      By Akshay Gupta  ~  2 years ago
      reply Reply

      Yup, shouldn't change much in terms of structure. More or less like a talk/demonstration still, but it can be hands-on – people can participate in the demo and contribute opinions, we can change things around as it would be a fairly malleable system. Should work better as a hour-long workshop-cum-talk type thing, as Tejas said.


  • Liked Morten Kromberg
    keyboard_arrow_down

    Morten Kromberg - Array Oriented Functional Programming with Dyalog

    Morten Kromberg
    Morten Kromberg
    CXO
    Dyalog
    Jay Foad
    Jay Foad
    Developer
    Dyalog Ltd
    schedule 2 years ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    Dyalog is a modern, array-first, multi-paradigm programming language, which supports functional, object-oriented and imperative programming based on an APL language kernel. During the morning and early afternoon, Dyalog's functional language guru John Scholes will introduce the fundamentals of functional programming in APL. At the end of the day, Dyalog CTO Morten Kromberg will round off with an application development session, showing how a simple web application with a HTML5/JS front-end and a RESTful web service can be used to deliver the power of APL to end users, or as embeddable components for other application tools. The "hands on" sections of the workshop can be followed under Mac OS X, Linux, or Microsoft Windows.

  • Liked Abhinav Sarkar
    keyboard_arrow_down

    Abhinav Sarkar - Introduction to Concurrency in Haskell

    45 mins
    Demonstration
    Intermediate

    Haskell is widely known as a functional programming language which is great to write pure (non I/O) code and immutable data structures. But it is not well known that it has a great support for a wide variety of concurrency models too.

    In this talk we shall explore the various concurrency models and primitives supported by Haskell including green threads, mutable variables, channels and software transactional memory. We shall see how the purity, laziness, and the strong type system of Haskell come together, and help write clean and modular concurrent code. We shall explore these by writing a mutli-user chat server with support for private and channel chatting.

     

  • Liked Tanmai Gopal
    keyboard_arrow_down

    Tanmai Gopal - Haskell In Production

    Tanmai Gopal
    Tanmai Gopal
    Engineering Head, Founder
    34 Cross
    schedule 2 years ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    From the perspective of one of the few companies in the country using Haskell in production (hasura.io), we will talk about the journey of using Haskell as a programming language to build and maintain real world software.
    From the burden of choosing a language that probably has a single digit number of active developers in the country, to
    the joys of a small team churning out code that just keeps on working.

    We'll go over how we created a team, organized work ("software development practices"), architecting code, tooling up, testing, performance tweaking and finally deploying a Haskell codebase. We hope that Haskell developers with some prior experience will be able to take away
    something new about the Haskell ecosystem, and those interested in just functional programming will see tangible value in how certain features unique to Haskell have an impact.

    Expect a case study, a review of some cutting-edge tools and how to use them, and some Haskell code samples that will make you grunt in geeky approval.

  • Liked Ravi Mohan
    keyboard_arrow_down

    Ravi Mohan - Building a General Game Playing Engine with OCaml and Erlang

    Ravi Mohan
    Ravi Mohan
    CEO
    AxiomChoice
    schedule 2 years ago
    Sold Out!
    90 mins
    Experience Report
    Beginner

    "General Game Playing  is the design of artificial intelligence programs that play more than one game"  Wikipedia [1]

    In other words, one program has to be able to play (and win!) multiple games ( Chess, checkers, Go, Othello etc).

    Summary: This experience report is about my  "Forever Project" [2,3,4] to build such a system in OCaml, and the problems (theoretical and practical) and will include a demo of the program, warts and all.

    Detail: GGP is an active CS research area [5]Annual competitions are held every year where  ggp programs compete against each other[6], playing games which they have never seen before (the rules are supplied as an input file for the system to consume five minutes before the match begins).

    Over the last year, in my non-existent spare time, I've been building a GGP engine to compete in a GGP competition.

    Most such engines are written in Java, C++  etc.. I'm using OCaml for the game playing agent, and Manoj is building the backend in Erlang.

    As mentioned above this is our "Forever Project"  We've made some decent progress.

    Come, see, laugh, jeer!

  • Liked Nivedita Priyadarshini
    keyboard_arrow_down

    Nivedita Priyadarshini - Building an Experimentation Platform in Clojure

    45 mins
    Experience Report
    Intermediate

    Here's a markdown version of this proposal: https://gist.github.com/ssrihari/1cad915e7ef22ce61b54

    Over the last year and half at Staples SparX, we built a multivariate testing platform as a service. It satisfies an SLA of 10ms at 99.9th percentile, services all of Staples' experimentation from a single machine, is simulation tested, and is written in Clojure.

    We'll give an introduction to the Experimentation domain, design of experiments and our battle in attaining statistical significance with constrained traffic. We will share our experiences in loading and reporting over months of data in Datomic, using Clojure to grow a resilient postgres cluster, using a homegrown jdbc driver, interesting anecdotes, and OLAP solutions with ETL built in Clojure using core.async. Expect to see references to google white papers, latency and network graphs, histograms, comparison tables and an eyeful of clojure code.

  • Liked Srihari Sriraman
    keyboard_arrow_down

    Srihari Sriraman - Carnatic music synthesis in Clojure

    Srihari Sriraman
    Srihari Sriraman
    Partner, Engineer
    nilenso
    schedule 2 years ago
    Sold Out!
    45 mins
    Demonstration
    Beginner

    Here's a markdown version of this proposal: https://gist.github.com/ssrihari/b25ddda331ec220663de

    I'm building a carnatic music (south indian classical music) synthesizer using Clojure and Overtone. Here's the library I'm building.

    Carnatic music is different fundamentally in the movements of pitch ('Gamakams'), and needs to be modelled differently than most MIDI music out there today. The library now, can understand prescriptive carnatic music notation, build finer abstractions from that, and play back a not-so-terrible synth version with a plethora of configurable inputs.

    The concept of 'Gamakam' (the transitions between notes) is central, so I'll go into it's guts and show how the library piggybacks on recent work on PASR (pitch-attack-sustain-release) and Dual-PASR ('stage' and 'dance' PASR components) transcriptions. I'll also put forth a path I see for fully synthetic carnatic music based on machine learning 'Ragams' (melodic modes).

    I'll give a detailed demo of the synthetic music on the repl through the talk, and do a live voice vs synth comparison in the end.