schedule 04:15 PM - 05:00 PM place Grand Ballroom 2

Here's a markdown version of this proposal: https://gist.github.com/ssrihari/1cad915e7ef22ce61b54

Over the last year and half at Staples SparX, we built a multivariate testing platform as a service. It satisfies an SLA of 10ms at 99.9th percentile, services all of Staples' experimentation from a single machine, is simulation tested, and is written in Clojure.

We'll give an introduction to the Experimentation domain, design of experiments and our battle in attaining statistical significance with constrained traffic. We will share our experiences in loading and reporting over months of data in Datomic, using Clojure to grow a resilient postgres cluster, using a homegrown jdbc driver, interesting anecdotes, and OLAP solutions with ETL built in Clojure using core.async. Expect to see references to google white papers, latency and network graphs, histograms, comparison tables and an eyeful of clojure code.

 
14 favorite thumb_down thumb_up 2 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

We'll be talking about these topics in detail:

Introduction to Experimentation [10 minutes]
Loading Datomic [5 minutes]
Reporting on Datomic [5 minutes]
A homegrown JDBC driver [5 minutes]
Simulation Testing [15 minutes]
Anecdotes [5 minutes]

Learning Outcome

We also have a more detailed version of this content, and each section here has links to relevant portions of that version.

Introduction to Experimentation

  • How do we provide statistically sound testing of hypotheses for complex multi-variable systems?
  • What are non-overlapping and overlapping experiments?
  • How do we tradeoff between precise measurement and splitting traffic between experiments?
  • How do 'Overlapping Experiments' work?

We'll get into the guts of these concepts because they are central to the service.

More: https://gist.github.com/nid90/5a5be8586b41949e811a#experimentation

Loading datomic

If you've tried to profile your application that uses datomic, chances are you wanted to load it with data first. There isn't first class support for this. So we wrote the plumbing to do this effectively. We'll go through how we approached this problem and discuss the solutions in code.

More: https://gist.github.com/nid90/5a5be8586b41949e811a#loading-datomic

Reporting on datomic

Unlike on an relational database, there aren't established practices on reporting on datomic. How to write optimized datalog query plans for large datasets? Does it make a difference if these are spread over many months? Do we use the datoms api? We'll discuss the plethora of solutions we tried and the one that worked for us.

More: https://gist.github.com/nid90/5a5be8586b41949e811a#reporting-on-datomic

A homegrown JDBC driver

We wrote our own database driver in clojure because clojure/jdbc wasn't fast enough for us. We'll explain why, give code samples of the driver, how we use it, and compare timings with clojure/jdbc.

More: https://gist.github.com/nid90/5a5be8586b41949e811a#a-homegrown-jdbc-driver

Simulation testing

We have a simulation testing tool written in Clojure using Datomic, Simulant and Causatum that runs various scenarios to test the integrity of the experimentation platform. This is probably an entire talk by itself. But again, it's a talk that has already been given by other team members ;) Without going into details of how we did simulation testing, we'll explain what tests we wrote and how it helped discover critical bugs in our domain logic.

More: https://gist.github.com/nid90/5a5be8586b41949e811a#simulation-testing

On postgres

Depending on time availability, we'll answer the following questions:

  • What were the compelling arguments for us to move to Postgres?
  • How were loading, querying, and reporting different?
  • What were the difficulties we faced in migrating our code to Postgres?

More: https://gist.github.com/nid90/5a5be8586b41949e811a#on-postgres

Anecdotes

The "Out Of Memory" story

The application would crash randomly with out-of-memory errors. Profiling it showed the datomic objects consuming most of the memory! We tried tweaking the GC config, the datomic config but to no avail. What really happened? Did we do something utterly stupid? Or is it something that any clojure dev can be bitten by? Thriller finish guaranteed.

More: https://gist.github.com/nid90/5a5be8586b41949e811a#the-out-of-memory-story

The weird network issue

We had odd looking spikes of 40ms, when our application was normally at 2ms or 3ms. A really weird latency graph. And TCP resets! We spent almost a week diagnosing the network issue which was basically around 10 lines of HTTP client code written in Clojure. What was wrong? We worked around the problem in the end, but it's a story worth sharing.

More: https://gist.github.com/nid90/5a5be8586b41949e811a#the-weird-network-issue

Other things we can talk about

  • How we built a postgres cluster using repmgr
  • How we reduced report times 30x by making a non-obvious change to a single clause
  • ETL written using core.async

More: https://gist.github.com/nid90/5a5be8586b41949e811a#other-things-we-can-talk-about

Target Audience

People interested in an experience report of using Clojure in production with low latency requirements.

schedule Submitted 2 years ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Naresh Jain
    By Naresh Jain  ~  2 years ago
    reply Reply

    Thanks Nivedita. This looks like a great experience report. Can you please update your proposal with a link to a video, where we can see your on-stage presentation skills? Thanks.

    • Nivedita Priyadarshini
      By Nivedita Priyadarshini  ~  2 years ago
      reply Reply

      Hey Naresh,

      Neither of us has given any talks before this. We both have given a couple lightning talks in RubyconfIndia Goa, RubyconfIndia Pune, GardenCityRubyConf, and we've both talked in the local ruby user group meetings, but I don't have a video link for any of those, and not sure if those really count.


  • Liked Morten Kromberg
    keyboard_arrow_down

    Morten Kromberg - Parallel Programming in Dyalog using Futures and Isolates

    Morten Kromberg
    Morten Kromberg
    CXO
    Dyalog
    schedule 2 years ago
    Sold Out!
    45 mins
    Keynote
    Beginner

    Dyalog is an array first multi-paradigm language based on an APL language kernel. In APL, it is common for the application of primitive functions to arrays of conforming shape to imply a map; the function is applied item-wise to corresponding elements of the arguments, producing a result of the same shape. APL also provides a variety of explicit operators that provide different mappings of user-defined functions, or more generally functional compositions, to multi-dimensional and nested arrays.

    One might expect that such language features would provide a complete framework for parallel programming. In practice, writing programs that can be efficiently executed in parallel on current hardware is extremely difficult, if implicit or explicit maps are the only parallel language feature available and an interpreter or compiler needs to deduce where to introduce the asynchronicity that is required in order to take advantage of the hardware.

    This talk will present Futures and Isolates, and related parallel operators constructed upon them, which were added to Dyalog version 14.0 in 2014. The goal has been to provide explicit forms of deterministic asynchronous execution, designed to be very familiar to APL users, and continue to allow them to view the language as an executable mathematical notation, while taking advantage of parallel hardware that is now readily available.

  • Liked Morten Kromberg
    keyboard_arrow_down

    Morten Kromberg - Array Oriented Functional Programming with Dyalog

    Morten Kromberg
    Morten Kromberg
    CXO
    Dyalog
    Jay Foad
    Jay Foad
    Developer
    Dyalog Ltd
    schedule 2 years ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    Dyalog is a modern, array-first, multi-paradigm programming language, which supports functional, object-oriented and imperative programming based on an APL language kernel. During the morning and early afternoon, Dyalog's functional language guru John Scholes will introduce the fundamentals of functional programming in APL. At the end of the day, Dyalog CTO Morten Kromberg will round off with an application development session, showing how a simple web application with a HTML5/JS front-end and a RESTful web service can be used to deliver the power of APL to end users, or as embeddable components for other application tools. The "hands on" sections of the workshop can be followed under Mac OS X, Linux, or Microsoft Windows.

  • Liked Abhinav Sarkar
    keyboard_arrow_down

    Abhinav Sarkar - Introduction to Concurrency in Haskell

    45 mins
    Demonstration
    Intermediate

    Haskell is widely known as a functional programming language which is great to write pure (non I/O) code and immutable data structures. But it is not well known that it has a great support for a wide variety of concurrency models too.

    In this talk we shall explore the various concurrency models and primitives supported by Haskell including green threads, mutable variables, channels and software transactional memory. We shall see how the purity, laziness, and the strong type system of Haskell come together, and help write clean and modular concurrent code. We shall explore these by writing a mutli-user chat server with support for private and channel chatting.

     

  • Liked Shantanu Kumar
    keyboard_arrow_down

    Shantanu Kumar - Production Clojure: An Experience Report

    Shantanu Kumar
    Shantanu Kumar
    Principal Engineer
    Concur
    schedule 2 years ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    This talk is about our experience of using Clojure in production at Concur for building a SaaS platform.

    To set the perspective, Concur is a 22 years old, USD 8.3 billion enterprise, now a part of SAP. More than 70% of Fortune 500 companies use Concur for travel and expense solutions. Concur's SaaS platform serves 25,000 customers with 25 million users. The Concur SaaS platform was reaching its limits and needed "re-architecting". This talk is about why we chose Clojure at Concur and how we are building one of the world's largest B2B SaaS platforms in Clojure.

    We will discuss the practices we established, experiments we did and the lessons learnt. We have now built a platform that follows open standards, is rigorously tested, tolerates failure and performs well. We will go over how we reached there, how we deployed the code to production and how we monitor those today.

    If you are considering using Clojure in production or even simply interested in Clojure, you may get useful insights to take away from this talk.

  • Liked Tanmai Gopal
    keyboard_arrow_down

    Tanmai Gopal - Haskell In Production

    Tanmai Gopal
    Tanmai Gopal
    Engineering Head, Founder
    34 Cross
    schedule 2 years ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    From the perspective of one of the few companies in the country using Haskell in production (hasura.io), we will talk about the journey of using Haskell as a programming language to build and maintain real world software.
    From the burden of choosing a language that probably has a single digit number of active developers in the country, to
    the joys of a small team churning out code that just keeps on working.

    We'll go over how we created a team, organized work ("software development practices"), architecting code, tooling up, testing, performance tweaking and finally deploying a Haskell codebase. We hope that Haskell developers with some prior experience will be able to take away
    something new about the Haskell ecosystem, and those interested in just functional programming will see tangible value in how certain features unique to Haskell have an impact.

    Expect a case study, a review of some cutting-edge tools and how to use them, and some Haskell code samples that will make you grunt in geeky approval.

  • Liked Akshay Gupta
    keyboard_arrow_down

    Akshay Gupta - Modeling a game with melody recognition and machine training using ClojureScript and real instruments

    Akshay Gupta
    Akshay Gupta
    Ingénieur
    Nilenso Software LLP
    schedule 2 years ago
    Sold Out!
    45 mins
    Demonstration
    Beginner

    Being great at a first-person shooter game requires a lot of hand-eye coordination and skill with the mouse. Being great at an MMORPG requires a working knowledge of a vast variety of keyboard shortcuts. Being great at a fighting game requires speed and skill with the gamepad.

    The underlying philosophy behind this experiment is to showcase that you can potentially take the input to any stringed instrument (with a pickup) lying around in your house, to use to design a game that requires as much skill (if not more) as playing any other game with any other kind of input device. By treating the instrument as a game controller, and moving the onus of deriving quality from the playing into the system, we can harness some of the basic qualities – speed and melody – of the instrument to make our game more interesting and a lot more unique and fun.

    The idea is to go as far away from being a Guitar Hero derivative as possible, and closer to a system that treats it as just another input device. The game should not be about playing guitars or any kind of music, but should be about playing any other regular game. Unlike other Musical Instrument Games™, we want ours to work for anyone, anywhere. So we're going to use a simple semi-electric guitar plugged into the microphone line-in and demonstrate this on the browser using the Web Audio Interface.

    We'll go through a few papers and look at implementing some of the algorithms to pitch / note detect in near real-time, process the input to make sense of it, run it through a few simple constraints and algorithms to recognise what a melody might be like and eventually explore how we can make the system intelligent enough to train itself based on the player and the styles so that we can generate smarter and harder levels for our game.

  • Liked Mayank Jain
    keyboard_arrow_down

    Mayank Jain - Testing business logic using DSLs in Clojure

    Mayank Jain
    Mayank Jain
    Software Artisan
    Helpshift
    schedule 2 years ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    Testing business logic quickly becomes tricky, as applications grow and scale. Example based unit and integration tests, and exploratory tests become poor choices to check and verify a large state space. Also such methods are not  well-suited to clearly describe the state space / transitions we want to test. 

    Since we use Clojure to write test suites and tools at Helpshift, we tried experimenting with DSLs to express some of our testing problems. We've found that mini DSLs can indeed become useful to describe and test fairly complicated business logic.

    In this talk I will cover some of the mini-DSL approaches we've tried, demonstrate one of them by example, and discuss the benefits and drawbacks based on current experience.

  • Liked Viral B. Shah
    keyboard_arrow_down

    Viral B. Shah - Why Julia?

    60 mins
    Keynote
    Beginner

    In this talk, I will talk about our motivation for creating Julia. Julia is a high-level, high-performance dynamic programming language. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. Julia’s Base library, largely written in Julia itself, also integrates mature, best-of-breed open source C and Fortran libraries for linear algebra, random number generation, signal processing, and string processing. In addition, the Julia developer community is contributing a number of external packages through Julia’s built-in package manager at a rapid pace. This is why Julia is seeing rapid adoption in universities for teaching and research, as well as in businesses.

    I will discuss what makes Julia fast. Julia's ability to combine these levels of performance and productivity in a single language stems from the choice of a number of features that work well with each other:

    1. An expressive parametric type system, allowing optional type annotations;
    2. Multiple dispatch using those types to select implementations;
    3. A dynamic dataflow type inference algorithm allowing types of most expressions to be inferred;
    4. Aggressive code specialization against run-time types;
    5. Metaprogramming;
    6. Just-In-Time compilation using the LLVM compiler framework; and
    7. Careful design of the language and standard library to be amenable to type analysis;

    I will also touch upon how the language design has made it possible to explore many parallel programming models in Julia, completely within the language.

    See: http://www.julialang.org/

  • Liked Shashi Gowda
    keyboard_arrow_down

    Shashi Gowda - Julia - A Lisp for Fast Number Crunching

    240 mins
    Workshop
    Beginner

    Julia is a programming language for data science and numerical computing. Julia is a high-level, high-performance dynamic language, it uses just-in-time compilation to provide fast machine code - dynamic code runs at about half the speed of C, and orders of magnitude faster than traditional numerical computing tools.

    Julia borrows two main ideas from Lisp lore:

    1. Multiple-dispatch: a method is identified by a name and the types of all of its arguments (traditional OOP only uses the type of a single argument) multiple-dispatch allows for a natural programming model suitable for mathematics and general purpose programming. Interestingly, the same feature is responsible for Julia's ability to generate really fast machine code.
    2. Macros: Julia has syntax that is familiar to users of other technical computing environments, however, Julia expressions are homoiconic -- Julia code can be represented as Julia data structures, and transformed from one form to another via hygienic macros. There are some very interesting uses of macros to create domain-specific languages for effective programming. 

    (I will be presenting this workshop along with Viral B Shah - co-creator of Julia)

  • Liked John Thomas
    keyboard_arrow_down

    John Thomas - A Introduction to Functional Design Patterns in scala

    90 mins
    Workshop
    Beginner

    First notorious word, every functional programming encounter (without mathematical background ) during initial days  is "Category theory " . Our aim is to introduce the very minimum concept a functional developer should know in category theorem .

    The goal is to introduce basics of category theory and develop a small prototype library to validate and process data  , out a few principles from category theory.

    We will also be introducing and implementing representations of Functors, SemiGroup, Applicative, Monoid and Monad. We also make use of some interesting Scala features like ADT - Algebraic data type , Implicit conversion, Types . 

     

     

     

  • Liked Ravi Mohan
    keyboard_arrow_down

    Ravi Mohan - Building a General Game Playing Engine with OCaml and Erlang

    Ravi Mohan
    Ravi Mohan
    CEO
    AxiomChoice
    schedule 2 years ago
    Sold Out!
    90 mins
    Experience Report
    Beginner

    "General Game Playing  is the design of artificial intelligence programs that play more than one game"  Wikipedia [1]

    In other words, one program has to be able to play (and win!) multiple games ( Chess, checkers, Go, Othello etc).

    Summary: This experience report is about my  "Forever Project" [2,3,4] to build such a system in OCaml, and the problems (theoretical and practical) and will include a demo of the program, warts and all.

    Detail: GGP is an active CS research area [5]Annual competitions are held every year where  ggp programs compete against each other[6], playing games which they have never seen before (the rules are supplied as an input file for the system to consume five minutes before the match begins).

    Over the last year, in my non-existent spare time, I've been building a GGP engine to compete in a GGP competition.

    Most such engines are written in Java, C++  etc.. I'm using OCaml for the game playing agent, and Manoj is building the backend in Erlang.

    As mentioned above this is our "Forever Project"  We've made some decent progress.

    Come, see, laugh, jeer!

  • Liked Srihari Sriraman
    keyboard_arrow_down

    Srihari Sriraman - Carnatic music synthesis in Clojure

    Srihari Sriraman
    Srihari Sriraman
    Partner, Engineer
    nilenso
    schedule 2 years ago
    Sold Out!
    45 mins
    Demonstration
    Beginner

    Here's a markdown version of this proposal: https://gist.github.com/ssrihari/b25ddda331ec220663de

    I'm building a carnatic music (south indian classical music) synthesizer using Clojure and Overtone. Here's the library I'm building.

    Carnatic music is different fundamentally in the movements of pitch ('Gamakams'), and needs to be modelled differently than most MIDI music out there today. The library now, can understand prescriptive carnatic music notation, build finer abstractions from that, and play back a not-so-terrible synth version with a plethora of configurable inputs.

    The concept of 'Gamakam' (the transitions between notes) is central, so I'll go into it's guts and show how the library piggybacks on recent work on PASR (pitch-attack-sustain-release) and Dual-PASR ('stage' and 'dance' PASR components) transcriptions. I'll also put forth a path I see for fully synthetic carnatic music based on machine learning 'Ragams' (melodic modes).

    I'll give a detailed demo of the synthetic music on the repl through the talk, and do a live voice vs synth comparison in the end.