Production Clojure: An Experience Report

This talk is about our experience of using Clojure in production at Concur for building a SaaS platform.

To set the perspective, Concur is a 22 years old, USD 8.3 billion enterprise, now a part of SAP. More than 70% of Fortune 500 companies use Concur for travel and expense solutions. Concur's SaaS platform serves 25,000 customers with 25 million users. The Concur SaaS platform was reaching its limits and needed "re-architecting". This talk is about why we chose Clojure at Concur and how we are building one of the world's largest B2B SaaS platforms in Clojure.

We will discuss the practices we established, experiments we did and the lessons learnt. We have now built a platform that follows open standards, is rigorously tested, tolerates failure and performs well. We will go over how we reached there, how we deployed the code to production and how we monitor those today.

If you are considering using Clojure in production or even simply interested in Clojure, you may get useful insights to take away from this talk.

 
11 favorite thumb_down thumb_up 2 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

A brief overview of the premise, followed by an account of what problems we set out to solve, what we tried and what we chose to fix the problem. We cover all stages - from choosing Clojure, developing code, testing and profiling/performance tuning to deploying code in production, and everything in the middle.

We will cover the tools, libraries and infrastructure we used for various jobs, why we chose those and share screenshots/code snippets for illustration. We will also cover how we iterated within the team to arrive at cleaner abstractions and libraries.

Learning Outcome

People will see an example of how to take Clojure to production, with information on the aspects involved. Beginner and intermediate level Clojure enthusiasts will learn about the factors/tools/libraries to consider for taking Clojure code to production.

Non-engineering managers/people can understand the high level stakes in adopting Clojure.

Target Audience

People interested in taking Clojure to production

schedule Submitted 2 years ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Naresh Jain
    By Naresh Jain  ~  2 years ago
    reply Reply

    Thanks Shantanu. This looks like a great experience report. Can you please update your proposal with a link to a video, where we can see your on-stage presentation skills? Thanks.

    • Shantanu Kumar
      By Shantanu Kumar  ~  2 years ago
      reply Reply

      Hi Naresh, I do not have any recorded talk to share. I have spoken several times at tech meetups, however, none of those were recorded.


  • Liked Morten Kromberg
    keyboard_arrow_down

    Parallel Programming in Dyalog using Futures and Isolates

    Morten Kromberg
    Morten Kromberg
    schedule 2 years ago
    Sold Out!
    45 mins
    Keynote
    Beginner

    Dyalog is an array first multi-paradigm language based on an APL language kernel. In APL, it is common for the application of primitive functions to arrays of conforming shape to imply a map; the function is applied item-wise to corresponding elements of the arguments, producing a result of the same shape. APL also provides a variety of explicit operators that provide different mappings of user-defined functions, or more generally functional compositions, to multi-dimensional and nested arrays.

    One might expect that such language features would provide a complete framework for parallel programming. In practice, writing programs that can be efficiently executed in parallel on current hardware is extremely difficult, if implicit or explicit maps are the only parallel language feature available and an interpreter or compiler needs to deduce where to introduce the asynchronicity that is required in order to take advantage of the hardware.

    This talk will present Futures and Isolates, and related parallel operators constructed upon them, which were added to Dyalog version 14.0 in 2014. The goal has been to provide explicit forms of deterministic asynchronous execution, designed to be very familiar to APL users, and continue to allow them to view the language as an executable mathematical notation, while taking advantage of parallel hardware that is now readily available.

  • Liked Morten Kromberg
    keyboard_arrow_down

    Array Oriented Functional Programming with Dyalog

    Morten Kromberg
    Morten Kromberg
    Jay Foad
    Jay Foad
    schedule 2 years ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    Dyalog is a modern, array-first, multi-paradigm programming language, which supports functional, object-oriented and imperative programming based on an APL language kernel. During the morning and early afternoon, Dyalog's functional language guru John Scholes will introduce the fundamentals of functional programming in APL. At the end of the day, Dyalog CTO Morten Kromberg will round off with an application development session, showing how a simple web application with a HTML5/JS front-end and a RESTful web service can be used to deliver the power of APL to end users, or as embeddable components for other application tools. The "hands on" sections of the workshop can be followed under Mac OS X, Linux, or Microsoft Windows.

  • Tanmai Gopal
    Tanmai Gopal
    schedule 2 years ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    From the perspective of one of the few companies in the country using Haskell in production (hasura.io), we will talk about the journey of using Haskell as a programming language to build and maintain real world software.
    From the burden of choosing a language that probably has a single digit number of active developers in the country, to
    the joys of a small team churning out code that just keeps on working.

    We'll go over how we created a team, organized work ("software development practices"), architecting code, tooling up, testing, performance tweaking and finally deploying a Haskell codebase. We hope that Haskell developers with some prior experience will be able to take away
    something new about the Haskell ecosystem, and those interested in just functional programming will see tangible value in how certain features unique to Haskell have an impact.

    Expect a case study, a review of some cutting-edge tools and how to use them, and some Haskell code samples that will make you grunt in geeky approval.

  • Mayank Jain
    Mayank Jain
    schedule 2 years ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    Testing business logic quickly becomes tricky, as applications grow and scale. Example based unit and integration tests, and exploratory tests become poor choices to check and verify a large state space. Also such methods are not  well-suited to clearly describe the state space / transitions we want to test. 

    Since we use Clojure to write test suites and tools at Helpshift, we tried experimenting with DSLs to express some of our testing problems. We've found that mini DSLs can indeed become useful to describe and test fairly complicated business logic.

    In this talk I will cover some of the mini-DSL approaches we've tried, demonstrate one of them by example, and discuss the benefits and drawbacks based on current experience.

  • Viral B. Shah
    Viral B. Shah
    schedule 2 years ago
    Sold Out!
    60 mins
    Keynote
    Beginner

    In this talk, I will talk about our motivation for creating Julia. Julia is a high-level, high-performance dynamic programming language. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. Julia’s Base library, largely written in Julia itself, also integrates mature, best-of-breed open source C and Fortran libraries for linear algebra, random number generation, signal processing, and string processing. In addition, the Julia developer community is contributing a number of external packages through Julia’s built-in package manager at a rapid pace. This is why Julia is seeing rapid adoption in universities for teaching and research, as well as in businesses.

    I will discuss what makes Julia fast. Julia's ability to combine these levels of performance and productivity in a single language stems from the choice of a number of features that work well with each other:

    1. An expressive parametric type system, allowing optional type annotations;
    2. Multiple dispatch using those types to select implementations;
    3. A dynamic dataflow type inference algorithm allowing types of most expressions to be inferred;
    4. Aggressive code specialization against run-time types;
    5. Metaprogramming;
    6. Just-In-Time compilation using the LLVM compiler framework; and
    7. Careful design of the language and standard library to be amenable to type analysis;

    I will also touch upon how the language design has made it possible to explore many parallel programming models in Julia, completely within the language.

    See: http://www.julialang.org/

  • Liked Shashi Gowda
    keyboard_arrow_down

    Julia - A Lisp for Fast Number Crunching

    Shashi Gowda
    Shashi Gowda
    Viral B. Shah
    Viral B. Shah
    schedule 2 years ago
    Sold Out!
    240 mins
    Workshop
    Beginner

    Julia is a programming language for data science and numerical computing. Julia is a high-level, high-performance dynamic language, it uses just-in-time compilation to provide fast machine code - dynamic code runs at about half the speed of C, and orders of magnitude faster than traditional numerical computing tools.

    Julia borrows two main ideas from Lisp lore:

    1. Multiple-dispatch: a method is identified by a name and the types of all of its arguments (traditional OOP only uses the type of a single argument) multiple-dispatch allows for a natural programming model suitable for mathematics and general purpose programming. Interestingly, the same feature is responsible for Julia's ability to generate really fast machine code.
    2. Macros: Julia has syntax that is familiar to users of other technical computing environments, however, Julia expressions are homoiconic -- Julia code can be represented as Julia data structures, and transformed from one form to another via hygienic macros. There are some very interesting uses of macros to create domain-specific languages for effective programming. 

    (I will be presenting this workshop along with Viral B Shah - co-creator of Julia)

  • Liked John Thomas
    keyboard_arrow_down

    A Introduction to Functional Design Patterns in scala

    John Thomas
    John Thomas
    Shyam Sundar
    Shyam Sundar
    schedule 2 years ago
    Sold Out!
    90 mins
    Workshop
    Beginner

    First notorious word, every functional programming encounter (without mathematical background ) during initial days  is "Category theory " . Our aim is to introduce the very minimum concept a functional developer should know in category theorem .

    The goal is to introduce basics of category theory and develop a small prototype library to validate and process data  , out a few principles from category theory.

    We will also be introducing and implementing representations of Functors, SemiGroup, Applicative, Monoid and Monad. We also make use of some interesting Scala features like ADT - Algebraic data type , Implicit conversion, Types . 

     

     

     

  • Liked Ravi Mohan
    keyboard_arrow_down

    Building a General Game Playing Engine with OCaml and Erlang

    Ravi Mohan
    Ravi Mohan
    schedule 2 years ago
    Sold Out!
    90 mins
    Experience Report
    Beginner

    "General Game Playing  is the design of artificial intelligence programs that play more than one game"  Wikipedia [1]

    In other words, one program has to be able to play (and win!) multiple games ( Chess, checkers, Go, Othello etc).

    Summary: This experience report is about my  "Forever Project" [2,3,4] to build such a system in OCaml, and the problems (theoretical and practical) and will include a demo of the program, warts and all.

    Detail: GGP is an active CS research area [5]Annual competitions are held every year where  ggp programs compete against each other[6], playing games which they have never seen before (the rules are supplied as an input file for the system to consume five minutes before the match begins).

    Over the last year, in my non-existent spare time, I've been building a GGP engine to compete in a GGP competition.

    Most such engines are written in Java, C++  etc.. I'm using OCaml for the game playing agent, and Manoj is building the backend in Erlang.

    As mentioned above this is our "Forever Project"  We've made some decent progress.

    Come, see, laugh, jeer!

  • Nivedita Priyadarshini
    Nivedita Priyadarshini
    Srihari Sriraman
    Srihari Sriraman
    schedule 2 years ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    Here's a markdown version of this proposal: https://gist.github.com/ssrihari/1cad915e7ef22ce61b54

    Over the last year and half at Staples SparX, we built a multivariate testing platform as a service. It satisfies an SLA of 10ms at 99.9th percentile, services all of Staples' experimentation from a single machine, is simulation tested, and is written in Clojure.

    We'll give an introduction to the Experimentation domain, design of experiments and our battle in attaining statistical significance with constrained traffic. We will share our experiences in loading and reporting over months of data in Datomic, using Clojure to grow a resilient postgres cluster, using a homegrown jdbc driver, interesting anecdotes, and OLAP solutions with ETL built in Clojure using core.async. Expect to see references to google white papers, latency and network graphs, histograms, comparison tables and an eyeful of clojure code.

  • Liked Srihari Sriraman
    keyboard_arrow_down

    Carnatic music synthesis in Clojure

    Srihari Sriraman
    Srihari Sriraman
    schedule 2 years ago
    Sold Out!
    45 mins
    Demonstration
    Beginner

    Here's a markdown version of this proposal: https://gist.github.com/ssrihari/b25ddda331ec220663de

    I'm building a carnatic music (south indian classical music) synthesizer using Clojure and Overtone. Here's the library I'm building.

    Carnatic music is different fundamentally in the movements of pitch ('Gamakams'), and needs to be modelled differently than most MIDI music out there today. The library now, can understand prescriptive carnatic music notation, build finer abstractions from that, and play back a not-so-terrible synth version with a plethora of configurable inputs.

    The concept of 'Gamakam' (the transitions between notes) is central, so I'll go into it's guts and show how the library piggybacks on recent work on PASR (pitch-attack-sustain-release) and Dual-PASR ('stage' and 'dance' PASR components) transcriptions. I'll also put forth a path I see for fully synthetic carnatic music based on machine learning 'Ragams' (melodic modes).

    I'll give a detailed demo of the synthetic music on the repl through the talk, and do a live voice vs synth comparison in the end.

  • Liked Bhasker Kode
    keyboard_arrow_down

    Workshop: Erlang by example

    Bhasker Kode
    Bhasker Kode
    schedule 2 years ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    From fundamentals to deploying Erlang micro-service.