Realtime Distributed computing: dealing with Time and Failure in the wild

There is a growing need for scalable, realtime business systems that are continuously running and highly-available. Two very different frameworks/approaches you could use to build such systems are Storm and Akka.

Systems created with Storm or Akka are distributed, at runtime, on as many machines as you choose. The inherent concurrency implied by this brings the issues of State, Time and Failure into sharp focus. Functional programming has much to say about dealing with state and time; not surprisingly, both Storm and Akka have strong roots in functional languages (for Storm it is Clojure, and for Akka, Scala).

In this talk we'll explore the core concepts and challenges of distributed computation; the role of functional programming in concurrent distributed computing; we'll take a look at Storm and Akka, by example, and see that as different as these 2 approaches are, the underlying difficulties of distributed computation remains evident in both: dealing with time, and dealing with failure.

 
7 favorite thumb_down thumb_up 0 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

* Introduction to Distributed programming

* explore Storm/Akka, by example

* dealing with State, Time (sequencing) and Failure

* functional programming and concurrent distributed computation

Learning Outcome

* get an idea of the fundamentals of Distributed Systems and the challenges of programming in this paradigm

* become sensitised to the issues of State, Time, Failure

* learn about 2 diverse approaches: Storm (realtime, distributed stream processing), Akka (distributed actor based runtime)

* learn why and when to consider incurring the cost of writing Distributed code

* learn about the ways in which functional programming has been applied to concurrent, distributed systems

Target Audience

Developers, technology decision makers

schedule Submitted 2 years ago

Comments Subscribe to Comments

comment Comment on this Proposal

    • Liked Morten Kromberg
      keyboard_arrow_down

      Pragmatic Functional Programming using Dyalog

      Morten Kromberg
      Morten Kromberg
      schedule 2 years ago
      Sold Out!
      60 mins
      Demonstration
      Beginner

      APL is a member of the family of languages that are approaching middle age (Ken Iverson’s book titled “A Programming Language” was published in 1962). APL was very influential in the 60’s and 70’s, and widely used to deliver “end user computing” - but although the REPL, dynamic scope and lack of a type system endeared APL to domain experts, it also drew fire from computer scientists, most famously when Edsger Dijkstra declared that “APL is a mistake, carried through to perfection. It is the language of the future for the programming techniques of the past it creates a new generation of coding bums.”

      Dyalog is a modern, array-first, multi-paradigm programming language, which supports functional, object-oriented and imperative programming based on an APL language kernel. Dyalog allows people with good ideas – from bright high school students to PhDs – to contribute directly to the software development process using a notation which fits comfortably with those used in their own domains. Subject matter experts can write prototypes or, with suitable training and/or support, highly efficient, parallel and robust code that can be embedded in high-performance production applications.

    • Liked Debasish Ghosh
      keyboard_arrow_down

      Property based testing for functional domain models

      Debasish Ghosh
      Debasish Ghosh
      schedule 2 years ago
      Sold Out!
      45 mins
      Talk
      Intermediate

      Manual testing is something that's error prone, incomplete and impossible to replicate on a large scale. We have instead been using xUnit style of testing for quite some time now. This approach has a number of drawbacks like (a) We need to write test cases by hand which again doesn't scale for large systems (b) We may miss out some of the edge cases (c) Safeguarding missing cases with coverage metrics doesn't help, since metrics are mostly based on heuristics (d) maintaining test cases and test data is a real pain.

      In property based testing we write properties and not low level test cases. And let the system generate test cases which validate such properties. There are 2 main advantages with this approach:

      1. You think in terms of properties (or specifications) of the domain model which is the right granularity to think about
      2. You don't need to manage test cases, which is completely done by the system that generates a large collection of test data

      This approach is ideal for the functional programming paradigm, which focuses on pure functions. Using functional programming it's easier to reason about your model - hence it's easier to test functional programs using properties. In this talk I will take some real world examples of property validation and verification using scalacheck (the property based testing library for Scala) and a real world domain model.

    • Dhaval Dalal
      Dhaval Dalal
      Ryan Lemmer
      Ryan Lemmer
      schedule 2 years ago
      Sold Out!
      60 mins
      Demonstration
      Beginner

      In Indian classical music, we have Jugalbandi, where two lead musicians or vocalist engage in a playful competition. There is jugalbandi between Flutist and a Percussionist (say using Tabla as the instrument). Compositions rendered by flutist will be heard by the percussionist and will replay the same notes, but now on Tabla and vice-versa is also possible.

      In a similar way, we will perform Code Jugalbandi to see how the solution looks using different programming languages and paradigms.

      During the session, Dhaval and Ryan will take turns at coding the same problem using different languages and paradigms. There would be multiple such rounds during the Jugalbandi.

    • Mushtaq Ahmed
      Mushtaq Ahmed
      schedule 2 years ago
      Sold Out!
      60 mins
      Demonstration
      Intermediate

      Sync, Async, Blocking, Non-Blocking, Streaming are the buzzwords in the reactive programming world. This talk will attempt to attach some meaning to them. It will also demo the performance and resource consumption patterns for blocking-io, Scala Futures and RxJava Observables for comparable programs. Finally, a command line application that consumes twitter streams API will demo what is possible using the new reactive abstractions.

    • Liked Ramakrishnan Muthukrishnan
      keyboard_arrow_down

      An introduction to Continuation Passing Style (CPS)

      Ramakrishnan Muthukrishnan
      Ramakrishnan Muthukrishnan
      schedule 2 years ago
      Sold Out!
      60 mins
      Tutorial
      Intermediate

      Traditionally functions return some value. Someone is waiting for that value and does some computation with it. This "someone" is called the continuation of this value. In a normal functional call, the continuation is "implicit". In the "continuation passing style" (hence forth called with the short form, CPS), we make the continuations explicit. In this style, function definitions take an extra argument called "continuation" and it never return. The "return value" of the function 'continues' by passing this value as an argument to the continuation. Continuations are sometimes called "gotos with arguments".

      CPS is used as an intermediate stage while compiling a program since it makes the control structure of the program explicit and hence can be converted easily to machine code. Another feature of a CPS-transformed function is that it is tail-recursive even if the original function was not written in a tail-recursive style.

      Continuations enable a programmer to build new control operators (if the language's built-in operators does not already provide the control operators the programmer need).

    • Shashi Gowda
      Shashi Gowda
      schedule 3 years ago
      Sold Out!
      90 mins
      Tutorial
      Beginner

      Code used during the talk: https://github.com/shashi/fuconf-talk
      Slides: https://docs.google.com/presentation/d/16Xfqd-xU8y2JEN0TIcacDoYnp0b5-W7ESDB5v1SmcXs/edit#slide=id.p

       

      Elm is a strongly typed functional reactive programming (FRP) language that compiles to HTML, CSS, and Javascript. In Elm, the Signal type represents a time-varying value--things like mouse position, keys pressed, current time are signals. With Signals, one can write terse code that is isomorphic to a dataflow diagram of the app. The code hence feels natural and is 100% callback free. All this, with powerful type inference.

      This talk is an introduction to FRP. It explores functionally composing graphics and UIs, and creating interactions and animations with the Signal type. There will also be an overview of Elm’s execution mechanism and the time traveling debugger: a consequence of Elm's purely functional approach.

      While instructive, it will be good fun too, in the spirit of Elm.

    • Liked Vagmi Mudumbai
      keyboard_arrow_down

      Clojurescript and Om - Pragmatic functional programming in the Javascript Land

      Vagmi Mudumbai
      Vagmi Mudumbai
      schedule 3 years ago
      Sold Out!
      60 mins
      Demonstration
      Beginner

      Javascript programmers have had a lot of choices when it comes to programming. There were days of mootools, scriptaculous and jQuery and then there are now days of Angular, Ember, Knockout and the like. As a javascript programmer myself, I find that Clojurescript/React as Om offers a fresh perspective into building performant Javascript UIs that are easy to write.

      The talk will introduced concepts of React, immutable datastructures in Clojure and live code an application that demonstrates the concepts.

       

    • Liked Mohit Thatte
      keyboard_arrow_down

      Purely functional data structures demystified

      Mohit Thatte
      Mohit Thatte
      schedule 3 years ago
      Sold Out!
      45 mins
      Talk
      Beginner

      Immutable, persistent data structures form a big part of the value proposition of most functional programming languages.

      It is important to understand why these data structures are useful and how they make it easier to reason about your program. 

      It is also instructive to see how these data structures are implemented to get a greater appreciation for the inherent tradeoffs between performance and immutability.

      In this talk I will do a walkthrough of some of these data structures drawing from the work of Chris Okasaki[1], and attempt to explain the essential ideas in a simple way. 


      [1] Purely Functional Data Structures, Chris Okasaki, School of Computer Science, Carnegie Mellon University 

    • Liked Bhasker Kode
      keyboard_arrow_down

      Writing and improving tail recursive functions

      Bhasker Kode
      Bhasker Kode
      schedule 3 years ago
      Sold Out!
      45 mins
      Talk
      Beginner

      Brief history of recursion 

      Snippets from a few languages

      What is tail recursion?

      Design choices around recursion

      The importance of tail recursion in erlang

      How do you profile such improvements?

       

       

       

    • Liked Premanand Chandrasekaran
      keyboard_arrow_down

      Functional Programming in Java

      Premanand Chandrasekaran
      Premanand Chandrasekaran
      schedule 3 years ago
      Sold Out!
      60 mins
      Workshop
      Beginner

      Functional programming has started (re)gaining prominence in recent years, and with good reason too. Functional programs lend an elegant solution to the concurrency problem, result in more modular systems, are more concise and are easier to test. While modern languages like Scala and Clojure have embraced the functional style whole-heartedly, Java has lagged a bit behind in its treatment of functions as first-class citizens. With the advent of Java 8 and its support for lambdas, however, Java programmers can finally start reaping the power of functional programs as well. Even without Java 8, it is possible to adopt a functional style with the aid of excellent libraries such as Guava.

      This talk will explore how to apply functional concepts using the Java programming language and demonstrate how it can result in simpler, more elegant designs. We will conduct this in a hands-on workshop style with attendants being encouraged to code-along. So bring your favorite Java 8 aware IDE, an open mind and prepare to have a lot of fun.

    • Liked Venkat Subramaniam
      keyboard_arrow_down

      Haskell for Everyday Programmers

      Venkat Subramaniam
      Venkat Subramaniam
      schedule 2 years ago
      Sold Out!
      90 mins
      Talk
      Intermediate

      I learn different languages not to make use of them, but to program in my current languages in a better way. As we adapt functional style of programming in mainstream languages, like Java, C#, and C++, we can learn a great deal from a language that is touted as a purely functional language.

      Haskell is statically typed, but not in a way like Java, C#, or C++. Its static typing does not get in the way of productivity. Haskell quietly does lazy evaluation and enforces functional purity for greater good. Everyday programmers, like your humble speaker, who predominantly code in mainstream languages, can greatly benefit from learning the idioms and style of this elegant language. The next time we sit down to crank out some code in just about any language, we can make use of some of those styles, within the confines of the languages, and move towards a better, functional style.

    • Liked Kishore Nallan
      keyboard_arrow_down

      Applying functional programming principles to large scale data processing

      Kishore Nallan
      Kishore Nallan
      schedule 2 years ago
      Sold Out!
      45 mins
      Talk
      Intermediate
      At Indix, we deal with a stream of unstructured and constantly changing data. This data is processed through a series of systems before being fed as structured input to our analytics system. In this talk, I will walk through our experience of building a large scale data processing system using Hadoop that's focused on immutability, composition and other functional programming principles.
    • Liked Aditya Godbole
      keyboard_arrow_down

      Learning (from) Haskell - An experience report

      Aditya Godbole
      Aditya Godbole
      schedule 3 years ago
      Sold Out!
      45 mins
      Experience Report
      Beginner

      Functional programming as a programming style and discipline is useful even in languages which are not pure functional languages. By practising programming in a pure functional language like Haskell, programmers can drastically improve the quality of code when coding in other languages as well.

      The talk is based on first hand experience of using Haskell in internal courses in our organisation to improve code quality.

      This talk will cover Gofer (one of the earliest variants of Haskell) as a teaching tool, including the choice of the language, the features from Haskell that should (and shouldn't) be covered and the obstacles and benefits of the exercise.

       

    • Venkat Subramaniam
      Venkat Subramaniam
      schedule 2 years ago
      Sold Out!
      60 mins
      Keynote
      Intermediate

      It's been around for a long time, but everyone's talking about it all of a sudden. But why and why now? We've been

      programming in languages like Java for a while, quite well. Now we're asked to change and the languages themselves

      are changing towards this style of programming. In this keynote, a passionate polyglot programmer and author of

      "Functional Programming in Java: Harnessing the Power of Java 8 Lambda Expressions" will share the reasons

      we need to make the paradigm shift and the pure joy—the benefits—we will reap from it.

    • Liked Bodil Stokke
      keyboard_arrow_down

      Programming, Only Better

      Bodil Stokke
      Bodil Stokke
      schedule 2 years ago
      Sold Out!
      45 mins
      Talk
      Beginner

      The programming paradigms that served us so well through the 80s and 90s are failing us. Building systems the way we're used to building them always seems to end in the inevitable death march towards exponential complexity. But once you stop to ask the right question- "what's really causing all this complexity?" - you realise the answers have really been staring you in the face all along. Debugging is only hard when you can't reason about your code. Concurrency is only hard when you can't predict the state of your code. Reusability is only hard when your components aren't naturally composable.

      Fortunately, languages addressing these issues specifically are popping up all over the place. In many cases, it turns out we've had the solutions to our problems for a long time, we've just forgotten about them, or never really bothered to look. Let's take a moment to explore some of these languages, not as exercises in syntactic details, but looking at the inherent properties in their design that enable us to defy decades of OO tradition and write honest-to-Dijkstra bug free, fault tolerant software without even trying. After half a century in the wilderness, functional programming seems to finally be gaining some ground on the barbarians. Let's examine why.