Realtime Distributed computing: dealing with Time and Failure in the wild
There is a growing need for scalable, realtime business systems that are continuously running and highly-available. Two very different frameworks/approaches you could use to build such systems are Storm and Akka.
Systems created with Storm or Akka are distributed, at runtime, on as many machines as you choose. The inherent concurrency implied by this brings the issues of State, Time and Failure into sharp focus. Functional programming has much to say about dealing with state and time; not surprisingly, both Storm and Akka have strong roots in functional languages (for Storm it is Clojure, and for Akka, Scala).
In this talk we'll explore the core concepts and challenges of distributed computation; the role of functional programming in concurrent distributed computing; we'll take a look at Storm and Akka, by example, and see that as different as these 2 approaches are, the underlying difficulties of distributed computation remains evident in both: dealing with time, and dealing with failure.
Outline/Structure of the Experience Report
* Introduction to Distributed programming
* explore Storm/Akka, by example
* dealing with State, Time (sequencing) and Failure
* functional programming and concurrent distributed computation
Learning Outcome
* get an idea of the fundamentals of Distributed Systems and the challenges of programming in this paradigm
* become sensitised to the issues of State, Time, Failure
* learn about 2 diverse approaches: Storm (realtime, distributed stream processing), Akka (distributed actor based runtime)
* learn why and when to consider incurring the cost of writing Distributed code
* learn about the ways in which functional programming has been applied to concurrent, distributed systems
Target Audience
Developers, technology decision makers
Links
http://lanyrd.com/profile/ryanlemmer/
http://lanyrd.com/2013/euroclojure/scrdpg/
http://www.slideshare.net/ryanlemmer/lets-talk-about-sets
http://www.youtube.com/watch?v=GaSwjqU28-U
schedule Submitted 5 years ago
People who liked this proposal, also liked:
-
keyboard_arrow_down
Venkat Subramaniam - Keynote: The Joy of Functional Programming
60 Mins
Keynote
Intermediate
It's been around for a long time, but everyone's talking about it all of a sudden. But why and why now? We've been
programming in languages like Java for a while, quite well. Now we're asked to change and the languages themselves
are changing towards this style of programming. In this keynote, a passionate polyglot programmer and author of
"Functional Programming in Java: Harnessing the Power of Java 8 Lambda Expressions" will share the reasons
we need to make the paradigm shift and the pure joy—the benefits—we will reap from it.
-
keyboard_arrow_down
Morten Kromberg - Pragmatic Functional Programming using Dyalog
60 Mins
Demonstration
Beginner
APL is a member of the family of languages that are approaching middle age (Ken Iverson’s book titled “A Programming Language” was published in 1962). APL was very influential in the 60’s and 70’s, and widely used to deliver “end user computing” - but although the REPL, dynamic scope and lack of a type system endeared APL to domain experts, it also drew fire from computer scientists, most famously when Edsger Dijkstra declared that “APL is a mistake, carried through to perfection. It is the language of the future for the programming techniques of the past it creates a new generation of coding bums.”
Dyalog is a modern, array-first, multi-paradigm programming language, which supports functional, object-oriented and imperative programming based on an APL language kernel. Dyalog allows people with good ideas – from bright high school students to PhDs – to contribute directly to the software development process using a notation which fits comfortably with those used in their own domains. Subject matter experts can write prototypes or, with suitable training and/or support, highly efficient, parallel and robust code that can be embedded in high-performance production applications.
-
keyboard_arrow_down
Debasish Ghosh - Property based testing for functional domain models
45 Mins
Talk
Intermediate
Manual testing is something that's error prone, incomplete and impossible to replicate on a large scale. We have instead been using xUnit style of testing for quite some time now. This approach has a number of drawbacks like (a) We need to write test cases by hand which again doesn't scale for large systems (b) We may miss out some of the edge cases (c) Safeguarding missing cases with coverage metrics doesn't help, since metrics are mostly based on heuristics (d) maintaining test cases and test data is a real pain.
In property based testing we write properties and not low level test cases. And let the system generate test cases which validate such properties. There are 2 main advantages with this approach:
- You think in terms of properties (or specifications) of the domain model which is the right granularity to think about
- You don't need to manage test cases, which is completely done by the system that generates a large collection of test data
This approach is ideal for the functional programming paradigm, which focuses on pure functions. Using functional programming it's easier to reason about your model - hence it's easier to test functional programs using properties. In this talk I will take some real world examples of property validation and verification using scalacheck (the property based testing library for Scala) and a real world domain model.
-
keyboard_arrow_down
Dhaval Dalal / Ryan Lemmer - Code Jugalbandi
Dhaval DalalSoftware ArtisanCalixir Consultants Pvt. Ltd.Ryan LemmerProgrammer, code-coachIndependent consultant, Cape Townschedule 5 years ago
60 Mins
Demonstration
Beginner
In Indian classical music, we have Jugalbandi, where two lead musicians or vocalist engage in a playful competition. There is jugalbandi between Flutist and a Percussionist (say using Tabla as the instrument). Compositions rendered by flutist will be heard by the percussionist and will replay the same notes, but now on Tabla and vice-versa is also possible.
In a similar way, we will perform Code Jugalbandi to see how the solution looks using different programming languages and paradigms.
During the session, Dhaval and Ryan will take turns at coding the same problem using different languages and paradigms. There would be multiple such rounds during the Jugalbandi. -
keyboard_arrow_down
Mushtaq Ahmed - Demystify the Reactive Jargons
60 Mins
Demonstration
Intermediate
Sync, Async, Blocking, Non-Blocking, Streaming are the buzzwords in the reactive programming world. This talk will attempt to attach some meaning to them. It will also demo the performance and resource consumption patterns for blocking-io, Scala Futures and RxJava Observables for comparable programs. Finally, a command line application that consumes twitter streams API will demo what is possible using the new reactive abstractions.
-
keyboard_arrow_down
Ramakrishnan Muthukrishnan - An introduction to Continuation Passing Style (CPS)
60 Mins
Tutorial
Intermediate
Traditionally functions return some value. Someone is waiting for that value and does some computation with it. This "someone" is called the continuation of this value. In a normal functional call, the continuation is "implicit". In the "continuation passing style" (hence forth called with the short form, CPS), we make the continuations explicit. In this style, function definitions take an extra argument called "continuation" and it never return. The "return value" of the function 'continues' by passing this value as an argument to the continuation. Continuations are sometimes called "gotos with arguments".
CPS is used as an intermediate stage while compiling a program since it makes the control structure of the program explicit and hence can be converted easily to machine code. Another feature of a CPS-transformed function is that it is tail-recursive even if the original function was not written in a tail-recursive style.
Continuations enable a programmer to build new control operators (if the language's built-in operators does not already provide the control operators the programmer need).
-
keyboard_arrow_down
Shashi Gowda - Functional Reactive UIs with Elm
90 Mins
Tutorial
Beginner
Code used during the talk: https://github.com/shashi/fuconf-talk
Slides: https://docs.google.com/presentation/d/16Xfqd-xU8y2JEN0TIcacDoYnp0b5-W7ESDB5v1SmcXs/edit#slide=id.pElm is a strongly typed functional reactive programming (FRP) language that compiles to HTML, CSS, and Javascript. In Elm, the Signal type represents a time-varying value--things like mouse position, keys pressed, current time are signals. With Signals, one can write terse code that is isomorphic to a dataflow diagram of the app. The code hence feels natural and is 100% callback free. All this, with powerful type inference.
This talk is an introduction to FRP. It explores functionally composing graphics and UIs, and creating interactions and animations with the Signal type. There will also be an overview of Elm’s execution mechanism and the time traveling debugger: a consequence of Elm's purely functional approach.
While instructive, it will be good fun too, in the spirit of Elm.
-
keyboard_arrow_down
Vagmi Mudumbai - Clojurescript and Om - Pragmatic functional programming in the Javascript Land
60 Mins
Demonstration
Beginner
Javascript programmers have had a lot of choices when it comes to programming. There were days of mootools, scriptaculous and jQuery and then there are now days of Angular, Ember, Knockout and the like. As a javascript programmer myself, I find that Clojurescript/React as Om offers a fresh perspective into building performant Javascript UIs that are easy to write.
The talk will introduced concepts of React, immutable datastructures in Clojure and live code an application that demonstrates the concepts.
-
keyboard_arrow_down
Mohit Thatte - Purely functional data structures demystified
45 Mins
Talk
Beginner
Immutable, persistent data structures form a big part of the value proposition of most functional programming languages.
It is important to understand why these data structures are useful and how they make it easier to reason about your program.
It is also instructive to see how these data structures are implemented to get a greater appreciation for the inherent tradeoffs between performance and immutability.
In this talk I will do a walkthrough of some of these data structures drawing from the work of Chris Okasaki[1], and attempt to explain the essential ideas in a simple way.
[1] Purely Functional Data Structures, Chris Okasaki, School of Computer Science, Carnegie Mellon University
-
keyboard_arrow_down
Bhasker Kode - Writing and improving tail recursive functions
45 Mins
Talk
Beginner
Brief history of recursion
Snippets from a few languages
What is tail recursion?
Design choices around recursion
The importance of tail recursion in erlang
How do you profile such improvements?
-
keyboard_arrow_down
Premanand Chandrasekaran - Functional Programming in Java
60 Mins
Workshop
Beginner
Functional programming has started (re)gaining prominence in recent years, and with good reason too. Functional programs lend an elegant solution to the concurrency problem, result in more modular systems, are more concise and are easier to test. While modern languages like Scala and Clojure have embraced the functional style whole-heartedly, Java has lagged a bit behind in its treatment of functions as first-class citizens. With the advent of Java 8 and its support for lambdas, however, Java programmers can finally start reaping the power of functional programs as well. Even without Java 8, it is possible to adopt a functional style with the aid of excellent libraries such as Guava.
This talk will explore how to apply functional concepts using the Java programming language and demonstrate how it can result in simpler, more elegant designs. We will conduct this in a hands-on workshop style with attendants being encouraged to code-along. So bring your favorite Java 8 aware IDE, an open mind and prepare to have a lot of fun. -
keyboard_arrow_down
Venkat Subramaniam - Haskell for Everyday Programmers
90 Mins
Talk
Intermediate
I learn different languages not to make use of them, but to program in my current languages in a better way. As we adapt functional style of programming in mainstream languages, like Java, C#, and C++, we can learn a great deal from a language that is touted as a purely functional language.
Haskell is statically typed, but not in a way like Java, C#, or C++. Its static typing does not get in the way of productivity. Haskell quietly does lazy evaluation and enforces functional purity for greater good. Everyday programmers, like your humble speaker, who predominantly code in mainstream languages, can greatly benefit from learning the idioms and style of this elegant language. The next time we sit down to crank out some code in just about any language, we can make use of some of those styles, within the confines of the languages, and move towards a better, functional style.
-
keyboard_arrow_down
Kishore Nallan - Applying functional programming principles to large scale data processing
45 Mins
Talk
Intermediate
At Indix, we deal with a stream of unstructured and constantly changing data. This data is processed through a series of systems before being fed as structured input to our analytics system. In this talk, I will walk through our experience of building a large scale data processing system using Hadoop that's focused on immutability, composition and other functional programming principles. -
keyboard_arrow_down
Aditya Godbole - Learning (from) Haskell - An experience report
45 Mins
Experience Report
Beginner
Functional programming as a programming style and discipline is useful even in languages which are not pure functional languages. By practising programming in a pure functional language like Haskell, programmers can drastically improve the quality of code when coding in other languages as well.
The talk is based on first hand experience of using Haskell in internal courses in our organisation to improve code quality.
This talk will cover Gofer (one of the earliest variants of Haskell) as a teaching tool, including the choice of the language, the features from Haskell that should (and shouldn't) be covered and the obstacles and benefits of the exercise.
-
keyboard_arrow_down
Bodil Stokke - Programming, Only Better
45 Mins
Talk
Beginner
The programming paradigms that served us so well through the 80s and 90s are failing us. Building systems the way we're used to building them always seems to end in the inevitable death march towards exponential complexity. But once you stop to ask the right question- "what's really causing all this complexity?" - you realise the answers have really been staring you in the face all along. Debugging is only hard when you can't reason about your code. Concurrency is only hard when you can't predict the state of your code. Reusability is only hard when your components aren't naturally composable.
Fortunately, languages addressing these issues specifically are popping up all over the place. In many cases, it turns out we've had the solutions to our problems for a long time, we've just forgotten about them, or never really bothered to look. Let's take a moment to explore some of these languages, not as exercises in syntactic details, but looking at the inherent properties in their design that enable us to defy decades of OO tradition and write honest-to-Dijkstra bug free, fault tolerant software without even trying. After half a century in the wilderness, functional programming seems to finally be gaining some ground on the barbarians. Let's examine why.
Public Feedback