Why functional programming and BIG Data go hand in hand

My session will focus on why functional programming and big data go hand in hand covering the following aspects with use cases from my experience report:

  • Parallelize Code : Functional programming makes it easier to parallelize code, so even if a single operation may take more time to run in one thread, the overall performance can be better due to parallelism.
  • Seamless integration: Modern programming paradigms like functional programming, proven to provide the best scalability for cloud computing, are necessary to ensure cost-efficient performance for big data. Functional programming
    can be efficiently parallelised owing to it’s mathematical syntax thus allowing a seamless integration between algorithms
    and economic models.
  • Scalable Architecture: Functional languages encourage scalable software architectures is because of the fact that they, from the bottom up, eliminate as much shared state as possible from a language. This leads to scalable "micro" components.  Those micro components then scale naturally into larger components.
  • Efficient distributed code: Functional programming makes it much easier to write correct and efficient distributed code due to immutable data structures and statelessness.
 
2 favorite thumb_down thumb_up 3 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

1. Intro (5mins)

2. Quick run through of BIG Data challenges (10 mins)

3. Review optimizations using functional programming ( 25 mins)

4. Q & A ( 5 mins)

Learning Outcome

Key outcome includes big data code optimization using functional programming with a focus on parallelize code, seamless integration, scalable architecture and writing efficient distributed code.

Target Audience

Developers, Big Data Architects

schedule Submitted 3 years ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • uday
    By uday  ~  3 years ago
    reply Reply

    Just to add the optimizations are focused around MapReduce code.

  • uday
    By uday  ~  3 years ago
    reply Reply

    Thank you  Naresh for your inputs. I plan to change it to 45 mins - 60 mins as I realized itz difficult to cover challenges and optimizations in 20 mins.  Regarding the optimizations the code snippets are not on any public domain. Will cover them as a part of my presentation. Hope that works

  • Naresh Jain
    By Naresh Jain  ~  3 years ago
    reply Reply

    Thanks for the proposal, Uday. This looks like a very interesting experience report. I'm a little worried that 30 mins might not do justice to this topic. I would suggest 45 mins.

    Also would it be possible to highlight some of the key code optimizations you guys have done for big data? May be a link where you've already described them.

    Thanks.


  • Liked Bodil Stokke
    keyboard_arrow_down

    Bodil Stokke - The Mess We've Made

    60 Mins
    Keynote
    Beginner

    In this talk, we'll explore why some technologies end up inexplicably dominating our field while others fade away despite their obvious merits. What will it take for the languages we care about to succeed, and what are the consequences of success? Let's apply a broad perspective to the mess we've made of modern computer science, and explore our options for getting the situation back on track through a careful examination of history, science and ponies.

  • Liked Debasish Ghosh
    keyboard_arrow_down

    Debasish Ghosh - Property based testing for functional domain models

    Debasish Ghosh
    Debasish Ghosh
    Consultant
    IRadix Inc
    schedule 3 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Manual testing is something that's error prone, incomplete and impossible to replicate on a large scale. We have instead been using xUnit style of testing for quite some time now. This approach has a number of drawbacks like (a) We need to write test cases by hand which again doesn't scale for large systems (b) We may miss out some of the edge cases (c) Safeguarding missing cases with coverage metrics doesn't help, since metrics are mostly based on heuristics (d) maintaining test cases and test data is a real pain.

    In property based testing we write properties and not low level test cases. And let the system generate test cases which validate such properties. There are 2 main advantages with this approach:

    1. You think in terms of properties (or specifications) of the domain model which is the right granularity to think about
    2. You don't need to manage test cases, which is completely done by the system that generates a large collection of test data

    This approach is ideal for the functional programming paradigm, which focuses on pure functions. Using functional programming it's easier to reason about your model - hence it's easier to test functional programs using properties. In this talk I will take some real world examples of property validation and verification using scalacheck (the property based testing library for Scala) and a real world domain model.

  • Liked Mushtaq Ahmed
    keyboard_arrow_down

    Mushtaq Ahmed - Demystify the Reactive Jargons

    Mushtaq Ahmed
    Mushtaq Ahmed
    Mr Scala
    ThoughtWorks
    schedule 3 years ago
    Sold Out!
    60 Mins
    Demonstration
    Intermediate

    Sync, Async, Blocking, Non-Blocking, Streaming are the buzzwords in the reactive programming world. This talk will attempt to attach some meaning to them. It will also demo the performance and resource consumption patterns for blocking-io, Scala Futures and RxJava Observables for comparable programs. Finally, a command line application that consumes twitter streams API will demo what is possible using the new reactive abstractions.

  • Liked Ramakrishnan Muthukrishnan
    keyboard_arrow_down

    Ramakrishnan Muthukrishnan - An introduction to Continuation Passing Style (CPS)

    60 Mins
    Tutorial
    Intermediate

    Traditionally functions return some value. Someone is waiting for that value and does some computation with it. This "someone" is called the continuation of this value. In a normal functional call, the continuation is "implicit". In the "continuation passing style" (hence forth called with the short form, CPS), we make the continuations explicit. In this style, function definitions take an extra argument called "continuation" and it never return. The "return value" of the function 'continues' by passing this value as an argument to the continuation. Continuations are sometimes called "gotos with arguments".

    CPS is used as an intermediate stage while compiling a program since it makes the control structure of the program explicit and hence can be converted easily to machine code. Another feature of a CPS-transformed function is that it is tail-recursive even if the original function was not written in a tail-recursive style.

    Continuations enable a programmer to build new control operators (if the language's built-in operators does not already provide the control operators the programmer need).

  • Liked Mohit Thatte
    keyboard_arrow_down

    Mohit Thatte - Purely functional data structures demystified

    Mohit Thatte
    Mohit Thatte
    Programmer
    Helpshift Inc.
    schedule 3 years ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Immutable, persistent data structures form a big part of the value proposition of most functional programming languages.

    It is important to understand why these data structures are useful and how they make it easier to reason about your program. 

    It is also instructive to see how these data structures are implemented to get a greater appreciation for the inherent tradeoffs between performance and immutability.

    In this talk I will do a walkthrough of some of these data structures drawing from the work of Chris Okasaki[1], and attempt to explain the essential ideas in a simple way. 


    [1] Purely Functional Data Structures, Chris Okasaki, School of Computer Science, Carnegie Mellon University 

  • Liked Venkat Subramaniam
    keyboard_arrow_down

    Venkat Subramaniam - Haskell for Everyday Programmers

    90 Mins
    Talk
    Intermediate

    I learn different languages not to make use of them, but to program in my current languages in a better way. As we adapt functional style of programming in mainstream languages, like Java, C#, and C++, we can learn a great deal from a language that is touted as a purely functional language.

    Haskell is statically typed, but not in a way like Java, C#, or C++. Its static typing does not get in the way of productivity. Haskell quietly does lazy evaluation and enforces functional purity for greater good. Everyday programmers, like your humble speaker, who predominantly code in mainstream languages, can greatly benefit from learning the idioms and style of this elegant language. The next time we sit down to crank out some code in just about any language, we can make use of some of those styles, within the confines of the languages, and move towards a better, functional style.

  • Liked Aditya Godbole
    keyboard_arrow_down

    Aditya Godbole - Learning (from) Haskell - An experience report

    Aditya Godbole
    Aditya Godbole
    CTO
    Vertis Microsystems
    schedule 3 years ago
    Sold Out!
    45 Mins
    Experience Report
    Beginner

    Functional programming as a programming style and discipline is useful even in languages which are not pure functional languages. By practising programming in a pure functional language like Haskell, programmers can drastically improve the quality of code when coding in other languages as well.

    The talk is based on first hand experience of using Haskell in internal courses in our organisation to improve code quality.

    This talk will cover Gofer (one of the earliest variants of Haskell) as a teaching tool, including the choice of the language, the features from Haskell that should (and shouldn't) be covered and the obstacles and benefits of the exercise.

     

  • Liked Shakthi Kannan
    keyboard_arrow_down

    Shakthi Kannan - Lambdaaaaaaaaaa Calculus

    Shakthi Kannan
    Shakthi Kannan
    DevOps Engineer
    Self
    schedule 3 years ago
    Sold Out!
    60 Mins
    Talk
    Beginner

    This talk is an introduction on lambda calculus and will address the foundations of functional programming languages.

    We will learn the building blocks of lambda calculus - syntax, rules, and application.

  • Liked Venkat Subramaniam
    keyboard_arrow_down

    Venkat Subramaniam - Designing with Lambda Expressions in Java

    90 Mins
    Talk
    Intermediate

    As Java programmers we have used a number of design patterns and design techniques. With the introduction of lambda expressions, we now have some more sharper tools in our design toolbox. Come to this presentation to learn how we can implement some elegant design ideas with lambda expressions. We will learn about these design techniques, not using diagrams, but by realizing the ideas in code.

  • Liked Mushtaq Ahmed
    keyboard_arrow_down

    Mushtaq Ahmed - Typeclasses as objects and implicits

    Mushtaq Ahmed
    Mushtaq Ahmed
    Mr Scala
    ThoughtWorks
    schedule 3 years ago
    Sold Out!
    60 Mins
    Tutorial
    Intermediate

    Haskell has populairzed typelcasses a principled way to add ad-hoc extensions on existing data types. They allow you to 1) add new operations on existing data types and 2) support new data types on existing operations, and thus solve the famous "expression problem".

    There is a lot of similarily between typeclasses and Java good practices of programming to interfaces and preferring composition over inheritance. The missing link is the implicit dictionary passing which allows haskell to be much more concise and expressive.

    In this tutorial, we will look at how Scala adopts typeclasses by adding the missing link of implicits.

  • Liked Amit Dev
    keyboard_arrow_down

    Amit Dev - A practical introduction to Monads

    Amit Dev
    Amit Dev
    Technical Leader
    Cisco Systems
    schedule 3 years ago
    Sold Out!
    30 Mins
    Demonstration
    Beginner

    Some people find it hard to understand Monads (like pointers in C). But it is actually a simple concept (like pointers in C). In this session we try to understand Monads and why they are useful through examples. It is purely from a practical point of view and no theory is covered.

  • Liked Mushtaq Ahmed
    keyboard_arrow_down

    Mushtaq Ahmed - Functions as objects

    Mushtaq Ahmed
    Mushtaq Ahmed
    Mr Scala
    ThoughtWorks
    schedule 3 years ago
    Sold Out!
    60 Mins
    Tutorial
    Beginner

    Objected oriented languages like Scala support the paradigm of programming with functions. How does it work? The language has to conceptually map functions to objects. This tutorial will explain this idea by starting with a simple Java-like code and progressively refactor it to make use of higher order functions. As a result, you will learn about a few syntax sugars and also the cost implication of using objects to represent functions.