High Performance Data Mart using Event Plots

schedule Nov 17th 05:00 PM - 05:20 PM place Regal people 3 Attending

Reporting data for finite state machine transactions like calls require exhaustive data mining into 10s and 100s of parameters for 1000s of calls per hour. These postmortems require heavy processing and can demand millions of maps per reduction. The presentation is a case study and demonstration of near-realtime asynchronous event processing based plotting of events and states of FSMs. Thus creating semi-cooked event plots delivering lightening fast reports with extensive customizability.

 
3 favorite thumb_down thumb_up 8 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

The session would start with the problem definition of the case study, followed by the solution design and a demonstration of the technique at the end the session.

  1. Presentation (10 min ~ 15 min)
    1. Domain overview - Cloud-based Call Center Solution
    2. Problem Definition - Reports using traditional Datamart approach - short falls - challenges
    3. Overview of Data Plots - Erlang Messaging - Event Maps - Mnesia/ETS (memory & disc) - Mnesia/eleveldb(disc only)
    4. Reporting - Scanning Events - Data Aggregation - publish - stream - MapReduce
  2. Demonstration (10 min ~ 15 min)
    1. Pre-populated call center event plots (mnesia, 200 GB offline / 1TB online)
    2. Access the web based report dashboard and demonstrate performance of existing reports
    3. Modify any existing report to demonstrate customization
    4. Simulate a bunch of call events and demonstrate the report streaming
  3. Questions & Discussions (5 min ~ offline)

Learning Outcome

The participant would be able to appreciate the technique, the functional design aspect and the Erlang/OTP efficiency.

The participant can also get the exposure to the data mart and reporting solutions.

Target Audience

Functional Programmers, Architects and Data Mart Engineers

Prerequisite

The participants should be aware of async messaging, map-reduce algorithm and basics of functional programming.

schedule Submitted 4 months ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Naresh Jain
    By Naresh Jain  ~  3 months ago
    reply Reply

    Hi Chaitanya,

    Thanks for updating your proposal based on the feedback.

    Can you please share a video link to your past presentations? This will help the program committee understand your presentation style.

    If you don't have any links, can you please records a 1-2 mins trailer of your talk from your laptop and share a link to that?

    • Chaitanya Chalasani
      By Chaitanya Chalasani  ~  3 months ago
      reply Reply

      Added a short video on the same topic. Recorded on my laptop.

  • Debasish Ghosh
    By Debasish Ghosh  ~  3 months ago
    reply Reply

    Can we have some more details on the Outline of the presentation ?

    • Chaitanya Chalasani
      By Chaitanya Chalasani  ~  3 months ago
      reply Reply

      I updated the presentation plan. Let me know if you are looking for something else.

      • Dhaval Dalal
        By Dhaval Dalal  ~  3 months ago
        reply Reply

        Thanks Chaitanya!

  • Chaitanya Chalasani
    By Chaitanya Chalasani  ~  3 months ago
    reply Reply

    Did remove the telecom domain pre-req smile

  • Dhaval Dalal
    By Dhaval Dalal  ~  3 months ago
    reply Reply

    Hi Chaitanya,

       In the pre-req section, you have mentioned  - 'should be aware of telecom domain', what aspects of it is a mandatory requirement, can you elaborate?   I'd say if you are already establishing the context in Domain overview and  problem definition, do you still require that pre-req?

    thanks,

    Dhaval.

    • Chaitanya Chalasani
      By Chaitanya Chalasani  ~  3 months ago
      reply Reply

      Hi Dhaval,

      The domain context here is only to understand what does call offered, answered, rejected, abandoned mean. Yes I would be adding a slide explaining these terms. 

      If thats sufficient, I can remove the domain pre-req.


  • 45 Mins
    Demonstration
    Beginner

    In Indian classical music, we have Jugalbandi, where two lead musicians or vocalist engage in a playful competition. There is jugalbandi between Flutist and a Percussionist (say using Tabla as the instrument). Compositions rendered by flutist will be heard by the percussionist and will replay the same notes, but now on Tabla and vice-versa is also possible.

    In a similar way, we will perform Code Jugalbandi to see how the solution looks using different programming languages and paradigms.This time the focus of code jugalbandi will be on solutioning in different paradigms - object-oriented or functional programming and array-oriented paradigms.

    During the session, Morten and Dhaval will take turns at coding the same problem using different languages and paradigms. There would be multiple such rounds during the Jugalbandi.

  • Liked Eric Torreborre
    keyboard_arrow_down

    Eric Torreborre - Practical eff monad for micro-services

    Eric Torreborre
    Eric Torreborre
    Senior Software Engineer
    Zalando
    schedule 10 months ago
    Sold Out!
    45 Mins
    Talk
    Advanced

    This talk presents how we use the Eff monad at Zalando to structure a REST microservice. Even when programming with functions, modules and components still matter, and I will present a new approach for dependency injection using the Reader monad (classic) and tree rewriting (what?). Then we will see how to specify more precisely several components (authentication, request to other services and database access) with the Eff monad and support additional concerns like logging or asynchronous execution.

    Once you get the basic idea, the possibilities are endless! Rate limiting is just another component and another set of effects involving time and state, testing becomes really clean and easy.

  • Liked Aaron W Hsu
    keyboard_arrow_down

    Aaron W Hsu - Design Patterns vs. Anti-pattern in APL

    Aaron W Hsu
    Aaron W Hsu
    Computer Scientist
    Indiana University
    schedule 3 months ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    APL is a notorious language with a reputation for being "write only" and difficult to learn. Yet, many people consider APL to be their single most productive, advantageous tool that they hope never to give up. In particular, it is generally observed that many computer scientists have a poor opinion of the usability of APL, while many data scientists and domain experts find the language to be exceptionally usable.

    This session focuses on the experienced programmer's struggle to gain real facility with APL beyond the basic, trivial understanding of its semantics and syntax (which can be taught in a few hours). We will observe this struggle through a "human-centered" experience analysis identifying a set of Pattern/Anti-pattern tensions that are at the heart of the intermediate APL learning "wall." By examining the experience of thinking and working with APL code versus regular code, we can identify principles of APL coding practice that directly oppose the widely taught and embraced best practices of the broader programming community. We'll see not only what these principles are, but also why they survive in APL and how they contribute to the positive development cycle of an experienced APL programmer, instead of the negative impact such practices usually have when writing in other languages.

    Understanding these practices will provide a focus point for the discussion of programming experience design and the use of concise array language, as well as provide a structure for becoming better able to write, read, and think in APL expert.

  • Liked Aloïs Cochard
    keyboard_arrow_down

    Aloïs Cochard / Tony Morris - Introduction to Functional Programming using Haskell

    Aloïs Cochard
    Aloïs Cochard
    Computer Science Research
    Orexio / Swisscom
    Tony Morris
    Tony Morris
    Senior Functional Programming Engineer
    Data61 / CSIRO
    schedule 4 months ago
    Sold Out!
    480 Mins
    Workshop
    Beginner

    We will be spending the day learning the fundamentals of Functional Programming (FP) using the Haskell programming language. The exercise material will be a condensed selection of the NICTA/course which is regularly held in Australia over three days.

    This one day session is targeted to experienced industry programmers who are looking to break into Functional Programming and develop the rudimentary skills and techniques that enable continued independent study. A refresher on Haskell syntax will be provided, however, it is highly recommended to practice with the syntax and development tools prior to obtain the best outcome for the day.

    You will be required to bring a suitable development machine (portable) for working through the exercises. You will also need to install Glasgow Haskell Compiler (http://www.haskell.org/ghc/) version 7.8 or higher on that machine prior to the day.

  • Liked Shri Vignesh Balasubramaniam
    keyboard_arrow_down

    Shri Vignesh Balasubramaniam - Understanding some basic FP constructs by writing a simple Programming Language

    45 Mins
    Demonstration
    Beginner

    What I cannot create, I do not understand - Richard Feynman

    Most of the time, we end up being consumers of a language. Language creation is not something which many of us do as part of our daily job. And as such, we become consumers of using various concepts like types of a language, functions-as-first-class citizens, closures, let expressions, etc. But what are these? What is semantics behind them? And how do language creators enable a language to have these? Once we understand what these mean, we go beyond the language specific implementations and can be true polyglots who can think in terms of these composable abstractions.

    This beginner focussed talk+demonstration hopes to clarify few fundamental concepts by building a language from the ground up. This talk has its foundations in a couple of academic courses which I got to be part of during 2012/2013.

    Using Racket(a variant of Scheme) as a vehicle, a hypothesized programming language (MathL) is introduced with its specifications, syntax, and rules. And slowly this language is built up with more features.The language is realized via an interpreter/executor. [We would not be discussing compilers here].
    Some tests also would be done which would help to verify the language implementation.

     

     

     

  • Liked Tony Morris
    keyboard_arrow_down

    Tony Morris - Functional Programming in Aviation

    45 Mins
    Case Study
    Beginner

    In this talk & demo, we have a look at some of the low-hanging problems in general aviation and how functional programming can be applied to provide significant improvements in efficiency and air safety. The current solutions to problems such as navigation, traffic/terrain collision avoidance and weight/balance calculations will be demonstrated to the audience, mostly for amusement. More seriously, we will have a look at the legacy that has led to the way things are, and how to improve by applying our programming skills.

    We will look at:

    • how aviation safety is regulated.
    • how aeronautical services are provided to flight operators.
    • how aeronautical navigation is conducted and regulated.
    • how the weight and balance for a flight is conducted.
    • the methods by which aircraft and ground coordinate between each other.

    We will see:

    • some real (and basic) data management problems in aviation, that very obviously threaten safety, then solve them, using programming.
    • we will see a live demonstration of aeronautical navigation methods, investigate incident reports where lives were lost as a result, and consider how our programming skills can yield improvements, possibly even save lives.
    • we will conduct a real weight&balance calculation for a flight, then once hilarity inevitably ensues, we will look at the problems that arise by this method, then solve them using data structures and functional programming. Some
      of these practical problems are obvious, even to a non-aviator, and the predictable incident reports are the end result.
    • finally, we will have a look at a live demonstration of a software defined radio (SDR), receiving ADS-B transmissions from aircraft (live), an AHRS implementation and GNSS receiver using off-the-shelf, low-cost parts. We will look at why these instruments are helpful to aircraft pilots and interact with that device using the Haskell programming language.
  • Liked Manoj Govindan
    keyboard_arrow_down

    Manoj Govindan - Learning Computer Architecture

    Manoj Govindan
    Manoj Govindan
    Director, Engineering
    Perfios
    schedule 4 months ago
    Sold Out!
    45 Mins
    Experience Report
    Intermediate

    Computer Architecture begins with electronics. As computers are built layer upon layer starting with primitive gates baked into silicon, followed by chips and logic gates, hardware platform including ALU, RAM, cache etc., and so on, so is the study of computer architecture a study of each of these layers of abstraction.

    Students of computer architecture usually rely on simulators, often written in Java or similar imperative/object oriented languages, to aid their learning. Here are some examples of such a program:

    Simulating XOR gate

    As an Erlang/functional programming enthusiast studying computer architecture I decided to write my own equivalent tools in Erlang and TypeScript as I went about my learning. I am studying computer architecture and simultaneously writing the required simulators and other tooling in Erlang and TypeScript.

    Building such emulators and tools in Erlang and Typescript results in distinctly different architectures. I'll be demoing the simulator I built and talking about lessons learned doing this.

  • Liked Abhiroop Sarkar
    keyboard_arrow_down

    Abhiroop Sarkar - Wormholes and teleporting with Transient.

    Abhiroop Sarkar
    Abhiroop Sarkar
    Student
    University of Nottingham
    schedule 6 months ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    The transient library in Haskell/Eta composes arbitrary pieces of software in concurrent, parallel and sequential settings using standard Haskell/Eta combinators like applicative, alternative, bind etc. It provides algebraic and monadically composable primitives for concurrency, parallelism, event handling, transactions and multithreading

    It liberates the programmers from dealing with state machines, objects frameworks, routes, configurations and callbacks. Transient expresses business functionalities as compact category theoretical expressions which compose. These compositions are verified during compilation, although the code might be executing in different machines.

    It presents a fresh new approach to programming concurrent and distributed systems.

     

  • Liked Ryan Lemmer
    keyboard_arrow_down

    Ryan Lemmer - Haskell Design Patterns

    Ryan Lemmer
    Ryan Lemmer
    Programmer, code-coach
    Independent consultant, Cape Town
    schedule 6 months ago
    Sold Out!
    480 Mins
    Workshop
    Intermediate

    This is an experiential workshop, we will code together, and explore ideas from first principles. You will need GHC 8.*

    The workshop is pitched at Haskell enthusiasts (you don't have to be hardcore at Haskell, I am not!). We'll be writing code together and apart, reflecting on code and ideas, and getting into and out of trouble.

    Patterns for IO

    Experience the 3 styles of IO programming in Haskell, in the order in which they evolved:

    * imperative style (the status quo)
    * Lazy IO and its limitations
    * Iteratee IO, the solution to Lazy IO

    As we compare the different styles, we will pay attention to

    * what controls evaluation
    * space and resource management
    * the relationship between producer and consumer

    From Foldable, Traversable to Lenses

    We follow the evolution of fold and map on lists, to their generalisation over arbitrary types with the Foldable and Traversable type-classes.

    The Lens library raises Foldable and Traversable to an even higher level of abstraction. We will see how the Lens library gives a unified vocabulary for traversal of arbitrary data structures.

    Functor, Applicative, Arrow, Monad

    These data types fall on a spectrum of power and generality (Functor being the most general, and Monad the most powerful). By experiencing how these types differ in the way they compose, we can gain a clearer understanding of where and how each type may be used.

    Dependent Haskell / Type-level programming 101 (bonus material)

    In modern Haskell, the line between types and functions have blurred, types have become as powerful and expressive as functions. Since we can write functions on type-level, we can do whatever we like with types! Almost.

    Let's write some type-level code, and see why Dependently Typed Programming matters.

    Datatype Generic Programming (bonus material)

    This style of Haskell programming is based on a simple technique: instead of defining functions for ad-hoc types, we deconstruct our types into a more fundamental type representation, and then write generic functions against the lower-level type. These generic functions are impervious to changes in the original, higher-level data types.

    We will experience datatype generic programming from first principles, and along the way uncover the key ingredient in the magic sauce called GHC.Generics.