Designing powerful real-time systems using Scala and Akka

Sky BML (Beat/Meet/Lose) is a streaming system that produces real time Competitive Intelligence about our Lodging Supply.

This competitive information is being used, among others, for Market Managers to negotiate with hotels for better rates and avail. Competitiveness is proportional to the share of BML Losses that Expedia incurs. This is why Market Managers focus their energies on minimising losses.

However, it is not easy to understand why losses occur and consequently their resolution takes proportionately long.Our system tries to simplify this and hence accelerate resolution of losses.

To develop this system, we fully leveraged the power of different functional features of Scala and used Structured Spark Streaming as the processing engine in combination with Clustering(for better app supervision and lifecycle management of the nodes).

Similarly, some of the complex challenges are solved very naturally using functional programming features of easily. We had this complex problem of traversing a decision tree (based on conditional expressions) and then evaluating the child nodes(which are also expressions) with different lookup services responsible for mapping the symbols used in Expressions.

I will be discussing in detail on how we tactically use Functional Programming concepts like Implicits, currying, High-Order Functions to get rolling with this problem with ease.

Similarly, I will discuss on the way we have apply OOP principles to handle different kinds of inputs(streaming/batch) with minimal code duplication.

Overall, it will be lot of fun to discuss the way to think about our problems and how our thinking is shaped by different concepts available in Scala.

 
 

Outline/Structure of the Talk

1) Visuals of our real-time data processing engine and its capabilities in delivering business value and churning out competitive info.

2) Deep dive into one of the critical components of our system.

3) How we have improved our app supervision capabilities using Clustering.

4) How we use Scala constructs and capabilities to solve our business problem with ease.. Expect some crisp discussions on implicits, HOF(s), currying and many others (in the context of our problem)

5) How we handle the growing variety of data using Scala and make our app versatile.'

6) (New addition) Walking through the concepts of fixtures in Scala and how we apply it in all our integration tests here at Expedia to build our testing frameworks..

7) (New addition) (OPEX-driven development) : Achieving Operational Excellence in out coding style

8) Q&A (lots if them!)

Also, in between the talk and the slides(and ofcourse during the Q&A).. I will try to draw out some diagrams using Paint or Miro..though it is presentation but I really want to give it the feel of a demo wherein . many explanations and diagrams are made/presented right at the moment,

Learning Outcome

Key learnings :

1) I will be live coding many things ...(updated the ppt to reflect the exact slides at which I will do so)..e.g. coding up a Clustered application from scratch, writing a small Integration test using Scala fixtures(something new that we are doing in all our projects now)...so people can directly apply these learnings to their own projects.

2) There are many features like implicits, high-order functions etc which we study about but forget... I will explicitly focus on those features while explaining how we solved the decision-tree expression evaluation problem... which will help us understand them better and also start thinking of problems in terms of functional programming.

3)The audience will get to know how we build/monitor systems here at Expedia e.g. there is a separate section on monitoring the real-time systems(OPEX.)

Additional Notes :

I would love to see people asking lot of questions.. that would be proof that they are actually involved in what's going on..

Secondly, they should walk out with easy-to-grasp , new ideas that they can discuss with their team right away and start building on that..

Lastly, it will be great learning for me as well; to get feedback/questions from people and carry it back to my place :)

Target Audience

Everyone who loves Scala and wants to see some new kinds of use-cases getting solved very naturally by harnessing its power.

Prerequisites for Attendees

Basic knowledge of Scala would be good...

Mainly, i need zeal and am looking forward to many questions

schedule Submitted 10 months ago

Public Feedback


    • Liked Anupam Jain
      keyboard_arrow_down

      Anupam Jain - Typesafe Functional SQL

      45 Mins
      Talk
      Intermediate

      We'll discuss how to use strongly typed functional programming to build a typesafe and highly declarative DSL to generate SQL, that allows better abstractions and minimises runtime errors. We'll focus on type systems, and build an intuition for row types and how it guarantees correctness for SQL. I'll cover some parts of actual SQL generation.

      A similar talk was given at IIIT Delhi earlier, however that was more geared towards beginners to Functional Programming (link below). I plan to give a more intermediate level talk this time.

    • Liked Anish George
      keyboard_arrow_down

      Anish George - Demystifying Function Sub-typing

      Anish George
      Anish George
      UI Developer
      ThoughtWorks
      schedule 1 year ago
      Sold Out!
      20 Mins
      Talk
      Intermediate

      What is a Type for a function? When can you say that a function is a sub-type of another function? Why should you even bother?

      Function sub-typing is one of the most confusing areas of FP. In this talk, I'll take a deep dive and explain the core concepts of function types which applies to many different languages. More than just explaining the mere rules, I will be deriving those rules in a step-by-step fashion till the eureka moment so that you will never be confused again.

      The topics covered:

      1. Sub-typing basics
      2. Covariance & Contravariance
      3. Function sub-typing
      4. Implementation gotchas in different languages (Java, Scala, Haskell, JavaScript)
    • Liked Siddharth Bhat
      keyboard_arrow_down

      Siddharth Bhat - Monad-bayes: Probabilistic programming in haskell, tutorial & implementation

      Siddharth Bhat
      Siddharth Bhat
      Student
      IIIT Hyderabad
      schedule 10 months ago
      Sold Out!
      20 Mins
      Case Study
      Intermediate

      We take a dive into probabilistic programming, beginning with a high-level explanation of what probabilistic programming is. We then continue to see how to use the monad-bayes library for performing tasks such as regression in a bayesian formalism. We also look into the implementation of the library, which involves performing an interesting sampling method (markov-chain-monte-carlo) on a very unique space (the space of computational traces of the program). We finally end with next-steps for the haskell community to improve in the probablistic programming space, ala pyro for python

      Here is our tiny re-implementation of monad-bayes, boiled down to the essentials: https://github.com/bollu/shakuni

      Paper on which the talk is based on: http://mlg.eng.cam.ac.uk/pub/pdf/SciGhaGor15.pdf

    • Liked Harmeet Singh
      keyboard_arrow_down

      Harmeet Singh - Type System: The Beauty And The Beast

      45 Mins
      Talk
      Intermediate

      The type system plays an important role in building type-safe applications that reduce runtime exceptions and developer mistakes. This sophisticated system is leveraged by Scala’s most famous functional libraries, such as Scala-Cats and ScalaZ. This feature-rich system allows implementing pure functional programming on the JVM. It is a beast due to its complex syntax, as it prevents developers from exploring its beautiful aspects like partially applying types in Scala. In this talk, we will tame the beast and use its beauty to solve real-life issues faced during coding. Our examples will be based on generics, constraints, type lambdas, and the kind projector plugin.

      The following are the learning objectives of this talk:

      1. Practical use cases of the type system using real-life coding.

      2. Implementing custom higher kinded types and handling partially applied types via type lambdas and the kind projector plugin.

    • Liked ganesan arunachalam
      keyboard_arrow_down

      ganesan arunachalam - What the hell is monad? Why should you care?

      ganesan arunachalam
      ganesan arunachalam
      Developer
      ThoughtWorks
      schedule 10 months ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      Real world applications are hard to imagine without side effects such as writing to a DB or UI. IO-monads were introduced into the Haskell language to write programs in a robust way. And we have been using monads in our day to day job without even knowing those are monads such as IEnumerable/IObservable in languages like C#, Java.

      This talk is to give a basic understanding about monads and why should we care about that, and a little bit about onion/clean architecture.