Designing powerful real-time systems using Scala and Akka

Sky BML (Beat/Meet/Lose) is a streaming system that produces real time Competitive Intelligence about our Lodging Supply.

This competitive information is being used, among others, for Market Managers to negotiate with hotels for better rates and avail. Competitiveness is proportional to the share of BML Losses that Expedia incurs. This is why Market Managers focus their energies on minimising losses.

However, it is not easy to understand why losses occur and consequently their resolution takes proportionately long.Our system tries to simplify this and hence accelerate resolution of losses.

To develop this system, we fully leveraged the power of different functional features of Scala and used Structured Spark Streaming as the processing engine in combination with Clustering(for better app supervision and lifecycle management of the nodes).

Similarly, some of the complex challenges are solved very naturally using functional programming features of easily. We had this complex problem of traversing a decision tree (based on conditional expressions) and then evaluating the child nodes(which are also expressions) with different lookup services responsible for mapping the symbols used in Expressions.

I will be discussing in detail on how we tactically use Functional Programming concepts like Implicits, currying, High-Order Functions to get rolling with this problem with ease.

Similarly, I will discuss on the way we have apply OOP principles to handle different kinds of inputs(streaming/batch) with minimal code duplication.

Overall, it will be lot of fun to discuss the way to think about our problems and how our thinking is shaped by different concepts available in Scala.

 
 

Outline/Structure of the Talk

1) Visuals of our real-time data processing engine and its capabilities in delivering business value and churning out competitive info.

2) Deep dive into one of the critical components of our system.

3) How we have improved our app supervision capabilities using Clustering.

4) How we use Scala constructs and capabilities to solve our business problem with ease.. Expect some crisp discussions on implicits, HOF(s), currying and many others (in the context of our problem)

5) How we handle the growing variety of data using Scala and make our app versatile.'

6) (New addition) Walking through the concepts of fixtures in Scala and how we apply it in all our integration tests here at Expedia to build our testing frameworks..

7) (New addition) (OPEX-driven development) : Achieving Operational Excellence in out coding style

8) Q&A (lots if them!)

Also, in between the talk and the slides(and ofcourse during the Q&A).. I will try to draw out some diagrams using Paint or Miro..though it is presentation but I really want to give it the feel of a demo wherein . many explanations and diagrams are made/presented right at the moment,

Learning Outcome

Key learnings :

1) I will be live coding many things ...(updated the ppt to reflect the exact slides at which I will do so)..e.g. coding up a Clustered application from scratch, writing a small Integration test using Scala fixtures(something new that we are doing in all our projects now)...so people can directly apply these learnings to their own projects.

2) There are many features like implicits, high-order functions etc which we study about but forget... I will explicitly focus on those features while explaining how we solved the decision-tree expression evaluation problem... which will help us understand them better and also start thinking of problems in terms of functional programming.

3)The audience will get to know how we build/monitor systems here at Expedia e.g. there is a separate section on monitoring the real-time systems(OPEX.)

Additional Notes :

I would love to see people asking lot of questions.. that would be proof that they are actually involved in what's going on..

Secondly, they should walk out with easy-to-grasp , new ideas that they can discuss with their team right away and start building on that..

Lastly, it will be great learning for me as well; to get feedback/questions from people and carry it back to my place :)

Target Audience

Everyone who loves Scala and wants to see some new kinds of use-cases getting solved very naturally by harnessing its power.

Prerequisites for Attendees

Basic knowledge of Scala would be good...

Mainly, i need zeal and am looking forward to many questions

schedule Submitted 3 months ago

Public Feedback

comment Suggest improvements to the Speaker
  • Ankit Jain
    By Ankit Jain  ~  2 months ago
    reply Reply

    Short preview video for Currying/HOF(s) : https://drive.google.com/file/d/1RaxIyPE_t94wja1k8-JsepbuVIWrTqax/view . (Also added this as the preview video)

    • Ankit Jain
      By Ankit Jain  ~  2 months ago
      reply Reply

      Hi guys..Any feedback/update on this video?

  • Ankit Jain
    By Ankit Jain  ~  2 months ago
    reply Reply

    Updated my preview video.. (Preview video contains general outline of the session plus the topic of Integration tests explained, with a special focus on how things work inside )..

    Suggestions/feedback are welcome :)

  • Deepti Tomar
    By Deepti Tomar  ~  2 months ago
    reply Reply

    Hello Ankit,

    Thanks for your proposal! This is a fairly interesting topic. 

    Could you throw more light on the learning outcome and update the proposal with more details on it? It would be interesting to know what can the audience expect to gain out of the talk from the technical aspect along with the points that you have already mentioned.

    Would you also plan to run a code walkthrough in your session, along with the concepts?

    Also, to help the program committee to understand your presentation style, can you please record a small 1 mins trailer of your talk and share the link to the same?

    • Ankit Jain
      By Ankit Jain  ~  2 months ago
      reply Reply

      Hi Deepti,

      Thanks for your feedback.. 

      Yes I have added a demo.. (Section : Video)

      I have updated the key learnings for the audience..(See section : Learning Outcome)

      And also I have added some improvements for my ppt(plus new sections for fixture-based integration testing and OPEX)..

      And yes, I plan to do some live-coding/code-demos alongside and I have updated my ppt to reflect the same

      Please have a look..

    • Naresh Jain
      By Naresh Jain  ~  2 months ago
      reply Reply

      Hi Ankit, can you please reply to Deepti's comment?

      • Ankit Jain
        By Ankit Jain  ~  2 months ago
        reply Reply

        Hi Naresh, 

        Done. I have updated my proposal with all the improvements mentioned by Deepti.

         

  • Ankit Jain
    By Ankit Jain  ~  3 months ago
    reply Reply

    @Program Commitee : When can we expect to know the final results of the submissions.. and also would love to know the feedback of the Committe on this proposal :) (like how I can make it better or in line with what is needed for the acceptance :))


  • Liked Anupam Jain
    keyboard_arrow_down

    Anupam Jain - Typesafe Functional SQL

    45 Mins
    Talk
    Intermediate

    We'll discuss how to use strongly typed functional programming to build a typesafe and highly declarative DSL to generate SQL, that allows better abstractions and minimises runtime errors. We'll focus on type systems, and build an intuition for row types and how it guarantees correctness for SQL. I'll cover some parts of actual SQL generation.

    A similar talk was given at IIIT Delhi earlier, however that was more geared towards beginners to Functional Programming (link below). I plan to give a more intermediate level talk this time.

  • Liked Anish George
    keyboard_arrow_down

    Anish George - Demystifying Function Sub-typing

    Anish George
    Anish George
    UI Developer
    ThoughtWorks
    schedule 5 months ago
    Sold Out!
    20 Mins
    Talk
    Intermediate

    What is a Type for a function? When can you say that a function is a sub-type of another function? Why should you even bother?

    Function sub-typing is one of the most confusing areas of FP. In this talk, I'll take a deep dive and explain the core concepts of function types which applies to many different languages. More than just explaining the mere rules, I will be deriving those rules in a step-by-step fashion till the eureka moment so that you will never be confused again.

    The topics covered:

    1. Sub-typing basics
    2. Covariance & Contravariance
    3. Function sub-typing
    4. Implementation gotchas in different languages (Java, Scala, Haskell, JavaScript)
  • Liked Siddharth Bhat
    keyboard_arrow_down

    Siddharth Bhat - Monad-bayes: Probabilistic programming in haskell, tutorial & implementation

    Siddharth Bhat
    Siddharth Bhat
    Student
    IIIT Hyderabad
    schedule 3 months ago
    Sold Out!
    20 Mins
    Case Study
    Intermediate

    We take a dive into probabilistic programming, beginning with a high-level explanation of what probabilistic programming is. We then continue to see how to use the monad-bayes library for performing tasks such as regression in a bayesian formalism. We also look into the implementation of the library, which involves performing an interesting sampling method (markov-chain-monte-carlo) on a very unique space (the space of computational traces of the program). We finally end with next-steps for the haskell community to improve in the probablistic programming space, ala pyro for python

    Here is our tiny re-implementation of monad-bayes, boiled down to the essentials: https://github.com/bollu/shakuni

    Paper on which the talk is based on: http://mlg.eng.cam.ac.uk/pub/pdf/SciGhaGor15.pdf

  • Liked Harmeet Singh
    keyboard_arrow_down

    Harmeet Singh - Type System: The Beauty And The Beast

    45 Mins
    Talk
    Intermediate

    The type system plays an important role in building type-safe applications that reduce runtime exceptions and developer mistakes. This sophisticated system is leveraged by Scala’s most famous functional libraries, such as Scala-Cats and ScalaZ. This feature-rich system allows implementing pure functional programming on the JVM. It is a beast due to its complex syntax, as it prevents developers from exploring its beautiful aspects like partially applying types in Scala. In this talk, we will tame the beast and use its beauty to solve real-life issues faced during coding. Our examples will be based on generics, constraints, type lambdas, and the kind projector plugin.

    The following are the learning objectives of this talk:

    1. Practical use cases of the type system using real-life coding.

    2. Implementing custom higher kinded types and handling partially applied types via type lambdas and the kind projector plugin.

  • Liked ganesan arunachalam
    keyboard_arrow_down

    ganesan arunachalam - What the hell is monad? Why should you care?

    ganesan arunachalam
    ganesan arunachalam
    Developer
    ThoughtWorks
    schedule 3 months ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Real world applications are hard to imagine without side effects such as writing to a DB or UI. IO-monads were introduced into the Haskell language to write programs in a robust way. And we have been using monads in our day to day job without even knowing those are monads such as IEnumerable/IObservable in languages like C#, Java.

    This talk is to give a basic understanding about monads and why should we care about that, and a little bit about onion/clean architecture.