Designing powerful real-time systems using Scala and Akka
Sky BML (Beat/Meet/Lose) is a streaming system that produces real time Competitive Intelligence about our Lodging Supply.
This competitive information is being used, among others, for Market Managers to negotiate with hotels for better rates and avail. Competitiveness is proportional to the share of BML Losses that Expedia incurs. This is why Market Managers focus their energies on minimising losses.
However, it is not easy to understand why losses occur and consequently their resolution takes proportionately long.Our system tries to simplify this and hence accelerate resolution of losses.
To develop this system, we fully leveraged the power of different functional features of Scala and used Structured Spark Streaming as the processing engine in combination with Clustering(for better app supervision and lifecycle management of the nodes).
Similarly, some of the complex challenges are solved very naturally using functional programming features of easily. We had this complex problem of traversing a decision tree (based on conditional expressions) and then evaluating the child nodes(which are also expressions) with different lookup services responsible for mapping the symbols used in Expressions.
I will be discussing in detail on how we tactically use Functional Programming concepts like Implicits, currying, High-Order Functions to get rolling with this problem with ease.
Similarly, I will discuss on the way we have apply OOP principles to handle different kinds of inputs(streaming/batch) with minimal code duplication.
Overall, it will be lot of fun to discuss the way to think about our problems and how our thinking is shaped by different concepts available in Scala.
Outline/Structure of the Talk
1) Visuals of our real-time data processing engine and its capabilities in delivering business value and churning out competitive info.
2) Deep dive into one of the critical components of our system.
3) How we have improved our app supervision capabilities using Clustering.
4) How we use Scala constructs and capabilities to solve our business problem with ease.. Expect some crisp discussions on implicits, HOF(s), currying and many others (in the context of our problem)
5) How we handle the growing variety of data using Scala and make our app versatile.'
6) (New addition) Walking through the concepts of fixtures in Scala and how we apply it in all our integration tests here at Expedia to build our testing frameworks..
7) (New addition) (OPEX-driven development) : Achieving Operational Excellence in out coding style
8) Q&A (lots if them!)
Also, in between the talk and the slides(and ofcourse during the Q&A).. I will try to draw out some diagrams using Paint or Miro..though it is presentation but I really want to give it the feel of a demo wherein . many explanations and diagrams are made/presented right at the moment,
Key learnings :
1) I will be live coding many things ...(updated the ppt to reflect the exact slides at which I will do so)..e.g. coding up a Clustered application from scratch, writing a small Integration test using Scala fixtures(something new that we are doing in all our projects now)...so people can directly apply these learnings to their own projects.
2) There are many features like implicits, high-order functions etc which we study about but forget... I will explicitly focus on those features while explaining how we solved the decision-tree expression evaluation problem... which will help us understand them better and also start thinking of problems in terms of functional programming.
3)The audience will get to know how we build/monitor systems here at Expedia e.g. there is a separate section on monitoring the real-time systems(OPEX.)
Additional Notes :
I would love to see people asking lot of questions.. that would be proof that they are actually involved in what's going on..
Secondly, they should walk out with easy-to-grasp , new ideas that they can discuss with their team right away and start building on that..
Lastly, it will be great learning for me as well; to get feedback/questions from people and carry it back to my place :)
Everyone who loves Scala and wants to see some new kinds of use-cases getting solved very naturally by harnessing its power.
Prerequisites for Attendees
Basic knowledge of Scala would be good...
Mainly, i need zeal and am looking forward to many questions
schedule Submitted 9 months ago
People who liked this proposal, also liked:
Anupam Jain - Typesafe Functional SQLAnupam JainAssociate Director, AnalyticsS&P Global
schedule 9 months agoSold Out!
We'll discuss how to use strongly typed functional programming to build a typesafe and highly declarative DSL to generate SQL, that allows better abstractions and minimises runtime errors. We'll focus on type systems, and build an intuition for row types and how it guarantees correctness for SQL. I'll cover some parts of actual SQL generation.
A similar talk was given at IIIT Delhi earlier, however that was more geared towards beginners to Functional Programming (link below). I plan to give a more intermediate level talk this time.
Anish George - Demystifying Function Sub-typingAnish GeorgeUI DeveloperThoughtWorks
schedule 11 months agoSold Out!
What is a Type for a function? When can you say that a function is a sub-type of another function? Why should you even bother?
Function sub-typing is one of the most confusing areas of FP. In this talk, I'll take a deep dive and explain the core concepts of function types which applies to many different languages. More than just explaining the mere rules, I will be deriving those rules in a step-by-step fashion till the eureka moment so that you will never be confused again.
The topics covered:
- Sub-typing basics
- Covariance & Contravariance
- Function sub-typing
Siddharth Bhat - Monad-bayes: Probabilistic programming in haskell, tutorial & implementationSiddharth BhatStudentIIIT Hyderabad
schedule 9 months agoSold Out!
We take a dive into probabilistic programming, beginning with a high-level explanation of what probabilistic programming is. We then continue to see how to use the monad-bayes library for performing tasks such as regression in a bayesian formalism. We also look into the implementation of the library, which involves performing an interesting sampling method (markov-chain-monte-carlo) on a very unique space (the space of computational traces of the program). We finally end with next-steps for the haskell community to improve in the probablistic programming space, ala pyro for python
Here is our tiny re-implementation of monad-bayes, boiled down to the essentials: https://github.com/bollu/shakuni
Paper on which the talk is based on: http://mlg.eng.cam.ac.uk/pub/pdf/SciGhaGor15.pdf
Harmeet Singh - Type System: The Beauty And The BeastHarmeet SinghModule Lead Software ConsultantKnoldus
schedule 10 months agoSold Out!
The type system plays an important role in building type-safe applications that reduce runtime exceptions and developer mistakes. This sophisticated system is leveraged by Scala’s most famous functional libraries, such as Scala-Cats and ScalaZ. This feature-rich system allows implementing pure functional programming on the JVM. It is a beast due to its complex syntax, as it prevents developers from exploring its beautiful aspects like partially applying types in Scala. In this talk, we will tame the beast and use its beauty to solve real-life issues faced during coding. Our examples will be based on generics, constraints, type lambdas, and the kind projector plugin.
The following are the learning objectives of this talk:
Practical use cases of the type system using real-life coding.
Implementing custom higher kinded types and handling partially applied types via type lambdas and the kind projector plugin.
ganesan arunachalam - What the hell is monad? Why should you care?ganesan arunachalamDeveloperThoughtWorks
schedule 9 months agoSold Out!
Real world applications are hard to imagine without side effects such as writing to a DB or UI. IO-monads were introduced into the Haskell language to write programs in a robust way. And we have been using monads in our day to day job without even knowing those are monads such as IEnumerable/IObservable in languages like C#, Java.
This talk is to give a basic understanding about monads and why should we care about that, and a little bit about onion/clean architecture.