Taming and Composing high performance Stream-Oriented Processing Systems for Big Data
Real time applications are dominating the industry! Data is the main ingredient in Internet-based, social media and Internet of things (IoT) systems, which generate continuous streams of events used for real time analytics. This poses a tremendous challenge due to the massive volume of data collected and processed. These event-based Real-time analysis systems can easily process millions of messages per second through new generation solutions by simply defining small flows and then combining them together to create processing graphs. In this talk, will cover the concepts behind high-performance streamed-oriented big data processing systems. We will explore messaging queue systems like Kafka and Akka Streams which let developers define their process workflows at a higher level to define a graph system enabling a high throughput. You will learn how to integrate high performance stream message queues and how to define process workflows in C# and F#.
Outline/Structure of the Experience Report
The problem of realtime analysis for BigData
The solution : Intro and concept about Event-Streaming
Use case of analyzing 6 millions messages per second using Kafka and Akka Stream
Pro and Cons
Patterns and designs to handle millions of event messages per seconds.
Composing and event stream analysis
Developers that are interested in Real-Time event analysis and high performance systems