Organising code as chained combination of data processing steps is a common pattern. Functional programming has made it ubiquitous due to concise lambda notations. But there are wide variations in how these pipelines are implemented. In this talk we will explore important properties of lazy data streams such as: push vs pull, hot vs cold, linear-flows vs graphs, perishable vs reusable, unicast vs multicast, synchronous vs asynchronous back-pressured etc.

We will then explain "reactive-streams" protocol which guarantees asynchronous back-pressure for push based data streams. We will conclude with a demo using Akka-Streams which is an implementation of this protocol. The demo will involve a) streaming images from server to the browser over websockets, b) transferring large files from one source server to multiple destination servers.


Outline/structure of the Session

- Motivate with use cases

- Explain basic concepts around data streams

- Compare and Contrast different implementation approaches

- Demo running code for each atomic concept

- Final demo using Akka-Http

Learning Outcome

Deeper understanding about differences between push vs pull, lazy vs eager, hot vs cold and other properties of data pipelines. Appreciating why push-based and back-pressured "reactive-streams" protocol is essential for IO based programming

Target Audience

Developers, Tech leads, Architects

schedule Submitted 1 year ago

Comments Subscribe to Comments

comment Comment on this Proposal