Background Processing in Elixir with GenStage
We recently started working on a new Elixir project and had the need for a background processing system.
We decided to build our own background processing system in Elixir.
This talk describes why and how we built our own background processing system in Elixir.
How we use Elixir's GenStage package for adding back-pressure between the producer and consumer stages and how we add support for custom rate-limiting between several pipelines.
Outline/Structure of the Talk
The talk covers the following topics:
- Why did we build our own background processing system.
- How we use Elixir's GenStage to build producer/consumer pipelines.
- How we add custom rate-limits to each pipeline.
- How we use Redis for persisting jobs to be processed later.
- How we exponentially retry failed jobs.
- How we add job scheduling.
- Any issues/roadblocks we faced while working on our current solutions.
- The road ahead.
The talk will give the audience some familiarity with background processing systems and the inner workings of Elixir's GenStage library.
The audience will also learn some basics of Redis and its data structures.
Anyone interested in exploring & building solutions with Elixir/Erlang
schedule Submitted 5 months ago
People who liked this proposal, also liked:
Abhinav Sarkar - Many Ways to ConcurAbhinav SarkarPrincipal EngineerFlipkart
schedule 5 months agoSold Out!
Easy concurrency is one of the main prophesied benefits of the modern functional programming (FP) languages. But the implementation of concurrency differs widely between different FP languages. In this talk, we shall explore the methods and primitives of concurrency across three FP languages: Haskell, Erlang, and Clojure (with core.async).
We shall learn about and compare the trade-offs between
- the green threads and STM channels oriented concurrency of Haskell
- everything-is-a-process and message-passing actor pattern of Erlang
- macro-based CSP transformation of Clojure/core.async
Kiran Gangadharan / Udit Kumar - The Road to a Masterless Multi-node Distributed System in Elixir
In this talk, we will cover our journey from the base to the summit of a masterless multi-node distributed system in Elixir. We’ll talk about the challenges faced, decisions made and how they shaped/evolved the architecture.