Background Processing in Elixir with GenStage

location_city Bengaluru schedule Dec 13th 04:25 - 05:10 PM place Senate people 10 Interested

We recently started working on a new Elixir project and had the need for a background processing system.

We decided to build our own background processing system in Elixir.

This talk describes why and how we built our own background processing system in Elixir.

How we use Elixir's GenStage package for adding back-pressure between the producer and consumer stages and how we add support for custom rate-limiting between several pipelines.


Outline/Structure of the Talk

The talk covers the following topics:

  • Why did we build our own background processing system.
  • How we use Elixir's GenStage to build producer/consumer pipelines.
  • How we add custom rate-limits to each pipeline.
  • How we use Redis for persisting jobs to be processed later.
  • How we exponentially retry failed jobs.
  • How we add job scheduling.
  • Any issues/roadblocks we faced while working on our current solutions.
  • The road ahead.

Learning Outcome

The talk will give the audience some familiarity with background processing systems and the inner workings of Elixir's GenStage library.

The audience will also learn some basics of Redis and its data structures.

Target Audience

Anyone interested in exploring & building solutions with Elixir/Erlang

Prerequisites for Attendees

* Erlang/Elixir


schedule Submitted 2 years ago

  • Abhinav Sarkar
    Abhinav Sarkar
    Software Engineer
    schedule 2 years ago
    Sold Out!
    45 Mins

    Easy concurrency is one of the main prophesied benefits of the modern functional programming (FP) languages. But the implementation of concurrency differs widely between different FP languages. In this talk, we shall explore the methods and primitives of concurrency across three FP languages: Haskell, Erlang, and Clojure (with core.async).

    We shall learn about and compare the trade-offs between

    • the green threads and STM channels oriented concurrency of Haskell
    • everything-is-a-process and message-passing actor pattern of Erlang
    • macro-based CSP transformation of Clojure/core.async
  • Kiran Gangadharan

    Kiran Gangadharan / Udit Kumar - The Road to a Masterless Multi-node Distributed System in Elixir

    45 Mins

    In this talk, we will cover our journey from the base to the summit of a masterless multi-node distributed system in Elixir. We’ll talk about the challenges faced, decisions made and how they shaped/evolved the architecture.