Background Processing in Elixir with GenStage

schedule Dec 13th 04:25 PM - 05:10 PM place Senate people 10 Interested

We recently started working on a new Elixir project and had the need for a background processing system.

We decided to build our own background processing system in Elixir.

This talk describes why and how we built our own background processing system in Elixir.

How we use Elixir's GenStage package for adding back-pressure between the producer and consumer stages and how we add support for custom rate-limiting between several pipelines.

 
3 favorite thumb_down thumb_up 0 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

The talk covers the following topics:

  • Why did we build our own background processing system.
  • How we use Elixir's GenStage to build producer/consumer pipelines.
  • How we add custom rate-limits to each pipeline.
  • How we use Redis for persisting jobs to be processed later.
  • How we exponentially retry failed jobs.
  • How we add job scheduling.
  • Any issues/roadblocks we faced while working on our current solutions.
  • The road ahead.

Learning Outcome

The talk will give the audience some familiarity with background processing systems and the inner workings of Elixir's GenStage library.

The audience will also learn some basics of Redis and its data structures.

Target Audience

Anyone interested in exploring & building solutions with Elixir/Erlang

Prerequisite

* Erlang/Elixir

* OTP

schedule Submitted 2 months ago

Comments Subscribe to Comments

comment Comment on this Submission

  • Liked Abhinav Sarkar
    keyboard_arrow_down

    Abhinav Sarkar - Many Ways to Concur

    Abhinav Sarkar
    Abhinav Sarkar
    Principal Engineer
    Flipkart
    schedule 3 months ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Easy concurrency is one of the main prophesied benefits of the modern functional programming (FP) languages. But the implementation of concurrency differs widely between different FP languages. In this talk, we shall explore the methods and primitives of concurrency across three FP languages: Haskell, Erlang, and Clojure (with core.async).

    We shall learn about and compare the trade-offs between

    • the green threads and STM channels oriented concurrency of Haskell
    • everything-is-a-process and message-passing actor pattern of Erlang
    • macro-based state machine code transformation of Clojure/core.async
  • Liked Kiran Gangadharan
    keyboard_arrow_down

    Kiran Gangadharan / Udit Kumar - The Road to a Masterless Multi-node Distributed System in Elixir

    45 Mins
    Talk
    Intermediate

    In this talk, we will cover our journey from the base to the summit of a masterless multi-node distributed system in Elixir. We’ll talk about the challenges faced, decisions made and how they shaped/evolved the architecture.