Streams, effects and beautiful folds, a winning trilogy

Most applications are just reading data, transforming it and writing it somewhere else. And there are great libraries in the Scala eco-system to support these use cases: Akka-Stream, fs2, Monix,... But if you look under the hood and try to understand how those libraries work you might be a bit scared by their complexity!

In this talk you will learn how to build a very minimal "streaming library" where all the difficult concerns are left to other libraries: eff for asynchronous computations and resources management, origami for extracting useful data out of the stream. Then you will decide how to spend your complexity budget and when you should pay for more powerful abstractions.

 
3 favorite thumb_down thumb_up 0 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

This talk presents requirements for stream programming then shows how they could be decoupled (contrary to existing libraries) into

1. a datatype for producing values

what can we do with a simple Producer[M[_], A] where M is a monad? A LOT!

2. a library for managing effects like eff

for resource safety, concurrency, errors

3. a library for outputting values

how "monadic folds" can be composed, reused with many streaming abstractions and implement a variety of useful processing

Then what would be some required extensions to fully recreate existing libraries:

  •  asynchronous queues
  •  unboxing
  • a Reader effect for FRP like API

 

Learning Outcome

  • a better understanding of the requirements around streaming libraries
  • a new proposal to decompose them into reusable pieces
  • some uses of the Eff monad
  • "beautiful folds" as a way to structure streaming outputs 

Target Audience

Scala developers

schedule Submitted 9 months ago

Comments Subscribe to Comments

comment Comment on this Proposal

  • Liked Manuel Chakravarty
    keyboard_arrow_down

    Manuel Chakravarty - Haskell SpriteKit - a Purely Functional API for a Stateful Animation System and Physics Engine

    45 mins
    Demonstration
    Intermediate

    Graphics, animation, and games programming in Haskell faces a dilemma. We can either use existing frameworks with their highly imperative APIs (such as OpenGL, Cocos2D, or SpriteKit) or we waste a lot of energy trying to re-engineer those rather complex systems from scratch. Or, maybe, we can escape the dilemma. Instead of a Haskell program directly manipulating the mutable object-graph of existing high-level frameworks, we provide an API for purely functional transformations of a Haskell data structure, together with an adaptation layer that transcribes those transformations into edits of the mutable object-graph.

    I this talk, I will explain how I used this approach to architect a Haskell binding to the animation system and physics engine of Apple’s SpriteKit framework. I will discuss both how the binding is structured and how it achieves the translation of Haskell side changes to SpriteKit and vice versa, such that it is sufficiently efficient. Moreover, I will demonstrate by example how to use the Haskell library to implement a simple game.

  • Liked Manuel Chakravarty
    keyboard_arrow_down

    Manuel Chakravarty - Functional Programming in Swift

    Manuel Chakravarty
    Manuel Chakravarty
    Lambda Scientist
    Applicative & Tweag I/O
    schedule 4 months ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    Swift’s strong functional programming support generates a lot of interest in adapting functional programming techniques to improve designing, developing, and maintaining Cocoa applications. This workshop will help you unlock this potential.

    We will explore the foundations of functional programming and how functional programming helps to improve software development. We will start by discussing the main goals of functional programming, followed by surveying Swift’s language features that support functional programming. Then, we will look at how to put functional programming principles into practice using Swift. We will focus on immutable data structures, value types, and functional reactive programming.

    = Value types 

    One of the biggest changes in the transition from Objective-C to Swift is the new type system. This change is at the heart of Swift’s language design, from optionals over value types to constrained generics, and to truly understand the language, a thorough understanding of its type system is indispensable. We will survey these type system features with special attention to value types. We will discuss how structs and enums in Swift can be used to help avoid bugs by design and how immutable data structures help to avoid unwelcome side effects.

    = Functional reactive programming

    Functional reactive programming (FRP) promises to replace unstructured side effects by structured change propagation in response to user input, network events, sensor activity, and similar. Typically explained by way of sophisticated, but often also complex functional reactive programming frameworks, such as ReactiveCocoa and RxSwift, it can be difficult to see the simple principles underlying functional reactive programming. In fact, these principles are sufficiently simple that we can explain them by building a functional reactive programming library from scratch.

    We will discuss how we can construct our own lightweight functional reactive programming library out of simple components using Swift’s support for protocol-oriented programming. Moreover, we will discuss how to use this library to design and implement the change propagation between the model and view components of a Cocoa app. 

    All of this is illustrated with hands-on coding exercises based on simple iPhone application that incorporates the discussed techniques.

  • Liked Eric Torreborre
    keyboard_arrow_down

    Eric Torreborre - Practical eff monad for micro-services

    Eric Torreborre
    Eric Torreborre
    Senior Software Developer
    Zalando
    schedule 9 months ago
    Sold Out!
    45 mins
    Talk
    Advanced

    This talk presents how we use the Eff monad at Zalando to structure a REST microservice. Even when programming with functions, modules and components still matter, and I will present a new approach for dependency injection using the Reader monad (classic) and tree rewriting (what?). Then we will see how to specify more precisely several components (authentication, request to other services and database access) with the Eff monad and support additional concerns like logging or asynchronous execution.

    Once you get the basic idea, the possibilities are endless! Rate limiting is just another component and another set of effects involving time and state, testing becomes really clean and easy.

  • Liked Aloïs Cochard
    keyboard_arrow_down

    Aloïs Cochard / Tony Morris - Introduction to Functional Programming using Haskell

    Aloïs Cochard
    Aloïs Cochard
    Computer Science Research
    Orexio / Swisscom
    Tony Morris
    Tony Morris
    Senior Functional Programming Engineer
    Data61 / CSIRO
    schedule 3 months ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    We will be spending the day learning the fundamentals of Functional Programming (FP) using the Haskell programming language. The exercise material will be a condensed selection of the NICTA/course which is regularly held in Australia over three days.

    This one day session is targeted to experienced industry programmers who are looking to break into Functional Programming and develop the rudimentary skills and techniques that enable continued independent study. A refresher on Haskell syntax will be provided, however, it is highly recommended to practice with the syntax and development tools prior to obtain the best outcome for the day.

    You will be required to bring a suitable development machine (portable) for working through the exercises. You will also need to install Glasgow Haskell Compiler (http://www.haskell.org/ghc/) version 7.8 or higher on that machine prior to the day.

  • Liked Mushtaq Ahmed
    keyboard_arrow_down

    Mushtaq Ahmed / Unmesh Joshi - Service Discovery using CRDTs

    45 mins
    Experience Report
    Intermediate

    Service Discovery is an important aspect of microservices, particularly when using containers on elastic infrastructure. Most popular approaches for service discovery require a standalone fault tolerant cluster like Consul, Zookeeper or etcd.

    If you are already using Akka as a clustering platform for your services, there is an interesting alternative available for Service Discovery. CRDTs, or Conflict-free Replicated Data Types are a relatively new primitive in distributed systems. CRDTs allows concurrent updates without coordination and has provisions for tuning consistency levels. 'Distributed Data' extension natively supports CRDTs on top of Akka-Cluster.

    We used CRDTs in Akka to build the service registration and discovery mechanism for TMT's (Thirty Meter Telescope) software platform. TMT will be world's largest optical telescope once operational. The ongoing software work is open source and available on github. We would like to share the details of our implementation and also compare it existing approaches.

  • Debasish Ghosh
    Debasish Ghosh
    Consultant
    IRadix Inc
    schedule 4 months ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    Using a programming language that supports first class functional programming doesn’t necessarily imply that the code you write is functional. Functional programming has the nice properties of being compositional - you design smaller abstractions and then reuse them to build larger ones. You use the power of algebraic properties that these abstractions offer to build larger elements of your model out of smaller ones.

     However often we find that many code bases designed in a functional programming language are inherently imperative, especially when you are using a multi-paradigm language like Scala or Java. There’s nothing wrong with imperative code, but it lacks the niceties of expression based composition that makes reasoning about your code so much easier. Also assignments used in imperative code are inherent side-effects that are best avoided (or at least minimized) in an expression oriented language.

     In this session I will discuss some of my experiences of mining imperative code and refactoring them into functional patterns. You will be surprised to see how much functional goodness can be derived through functional thinking. Algebraic structures like Monoids, Monads and Kleislis are just buried within the reams of imperative statements crying to come out. In an exercise of half an hour we will liberate them and see how the code quality improves from the point of view of modularity, compositionality and simplicity.

  • Liked Eric Torreborre
    keyboard_arrow_down

    Eric Torreborre - The Eff monad, one monad to rule them all!

    Eric Torreborre
    Eric Torreborre
    Senior Software Developer
    Zalando
    schedule 9 months ago
    Sold Out!
    45 mins
    Talk
    Advanced

    Monad transformers are a neat way to compose effectful computations: pass some configuration, read/write from disk, track errors... However they can be a bit cumbersome to deal with, especially if you want to use different stacks sharing some common effects: for example a stack to interact with AWS and another one for Hadoop computations.

    This talk will shortly introduce monad transformers in Scala and their drawbacks then present the Eff monad.

    The Eff monad has some really interesting properties:

    • effects can be developed completely independently from each other
    • effects can be interpreted in different ways and the interpreters can be composed together
    • it is based on an efficient "Free monad" (avoiding the issue of quadratic flatMaps)
    • with the Eff monad you will never have to re-implement the Monad operations (point / flatMap) again!
  • Liked Martin Thompson
    keyboard_arrow_down

    Martin Thompson - High-Performance Messaging with Aeron

    Martin Thompson
    Martin Thompson
    High-Performance Computing Specialist
    Real Logic
    schedule 4 months ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    In this workshop you will learn how to build microservices with Aeron and SBE in Java. Microservices capable of handling millions of messages per second with very low and predictable response times. We will cover some networking theory and the design of Aeron so you know how to use it with mechanical sympathy.

    The core design of Aeron uses a new functional data structure and functional concepts. In this workshop we will explore how functional designs applied with mechanical sympathy can result in the highest performance messaging system available.

  • Liked Shad Amez
    keyboard_arrow_down

    Shad Amez - Functional Programming with Spark

    Shad Amez
    Shad Amez
    Senior Software Engineer
    Informatica
    schedule 6 months ago
    Sold Out!
    20 mins
    Demonstration
    Beginner

    Spark is a general purpose distributed computing platform, designed to handle both batch and streaming applications. It extends on the map reduce paradigm initially coined by the Google in it's 2004 research paper. It leverages functional programming paradigm for doing the transformations on the datasets residing in cluster's memory. 

     Matei Zaharia, creator of Spark mentioned the importance of using functional programming language  - " At the time we started, I really wanted a PL that supports a language-integrated interface (where people write functions inline, etc) because I thought that was the way people would want to program these applications after seeing research systems that had it .. "

     

  • Liked Maria Livia Chiorean
    keyboard_arrow_down

    Maria Livia Chiorean - The path to generic endpoints using Shapeless

    20 mins
    Talk
    Beginner

    Programming in Scala is sometimes very time consuming. Finding the most efficient way to solve a problem can end up in days of frustration. This talk is a story of trail and error with a twist at the end. It's a story of API endpoints, generic programming, Shapeless and what happens when they all come together. Everything with examples and plenty of code.

  • Liked Paweł Szulc
    keyboard_arrow_down

    Paweł Szulc - Going bananas with recursion schemes for fixed point data types

    Paweł Szulc
    Paweł Szulc
    Scala Developer
    Slam Data
    schedule 7 months ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    In 1991 Erik Meijer, Maarten Fokkinga, and Ross Paterson published "Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire." This paper is a classic one, widely recognizable in the FP community. Constructs described - known as recursion schemas - have real world applications. Strictly speaking, explicit recursion is the ‘goto’ of pure functional programming. Recursion schemas provide same level of sanity as loops did for structural programming back in the day.

    Over the years a lot of the progress have been made. Papers like "Recursion schemes from comonads" by Tarmo Uustalu, Varmo Vene & Alberto Pardo or "Unifying Structured Recursion Schemes" by Ralf Hinze, Nicolas Wu & Jeremy Gibbons - pushed the concept forward.

    This talk is about generalization of a very specific recursion schema (called catamorphism) over fixed point data type. After providing introduction the concept of catamorphism, we will jump straight to fix point data types trying to solve some real-world problems. Code examples are in Scala. Code examples use Matryoshka - which is an open sourced project design to generalize folds, unfolds, and traversals for fixed point data structures in Scala.

  • Liked Tony Morris
    keyboard_arrow_down

    Tony Morris - Functional Programming in Aviation

    45 mins
    Case Study
    Beginner

    In this talk & demo, we have a look at some of the low-hanging problems in general aviation and how functional programming can be applied to provide significant improvements in efficiency and air safety. The current solutions to problems such as navigation, traffic/terrain collision avoidance and weight/balance calculations will be demonstrated to the audience, mostly for amusement. More seriously, we will have a look at the legacy that has led to the way things are, and how to improve by applying our programming skills.

    We will look at:

    • how aviation safety is regulated.
    • how aeronautical services are provided to flight operators.
    • how aeronautical navigation is conducted and regulated.
    • how the weight and balance for a flight is conducted.
    • the methods by which aircraft and ground coordinate between each other.

    We will see:

    • some real (and basic) data management problems in aviation, that very obviously threaten safety, then solve them, using programming.
    • we will see a live demonstration of aeronautical navigation methods, investigate incident reports where lives were lost as a result, and consider how our programming skills can yield improvements, possibly even save lives.
    • we will conduct a real weight&balance calculation for a flight, then once hilarity inevitably ensues, we will look at the problems that arise by this method, then solve them using data structures and functional programming. Some
      of these practical problems are obvious, even to a non-aviator, and the predictable incident reports are the end result.
    • finally, we will have a look at a live demonstration of a software defined radio (SDR), receiving ADS-B transmissions from aircraft (live), an AHRS implementation and GNSS receiver using off-the-shelf, low-cost parts. We will look at why these instruments are helpful to aircraft pilots and interact with that device using the Haskell programming language.
  • Liked Abhiroop Sarkar
    keyboard_arrow_down

    Abhiroop Sarkar - Wormholes and teleporting with Transient.

    Abhiroop Sarkar
    Abhiroop Sarkar
    Student
    University of Nottingham
    schedule 5 months ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    The transient library in Haskell/Eta composes arbitrary pieces of software in concurrent, parallel and sequential settings using standard Haskell/Eta combinators like applicative, alternative, bind etc. It provides algebraic and monadically composable primitives for concurrency, parallelism, event handling, transactions and multithreading

    It liberates the programmers from dealing with state machines, objects frameworks, routes, configurations and callbacks. Transient expresses business functionalities as compact category theoretical expressions which compose. These compositions are verified during compilation, although the code might be executing in different machines.

    It presents a fresh new approach to programming concurrent and distributed systems.

     

  • Liked Aloïs Cochard
    keyboard_arrow_down

    Aloïs Cochard - The Tale of Troy

    45 mins
    Demonstration
    Beginner

    In recent years, the impact of functional programming on mainstream languages has grown significantly.

    After conquering server applications and distributed systems,
    the paradigm is now conquering the fields of web development and data analytics.

    So what remains? The land of system programming.

    When Mozilla did release the language Rust in 2012,
    it did immediately striked interest from functional programming enthusiasts due to it's strong emphasis on safety.

    Even though it currently lacks some advanced features like higher kinded types,
    it is built with functional programming principles as it's core.

    Like a Trojan horse (or the parasite mushroom after the language is named),
    Rust looks like the next C on the outside... while it might actually be closer to Haskell on the inside!

    Let's take a tour and discover an other side of system programming by learning together how to do functional programming in Rust.

  • Liked Ryan Lemmer
    keyboard_arrow_down

    Ryan Lemmer - From Haskell 98 to Modern Haskell - Making sense of the Haskell type system

    45 mins
    Talk
    Intermediate

    There are several great books and courses that serve as an introduction to functional programming in Haskell. Typically, these introductions focus on the original core of Haskell ("Haskell 98").

    However, going from there into real world Haskell systems can be a bewildering experience. This is because Haskell has evolved significantly since Haskell 98. In this talk we will investigate the key advances in the type-system that make up "modern Haskell".

    • type-system extensions: GADTs, multi-param type-classes, functional dependencies, ...
    • kind-system extensions: type functions, kind polymorphism, type promotion, ...

    By taking a historical perspective we will look at why new language extensions were introduced, and the context in which they first appeared. This approach gives us a layered view of seemingly disparate language features.

    We will see that the evolution of the type system has been a steady march from term-level programming to type-level programming, and ultimately dependently-typed programming in Haskell.

    (This talk will draw from the book "Haskell Design Patterns")

  • Liked Nikhil Tibrewal
    keyboard_arrow_down

    Nikhil Tibrewal / Rafał Wojdyła - Scio – Scala DSL for Apache Beam

    45 mins
    Talk
    Intermediate

    Apache Beam (based on Google’s Dataflow Model) provides a simple, unified programming model for both batch and streaming data processing. If only it wasn’t so unfamiliar and verbose for our Scala engineers. Learn how Scio leverages Scala’s type system, macros and functional paradigm to provide more engineer-friendly and type safe API.

  • Liked Arthur Kushka
    keyboard_arrow_down

    Arthur Kushka - Recursion schemes and fixed point data types

    Arthur Kushka
    Arthur Kushka
    Software developer
    AutoScout24 GmbH
    schedule 7 months ago
    Sold Out!
    45 mins
    Talk
    Advanced

    Its always challenge to build a good abstraction during work on library. Library codebase must be flexible and modular when user code must be clean and easy understandable. One of variants to build API between library and customer its Free evaluation approach. In that case you as developer is free to interpret customer actions as you want and customer is free to define actions as he want. In my talk I will show how to build API based on recursive abstract data types, describe why its cool and demonstrate some tricks on type system to make customer experience with library better.

     
  • Liked Oskar Wickström
    keyboard_arrow_down

    Oskar Wickström - The Power of Functional Programming and Static Type Systems in Server-Side Web Applications

    Oskar Wickström
    Oskar Wickström
    Programmer
    Empear
    schedule 10 months ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    Single-page web applications have been spreading like wildfire, with an endless amount of frameworks and libraries, and functional programming together with static types fueling the fire with great ideas like pure functions, monads, and strong type checking. But what happened to Progressive Enhancement? Some parts of our applications might require Javascript to function, but the majority could be built with ordinary links and forms.

    In this talk we will explore how we can build web applications using established web technology, and the power of functional programming on the server-side, leveraging what is already in our browsers.

  • Liked Tony Morris
    keyboard_arrow_down

    Tony Morris - Parametricity, Types are Documentation

    45 mins
    Talk
    Intermediate

    In this talk, we define the principle of functional programming, then go into
    detail about what becomes possible by following this principle. In particular,
    parametricity (Wadler, 1989) and exploiting types in API design are an essential
    property of productive software teams, especially teams composed of volunteers
    as in open-source. This will be demonstrated.

    Some of our most important programming tools are neglected, often argued away
    under a false compromise. Why then, are functional programming and associated
    consequences such as parametricity so casually disregarded? Are they truly so
    unimportant? In this talk, these questions are answered thoroughly and without
    compromise.

    We will define the principle of functional programming, then go into
    detail about common problems to all of software development. We will build the
    case from ground up and finish with detailed practical demonstration of a
    solution to these problems. The audience should expect to walk away with a
    principled understanding and vocabulary of why functional programming and
    associated techniques have become necessary to software development.

  • Liked Tony Morris
    keyboard_arrow_down

    Tony Morris - Explain List Folds to Yourself

    45 mins
    Talk
    Beginner

    In this talk, we go back to first principles, defining and examining the definition for a cons list, then take a look at the ubiquitous right and left fold functions on a list.

    The primary focus of this talk is to develop an intuition for how these functions work so that we can best decide when to apply them. Multiple programming languages will be used to help emphasise the independence of the gained intuition. This talk will generally be interactive with the audience as we solve problems and build confidence in our new knowledge.

    Knowing how to apply the various list fold functions is a common question by the student of FP. This talk aims to provide a solid, reliable answer to this question. No prior understanding of list folds is necessary.