Streamly: Declarative Concurrency and Dataflow Programming in Haskell

schedule Nov 15th 04:00 - 04:45 PM place Ball Room 1 people 4 Interested

Overview: Streamly is a Haskell library for writing programs in a high level, declarative data flow programming paradigm. It provides a simple API, very close to standard Haskell lists. A program is expressed as a composition of data processing pipes, generally known as streams. Streams can be generated, merged, chained, mapped, zipped, and consumed concurrently – enabling a high level, declarative yet concurrent composition of programs. Programs can be concurrent or non-concurrent without any significant change. Concurrency is auto scaled based on consumption rate. Programmers do not have to be aware of threads, locking or synchronization to write scalable concurrent programs. Streamly provides C like performance, orders of magnitude better compared to existing streaming libraries.

Motivation: Haskell has a beautiful abstraction capability. Haskell lists, several streaming and concurrency libraries are some examples of good programming abstractions. The problem is that concurrency is not handled elegantly and declaratively by these libraries. Moreover, the lack of good performance gives rise to special purpose performance centric libraries like text and bytestring creating more confusion. Streamly unifies, lists, streaming, concurrency, logic programming and reactive programming with unparalleled performance. We can finally have a unified string handling.

Audience: Streamly is designed to express the full spectrum of programs. Do not think that if you are writing a small and simple program it may not be for you. It expresses a small program with the same efficiency, simplicity and elegance as a large scale concurrent application. It unifies many different aspects of special purpose libraries into a single yet simple framework. It is especially suitable for high performance and scalable backend data processing systems. If you use or plan to use any real time streaming or data flow programming frameworks including Apache Flink, Apache Spark or asynchronous/reactive programming frameworks like ReactiveX/RxJs then you may be interested in Streamly.

 
 

Outline/Structure of the Talk

Functionality Overview: Streamly covers the functionality provided by Haskell lists as well as the functionality provided by streaming libraries like streaming, pipes, and conduit with a simpler API and better performance. Streamly provides advanced stream composition including various ways of appending, merging, zipping, splitting, grouping, distributing, partitioning and unzipping of streams with true streaming and with concurrency. Streamly subsumes the functionality of list transformer libraries like pipes or list-t and also the logic programming library logict. The grouping, splitting and windowing combinators in streamly can be compared to the window operators in Apache Flink. However, compared to Flink streamly has a pure functional, succinct and expressive API.

The concurrency capabilities of streamly are more advanced and powerful compared to the basic concurrency functionality provided by the async package. Streamly is a first class reactive programming library. If you are familiar with Reactive Extensions you will find that it is very similar. For most RxJs combinators you can find or write corresponding ones in streamly. Streamly can be used as an alternative to Yampa or Reflex as well.

Streamly focuses on practical engineering with high performance. From well written streamly programs one can expect performance competitive to C. High performance streaming eliminates the need for string and text libraries like bytestring, text and their lazy and strict flavors.

Structure of the Talk: In this talk we will introduce Streamly, its streaming API and how concurrency fits in the streaming framework. We will go through some examples and see how Streamly can express complex real world programs in a simple and concise manner. Broadly, the following topics will be touched:

  • Motivation for Streamly
  • Streams
  • Folds
  • Arrays
  • Concurrency
  • Managing State
  • Errors and Exceptions
  • Strings
  • File and Netwotk IO
  • Performance

Learning Outcome

You will learn about data flow/streaming programming paradigm and how you can express programs in a declarative manner. You will learn how concurrency can be expressed in a declarative manner and how it fits with data flow programming paradigm. You will understand where and how you can use Streamly to solve general programming problems. After this talk, and a bit of follow up reading, you should be able to write concurrent file/IO, networking (e.g. a web server) or reactive applications using Streamly (e.g. games).

Target Audience

Engineers and architects interested in designing scalable, concurrent, high performance and correct by construction systems with excellent composability and little maintenance overhead.

Prerequisites for Attendees

A general idea of basic functional programming constructs like map, fold and filter is needed. If you can work with Haskell lists then you can relate to this library well. If you know monads and monad transformers then you can definitely make sense out of almost everything that this talk covers.

schedule Submitted 1 month ago

Public Feedback

comment Suggest improvements to the Speaker

  • Liked Aaron W Hsu
    keyboard_arrow_down

    Aaron W Hsu - Programming Obesity: A Code Health Epidemic

    45 Mins
    Keynote
    Beginner

    Programs are getting fat. They're becoming slow. They're taking up more computing resources. They're getting harder to maintain and more complex from the ground up. Layer upon layer of sophistication is causing us to lose our ability to predict what software will do. Where's that bug? Why is everything going so slowly? Am I even using the right data structures? Where's that important point in the documentation again?

    What's happened to us? In this meta-dive into the nature of our approach to programming, we will explore some of the dangers of our current approaches to programming and the how/why of our current programming obesity problem. We will look at real case studies and see just how bad the situation can be.

    But we will also explore how we can battle these sources of obesity. In this passionate plea for code that we can gain control over again, we will look at examples of how we can return to a state of high-performance on all levels, from code size to code scalability. We will look at the principles that can help us to reach leaner, more efficient, more usable, less buggy code. We will hopefully find some light at the end of the tunnel, and how we can change our outlook on programming to push ourselves towards code that benefits not only ourselves, but also those that will come after us.

  • Liked Anne Ogborn
    keyboard_arrow_down

    Anne Ogborn - Introduction to Logic Programming and SWI-Prolog

    Anne Ogborn
    Anne Ogborn
    Logic Programmer
    DataChemist.com
    schedule 1 month ago
    Sold Out!
    480 Mins
    Workshop
    Beginner

    As machine learning matures, it is becoming obvious that we need explainable solutions. As functional programming matures it becomes obvious that we need inference and nondeterminism. And linked open data demands reasoning. This all day workshop will introduce the logic programming paradigm, in which programs are expressed as a set of logical rules and executed by finding proofs of queries.

    SWI-Prolog is the most highly developed and widely used language for logic programming.

    The language, which has been in continuous use in academic research settings since it's invention in 1972, provides unparalleled power. Many problems which would be difficult to express in other languages are simple in SWI-Prolog. SWI-Prolog is a 'batteries included' modern language, ready for real world tasks like web development and process control.

    In this dynamic hands on workshop we'll learn the basics of SWI-Prolog, and look at some of the amazing things it can do.

  • Liked Alexander Granin
    keyboard_arrow_down

    Alexander Granin - Hierarchical Free Monads and Software Design in Functional Programming

    Alexander Granin
    Alexander Granin
    Sr. Haskell Developer
    Juspay
    schedule 3 months ago
    Sold Out!
    45 Mins
    Talk
    Advanced

    Functional Programming is going through the same process of rethinking as Object Oriented Programming has passed already. FP today is not only something mystical, something barely applicable for day-to-day tasks, but rather a whole philosophy of building big, complex applications, with own tools, approaches and methodologies. We can see a lot of ideas about software design and architecture emerging from the FP community: talks, books, articles. We’re actively discussing Final Tagless, Free Monads, effect systems and other approaches, but it seems the theme not yet covered and we don’t know about possibilities we have.

    In this talk, I’ll tell you about the power of Free Monads in building complex applications in Haskell and PureScript. These applications will be satisfying the requirements like maintainability, testability, simplicity. You’ll know how to create a safe concurrent application state, how to divide the application into layers and how to keep the code sane. In fact, you can do the same with Final Tagless or extensible effects, but the approach with hierarchical Free Monads has some specific properties making it the best tool to separate concerns and create a very interesting eDSLs with different semantics.

    The talk is based on the ideas I’m describing in my book “Functional Design and Architecture”. I also gave several talks about this theme earlier (you can find all my talks here).

    I’ve used these ideas to build software for different companies, including Juspay (Bengaluru), Restaumatic (Poland), Enecuum (Hong Kong). We’ve created several interesting technologies that were not possible to make without Free Monads. Some of them are open sourced.

  • Liked Anupam Jain
    keyboard_arrow_down

    Anupam Jain - Typesafe Functional SQL

    45 Mins
    Talk
    Intermediate

    We'll discuss how to use strongly typed functional programming to build a typesafe and highly declarative DSL to generate SQL, that allows better abstractions and minimises runtime errors. We'll focus on type systems, and build an intuition for row types and how it guarantees correctness for SQL. I'll cover some parts of actual SQL generation.

    A similar talk was given at IIIT Delhi earlier, however that was more geared towards beginners to Functional Programming (link below). I plan to give a more intermediate level talk this time.

  • Liked Tony Morris
    keyboard_arrow_down

    Tony Morris - An Intuition for List Folds

    45 Mins
    Talk
    Beginner

    In this talk, we go back to first principles, defining and examining the definition for a cons list, then take a look at the ubiquitous right and left fold functions on a list.

    The primary focus of this talk is to develop an intuition for how these functions work so that we can best decide when to apply them. Multiple programming languages will be used to help emphasise the independence of the gained intuition. This talk will generally be interactive with the audience as we solve problems and build confidence in our new knowledge.

    Knowing how to apply the various list fold functions is a common question by the student of FP. This talk aims to provide a solid, reliable answer to this question. No prior understanding of list folds is necessary.

  • Liked Tony Morris
    keyboard_arrow_down

    Tony Morris - Zippers

    45 Mins
    Talk
    Intermediate

    The term zipper is a colloquial used to describe n-hole (most often, 1-hole) contexts. That is, a data structure that has a _hole_ or _pointer_ focused on a specific element with the ability to efficiently traverse to its neighbouring elements, providing an elegant solution for the need to efficiently traverse and _modify_ immutable data structures.

    In this talk, we will look at examples of zippers for canonical data structures such as lists and other products and sums. We will then define comonads and see the relationship between zippers and comonads.

    Most of this talk will be spent on the practical application of zippers in everyday programming. We will solve some simple problems using zippers. We will then contrast why we might use a zipper, compared to a lens.

    Finally, for a fun and interesting observation, we will look at the algebraic structure of products and sums, then compute the derivative of these structures. Turns out, a derivative of a data structure is its zipper ("McBride, Conor, et al (2005). ∂ for Data: Differentiating Data Structures").

  • Liked Mohd Maqbool Alam
    keyboard_arrow_down

    Mohd Maqbool Alam - Building a MySQL Database Driver in Elixir

    45 Mins
    Demonstration
    Intermediate

    Have you ever wondered what happens beneath the covers when you talk to your Database? Well, you are in for a treat! In this talk, we are going to uncover the dark magic behind Database Drivers. We will look at everything that is needed to talk to a database, query its data and transform it into the native data types in Elixir.

  • Liked Morten Kromberg
    keyboard_arrow_down

    Morten Kromberg - Delivering your APLs

    Morten Kromberg
    Morten Kromberg
    CXO
    Dyalog
    schedule 1 month ago
    Sold Out!
    45 Mins
    Tutorial
    Beginner

    Most talks on APL at FunctionalConf have focused on the way array oriented programming allows you to quickly deliver concise and efficient solutions to problems with an analytical core. This talk will focus on the development environment, and follow the life of an APL code snippet from it's interactive discovery, via testing and debugging, through to delivery as a web service and a shared object, embedded into a solution implemented in Python.

  • Liked Aaron W Hsu
    keyboard_arrow_down

    Aaron W Hsu - APL Training Wheels

    Aaron W Hsu
    Aaron W Hsu
    Computer Scientist
    Indiana University
    schedule 2 months ago
    Sold Out!
    45 Mins
    Tutorial
    Beginner

    APL is getting a lot of attention lately due to its potential for very high performance portability and suitability for both rapid prototyping of complex solutions as well as deployment of complex algorithms to high-speed, modern parallel hardware. It has the potential to vastly improve the speed, scalability, and size of your code bases. But APL has a reputation as an intimidating language to learn.

    In this back to the basics tutorial, we will explore the core of APL, and focus on those areas that usually trip up the beginner in learning APL. We will also walk you through how to approach an APL expression, how to reason about them, and how to read them efficiently. We will teach you the skills that the expert APLer has internalized, and how you can work through these skills externally and explicitly in a way that will help you to eventually internalize these critical skills in a way that makes you efficient at using APL on real world problems.

  • Liked Saurabh Nanda
    keyboard_arrow_down

    Saurabh Nanda - Getting property-based testing to work after struggling for 3 years

    Saurabh Nanda
    Saurabh Nanda
    Founder
    Vacation Labs
    schedule 1 month ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    I got excited about property-based testing after hearing John Hughes talk at Functional Conf 2016. I tried it that year with QuickCheck, but failed miserably (it almost derailed the entire project delivery). I cribbed about it in my talk at Functional Conf 2017. In 2018, Srihari's talk got me excited again. This time, I tried with Hedgehog, and got it to work!

    This talk is about this journey and its learnings. We'll talk about how Hedgehog was used to test [1]:

    • A Postgres backed task/job queue
    • A small Wai/Servant based webapp

    And no, we will not talk about the most common (and completely useless) example of reversing a list!

    [1] Both of these are part of an open-sourced task/job queue library.

  • Liked Raghu Ugare
    keyboard_arrow_down

    Raghu Ugare / Vijay Anant - Have you GADT ?

    45 Mins
    Talk
    Intermediate

    The type-system of Haskell is well-known as well as notorious for its rigour & strong mathematical foundations.
    In this talk, we would like to explain about the recent idea of GADT's and how they help eliminate runtime checks & issues.

    Our hope is that newbies to Functional Programming, and esp., Haskell are no longer intimidated by the term GADT's but instead see them as their ally in conquering some types of problems.


  • Liked Anupam Jain
    keyboard_arrow_down

    Anupam Jain - Supercharged Web Application development with Purescript

    45 Mins
    Workshop
    Beginner

    Purescript is a purely functional, strongly typed language, that compiles to Javascript.

    It provides several high quality libraries for frontend development, and comes with excellent tooling and editor support that makes writing code a snap!

    In this talk I will provide a quick introduction to some basics of Purescript, and then dive into an interactive demonstration of building a non-trivial web application from scratch. I will show how to interface with React bindings, and how to interface with some existing Javascript libraries (a React UI component lib).

    The presentation will be interactive and similar to an inpromptu talk I gave which is linked below. However, I will also demonstrate live coding of an actual purescript application which people can follow along.

    Bring your laptop to follow along.

  • Liked Edward Kmett
    keyboard_arrow_down

    Edward Kmett / Tony Morris - Let's Lens

    Edward Kmett
    Edward Kmett
    Chair of the Haskell Core Libraries Committee, Research Engineer
    Machine Intelligence Research Institute
    Tony Morris
    Tony Morris
    Sr. Software Engineer
    Queensland FP Lab, Data61
    schedule 1 month ago
    Sold Out!
    480 Mins
    Workshop
    Intermediate

    Let's Lens presents a series of exercises, in a similar format to the Data61 functional programming course material. The subject of the exercises is around the concept of lenses, initially proposed by Foster et al., to solve the view-update problem of relational databases.

    The theories around lenses have been advanced significantly in recent years, resulting in a library, implemented in Haskell, called lens.

    This workshop will take you through the basic definition of the lens data structure and its related structures such as traversals and prisms. Following this we implement some of the low-level lens library, then go on to discuss and solve a practical problem that uses all of these structures.

  • Liked Saurabh Nanda
    keyboard_arrow_down

    Saurabh Nanda - Why is Haskell so hard to learn? (and how to deal with it)

    Saurabh Nanda
    Saurabh Nanda
    Founder
    Vacation Labs
    schedule 1 month ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Haskell is notoriously hard to learn. I have struggled with the learning curve. I have seen others struggle with it. And I have seen developers struggling to wield the language effectively even after months of learning.

    We'll talk about five things that give Haskell this (understandably) bad rep, and how to effectively deal with them during your learning phase.

  • Liked Siddharth Bhat
    keyboard_arrow_down

    Siddharth Bhat - Monad-bayes: Probabilistic programming in haskell, tutorial & implementation

    Siddharth Bhat
    Siddharth Bhat
    Student
    IIIT Hyderabad
    schedule 1 month ago
    Sold Out!
    20 Mins
    Case Study
    Intermediate

    We take a dive into probabilistic programming, beginning with a high-level explanation of what probabilistic programming is. We then continue to see how to use the monad-bayes library for performing tasks such as regression in a bayesian formalism. We also look into the implementation of the library, which involves performing an interesting sampling method (markov-chain-monte-carlo) on a very unique space (the space of computational traces of the program). We finally end with next-steps for the haskell community to improve in the probablistic programming space, ala pyro for python

    Here is our tiny re-implementation of monad-bayes, boiled down to the essentials: https://github.com/bollu/shakuni

    Paper on which the talk is based on: http://mlg.eng.cam.ac.uk/pub/pdf/SciGhaGor15.pdf

  • Liked Anupam Jain
    keyboard_arrow_down

    Anupam Jain - ActiveTasx - A Haskell DSL for building Multi User, Strongly Typed, Flexible Workflows

    45 Mins
    Talk
    Beginner

    We introduce a Strongly Typed Workflow DSL for Haskell called ActiveTasx, which allows writing Multi-User, Interactive, and Effecful workflow applications at a very high level.

    We will discuss the benefits of building ActiveTasx as a DSL in a functional programming language.

    For example, ActiveTasx improves upon existing imperative Workflow engines by using Functional Programming and design principles. It allows you to compose smaller workflows together using combinators, to build progressively more complex workflows, and the resulting workflows behave predictably at every step.

    ActiveTasx also exploits Haskell's laziness to allow self referential and infinite workflows without any extra syntax.

    Since the Workflows are first class in the DSL, it's also possible to pass them around, creating "Higher Order Workflows". For example, it's possible to create a workflow that allows a user 'A' to choose a workflow to assign to a different user 'B'. Again this is all possible without having to build any special handling in the language for this.

    Workflows created with ActiveTasx also come with -

    1. An automatically generated web user interface with authentication, and wizard like steps for providing information.
    2. Websocket based fast communication between client and server.
    3. Automatic data persistence.