Beautiful Folds
A journey from statistics computations to composable streaming folds.
Come join me and learn by example about the beauty of generic folds and streaming data. I'll be revisiting these concepts, talking about its origins in Haskell and demonstrating the power of them by showing an example of a program that runs statistics computations using the Fs2 and Origami Scala libraries.
Inevitably, I'll also be talking about performance, concurrency, parallelism and some type classes such as Foldable, Applicative and Monoid, you won't be bored!
Outline/Structure of the Demonstration
- Statistics Computations
- Streams
- Applicative, and Monoid
- Beautiful Folds
- Composable Streaming Folds
Learning Outcome
- Understanding the power of generic Folds.
- Understanding when streams are good.
- Dealing with statistics computations in a generic way.
Target Audience
JVM developers (specially Scala) with interest in FP
Prerequisites for Attendees
Familiarity with Scala syntax will help. Other than that, you won't need anything else!
Links
All my public talks are published here https://gvolpe.github.io/talks/ (I haven't given this talk about Beautiful Folds yet, so there's no links)
I'm an active open source contributor: https://gvolpe.github.io/floss/ and maintainer of Fs2 Rabbit https://gvolpe.github.io/fs2-rabbit/
I enjoy sharing knowledge and motivate other people to get hands on pure functional programming concepts and I have some experience giving talks on both Scala and Haskell communities.
Some examples involving the topics of this talk: https://github.com/gvolpe/beautiful-folds
schedule Submitted 5 years ago
People who liked this proposal, also liked:
-
keyboard_arrow_down
Michael Snoyman - Functional Programming for the Long Haul
45 Mins
Keynote
Beginner
How do you decide whether a programming language is worth using or not? By necessity, such decisions are usually based on assessments that can be made relatively quickly: the ease of using the language, how productive you feel in the first week, and so on. Unfortunately, this tells us very little about the costs involved in continuing to maintain a project past that initial phase. And in reality, the vast majority of time spent on most projects is spent in those later phases.
I'm going to claim, based on my own experience and analysis of language features, that functional programming in general, and Haskell in particular, are well suited for improving this long tail of projects. We need languages and programming techniques that allow broad codebase refactorings, significant requirements changes, improving performance in hotspots of the code, and reduced debug time. I believe Haskell checks these boxes.
-
keyboard_arrow_down
Dhaval Dalal / Morten Kromberg / Ravindra Jaju - Code Jugalbandi - Exploring Concurrency
Dhaval DalalSoftware ArtisanCalixir Consultants Pvt. Ltd.Morten KrombergCXODyalogRavindra JajuSolutions ConsultantSahaj Softwareschedule 5 years ago
45 Mins
Demonstration
Beginner
In Indian classical music, we have Jugalbandi, where two lead musicians or vocalist engage in a playful competition. There is jugalbandi between Flutist and a Percussionist (say using Tabla as the instrument). Compositions rendered by flutist will be heard by the percussionist and will replay the same notes, but now on Tabla and vice-versa is also possible.
In a similar way, we will perform Code Jugalbandi (http://codejugalbandi.org) to see how the solution looks using different programming languages. This time the focus of Code Jugalbandi will be on exploring concurrency models in different languages. Functional Programming has made programming concurrency easier as compared to imperative programming. For deeper perspective on Code Jugalbandi, check out http://codejugalbandi.org/#essence-of-code-jugalbandi
-
keyboard_arrow_down
Ajay Viswanathan - Algebraic Data Types: Just my kinda Type
20 Mins
Talk
Beginner
Functional programming is built around a foundation of well-defined Types, and when you throw in Typeclasses into the mix, you get the love-child that is Algebraic Data Types. In this talk I aim to explore the mathematical foundations of Type theory and how it can be used practically in Scala for wide variety of applications like Machine Learning (Apache Spark MLlib), API design (using Vertx), DSLs and the like.
I will also be introducing the scalacheck library for Property-based testing and how you can quickly validate your ADT domains.
The talk will further deep-dive into how you can utilize the amazing Cats library and Shapeless to build generic libraries around your ADTs, having Circe as a case-study.
-
keyboard_arrow_down
Raghu Ugare / Vijay Anant - (Why) Should You know Category Theory ?
Raghu UgareArchitect, Designer, DeveloperLambdaMattersVijay AnantArchitect, Designer, DeveloperLambdaMattersschedule 5 years ago
45 Mins
Talk
Intermediate
Category Theory has been found to have a vast field of applications not limited to programming alone.
In this fun-filled talk (Yes! We promise!) , we want to make the audience fall in love with Math & Category Theory in general, and Haskell in particular.
We will address questions such as below:- What is the mysterious link between the abstract mathematical field of Category Theory and the concrete world of real-world Programming ? And why is it relevant especially in Functional Programming?
- Most of all, how can You benefit knowing Category Theory ? (Examples in Haskell)
-
keyboard_arrow_down
Tony Morris - Parametricity, Functional Programming, Types
45 Mins
Talk
Intermediate
In this talk, we define the principle of functional programming, then go into
detail about what becomes possible by following this principle. In particular,
parametricity (Wadler, 1989) and exploiting types in API design are an essential
property of productive software teams, especially teams composed of volunteers
as in open-source. This will be demonstrated.Some of our most important programming tools are neglected, often argued away
under a false compromise. Why then, are functional programming and associated
consequences such as parametricity so casually disregarded? Are they truly so
unimportant? In this talk, these questions are answered thoroughly and without
compromise.We will define the principle of functional programming, then go into
detail about common problems to all of software development. We will build the
case from ground up and finish with detailed practical demonstration of a
solution to these problems. The audience should expect to walk away with a
principled understanding and vocabulary of why functional programming and
associated techniques have become necessary to software development. -
keyboard_arrow_down
Michael Ho - Making the Switch: How We Transitioned from Java to Haskell
45 Mins
Case Study
Intermediate
In this case study presentation, SumAll's CTO, Todd Sundsted, and Senior Software Engineer, Michael Ho, will discuss the move from Java to Haskell along two parallel paths. First, the business/political story — how SumAll convinced the decision makers, fought the nay-sayers, and generally managed the people impacted by the transition. Second, the technical story — how they actually replaced their Java code with Haskell code. Along the way, they will address their hopes and expectations from transitioning from Java to Haskell, and will conclude with the results they've gained and seen to date.
-
keyboard_arrow_down
Anupam Jain - Purely Functional User Interfaces that Scale
45 Mins
Talk
Beginner
A virtual cottage industry has sprung up around Purely functional UI development, with many available libraries that are essentially just variants on two distinct approaches: Functional Reactive Programming (FRP), and some form of functional views like "The Elm Architecture". After having worked extensively with each of them, I have found that none of the approaches scale with program complexity. Either they are too difficult for beginners trying to build a hello world app, or they have unpredictable complexity curves with some simple refactorings becoming unmanageably complex, or they "tackle" the scaling problem by restricting developers to a safe subset of FP which becomes painful for experienced developers who start hitting the complexity ceiling.
In this talk I give an overview of the current Purely Functional UI Development Landscape, and then present "Concur", a rather unusual UI framework, that I built to address the shortcomings of the existing approaches. In particular, it completely separates monoidal composition in "space" (i.e. on the UI screen), from composition in "time" (i.e. state transitions), which leads to several benefits. It's also a general purpose approach, with Haskell and Purescript implementations available currently, and can be used to build user interfaces for the web or for native platforms.
The biggest advantage of Concur that has emerged is its consistent UI development experience that scales linearly with program complexity. Simple things are easy, complex things are just as complex as the problem itself, no more. Reusing existing widgets, and refactoring existing code is easy and predictable. This means that Concur is suitable for all levels of experience.
- For Learners - Concur provides a consistent set of tools which can be combined in predictable ways to accomplish any level of functionality. Due to its extremely gentle learning curve, Concur is well suited for learners of functional programming (replacing console applications for learners).
- For experienced folks - Assuming you are already familiar with functional programming, Concur will provide a satisfying development experience. Concur does not artificially constrain you in any form. You are encouraged to use your FP bag of tricks in predictable ways, and you are never going against the grain. It's a library in spirit, rather than a framework.
-
keyboard_arrow_down
Debasish Ghosh - Managing Effects in Domain Models - The Algebraic Way
45 Mins
Talk
Intermediate
When we talk about domain models, we talk about entities that interact with each other to accomplish specific domain functionalities. We can model these behaviors using pure functions. Pure functions compose to build larger behaviors out of smaller ones. But unfortunately the real world is not so pure. We need to manage exceptions that may occur as part of the interactions, we may need to write stuff to the underlying repository (that may again fail), we may need to log audit trails and there can be many other instances where the domain behavior does not guarantee any purity whatsoever. The substitution model of functional programming fails under these conditions, which we call side-effects.
In this session we talk about how to manage such impure scenarios using the power of algebraic effects. We will see how we can achieve function composition even in the presence of effects and keep our model pure and referentially transparent. We will use Scala as the implementation language.
In discussing effects we will look at some patterns that will ensure a clean separation between the algebra of our interface and the implementation. This has the advantage that we can compose algebras incrementally to build richer functionalities without committing to specific implementations. This is the tagless final approach that offers modularity and extensibility in designing pure and effectful domain models.
-
keyboard_arrow_down
Michael Snoyman - Applied Haskell Workshop
480 Mins
Workshop
Intermediate
This full day workshop will focus on applying Haskell to normal, everyday programming. We'll be focusing on getting comfortable with common tasks, libraries, and paradigms, including:
- Understanding strictness, laziness, and evaluation
- Data structures
- Structuring applications
- Concurrency and mutability
- Library recommendations
By the end of the workshop, you should feel confident in working on production Haskell codebases. While we obviously cannot cover all topics in Haskell in one day, the goal is to empower attendees with sufficient knowledge to continue developing their Haskell skillset through writing real applications.
-
keyboard_arrow_down
Brian McKenna - Starting Data61 Functional Programming Course
90 Mins
Workshop
Beginner
Following Tony and Alois' Introduction to Haskell syntax and tools, we will work through the first few modules of Data61's Functional Programming Course. These modules cover writing functions for the optional and list data types.
We will complete enough exercises to cover basic data types, functions and polymorphism. We'll practice the techniques of equational reasoning, parametricity and type/hole driven development. After completing these modules, you should be able to use the techniques to attempt most other exercises in the repository.
This workshop has the same requirements as Tony's introduction, along with a download of a recent version of the fp-course repository (https://github.com/data61/fp-course).
-
keyboard_arrow_down
Luka Jacobowitz - Testing in the world of Functional Programming
45 Mins
Demonstration
Intermediate
Testing is one of the most fundamental aspects of being a software developer. There are several movements and communities based on different methodologies with regards to testing such as TDD, BDD or design by contract. However, in the FP community testing is often not a large topic and is often glossed over. While it’s true that testing in functional programming tends to be less important, there should still be more resources on how to create tests that add actual value.
This talks aims to provide exactly that, with good examples on how to leverage property based testing, refinement types and the most difficult part: figuring out how to test code that interacts with the outside world.
-
keyboard_arrow_down
Mark Hibberd - Property Based Testing
90 Mins
Workshop
Beginner
Building on the earlier two introductions to functional programming with types, property based testing is the extra verification technique you need to ensure working software. We will work through the patterns of property based testing, starting with simple functions, working up to verification of a larger program.
By the end of this workshop participants will have a better understanding of the advantages of property based tests over example based tests, as well as acquiring the skills and confidence to start applying property based testing techniques to their current work.
This workshop has the same requirements as Tony's introduction, and will require a recent clone of the workshop repository available at https://github.com/markhibberd/property-based-testing-workshop.
-
keyboard_arrow_down
Debasish Ghosh - Power of Functions in a Typed World on the JVM
90 Mins
Tutorial
Beginner
John Carmac once mentioned on twitter that "Sometimes, the elegant implementation is just a function. Not a method. Not a class. Not a framework. Just a function." In this talk we will discuss the power of functions in modeling real world applications on the JVM. When we say functions, we mean "pure" functions as in the world of mathematics.
Functions model behaviors, functions compose to build larger functions, and combined with a powerful type system allow us to abstract over the generalities in defining real world domain models. The combination of functions along with algebraic data types has proven to be extremely useful in designing large scale systems that are modular and extensible.
Scala is a typed functional (well, almost) language on the JVM. In this session we will discuss how the functional features of Scala offer many of the above benefits in designing large scale systems. If you are coming from an OO background, you will appreciate how an alternative approach to programming can make your systems simpler to design, implement and maintain.
-
keyboard_arrow_down
Ravi Mohan - Experience Report: Building Shin - A Typed Functional Compiler For Computational Linear Algebra Problems.
45 Mins
Talk
Intermediate
Abstract: I wrote a distributed (mostly) Functional Compiler in Scheme, OCaml and Elixir that incorporates knowledge of Computational Linear Algebra and domain specific knowledge to generate highly optimized linear algebra code from specification of problems. This talk is about lessons learned in the process.
The problem:
In every domain that uses computational linear algebra (which is all of engineering and science), we encounter the 'how to optimize a linear algebra expression into an optimized sequence of BLAS (or LAPACK or $linear_algera library) kernel calls' problem.Example: (if the math equations make you want to tear your hair out and go jump off a cliff, don't worry, it is just an example, you don't have to grok it. Just skim the equations The basic problem being addressed here is that solving such equations with code takes up a lot of effort and time from experts in computational linear algebra)
Here is a linear algebra expression from a genetics problem , specifically GWAS -Genome Wide Association Studies, looking for significant associations for millions of genetic markers- where the essence of the problem [1] comes down to generating the most efficient algorithm possible that solves these equationsThis in turn involves solving a 2 dimensional sequence of Generalized Least Squared Problems of the form
The algorithms to solve these can be directly coded up in Matlab or Julia. But there are problems with this approach, with this specific problem.
1. For different input sizes, different algorithms give the most optimal performance. Which algorithm do you code up?
2. Even for a given input size, there are multiple algorithms that compute the same result, but have differing computational characteristics depending on the hardware etc. How do you generate the optimal algorithm for your hardware ?
3. Most importantly the structure of *this* specific problem allows optimizations that are specific to the problem which are not built into generic linear algebra routines. (Obviously, one can't expect MATLAB to incorporate problem specific information for every scientific/engineering problem ever). The GLS problems are connected to others, thus saving intermediate results can save hours of computation vs calculating every GLS problem from scratchIn practice, one needs to be an expert in Computational Linear Algebra to come up with the optimized algorithm for a domain specific problem, and then write (say Fortran) code to use BLAS, LAPACK etc optimally to actualize this algorithm, often with much iteration, often consuming 100s of hours.
The Solution:
Incorporating this 'expert knowledge' into a compiler speeds up the time taken to arrive at the best solution (often by a factor of 100 or 1000), and allows Computational Linear Algebra experts to do more interesting things, like focus on their research.For this particular problem, the above equations, and additional knowledge of the problem domain are the input into an expression compiler. The output is highly efficient and 'proved correct' code
In compiler terms, incorporating domain knowldege into the compilation process results in being able to apply optimizations to the generated Syntax Trees/Graphs, resulting in optimal algorithms. (note: the output of the compiler is a program in another language- say Matlab).
In essence, "Domain Specific Compilers" consume knowledge about the structure of a problem and generate optimized code that solves that problem.Shin is one such compiler. It consumes a problem description and outputs highly efficient Julia code that solves the problem.
This talk focuses on the engineering challenges I faced in building this compiler, with a special focus on the approaches that failed [5]
Trivia:
"Shin" is the Hebrew letter, not the English word meaning 'front of the leg between knee and ankle' ;-).
Every company uses names from a common theme to name their servers and components - Athena, Zeus, Hercules , or Thor, Loki, Odin, or Jedi, Sith, Skywalker etc. We use Hebrew words, so we have Ruach, Melekh, Malkuth etc..
-
keyboard_arrow_down
Markus Hauck - Free All The Things!
45 Mins
Talk
Intermediate
Have you ever asked yourself why we only free monads? Turns out there are a lot of other structures that want to be liberated from the constraints of their existence! In this talk, we will investigate what other poor (algebraic) structures we can free from the dirty hands of imperative programmers. Our journey starts with the well-known free monads, but after that we will have a look at all the other interesting structures that can be freed and of course we will also look at what we can do with them.
-
keyboard_arrow_down
Sean Chalmers - Condensed Applied FP Course
480 Mins
Workshop
Intermediate
Intermediate functional programmers often find it daunting to move from examples provided in books and blogs to developing their first fully functioning application. The Queensland Functional Programming Lab at Data61/CSIRO have produced a course for exactly this purpose. Building on the fundamentals, we work through the process of constructing a REST application covering the following topics:
- Package dependencies
- Project configuration
- Application testing & building
- Encoding / decoding messages
- Persistent storage integration
- App state & configuration management
- Error handling & reporting
-
keyboard_arrow_down
Emily Pillmore - A Radically New Functional Blockchain Architecture: Chainweb
45 Mins
Talk
Advanced
Proof-of-work blockchain networks like Bitcoin, Litecoin and Ethereum are characterized by low throughput (5-15 transactions per second). Efforts to improve throughput through protocol modifications, such as block size increases, have no hope of reaching levels required to take on modern fiat-currency payment networks. However, efforts that seek to replace Proof-of-Work (Proof-of-Stake and variants) or integrate it with off-chain networks and processes (payment channels, side chains) degrade assurance, censorship resistance or trustless-ness of the original design. Recovering and elaborating on early proposals for Bitcoin scaling, we present ChainWeb, a parallel-chain architecture which can combine hundreds to thousands of Proof-of-Work blockchains pushing throughput to 10,000 transactions per second and beyond. The network transacts a single currency, using atomic and trustless SPV (Simple Payment Verification) cross-chain transfers orchestrated at the application layer with capability and coroutine support in the Pact smart contract language. Chains incorporate each other’s Merkle tree receipts to enforce a single “super branch” offering an effective hash power that is the sum of each individual chain’s hash rate. In addition to massive throughput, other benefits accrue from having a truly parallelized smart-contract blockchain system.
-
keyboard_arrow_down
Nikhil Barthwal - Implementing Event-Driven Microservices architecture in Functional language
45 Mins
Talk
Advanced
Web services are typically stateless entities, that need to operate at scale at large. Functional paradigm can be used to model these web services work and offer several benefits like scalability, productivity, and correctness.
This talk describes how to implement Event-Driven Microservices with examples in F#. It starts with introducing Domain Driven Design to create Microservices boundaries. Using Discriminated Unions (F#'s Algebraic Data Types), the domain model can be captured as code eliminating the need for separate documentation. Moreover, using Computation expressions (F#'s Monads), one can model custom workflows easily.
It then introduces event-driven architecture, where every external action generates an event that the system responds to. Events act as the notification messages for any significant change in state and may generate other event(s) as services invoke each other. They are immutable by nature.
An explanation on why 2-phase commits cannot be used in Microservices having their own databases. Further the talk explains, how Event Driven Architecture solves this problem in an eventually consistent manner without sacrificing availability or partition tolerance. Distributed Sagas as a protocol for coordinating Microservices is introduced and its implementation in F# is also provided.
Event Sourcing can be used to model the system state. Event sourcing models the state of entity as a sequence of state-changing events. Whenever the state of a business entity changes, a new event is appended. List fold operation is ideal for implementing Event sourcing where the application reconstructs an entity's current state by replaying the events. An example with F#'s List.fold is provided.
Some aspects of evolutionary architecture are also discussed, particularly on how to evolve Microservices interface. F#'s Type providers can be used for the same though there are alternate approaches using Apache Thrift/Google Protobuf (They don't have support for F# but they do have support for C#, which F# code can leverage).
Events and their responses can be very easily modeled with Discriminated Unions. Data immutability captures the behavior of these events, since events are immutable by nature. A service can be thought of as a function that accepts an event (input) and gives back a response (output). A service may call other services, which is equivalent to a function calling other functions or even Higher-Order functions.
Immutability allows infinite scalability as it eliminated the need to worry about a mutex, a lock, or a race. As functional code is much more terse compares to object-oriented code, it provides productivity benefits. Its strict typing makes writing correct code easy as mismatch of types are caught at compile time.
Most of the services are implemented as set of pure functions. These functions which have no internal state, where outputs depend only on inputs and constants and it is very easy to test such functions. The absence of internal state means that there are no state transitions to test. The only testing left is to collect a bunch of inputs that tests for all the boundary conditions, pass each through the function under test and validate the output.
The objective of the talk is to show how to create a scalable & highly distributed web service in F#, and demonstrate how various characteristics of functional paradigm captures the behavior of such services architecture very naturally. -
keyboard_arrow_down
Andrew McMiddlin - Property-based State Machine Testing
45 Mins
Talk
Intermediate
Automated testing is key to ensuring the ongoing health and well-being of any software project,giving developers and users confidence that their software works as intended. Property based testing is a significant step forward compared to traditional unit tests, exercising code with randomly generated inputs to ensure that key properties hold. However, both of these techniques tend to be used at the level of individual functions. Many important properties of an application only appear at a higher level, and depend on the state of the application under test. The Haskell library hedgehog, a relative newcomer to the property based testing world, includes facilities for property-based state machine testing, giving developers a foundation on which to build these more complicated tests.
In this talk, Andrew will give you an introduction to state machine property testing using hedgehog. He'll start with a quick refresher on property based testing, followed by a brief introduction to state machines and the sorts of applications they can be used to model. From there, he'll take you on a guided tour of hedgehog's state machine testing facilities. Finally, Andrew will present a series of examples to show off what you can do and hopefully give you enough ideas to start applying this tool to your own projects. The first set of examples will test a web application written in Haskell. These tests will include: content creation and deletion, uniqueness constraints, authentication, and concurrent transactions. A second set of examples will test an application written in a language other than Haskell to demonstrate that this technique is not limited to applications written in Haskell.
An intermediate knowledge of Haskell and familiarity with property based testing will be beneficial,but not essential. The slides and demo application will be available after the talk for people to study in detail.
-
keyboard_arrow_down
Emily Pillmore - Pact: An Open Source Language for Smart Contracts
45 Mins
Demonstration
Intermediate
In this talk we'll discuss the design and implementation of a smart contract property verification tool for Pact.
The revolutionary idea of putting computer programs in a blockchain to create smart contracts has opened up a whole new world of possibilities. But these programs have very different characteristics from other software. This talk explores these differences, some of the challenges that have been encountered, and then discusses how Kadena is solving these problems with its smart contract language Pact. We'll discuss the design and implementation of a smart contract property verification tool for Pact. We leverage these (lack of) features to build a system capable of proving many properties of contracts via the Z3 SMT solver. We'll also give examples of real bugs caught by the system.