Transducing for fun and profit
Transducers -- composable algorithmic transformation decoupled from input or output sources -- are Clojure’s take on data transformation. In this talk we will look at what makes a transducer; push their composability to the limit chasing the panacea of building complex single-pass transformations out of reusable components (eg. calculating a bunch of descriptive statistics like sum, sum of squares, mean, variance, ... in a single pass without resorting to a spaghetti ball fold); explore how the fact they are decoupled from input and output traversal opens up some interesting possibilities as they can be made to work in both online and batch settings; all drawing from practical examples of using Clojure to analize “awkward-size” data.
Outline/Structure of the Talk
* Clojure at a glance
* Motivation/The problem
* Introducing transducers
* Anatomy of a transducer
** Transducers as wrappers for state
* Composing transducers
** Encoding computation with data and the lisp way
** The importance of single-pass algorithms and performance tradeoffs
* Using transducers in a streaming context
* How transducers change how I write code
** data science vs. engineering
Learning Outcome
* a peak into the dynamic side of the FP family
* a new way to think about datfa transformation and transformation composition
* food for thought about using FP in a data science context
Target Audience
Programers working with data and transformations thereof (especially numerical); programers curious about Clojure
Prerequisites for Attendees
Knowledge of Clojure is not required, but it helps if one is not afraid of parentheses.
Many of the examples will draw from statistics and ETL tasks, so a basic familiarity with the two helps to get a better idea of the context, but it is not mandatory.
Video
Links
Some of my past talks:
https://www.youtube.com/playlist?list=PL1apcF1OMBfp2S59e7OTRhKPp7mj9AhxD
http://slideshare.net/simonbelak
schedule Submitted 5 years ago
People who liked this proposal, also liked:
-
keyboard_arrow_down
Eugenia Cheng - Category Theory and Life
60 Mins
Keynote
Beginner
Category theory can be thought of as being "very abstract algebra". It is thought of as "too abstract" by some people, and as "abstract nonsense" by some others. In this talk I will show that while it is abstract, it is far from being nonsense. I will argue that the abstraction has a purpose and that broad applicability is one of the powerful consequences. To demonstrate this, I will show how I apply concepts of category theory to important questions of life such as prejudice, privilege, blame and responsibility. I will introduce the category theory concepts from scratch so no prior knowledge is needed. These concepts will include objects and morphisms, isomorphisms and universal properties.
-
keyboard_arrow_down
Yaron Minsky - Reactive Programming with Diff and Patch
60 Mins
Keynote
Intermediate
Writing reactive programs can be challenging. Such programs need to be able to react efficiently to changes as they stream in from the outside world, which often leads people to make their program logic incremental in an ad-hoc way, directly expressing their logic in terms of how to integrate a change, rather than as an all-at-oncecomputation.But such ad-hoc incrementalization is complicated, and the resulting programs can be hard to reason about. In this talk, I'll discuss an approach that leverages the fact that functional data structures can often be diffed efficiently. I'll show how the ability to diff and patch can be integrated into a more general reactive programmingframework to let you write programs which read like simple, all-at-once implementations of your program logic, but which perform like hand-optimized change-oriented programs. -
keyboard_arrow_down
Edward Kmett - Combinators Revisited
45 Mins
Invited Talk
Intermediate
Back in the 80's, one approach to compiling functional programming languages was to compile down to combinators such as SKI. John Hughes' initial work on supercombinators changed the way folks thought about compiling functional languages and caused folks to turn away from this approach by customizing the combinator set to your particular program. Then Lennart Augustsson's work on implementing supercombinators more efficiently sealed the deal. GHC's compilation technique is a descendant of this school of thought.
But what did we give up to get to where we are? Let's explore a bit of alternate history.
-
keyboard_arrow_down
Manuel Chakravarty - Welcome to FP Introductory Workshop
480 Mins
Introductory Workshop
Beginner
Functional programming has become inevitable. New programming languages draw inspiration from the functional paradigm; old programming languages retrofit support for functional programming; and development teams change their coding style to adopt the best functional programming idioms. We are clearly experiencing a paradigm shift in our industry.
Due to its academic roots, functional programming sometimes seems unapproachable, with unfamiliar jargon, obscure concepts, and bewildering theories. It doesn’t have to be like that.
In this one-day series of lectures and hands-on workshops, we will translate the jargon, demystify the concepts, and put the theories into practice. There is nothing inherently difficult about functional programming. In fact, its main aim is to simplify programming and to make it more widely accessible. Functional programming is about being able to understand one function without the million lines of code it is a part of. It is about code reuse. It is about modularity and keeping code easy to change and refactor. These are all goals of good program design that every developer appreciates. Based on this common ground, we will explore functional programming together and see how it can help us to achieve these design goals. In fact, by learning the fundamentals of functional programming in Haskell, we can improve program design in mainstream languages, such as Javascript and C++, and even more so, in hybrid languages, such as Scala and Swift.
Throughout the day, we will explain the most commonly used functional programming terminology. You will learn the fundamentals of Haskell, one of the most popular functional programming languages. In the process, we will look at a lot of concrete code to understand what functional programming is all about and how to use it in your own programs. In the workshops, you will have plenty of opportunity to write code yourself, experiment, and ask questions. It’ll be fun!
Bring your laptop and your curiosity and by the end of the day, functional programming will be another tool in your toolbox, and you will be ready to enjoy the main YOW! Lambda Jam conference.
-
keyboard_arrow_down
Yaron Minsky - Introduction to OCaml
480 Mins
Introductory Workshop
Intermediate
The goal of this workshop is to get you a basic familiarity with OCaml and the tools you'll need to be effective working in the language. It will be aimed at experienced programmers who don't know OCaml, and don't necessarily know any functional programming.The workshop is organized around a set of exercises that should take you through the basics of the language, up through building a simple client/server application using Async RPC.In addition to that, we hope to teach people how to use the latest tools for OCaml, including installing dependencies with opam , building your code with Dune, using Merlin for IDE-style functionality like type throwback and go-to-definition, and writing expect tests to visualize what your code is doing. We'll also show you how to write a simple web-game using js_of_ocaml. -
keyboard_arrow_down
Colin Fleming - Developing an IDE for Clojure code
30 Mins
Talk
Intermediate
Cursive is an IDE for Clojure, based on the IntelliJ framework. In contrast to the majority of Clojure development environments, Cursive uses static analysis of the source code to work its magic rather than using runtime inspection of a live system via the REPL. IntelliJ provides a sophisticated indexing infrastructure, and this in combination with static analysis allows many interesting features which are difficult or impossible to achieve with a traditional REPL-based environment. Essential code navigation tools such as Find Usages and refactorings such as Rename become possible, and using IntelliJ's code inspections can provide error marking and static analysis of code, right in the editor. This ability to see problems immediately provides an even shorter feedback loop than working in the REPL.
I'll discuss some of the challenges of developing a traditional IDE for a language as flexible as Clojure, including the implementation details of how to deal with syntax extensions from macros. I'll also talk about the various ups and downs of developing much of it in Clojure, and why some of it is also developed in Kotlin. I'll also talk about why I believe our editors should be syntax aware, not just text based.
-
keyboard_arrow_down
Mark Hibberd - Hanging on in Quiet Desperation: Time & Programming
30 Mins
Talk
Intermediate
Time has a profound impact on the complexity of the systems we build.
A significant amount of this software complexity comes from either an
inability to recall previous states or the inability to understand
how a state was arrived at.From the foundations of AI, LISP and functional programming [1], to
causality in distributed systems [2], to the more grungy practices of
immutable infrastructure, or the unreasonable effectiveness of
fact-based approaches to large scale data systems; the ability to
adequately cope with time, and the change and conflict it inevitable
creates, is a common thread to being able to build and reason about
these systems.This talk looks at the impact of time on system design. We will walk
through examples of large-scale systems and their battles with
complexity. At the end of the talk, the audience should start to see
the common spectre of time and have an appreciation of how
understanding time is fundamental to maintaining clarity, correctness
and reliability in systems.[1] Situations, Actions, and Causal Laws
John McCarthy
http://www.dtic.mil/dtic/tr/fulltext/u2/785031.pdf
[2] Times, Clocks and the Ordering of Events in a Distributed System
Leslie Lamport
https://amturing.acm.org/p558-lamport.pdf -
keyboard_arrow_down
Brian McKenna - Teaching functional programming at Atlassian
30 Mins
Talk
Beginner
My team works on Atlassian Marketplace, a project which started 10 years ago and uses Scala, Haskell and Nix. The Marketplace code uses concepts such as applicative functors, monad transformers and lenses. People joining the team mostly learned concepts as they needed to, through experimentation and asking questions.
Last year we started hosting functional programming classes for more fundamental and broad understanding. We've been using the Data61 (formerly NICTA) Functional Programming Course with a lot of success. This talk will describe how we teach FP, what we've learned about teaching FP and the challenges we face.
-
keyboard_arrow_down
Simon Belak - clojure.spec: a lisp-flavoured type system
30 Mins
Talk
Intermediate
With the introduction of spec, Clojure got its own distinct spin on a type system. Just as macros add another -time (runtime and compile time) where the full power of the language can be used, spec does to describing data. Ostensibly and implementation of regular expressions for data, it weaves the full power of the language into the task and at the same time -- in typical lisp fashion -- opens the internals to the end user. The result is an entire additional type system that is a first class citizen and accessible at runtime that facilitates validation, generative testing (a la QuickCheck), destructuring (pattern matching into deeply nested data), data macros (recursive transformations of data) and a pluginable error system. And then you can start building on top of it ...