More is different: Mistaking trees for Forest

Generally scientific approach is kept limited to only reductionist approach i.e. trying to understand the system by understanding its parts. Thats how we have been taught to think. However this construct fails in non trivial systems. Often the property of the system can not be reduced to any property of its constituent parts. To take an example “pressure” is not a property individual atoms/molecules but that of a gas. In any complex system it is important to identify relationship between the property of the system and property of its parts. This relationship is known as emergence in complex systems.

In this talk I will cover variety of techniques I’ve used to identify emergent properties. I will also cover difficulties in modelling abstraction and some heuristics that has helped me unravel knotty designs and truly understand “deep insight” aspect of domain driven design. I feel this topic is important in the context of functional programming as functions are touted “pure” abstractions: missing the point of interaction.

 
10 favorite thumb_down thumb_up 4 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

Structure of session

  • Using simple parts to construct complex systems: 

Take an example of a vehicle which turns 360 degrees. The entire rotation is managed by wheels which can travel either forward or backwards. So the arrangement of wheels is what makes the system to turn 360 degrees.

  • Central Inverse Law

It is easy to engineer than reverse engineer. I learnt about this watching interference pattern of light. From interference pattern it is difficult to figure out placement of light sources. If we apply this to software, from description of the system, it is very difficult to find out exact parts which make up the system. So the example of vehicle I gave above is very difficult to understand by just watching the vehicle without any a priory knowledge.

  • Heuristic to the rescue

Complex system theory gives us some tools to identify nature of components of a system. If the system is complex at scale the components should be simple and vice versa. We can use these heuristic to identify contours of the components. This is known as complexity profile of a system.

  • Examples of applying the heuristics technique

I will show how heuristics technique can be used with multiple examples including commonly known examples of Bank Account Transfers and File System.

 

Learning Outcome

I hope people think differently about composition and modularization in applications. I hope developers give a serious thought to use different composition techniques for different abstraction layer in their applications.

Target Audience

Anyone who is building complex system.

schedule Submitted 5 months ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Dhaval Dalal
    By Dhaval Dalal  ~  4 months ago
    reply Reply

    Hi Sandeep,

         Thank-you for submitting the proposal.  The abstract of the talk seems quite interesting.  However, when I go through the structure of the session, I find it quite abstract.  I'd like to encourage more case studies and/or examples/demo that go along with it, so that there is a strong takeaway for the attendees, something that they can relate to and apply straight on their day-today work.  Further, As I understand John Hughes in his paper Why functional programming matters says - "Our ability to de-compose a problem into parts depends directly on our ability to glue solutions together.  To support modular programming, a language must provide a good glue.  Functional programming languages provide two new kinds of glue — higher-order functions and lazy evaluation.  Using these glues one can modularize programs in new and useful ways..."

    What I'd like to see is your take on modularization and composition, especially when you say...applying entity thinking?  Also, please provide links to your past talks/slide etc...on the topic submitted, so that the committee can make an informed decision.

    • Shripad Agashe
      By Shripad Agashe  ~  4 months ago
      reply Reply

      Hi Dhaval,

      Composition and modularization are interlinked. In complex system it is called scale dependent properties. So what the complex system theory says that to achieve to achieve complexity at large scale, the components of the system have to be simple at fine scale. To explain this lets take example of Bank account. If we just have two operations on it, debit and credit, we can compose host of possible variations. Take case of Funds transfer where there is interaction between two accounts. In a system with a single account, this functionality would not exists. So Funds Transfer behaviour is a property of accounting system rather than an account(where mostly it is wrongly put in most cases). By looking at a single entity i.e. N=1, we can isolate behaviour of the part vs behaviour of the collective. 

      We can extend this further and do many to one or one to many funds transfer. But this example also teaches us that we need to model composition on its own and then determine which parts are suitable for it. The way entity thinking comes in is you have two distinct analysis modes: N=1 and N > 1 for any domain object. This helps in identifying boundaries of collective behaviour vs parts behaviour. So in the talk, I will be taking example of funds transfer between two accounts with the same branch, two accounts in different brach and two accounts in different banks and try and explain this point further.

      I had given a presentation in our internal event. Not very polished but illustrates the point.

      https://www.slideshare.net/ShripadAgashe/abstractions-76443153

      I gave https://www.slideshare.net/ShripadAgashe/abstractions-76443153

      • Dhaval Dalal
        By Dhaval Dalal  ~  4 months ago
        reply Reply

        Hi Shripad,

              Thanks for your explanation,  I can relate to the funds transfer example somewhat, however,  do you have additional examples for the "distinct" analysis modes that you are talking about?   Request you to please exemplify all concepts with pragmatic examples/scenarios that the audience can readily relate to, identify such places in the projects and apply it.

         

        thanks,

        Dhaval.

         

        • Shripad Agashe
          By Shripad Agashe  ~  3 months ago
          reply Reply

          I've updated the proposal, please have a look.


  • Liked Manuel Chakravarty
    keyboard_arrow_down

    Manuel Chakravarty - Haskell SpriteKit - a Purely Functional API for a Stateful Animation System and Physics Engine

    45 mins
    Demonstration
    Intermediate

    Graphics, animation, and games programming in Haskell faces a dilemma. We can either use existing frameworks with their highly imperative APIs (such as OpenGL, Cocos2D, or SpriteKit) or we waste a lot of energy trying to re-engineer those rather complex systems from scratch. Or, maybe, we can escape the dilemma. Instead of a Haskell program directly manipulating the mutable object-graph of existing high-level frameworks, we provide an API for purely functional transformations of a Haskell data structure, together with an adaptation layer that transcribes those transformations into edits of the mutable object-graph.

    I this talk, I will explain how I used this approach to architect a Haskell binding to the animation system and physics engine of Apple’s SpriteKit framework. I will discuss both how the binding is structured and how it achieves the translation of Haskell side changes to SpriteKit and vice versa, such that it is sufficiently efficient. Moreover, I will demonstrate by example how to use the Haskell library to implement a simple game.

  • Liked Manuel Chakravarty
    keyboard_arrow_down

    Manuel Chakravarty - Functional Programming in Swift

    Manuel Chakravarty
    Manuel Chakravarty
    Lambda Scientist
    Applicative & Tweag I/O
    schedule 4 months ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    Swift’s strong functional programming support generates a lot of interest in adapting functional programming techniques to improve designing, developing, and maintaining Cocoa applications. This workshop will help you unlock this potential.

    We will explore the foundations of functional programming and how functional programming helps to improve software development. We will start by discussing the main goals of functional programming, followed by surveying Swift’s language features that support functional programming. Then, we will look at how to put functional programming principles into practice using Swift. We will focus on immutable data structures, value types, and functional reactive programming.

    = Value types 

    One of the biggest changes in the transition from Objective-C to Swift is the new type system. This change is at the heart of Swift’s language design, from optionals over value types to constrained generics, and to truly understand the language, a thorough understanding of its type system is indispensable. We will survey these type system features with special attention to value types. We will discuss how structs and enums in Swift can be used to help avoid bugs by design and how immutable data structures help to avoid unwelcome side effects.

    = Functional reactive programming

    Functional reactive programming (FRP) promises to replace unstructured side effects by structured change propagation in response to user input, network events, sensor activity, and similar. Typically explained by way of sophisticated, but often also complex functional reactive programming frameworks, such as ReactiveCocoa and RxSwift, it can be difficult to see the simple principles underlying functional reactive programming. In fact, these principles are sufficiently simple that we can explain them by building a functional reactive programming library from scratch.

    We will discuss how we can construct our own lightweight functional reactive programming library out of simple components using Swift’s support for protocol-oriented programming. Moreover, we will discuss how to use this library to design and implement the change propagation between the model and view components of a Cocoa app. 

    All of this is illustrated with hands-on coding exercises based on simple iPhone application that incorporates the discussed techniques.

  • Liked Aloïs Cochard
    keyboard_arrow_down

    Aloïs Cochard / Tony Morris - Introduction to Functional Programming using Haskell

    Aloïs Cochard
    Aloïs Cochard
    Computer Science Research
    Orexio / Swisscom
    Tony Morris
    Tony Morris
    Senior Functional Programming Engineer
    Data61 / CSIRO
    schedule 3 months ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    We will be spending the day learning the fundamentals of Functional Programming (FP) using the Haskell programming language. The exercise material will be a condensed selection of the NICTA/course which is regularly held in Australia over three days.

    This one day session is targeted to experienced industry programmers who are looking to break into Functional Programming and develop the rudimentary skills and techniques that enable continued independent study. A refresher on Haskell syntax will be provided, however, it is highly recommended to practice with the syntax and development tools prior to obtain the best outcome for the day.

    You will be required to bring a suitable development machine (portable) for working through the exercises. You will also need to install Glasgow Haskell Compiler (http://www.haskell.org/ghc/) version 7.8 or higher on that machine prior to the day.

  • Liked Mushtaq Ahmed
    keyboard_arrow_down

    Mushtaq Ahmed / Unmesh Joshi - Service Discovery using CRDTs

    45 mins
    Experience Report
    Intermediate

    Service Discovery is an important aspect of microservices, particularly when using containers on elastic infrastructure. Most popular approaches for service discovery require a standalone fault tolerant cluster like Consul, Zookeeper or etcd.

    If you are already using Akka as a clustering platform for your services, there is an interesting alternative available for Service Discovery. CRDTs, or Conflict-free Replicated Data Types are a relatively new primitive in distributed systems. CRDTs allows concurrent updates without coordination and has provisions for tuning consistency levels. 'Distributed Data' extension natively supports CRDTs on top of Akka-Cluster.

    We used CRDTs in Akka to build the service registration and discovery mechanism for TMT's (Thirty Meter Telescope) software platform. TMT will be world's largest optical telescope once operational. The ongoing software work is open source and available on github. We would like to share the details of our implementation and also compare it existing approaches.

  • Liked Abhiroop Sarkar
    keyboard_arrow_down

    Abhiroop Sarkar - Wormholes and teleporting with Transient.

    Abhiroop Sarkar
    Abhiroop Sarkar
    Student
    University of Nottingham
    schedule 5 months ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    The transient library in Haskell/Eta composes arbitrary pieces of software in concurrent, parallel and sequential settings using standard Haskell/Eta combinators like applicative, alternative, bind etc. It provides algebraic and monadically composable primitives for concurrency, parallelism, event handling, transactions and multithreading

    It liberates the programmers from dealing with state machines, objects frameworks, routes, configurations and callbacks. Transient expresses business functionalities as compact category theoretical expressions which compose. These compositions are verified during compilation, although the code might be executing in different machines.

    It presents a fresh new approach to programming concurrent and distributed systems.

     

  • Debasish Ghosh
    Debasish Ghosh
    Consultant
    IRadix Inc
    schedule 4 months ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    Using a programming language that supports first class functional programming doesn’t necessarily imply that the code you write is functional. Functional programming has the nice properties of being compositional - you design smaller abstractions and then reuse them to build larger ones. You use the power of algebraic properties that these abstractions offer to build larger elements of your model out of smaller ones.

     However often we find that many code bases designed in a functional programming language are inherently imperative, especially when you are using a multi-paradigm language like Scala or Java. There’s nothing wrong with imperative code, but it lacks the niceties of expression based composition that makes reasoning about your code so much easier. Also assignments used in imperative code are inherent side-effects that are best avoided (or at least minimized) in an expression oriented language.

     In this session I will discuss some of my experiences of mining imperative code and refactoring them into functional patterns. You will be surprised to see how much functional goodness can be derived through functional thinking. Algebraic structures like Monoids, Monads and Kleislis are just buried within the reams of imperative statements crying to come out. In an exercise of half an hour we will liberate them and see how the code quality improves from the point of view of modularity, compositionality and simplicity.

  • Liked Eric Torreborre
    keyboard_arrow_down

    Eric Torreborre - The Eff monad, one monad to rule them all!

    Eric Torreborre
    Eric Torreborre
    Senior Software Developer
    Zalando
    schedule 9 months ago
    Sold Out!
    45 mins
    Talk
    Advanced

    Monad transformers are a neat way to compose effectful computations: pass some configuration, read/write from disk, track errors... However they can be a bit cumbersome to deal with, especially if you want to use different stacks sharing some common effects: for example a stack to interact with AWS and another one for Hadoop computations.

    This talk will shortly introduce monad transformers in Scala and their drawbacks then present the Eff monad.

    The Eff monad has some really interesting properties:

    • effects can be developed completely independently from each other
    • effects can be interpreted in different ways and the interpreters can be composed together
    • it is based on an efficient "Free monad" (avoiding the issue of quadratic flatMaps)
    • with the Eff monad you will never have to re-implement the Monad operations (point / flatMap) again!
  • Liked Martin Thompson
    keyboard_arrow_down

    Martin Thompson - High-Performance Messaging with Aeron

    Martin Thompson
    Martin Thompson
    High-Performance Computing Specialist
    Real Logic
    schedule 4 months ago
    Sold Out!
    480 mins
    Workshop
    Beginner

    In this workshop you will learn how to build microservices with Aeron and SBE in Java. Microservices capable of handling millions of messages per second with very low and predictable response times. We will cover some networking theory and the design of Aeron so you know how to use it with mechanical sympathy.

    The core design of Aeron uses a new functional data structure and functional concepts. In this workshop we will explore how functional designs applied with mechanical sympathy can result in the highest performance messaging system available.

  • Liked Maria Livia Chiorean
    keyboard_arrow_down

    Maria Livia Chiorean - The path to generic endpoints using Shapeless

    20 mins
    Talk
    Beginner

    Programming in Scala is sometimes very time consuming. Finding the most efficient way to solve a problem can end up in days of frustration. This talk is a story of trail and error with a twist at the end. It's a story of API endpoints, generic programming, Shapeless and what happens when they all come together. Everything with examples and plenty of code.

  • Liked Paweł Szulc
    keyboard_arrow_down

    Paweł Szulc - Going bananas with recursion schemes for fixed point data types

    Paweł Szulc
    Paweł Szulc
    Scala Developer
    Slam Data
    schedule 7 months ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    In 1991 Erik Meijer, Maarten Fokkinga, and Ross Paterson published "Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire." This paper is a classic one, widely recognizable in the FP community. Constructs described - known as recursion schemas - have real world applications. Strictly speaking, explicit recursion is the ‘goto’ of pure functional programming. Recursion schemas provide same level of sanity as loops did for structural programming back in the day.

    Over the years a lot of the progress have been made. Papers like "Recursion schemes from comonads" by Tarmo Uustalu, Varmo Vene & Alberto Pardo or "Unifying Structured Recursion Schemes" by Ralf Hinze, Nicolas Wu & Jeremy Gibbons - pushed the concept forward.

    This talk is about generalization of a very specific recursion schema (called catamorphism) over fixed point data type. After providing introduction the concept of catamorphism, we will jump straight to fix point data types trying to solve some real-world problems. Code examples are in Scala. Code examples use Matryoshka - which is an open sourced project design to generalize folds, unfolds, and traversals for fixed point data structures in Scala.

  • Liked Tony Morris
    keyboard_arrow_down

    Tony Morris - Functional Programming in Aviation

    45 mins
    Case Study
    Beginner

    In this talk & demo, we have a look at some of the low-hanging problems in general aviation and how functional programming can be applied to provide significant improvements in efficiency and air safety. The current solutions to problems such as navigation, traffic/terrain collision avoidance and weight/balance calculations will be demonstrated to the audience, mostly for amusement. More seriously, we will have a look at the legacy that has led to the way things are, and how to improve by applying our programming skills.

    We will look at:

    • how aviation safety is regulated.
    • how aeronautical services are provided to flight operators.
    • how aeronautical navigation is conducted and regulated.
    • how the weight and balance for a flight is conducted.
    • the methods by which aircraft and ground coordinate between each other.

    We will see:

    • some real (and basic) data management problems in aviation, that very obviously threaten safety, then solve them, using programming.
    • we will see a live demonstration of aeronautical navigation methods, investigate incident reports where lives were lost as a result, and consider how our programming skills can yield improvements, possibly even save lives.
    • we will conduct a real weight&balance calculation for a flight, then once hilarity inevitably ensues, we will look at the problems that arise by this method, then solve them using data structures and functional programming. Some
      of these practical problems are obvious, even to a non-aviator, and the predictable incident reports are the end result.
    • finally, we will have a look at a live demonstration of a software defined radio (SDR), receiving ADS-B transmissions from aircraft (live), an AHRS implementation and GNSS receiver using off-the-shelf, low-cost parts. We will look at why these instruments are helpful to aircraft pilots and interact with that device using the Haskell programming language.
  • Liked Aloïs Cochard
    keyboard_arrow_down

    Aloïs Cochard - The Tale of Troy

    45 mins
    Demonstration
    Beginner

    In recent years, the impact of functional programming on mainstream languages has grown significantly.

    After conquering server applications and distributed systems,
    the paradigm is now conquering the fields of web development and data analytics.

    So what remains? The land of system programming.

    When Mozilla did release the language Rust in 2012,
    it did immediately striked interest from functional programming enthusiasts due to it's strong emphasis on safety.

    Even though it currently lacks some advanced features like higher kinded types,
    it is built with functional programming principles as it's core.

    Like a Trojan horse (or the parasite mushroom after the language is named),
    Rust looks like the next C on the outside... while it might actually be closer to Haskell on the inside!

    Let's take a tour and discover an other side of system programming by learning together how to do functional programming in Rust.

  • Liked Ryan Lemmer
    keyboard_arrow_down

    Ryan Lemmer - From Haskell 98 to Modern Haskell - Making sense of the Haskell type system

    45 mins
    Talk
    Intermediate

    There are several great books and courses that serve as an introduction to functional programming in Haskell. Typically, these introductions focus on the original core of Haskell ("Haskell 98").

    However, going from there into real world Haskell systems can be a bewildering experience. This is because Haskell has evolved significantly since Haskell 98. In this talk we will investigate the key advances in the type-system that make up "modern Haskell".

    • type-system extensions: GADTs, multi-param type-classes, functional dependencies, ...
    • kind-system extensions: type functions, kind polymorphism, type promotion, ...

    By taking a historical perspective we will look at why new language extensions were introduced, and the context in which they first appeared. This approach gives us a layered view of seemingly disparate language features.

    We will see that the evolution of the type system has been a steady march from term-level programming to type-level programming, and ultimately dependently-typed programming in Haskell.

    (This talk will draw from the book "Haskell Design Patterns")

  • Liked Nikhil Tibrewal
    keyboard_arrow_down

    Nikhil Tibrewal / Rafał Wojdyła - Scio – Scala DSL for Apache Beam

    45 mins
    Talk
    Intermediate

    Apache Beam (based on Google’s Dataflow Model) provides a simple, unified programming model for both batch and streaming data processing. If only it wasn’t so unfamiliar and verbose for our Scala engineers. Learn how Scio leverages Scala’s type system, macros and functional paradigm to provide more engineer-friendly and type safe API.

  • Liked Arthur Kushka
    keyboard_arrow_down

    Arthur Kushka - Recursion schemes and fixed point data types

    Arthur Kushka
    Arthur Kushka
    Software developer
    AutoScout24 GmbH
    schedule 7 months ago
    Sold Out!
    45 mins
    Talk
    Advanced

    Its always challenge to build a good abstraction during work on library. Library codebase must be flexible and modular when user code must be clean and easy understandable. One of variants to build API between library and customer its Free evaluation approach. In that case you as developer is free to interpret customer actions as you want and customer is free to define actions as he want. In my talk I will show how to build API based on recursive abstract data types, describe why its cool and demonstrate some tricks on type system to make customer experience with library better.

     
  • Liked Eric Torreborre
    keyboard_arrow_down

    Eric Torreborre - Streams, effects and beautiful folds, a winning trilogy

    Eric Torreborre
    Eric Torreborre
    Senior Software Developer
    Zalando
    schedule 9 months ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    Most applications are just reading data, transforming it and writing it somewhere else. And there are great libraries in the Scala eco-system to support these use cases: Akka-Stream, fs2, Monix,... But if you look under the hood and try to understand how those libraries work you might be a bit scared by their complexity!

    In this talk you will learn how to build a very minimal "streaming library" where all the difficult concerns are left to other libraries: eff for asynchronous computations and resources management, origami for extracting useful data out of the stream. Then you will decide how to spend your complexity budget and when you should pay for more powerful abstractions.

  • Liked Oskar Wickström
    keyboard_arrow_down

    Oskar Wickström - The Power of Functional Programming and Static Type Systems in Server-Side Web Applications

    Oskar Wickström
    Oskar Wickström
    Programmer
    Empear
    schedule 10 months ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    Single-page web applications have been spreading like wildfire, with an endless amount of frameworks and libraries, and functional programming together with static types fueling the fire with great ideas like pure functions, monads, and strong type checking. But what happened to Progressive Enhancement? Some parts of our applications might require Javascript to function, but the majority could be built with ordinary links and forms.

    In this talk we will explore how we can build web applications using established web technology, and the power of functional programming on the server-side, leveraging what is already in our browsers.

  • Liked Tony Morris
    keyboard_arrow_down

    Tony Morris - Parametricity, Types are Documentation

    45 mins
    Talk
    Intermediate

    In this talk, we define the principle of functional programming, then go into
    detail about what becomes possible by following this principle. In particular,
    parametricity (Wadler, 1989) and exploiting types in API design are an essential
    property of productive software teams, especially teams composed of volunteers
    as in open-source. This will be demonstrated.

    Some of our most important programming tools are neglected, often argued away
    under a false compromise. Why then, are functional programming and associated
    consequences such as parametricity so casually disregarded? Are they truly so
    unimportant? In this talk, these questions are answered thoroughly and without
    compromise.

    We will define the principle of functional programming, then go into
    detail about common problems to all of software development. We will build the
    case from ground up and finish with detailed practical demonstration of a
    solution to these problems. The audience should expect to walk away with a
    principled understanding and vocabulary of why functional programming and
    associated techniques have become necessary to software development.

  • Liked Tony Morris
    keyboard_arrow_down

    Tony Morris - Explain List Folds to Yourself

    45 mins
    Talk
    Beginner

    In this talk, we go back to first principles, defining and examining the definition for a cons list, then take a look at the ubiquitous right and left fold functions on a list.

    The primary focus of this talk is to develop an intuition for how these functions work so that we can best decide when to apply them. Multiple programming languages will be used to help emphasise the independence of the gained intuition. This talk will generally be interactive with the audience as we solve problems and build confidence in our new knowledge.

    Knowing how to apply the various list fold functions is a common question by the student of FP. This talk aims to provide a solid, reliable answer to this question. No prior understanding of list folds is necessary.

  • Liked Calascibetta Romain
    keyboard_arrow_down

    Calascibetta Romain - Git under the hood with OCaml

    Calascibetta Romain
    Calascibetta Romain
    Independent
    OCaml Labs
    schedule 4 months ago
    Sold Out!
    45 mins
    Demonstration
    Intermediate

    Git is one of the most popular tool for a developper and a part of the power of Git is come from some functional concepts. So, it's obvious to implement the Git format with OCaml!

    This talk will present the functionnal concepts of Git (like the immutable tree) and show a practical implementation with OCaml step by step. Then, we will explain how can scale an OCaml code to compute a huge Git repository and understand by this way the power of OCaml (and the mix between the functionnal and the imperative way).

    At the end, we will show an example of the usability of Git with OCaml with a concrete application.