Fast and Curious: Benchmarking (Multicore) OCaml

location_city Online schedule Mar 26th 04:30 - 04:50 PM IST place Zoom people 25 Interested

An oft overlooked aspect of programming language development process is benchmarking. While popular CI/CD tools such as Azure pipelines, Gitlab CI, Circle CI, Drone.io and Github Actions are regularly used for continuous testing and deployment, there is a dearth of such tools for continuous benchmarking. This is because benchmarking well is challenging due to the care needed to choose appropriate benchmarks, the complexity of modern OSes and hardware that makes it hard to obtain reproducible results, and a powerful dashboard needed to surface resultant metrics in a useful manner that can be explored interactively by the developer.

For benchmarking the compiler of the OCaml [1] programming language, we have developed Sandmark [2]. Sandmark is a (a) suite of well chosen benchmarks aimed at exercising different performance axes including CPU, memory and IO (b) a tool to run the benchmarks, building the compiler and its dependencies under varying configuration settings and (c) a dashboard [3] to interactively explore the results of the benchmark runs. Sandmark was originally developed for supporting the Multicore OCaml [4] project, which aims to add native support for concurrency and parallelism to OCaml, and has undergone significant improvements since the initial release [5]. 

We have learnt several useful lessons in building a continuous benchmarking infrastructure that we would like to share in this talk, which may benefit developers who work on projects where performance is critical. In this talk, I will describe the Sandmark tool, illustrate the quirks of modern OSes and hardware and how we overcame them, highlighting useful takeaways for setting up continuous benchmarking for your own projects.

References:

[1] OCaml: https://ocaml.org/
[2] Sandmark: https://github.com/ocaml-bench/sandmark
[3] sandmark-nightly: https://github.com/ocaml-bench/sandmark-nightly
[4] Multicore OCaml: https://github.com/ocaml-multicore/ocaml-multicore/
[5] Tom Kelly. Benchmarking the OCaml compiler: our experience, OCaml Workshop 2019, https://icfp19.sigplan.org/details/ocaml-2019-papers/3/Benchmarking-the-OCaml-compiler-our-experience

 
 

Outline/Structure of the Experience Report

* Continuous Benchmarking (5 minutes)
  - The Challenge
  - Present-day Benchmarking
* What is Sandmark (5 minutes)
  - Introduction
  - Setup
  - Configuration
  - Features
  - Tuning
* Experience Report (10 minutes)
  - Case Studies
  - Conclusion and future work

Learning Outcome

* Continuous benchmarking for programming language implementations.
* Learn to use Sandmark for running benchmarks for OCaml and Multicore OCaml.
* Tuning of hardware for Multicore parallel programs.

Target Audience

Programming language enthusiasts who are interested in performance analysis.

Prerequisites for Attendees

None.

Video


schedule Submitted 9 months ago

Public Feedback


    • Aaron Hsu
      keyboard_arrow_down

      Aaron Hsu - DSLs, Architecture, and Structural Design in APL, 3 ways.

      Aaron Hsu
      Aaron Hsu
      Computer Researcher
      Dyalog Ltd.
      schedule 9 months ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      Beginning functional and APL programmers often express confusion about how to structure large software projects or larger pieces of code. Both APL and FP have a tendency to highlight their low-level features and de-emphasize system architecture patterns. This can leave programmers with a strong sense of how to write a set of small functions, but with less confidence or skill in designing, recognizing, and implementing more cohesive implicit system architectures that hold these lower level functions together. System architectures serve as a method for constraining the overall design of a system to give direction and focus to lower level implementation requirements. Especially in APL, where system architecture is often best implemented implicitly, it behooves the programmer to understand the ramifications of architecture and to implement them in their own systems. This talk unpacks a number of these "architecture level" questions within the framework of the APL programming language by exploring the same topic through 3 different architectural approaches, each of which has a very distinct flavor, presentation, and impact on the resulting source code. Particular attention is paid to the question of domain-specific languages, their design, and how they can interact with APL as tools for architectural exploration and guidance in APL source trees. 

    • Allister Beharry
      keyboard_arrow_down

      Allister Beharry - The Z3 SMT solver and functional programming

      Allister Beharry
      Allister Beharry
      Open-source developer
      Freelancer
      schedule 9 months ago
      Sold Out!
      45 Mins
      Demonstration
      Advanced

      Satisfiability modulo theories (SMT) solvers are extremely powerful tools that are indispensable in a number of applications of functional programming from mathematical analysis and optimization to computer security to program verification. SMT solvers allow you to determine if certain logical and mathematical formulas are satisfiable or (just as importantly) unsatisfiable in the context of theories like real arithmetic or set theory, and can provide definitive answers to commonly encountered programming problems involving logic, arithmetic, equations, and constraints.

      Z3 is one of the most popular SMT solvers available today with APIs available for many different programming languages. Although Z3 is most commonly used from Python, functional languages like F# provide powerful metaprogramming facilities that make it very easy to translate F# code and expressions to Z3 expressions without the need for custom types or operators.

      This presentation describes how the Z3 solver can be used from F# via quotations to quickly and easily solve common programming problems from Boolean formula satisfiability to arithmetic expression equality to linear programming to verifying fragments of source code.

    • Rodrigo Girão Serrão
      keyboard_arrow_down

      Rodrigo Girão Serrão - Why APL is a language worth knowing

      Rodrigo Girão Serrão
      Rodrigo Girão Serrão
      Consultant
      Dyalog Ltd.
      schedule 5 months ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      “A language that doesn't affect the way you think about programming, is not worth knowing.” ― Alan Perlis, in “Epigrams in Programming”

      Following Alan Perlis's words, this talk will show why APL is a language worth knowing. In other words, I will devote the talk to showcasing characteristics of APL that are likely to, on the one hand, influence the way you use other programming languages, and, on the other hand, understand concepts of computer science.

      By listening to this talk, I hope to convince you that learning a language that is radically different from all the other languages you know isn't harmful. Learning a language that is radically different from all other languages you know won't scatter your knowledge or spread your brain too thin. In fact, learning a language that is radically different from all other languages you know will cement your programming  knowledge, helping you build bridges between topics you didn't even know were connected.

      To drive my point home, we take a closer look at two characteristics of APL: the fact that Boolean values are represented by the integers 0 and 1, and the fact that APL is an array-oriented language. In studying these two things, we draw connections to the traditional if statement and to list comprehensions, deepening our understanding of those.

    • Sudha Parimala
      keyboard_arrow_down

      Sudha Parimala - OCaml Platform in 2022

      Sudha Parimala
      Sudha Parimala
      Software Engineer
      Tarides
      schedule 9 months ago
      Sold Out!
      45 Mins
      Demonstration
      Beginner

      OCaml - a functional programming language from the ML family, recently celebrated its 25th year. For a language to be around for that long is an achievement in itself. Furthermore, people are actively working on adding exciting features to the language and its ecosystem.

      Thanks to efforts from various developers and maintainers, the OCaml developer platform has come a long way and continues to evolve at a rapid phase. My talk will cover all the existing tooling that makes a OCaml developer's life easy.

      The talk will demonstrate end-to-end state-of-the-art developer tooling, right from installing a OCaml compiler, using VSCode and vscode-ocaml-platform which provides IDE like features, to publishing a library and its documentation.

    • Anupam Jain
      keyboard_arrow_down

      Anupam Jain - Type reification with PureScript

      Anupam Jain
      Anupam Jain
      UI Architect
      Arista Networks
      schedule 5 months ago
      Sold Out!
      45 Mins
      Talk
      Advanced

      PureScript is a Haskell-like language that compiles to JavaScript. If you like Haskell, you will find yourself completely at home with PureScript. However there are a few interesting typelevel things from Haskell that are not provided by the PureScript compiler. The Type reification machinery is one of them.

      In this talk, we'll understand the concept of type reification and its implementation using the Typeable typeclass. Essentially, using Typeable we can take a runtime value and get a value level representation of its type called a Typeref, this process is called type reification, which we can then compare with other known typerefs. We'll discuss indexed and unindexed typerefs and some real world problems that can be solved by this sort of type reification.

      A significant part of the talk will involve details of how Typeable is implemented in Haskell, and how it can be approximated in PureScript even without compiler support (available as a library https://github.com/ajnsit/purescript-typeable/). We'll also discuss some alternatives approaches and their pros/cons.

    • Siddharth Bhat
      keyboard_arrow_down

      Siddharth Bhat - minitt: ingredients for a dependently typed language

      Siddharth Bhat
      Siddharth Bhat
      Student
      IIIT Hyderabad
      schedule 6 months ago
      Sold Out!
      45 Mins
      Tutorial
      Advanced

      I explain the ingredients necessary to implement a minimal dependently typed language (https://github.com/bollu/minitt). I'll explain the ideas of normalization by evaluation on untyped lambda calculus, bidirectional type checking on simply typed lambda calculus, and then scale these up to a full, minimal dependently typed language, dubbed "mini-TT". At the end of the talk, you'll be fully equipped to implement a dependently typed language, and follow along other material such as "The little typer" for a deeper dive. 

    • Aaron Hsu
      keyboard_arrow_down

      Aaron Hsu - Learning APL through Combinatoric Idioms

      Aaron Hsu
      Aaron Hsu
      Computer Researcher
      Dyalog Ltd.
      schedule 9 months ago
      Sold Out!
      90 Mins
      Tutorial
      Beginner

      After learning the basics of APL, the question is often, "What do I do next?" The intermediate process of learning how to write good APL can feel especially challenging to those who are not used to operating in a dataflow, array-oriented style. One valuable resource is the large number of APL Idioms available for use. These are often referenced as a kind of "standard library" of APL functionality, especially for more experienced APL programmers. Studying the idioms is often a source of new insights for APL programmers, but for the uninitiated, it can be unclear what exactly you should learn when looking at an Idiom. Many users mistakenly see Idioms as just a short implementation of what would be a library call in another language, and thus equate Idioms and Libraries as serving the same purpose. In this talk, we will discuss and explore three combinatorial idioms in detail, not only explaining how they work, the thought process behind them, and the lessons that you can learn from them, but also the meta-level question of how to go about extracting these lessons for yourself when reading idioms on your own. 

    • Gopal S Akshintala
      keyboard_arrow_down

      Gopal S Akshintala - Fight Complexity with Functional Programming

      45 Mins
      Demonstration
      Intermediate

      A Metric-driven approach to reduce Cognitive Complexity in a code base, using Functional Programming, demoed **hands-on**, by solving a complex real-world ubiquitous design challenge - REST API Bulk Request Validation, with an extensible Framework that separates what-to-do (Validations) from how-to-do (Validation Orchestration). Let's do a case study of a successful implementation done by our team in the world's largest SaaS org, _Salesforce_, through our in-house baked FOSS library **Vader**.

    • Mourjo Sen
      keyboard_arrow_down

      Mourjo Sen - The metacircular impact of teaching FP

      Mourjo Sen
      Mourjo Sen
      Software Engineer
      Gojek
      schedule 5 months ago
      Sold Out!
      45 Mins
      Case Study
      Advanced

      Doordash recently made their software engineers deliver food to better understand how the product works. https://twitter.com/briannawu/status/1475953356644532230

      Likewise, organisations that hire junior engineers should also explore into teaching/mentoring to better understand how to not just build a team but to create a self-sustaining legacy.

      Over the last five years, I have taught programming to six people. The most consciously intentional effort being in the last eight months, where I taught programming to an experienced individual from a field that is completely unrelated to Computer Science. https://otee.dev

      In this talk, I wish to summarise my findings about what I thought would work, what actually worked, and what I could improve on in hindsight.

      It is also a critical appreciation of how functional programming concepts play a role in expediting the learner’s journey, even though it’s hard at first.

      Lastly, I also wish to introspect on how the act of teaching has changed the way I think: primarily on how teaching has made a better engineer at my day job. 

      Footnote: Learning by teaching, oddly resembles the metacircular evaluator popularised through SICP.

    • Akshay Kumar Arumugasamy
      keyboard_arrow_down

      Akshay Kumar Arumugasamy / Christopher Anand - Teaching Functional Programming to Children

      20 Mins
      Talk
      Beginner

      Is functional programming for experts? We have found that children learning programming also benefit!

      Over the last 5 years, we have taught 25K children through the McMaster University’s outreach program McMaster Start Coding (Canada), and started training undergraduates from peer institutions, including Vellore Institute of Technology, Cihan University (Kurdistan), and Narasu’s Sarathy Institute of Technology (India), so that they they can create their own programs.

      Our main motivation for teaching functional programming to children is the similarity between algebra and functional programming. Algebra is a barrier to secondary and postsecondary education, and helping children overcome this barrier can have a huge impact.

      We continued to develop our program because both children and teachers loved it. Children find it easy to learn about functions when the functions draw shapes on their screen using a compositional language they find easy to learn. Teachers love it, because children immediately understand concepts including Cartesian coordinates which are difficult to motivate for some.

      In this talk we will explain the design philosophy behind our open-source graphics library GraphicSVG for Elm, and why we think Elm (with pure functions, and well-typed standard library) is the best language to teach 10- to 14-year olds.

      We will give you a toolkit for advocating functional programming in your local schools, and show what children can create, including children from Vikas School in Hyderabad who were isolating at home with only a smartphone to learn programming on.

    help