Data-driven systems and machine learning continue to be a significant trend across our industry. However, most attempts at these systems face serious difficulties due the tension between the clean, controlled, lab environments where statisticians apply their skills, and the messy unpredictable, production environments where we want to apply their results at scale.

In this talk, we will provide an overview of the machine learning landscape, with an emphasis on the distinction between machine learning as a scientific practice and the larger concept of machine learning systems. Using this base, we will walk through the challenges of taking machine learning out of the lab and applying it successfully in an industrial setting.

By the conclusion of this talk, the audience should take away a better understanding of machine learning as a practice, together with an idea of what it takes to build and deploy machine-learning systems in an environment that deals with real customers and data at scale.

KEYWORDS

Machine Learning, Data, Scale, Deployment

Data-driven systems and machine learning continue to be a significant trend across our industry. However, most attempts at these systems face serious difficulties due the tension between the clean, controlled, lab environments where statisticians apply their skills, and the messy unpredictable, production environments where we want to apply their results at scale.

In this talk, we will provide an overview of the machine learning landscape, with an emphasis on the distinction between machine learning as a scientific practice and the larger concept of machine learning systems. Using this base, we will walk through the challenges of taking machine learning out of the lab and applying it successfully in an industrial setting.

By the conclusion of this talk, the audience should take away a better understanding of machine learning as a practice, together with an idea of what it takes to build and deploy machine-learning systems in an environment that deals with real customers and data at scale.

KEYWORDS

Machine Learning, Data, Scale, Deployment

Data-driven systems and machine learning continue to be a significant trend across our industry. However, most attempts at these systems face serious difficulties due the tension between the clean, controlled, lab environments where statisticians apply their skills, and the messy unpredictable, production environments where we want to apply their results at scale.

In this talk, we will provide an overview of the machine learning landscape, with an emphasis on the distinction between machine learning as a scientific practice and the larger concept of machine learning systems. Using this base, we will walk through the challenges of taking machine learning out of the lab and applying it successfully in an industrial setting.

By the conclusion of this talk, the audience should take away a better understanding of machine learning as a practice, together with an idea of what it takes to build and deploy machine-learning systems in an environment that deals with real customers and data at scale.

KEYWORDS

Machine Learning, Data, Scale, Deployment

Building robust, quality systems is hard. We trade off organizational issues against technical decisions; the ability to deliver quickly against our ability to change; and the ability to build systems easily against the ability to run those systems in production. However, good architectural decisions can free us to choose the right tools and techniques, allowing us to manage these challenges and concentrate on solving real problems rather than our made up ones.

In this talk, we will run through some stereotypical projects, come to terms with legacy systems, and look at the properties of robust architectures. In particular we are interested in how architectures lend themselves to experimentation and change in terms of both function and technology.

We will attempt to ground the discussion with examples from my past projects. Looking at where things have worked well and probably of more interest, where they really have not.

Building robust, quality systems is hard. We trade off organizational issues against technical decisions; the ability to deliver quickly against our ability to change; and the ability to build systems easily against the ability to run those systems in production. However, good architectural decisions can free us to choose the right tools and techniques, allowing us to manage these challenges and concentrate on solving real problems rather than our made up ones.

In this talk, we will run through some stereotypical projects, come to terms with legacy systems, and look at the properties of robust architectures. In particular we are interested in how architectures lend themselves to experimentation and change in terms of both function and technology.

We will attempt to ground the discussion with examples from my past projects. Looking at where things have worked well and probably of more interest, where they really have not.

Building robust, quality systems is hard. We trade off organizational issues against technical decisions; the ability to deliver quickly against our ability to change; and the ability to build systems easily against the ability to run those systems in production. However, good architectural decisions can free us to choose the right tools and techniques, allowing us to manage these challenges and concentrate on solving real problems rather than our made up ones.

In this talk, we will run through some stereotypical projects, come to terms with legacy systems, and look at the properties of robust architectures. In particular we are interested in how architectures lend themselves to experimentation and change in terms of both function and technology.

We will attempt to ground the discussion with examples from my past projects. Looking at where things have worked well and probably of more interest, where they really have not.

As we construct larger or more complex systems, failure and change are ever-present. We need to accept and even embrace these tensions to build software that works and keeps working.

This is a talk on building and operating reliable systems. We will look at how systems fail, particularly in the face of complexity or scale, and build up a set of principles and practices that will help us implement, understand and verify reliable systems.

As we construct larger or more complex systems, failure and change are ever-present. We need to accept and even embrace these tensions to build software that works and keeps working.

This is a talk on building and operating reliable systems. We will look at how systems fail, particularly in the face of complexity or scale, and build up a set of principles and practices that will help us implement, understand and verify reliable systems.

As we construct larger or more complex systems, failure and change are ever-present. We need to accept and even embrace these tensions to build software that works and keeps working.

This is a talk on building and operating reliable systems. We will look at how systems fail, particularly in the face of complexity or scale, and build up a set of principles and practices that will help us implement, understand and verify reliable systems.

Time has a profound impact on the complexity of the systems we build.

A significant amount of this software complexity comes from either an inability to recall previous states or the inability to understand how a state was arrived at.

From the foundations of AI, LISP and functional programming [1], to causality in distributed systems [2], to the more grungy practices of immutable infrastructure, or the unreasonable effectiveness of fact-based approaches to large scale data systems; the ability to adequately cope with time, and the change and conflict it inevitably creates, is a common thread to being able to build and reason about these systems.

This talk looks at the impact of time on system design. We will walk through examples of large-scale systems and their battles with complexity. At the end of the talk, the audience should start to see the common spectre of time and have an appreciation of how understanding time is fundamental to maintaining clarity, correctness and reliability in systems.

[1] Situations, Actions, and Causal Laws
John McCarthy
http://www.dtic.mil/dtic/tr/fulltext/u2/785031.pdf

[2] Times, Clocks and the Ordering of Events in a Distributed System
Leslie Lamport
https://amturing.acm.org/p558-lamport.pdf

Building on the earlier two introductions to functional programming with types, property based testing is the extra verification technique you need to ensure working software. We will work through the patterns of property based testing, starting with simple functions, working up to verification of a larger program.

By the end of this workshop participants will have a better understanding of the advantages of property based tests over example based tests, as well as acquiring the skills and confidence to start applying property based testing techniques to their current work.

This workshop has the same requirements as Tony's introduction, and will require a recent clone of the workshop repository available at https://github.com/markhibberd/property-based-testing-workshop.

Time has a profound impact on the complexity of the systems we build.

A significant amount of this software complexity comes from either an
inability to recall previous states or the inability to understand
how a state was arrived at.

From the foundations of AI, LISP and functional programming [1], to
causality in distributed systems [2], to the more grungy practices of
immutable infrastructure, or the unreasonable effectiveness of
fact-based approaches to large scale data systems; the ability to
adequately cope with time, and the change and conflict it inevitable
creates, is a common thread to being able to build and reason about
these systems.

This talk looks at the impact of time on system design. We will walk
through examples of large-scale systems and their battles with
complexity. At the end of the talk, the audience should start to see
the common spectre of time and have an appreciation of how
understanding time is fundamental to maintaining clarity, correctness
and reliability in systems.

[1] Situations, Actions, and Causal Laws
John McCarthy
http://www.dtic.mil/dtic/tr/fulltext/u2/785031.pdf
[2] Times, Clocks and the Ordering of Events in a Distributed System
Leslie Lamport
https://amturing.acm.org/p558-lamport.pdf

Data is a powerful tool. Data-driven systems leveraging modern analytical and predictive techniques can offer significant improvements over static or heuristic driven systems.

The question is:

• How much can you trust your data? Data collection, processing and aggregation is a challenging task.
• How do we build confidence in our data? Where did the data come from?
• How was it generated? What checks have or should be applied?
• What is affected when it all goes wrong?

This talk looks at the mechanics of maintaining data-quality at scale. Firstly looking at bad-data, what it is and where it comes from. Then diving into the techniques required to detect, avoid and ultimately deal with bad-data. At the end of this talk the audience should come away with an idea of how to design quality data-driven systems that ultimately build confidence and trust rather than inflate expectations.

Data is a powerful tool. Data-driven systems leveraging modern analytical and predictive techniques can offer significant improvements over static or heuristic driven systems.

The question is:

• How much can you trust your data? Data collection, processing and aggregation is a challenging task.
• How do we build confidence in our data? Where did the data come from?
• How was it generated? What checks have or should be applied?
• What is affected when it all goes wrong?

This talk looks at the mechanics of maintaining data-quality at scale. Firstly looking at bad-data, what it is and where it comes from. Then diving into the techniques required to detect, avoid and ultimately deal with bad-data. At the end of this talk the audience should come away with an idea of how to design quality data-driven systems that ultimately build confidence and trust rather than inflate expectations.