Big Data. It's complicated, right?

For the past 18 months or so, we've been developing a Big Data system that is used in production as part of our user facing application. I assumed this was the standard thing to do with Big Data and that there wouldn't really be much of a difference between using Big Data in user-facing production compared to using it for Data Science and Machine Learning.

As time passed I interviewed people for roles in our team, and talked to other teams, I found that the types of things that we were doing seemed to be different from what most other people were doing.

It may sound obvious, but it took me a while to work out that Big Data for Data Science requires different techniques to Big Data for user-facing products.

In this talk, I'll walk through what we're currently doing to solve our transformation problems, and explore some of the differences between user-facing and Data Science. I'll present some of the lessons we've learnt, the tech we've played with and invite you to help with some of the problems we continue to face.