Big Data in a production environment: Lessons Learnt
Big Data. It's complicated, right?
For the past 18 months or so, we've been developing a Big Data system that is used in production as part of our user facing application. I assumed this was the standard thing to do with Big Data and that there wouldn't really be much of a difference between using Big Data in user-facing production compared to using it for Data Science and Machine Learning.
As time passed I interviewed people for roles in our team, and talked to other teams, I found that the types of things that we were doing seemed to be different from what most other people were doing.
It may sound obvious, but it took me a while to work out that Big Data for Data Science requires different techniques to Big Data for user-facing products.
In this talk, I'll walk through what we're currently doing to solve our transformation problems, and explore some of the differences between user-facing and Data Science. I'll present some of the lessons we've learnt, the tech we've played with and invite you to help with some of the problems we continue to face.
Outline/Structure of the Talk
Presentation with some discussion:
- What is the Insights project
- Big Data for Data Science
- Big Data in a production, user-facing environment
- Lessons Learnt
- Problems still to solve
An understanding of some of the complexities of doing a Big Data project in a user-facing product
People with some interest in the challenges of doing Big Data
schedule Submitted 3 years ago
People who liked this proposal, also liked:
Lay Clough - Chinese WhisperLay CloughSr. Product OwnerMedAdvisor
schedule 3 years agoSold Out!
This is an interactive session
* Identify what are the constraints when we try to communicate / collaborate in an organisation - especially if you have distributed team
* Understand the frustration of getting the meaning of the message across
* Identify how communication can be improved