location_city Melbourne schedule Jun 29th 01:30 - 02:15 PM place EN 307 (C40) people 34 Interested

Big Data. It's complicated, right?

For the past 18 months or so, we've been developing a Big Data system that is used in production as part of our user facing application. I assumed this was the standard thing to do with Big Data and that there wouldn't really be much of a difference between using Big Data in user-facing production compared to using it for Data Science and Machine Learning.

As time passed I interviewed people for roles in our team, and talked to other teams, I found that the types of things that we were doing seemed to be different from what most other people were doing.

It may sound obvious, but it took me a while to work out that Big Data for Data Science requires different techniques to Big Data for user-facing products.

In this talk, I'll walk through what we're currently doing to solve our transformation problems, and explore some of the differences between user-facing and Data Science. I'll present some of the lessons we've learnt, the tech we've played with and invite you to help with some of the problems we continue to face.


Outline/Structure of the Talk

Presentation with some discussion:

  • What is the Insights project
  • Big Data for Data Science
  • Big Data in a production, user-facing environment
  • Lessons Learnt
  • Problems still to solve

Learning Outcome

An understanding of some of the complexities of doing a Big Data project in a user-facing product

Target Audience

People with some interest in the challenges of doing Big Data



schedule Submitted 3 years ago

  • Lay Clough
    Lay Clough
    Sr. Product Owner
    schedule 3 years ago
    Sold Out!
    45 Mins

    This is an interactive session

    * Identify what are the constraints when we try to communicate / collaborate in an organisation - especially if you have distributed team 

    * Understand the frustration of getting the meaning of the message across

    * Identify how communication can be improved