Big Data in a production environment: Lessons Learnt

schedule 01:30 PM - 02:15 PM place EN 307 (C40) people 34 Attending

Big Data. It's complicated, right?

For the past 18 months or so, we've been developing a Big Data system that is used in production as part of our user facing application. I assumed this was the standard thing to do with Big Data and that there wouldn't really be much of a difference between using Big Data in user-facing production compared to using it for Data Science and Machine Learning.

As time passed I interviewed people for roles in our team, and talked to other teams, I found that the types of things that we were doing seemed to be different from what most other people were doing.

It may sound obvious, but it took me a while to work out that Big Data for Data Science requires different techniques to Big Data for user-facing products.

In this talk, I'll walk through what we're currently doing to solve our transformation problems, and explore some of the differences between user-facing and Data Science. I'll present some of the lessons we've learnt, the tech we've played with and invite you to help with some of the problems we continue to face.

 
2 favorite thumb_down thumb_up 0 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

Presentation with some discussion:

  • What is the Insights project
  • Big Data for Data Science
  • Big Data in a production, user-facing environment
  • Lessons Learnt
  • Problems still to solve

Learning Outcome

An understanding of some of the complexities of doing a Big Data project in a user-facing product

Target Audience

People with some interest in the challenges of doing Big Data

schedule Submitted 2 months ago

Comments Subscribe to Comments

comment Comment on this Proposal

  • Liked Lay Clough
    keyboard_arrow_down

    Lay Clough - Chinese Whisper

    Lay Clough
    Lay Clough
    Product Owner
    Aconex
    schedule 3 months ago
    Sold Out!
    45 mins
    Workshop
    Beginner

    This is an interactive session

    * Identify what are the constraints when we try to communicate / collaborate in an organisation - especially if you have distributed team 

    * Understand the frustration of getting the meaning of the message across

    * Identify how communication can be improved