What's the big deal? Dealing with Big Data challenges

Big data is huge! with billions and billions of data sets and a need to analyze and apply that to real-life problem-solving is a challenge. Are traditional methods successful in solving big data problems? Traditional methods of applying water-fall has not been very successful, does Scrum work well? Let's take a look at what some industries have done to deal with big data challenges.

Let’s take a look at the current state of big data, if traditional methodologies are providing the necessary answers quick enough. Is Scrum a good framework to alleviate some of the challenges to deal with big data?

We will discuss the challenges of using traditional methodologies for dealing with big data and evaluate a framework like Scrum to deal with these challenges.


Outline/Structure of the Case Study

1. Introduction - Big Data in all industries - 5 min.

  • Google HiN1 Virus data
  • Government data - how big is it?
  • Healthcare
  • Financial
  • E-Commerce
  • Other

2. Four V's of Big Data. - 5 min.

  • Volume - Scale
  • Velocity - Streaming data
  • Variety - Different forms
  • Veracity - Uncertainty
  • Apply a 5th V - Values of Scrum - which value is important in this situation?

3. Big data complexity and how empirical process control theory can be applied? 15 min

  • We will look at the Four V's we discussed and how the empirical process control theory can be applied to alleviate these challenges.
  • How will chunking in terms of context of data helps?

4. Walkthrough my experiences in transforming big data projects using Scrum 20 min

  • Management buy-in
  • Experimenting with small teams
  • Transforming data teams using empiricism (transparency, inspection and adaptation)
  • Learning from industry trends
  • Metrics

5. Exercise/activity: 15 min

Scrum assessment canvas for big data. Participants will fill their current assessment on the canvas to assess current state and steps to take to deal with their big data projects using Scrum.

Canvas has the following sections to be filled:

  • Current challenges with big data in our organization
  • Top 3 things that worked in the past
  • Identify steps to improve current situation
  • Prioritize top 3 steps to try at work with my team
  • One take away from this talk
  • What I learnt from others (fill in during discussion & debrief)

Debrief - 10 min.

  • This will be an interactive discussion about the data the participants gathered on their canvas and share with others
  • Fill in what I learnt from others

Summary - 5 min.

Learning Outcome

Participants will have an understanding of dealing with the challenges of the big data industry and how some organizations have been using a framework like Scrum to alleviate those challenges. They will have a chance to take some information back after I share my experiences in working with several big data industries.

Participants will be able to do an assessment of their current state of big data projects and steps to take to deal with their big data projects using Scrum.

Target Audience

New to Big Data, Experienced Big Data Practitioners, ScrumMasters, Product Owners, Business Stakeholders, Managers, Developers, DBAs, Architects

Prerequisites for Attendees

Data knowledge any industry. Anyone who deals with data.

schedule Submitted 2 years ago

Public Feedback