Machine data: how to handle it better?

The rise of IoT and smart infrastructure has led to the generation of massive amounts of complex data. Traditional solutions struggle to cope with this shift, leading to a decrease in performance and an increase in cost. In this talk, we will take a look at this kind of data using a simulated Curiosity rover. Participants will learn how to create a data pipeline for ingestion and visualisation. By the end of this session, we will be able to set up a highly scalable data pipeline for complex time series data with real time query performance.

 
13 favorite thumb_down thumb_up 2 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/Structure of the Talk

High level outline of topics that will be covered in this presentation:

1. Growth of IoT and Sensor Data

2. Time-series data

3. Challenges that are posed by large volumes of time-series data

4. Showcasing and overcoming the problem: A case-study

5. Demo time: Curiosity rover simulation and data ingestion from the sensors and visualisation

Learning Outcome

By the end of this session, we will be able to set up a highly scalable data pipeline for complex time series data with real time query performance.

Target Audience

Developers, Managers, IoT Enthusiasts

Prerequisites for Attendees

Some knowledge of databases, data pipelines and containers will help the audiences to follow along and make the most of this talk.

schedule Submitted 1 month ago

Public Feedback

comment Suggest improvements to the Speaker
  • Kuldeep Jiwani
    By Kuldeep Jiwani  ~  6 days ago
    reply Reply

    Hi Tanay,

    IoT is an interesting space for ML enthusiasts, good that you are focusing on it.

    Just to understand more on your talk, will it be more focused on the ETL and data query/visualisations part of IoT data. Or will it also be covering some sensor event stream series / timeseries analysis of data and showcasing use of ML techniques on it.

    • Tanay Pant
      By Tanay Pant  ~  1 day ago
      reply Reply

      Hi Kuldeep,

      While application of ML techniques is out of scope for this talk, it will definitely be covering sensor stream series and time-series analysis of a massive amount of data while still being highly available. It will also cover some topic of visualisation of the data.


  • Liked Tanay Pant
    keyboard_arrow_down

    Tanay Pant - Tick Tock: What the heck is time-series data?

    Tanay Pant
    Tanay Pant
    Developer Advocate
    Crate.io
    schedule 1 month ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    The rise of IoT and smart infrastructure has led to the generation of massive amounts of complex data. In this session, we will talk about time-series data, the challenges of working with time series data, ingestion of this data using data from NYC cabs and running real time queries to gather insights. By the end of the session, we will have an understanding of what time-series data is, how to build streaming data pipelines for massive time series data using Flink, Kafka and CrateDB, and visualising all this data with the help of a dashboard.