Tick Tock: What the heck is time-series data?

The rise of IoT and smart infrastructure has led to the generation of massive amounts of complex data. In this session, we will talk about time-series data, the challenges of working with time series data, ingestion of this data using data from NYC cabs and running real time queries to gather insights. By the end of the session, we will have an understanding of what time-series data is, how to build streaming data pipelines for massive time series data using Flink, Kafka and CrateDB, and visualising all this data with the help of a dashboard.

 
 

Outline/Structure of the Talk

High level outline of topics that will be covered in this presentation:

1. Growth of IoT and Sensor Data

2. Time-series data

3. Challenges that are posed by large volumes of time-series data

4. Showcasing and overcoming the problem: A case-study

5. Demo time: Creating a highly available data pipeline with Kafka, Flink and CrateDB to visualise with Grafana. We will be ingesting ~4 million records of the NYC cab data

Learning Outcome

By the end of this session, we will be able to set up a highly scalable data pipeline for complex time series data with real time query performance.

Target Audience

Developers, Managers, IoT Enthusiasts

Prerequisites for Attendees

Some knowledge of databases, data pipelines and containers will help the audiences to follow along and make the most of this talk.

schedule Submitted 10 months ago

Public Feedback

comment Suggest improvements to the Speaker

  • Liked Nagendra Kumar
    keyboard_arrow_down

    Nagendra Kumar / siddharth sharma / Takeshi Takizawa - Increase confidence in code deployment with Canary and mirroring live traffic!

    45 Mins
    Talk
    Intermediate

    As your application grows, the effort required to test and deploy it also grows exponentially. There is an entire layer of errors that just can’t be found via automated and manual testing: concurrency, server environment specific bugs, bugs that occur from requests called in a particular order, and much more. Humans, browsers, and robots all do strange things that affect the frequency of requests, URL weighting, size of headers, etc.

    The idea is to have safe, reliable and stress free release with use of Canary and mirroring of production traffic

    1. Canary: Canary deployments are a popular technique for incrementally testing changes on real-world traffic - Kubernetes support multiple deployments strategies like rolling updates etc However, gradual rollouts of new feature version don’t come out of the box but can be achieved by adding ingress controller inside K8s and using weighted round robin scheme to route traffic between K8s services. During this session we will discuss about strategies and learning of Canary release implementations on Kubernetes.
    2. Traffic Mirroring: Use varieties of traffic generated in production, to check your new version of application before release. You should be able to record the traffic and replay it in a manner that is absolutely identical to the testing environment. Filter the traffic, limit the load of requests and then forward to multiple hosts would be key aspects in replay. At the end, you should see a comprehensive report that will really boost your and your client's confidence to upgrade the application successfully.
  • Liked Shama Ugale
    keyboard_arrow_down

    Shama Ugale - Testing your Bot!

    Shama Ugale
    Shama Ugale
    Sr. QA Consultant
    ThoughtWorks
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Chatbots are one of the most widely adopted AI/ML implementations in the business sector. A chatbot is an intelligent machine used to imitate human conversation through text and voice commands. Today bots are widely used as a personal assistant, customer service, HR, sales and marketing to name a few. In short, bots are everywhere and we rely on them to a certain extent, this makes it extremely important to assure the quality of the chatbots and test them thoroughly. They are built using NLU/NLP-Services (Natural language understanding and processing) and are subjected to constant training and improvement which has direct impact on tests. Voice based bots like Siri and Alexa depend on speech recognition technologies. As the chatbots user do not have any barriers and due to the unpredictable user’s behavior it becomes utmost difficult to verify the correctness on the output. In this talk, we will discuss how the chatbots are different as compared to the other applications and the challenges they bring onto the table while verifying their behavior, and focus on the testing strategies and automation testing of the bots.