Network Anomaly Detection and Root Cause Analysis

schedule Sep 1st 03:30 - 04:15 PM place Grand Ball Room 2 people 88 Interested

Modern telecommunication networks are complex, consist of several components, generate massive amounts of data in the form of logs (volume, velocity, variety), and are designed for high reliability as there is a customer expectation of always on network access. It can be difficult to detect network failures with typical KPIs as the problems may be subtle with mild symptoms (small degradation in performance). In this workshop on network anomaly detection we will present the application of multivariate unsupervised learning techniques for anomaly detection, and root cause analysis using finite state machines. Once anomalies are detected, the message patterns in the logs of the anomaly data are compared to those of the normal data to determine where the problems are occurring. Additionally, the error codes in the anomaly data are analyzed to better understand the underlying problems. The data preprocessing methodology and feature selection methods will also be presented to determine the minimum set of features that can provide information on the network state. The algorithms are developed and tested with data from a 4G network. The impact of applying such methods is the proactive detection and root cause analysis of network anomalies thereby improving network reliability and availability.

 
2 favorite thumb_down thumb_up 2 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/Structure of the Talk

Introduction

Anomaly Detection Overview

Methodology

Experimental Results

Summary

Learning Outcome

Participants will learn about data pre-processing for data sets with many variables (numeric and categorical), feature selection, application of principal component analysis, and application of finite state machine methods.

Target Audience

People interested in learning about the application of machine learning techniques to real problems.

Prerequisites for Attendees

Knowledge of basic machine learning techniques and statistics.

schedule Submitted 1 year ago

Public Feedback

comment Suggest improvements to the Speaker
  • Naresh Jain
    By Naresh Jain  ~  11 months ago
    reply Reply

    Veena, thank you for proposing this important session on Network Anomaly Detection and Root Cause Analysis.

    You've marked this as a workshop. However, from the outline, it appears to be a talk. Am I missing something? In a workshop, participants are expected to engage in a hands-on activity. If you do have an activity, can you please update the proposal to call that out and also mark any pre-reqs for the same.

    Also, in the title you've mentioned about Root Cause Analysis. However, in the proposal, I don't see much about RCA. Can you please explain?

    • Dr. Veena Mendiratta
      By Dr. Veena Mendiratta  ~  11 months ago
      reply Reply

      Hello  Naresh - Thanks for your comments. Based on your comments I have updated my proposal as follows: changed the category from Workshop to Talk; and added some detail on the root cause analysis methods. Let me know if you have additional comments or questions.


  • Liked Dr. Dakshinamurthy V Kolluru
    keyboard_arrow_down

    Dr. Dakshinamurthy V Kolluru - ML and DL in Production: Differences and Similarities

    45 Mins
    Talk
    Beginner

    While architecting a data-based solution, one needs to approach the problem differently depending on the specific strategy being adopted. In traditional machine learning, the focus is mostly on feature engineering. In DL, the emphasis is shifting to tagging larger volumes of data with less focus on feature development. Similarly, synthetic data is a lot more useful in DL than ML. So, the data strategies can be significantly different. Both approaches require very similar approaches to the analysis of errors. But, in most development processes, those approaches are not followed leading to substantial delay in production times. Hyper parameter tuning for performance improvement requires different strategies between ML and DL solutions due to the longer training times of DL systems. Transfer learning is a very important aspect to evaluate in building any state of the art system whether ML or DL. The last but not the least is understanding the biases that the system is learning. Deeply non-linear models require special attention in this aspect as they can learn highly undesirable features.

    In our presentation, we will focus on all the above aspects with suitable examples and provide a framework for practitioners for building ML/DL applications.

  • Liked Dr. Manish Gupta
    keyboard_arrow_down

    Dr. Manish Gupta / Radhakrishnan G - Driving Intelligence from Credit Card Spend Data using Deep Learning

    45 Mins
    Talk
    Beginner

    Recently, we have heard success stories on how deep learning technologies are revolutionizing many industries. Deep Learning has proven huge success in some of the problems in unstructured data domains like image recognition; speech recognitions and natural language processing. However, there are limited gain has been shown in traditional structured data domains like BFSI. This talk would cover American Express’ exciting journey to explore deep learning technique to generate next set of data innovations by deriving intelligence from the data within its global, integrated network. Learn how using credit card spend data has helped improve credit and fraud decisions elevate the payment experience of millions of Card Members across the globe.

  • Liked Srijak Bhaumik
    keyboard_arrow_down

    Srijak Bhaumik - Let the Machine THINK for You

    Srijak Bhaumik
    Srijak Bhaumik
    Sr. Staff Software Developer
    IBM
    schedule 11 months ago
    Sold Out!
    20 Mins
    Demonstration
    Beginner

    Every organization is now focused on the business or customer data and trying hard to get actionable insights out of it. Most of them are either hiring data scientists or up-skilling their existing developers. However, they do understand the domain or business, relevant data and the impact, but, not essentially excellent in data science programming or cognitive computing. To bridge this gap, IBM brings Watson Machine Learning (WML), which is a service for creating, deploying, scoring and managing machine learning models. WML’s machine learning model creation, deployment, and management capabilities are key components of cognitive applications. The essential feature is the “self-learning” capabilities, personalized and customized for specific persona - may it be the executive or business leader, project manager, financial expert or sales advisor. WML makes the need of cognitive prediction easy with model flow capabilities, where machine learning and prediction can be applied easily with just a few clicks, and to work seamlessly without bunch of coding - for different personas to mark boundaries between developers, data scientists or business analysts. In this session, WML's capabilities would be demonstrated by taking a specific case study to solve real world business problem, along with challenges faced. To align with the developers' community, the architecture of this smart platform would be highlighted to help aspiring developers be aware of the design of a large-scale product.

  • Liked Venkatraman J
    keyboard_arrow_down

    Venkatraman J - Detection and Classification of Fake news using Convolutional Neural networks

    Venkatraman J
    Venkatraman J
    Sr. Software engineer
    Metapack
    schedule 1 year ago
    Sold Out!
    20 Mins
    Talk
    Intermediate

    The proliferation of fake news or rumours in traditional news media sites, social media, feeds, and blogs have made it extremely difficult and challenging to trust any news in day to day life. There are wide implications of false information on both individuals and society. Even though humans can identify and classify fake news through heuristics, common sense and analysis there is a huge demand for an automated computational approach to achieve scalability and reliability. This talk explains how Neural probabilistic models using deep learning techniques are used to classify and detect fake news.

    This talk will start with an introduction to Deep learning, Tensor flow(Google's Deep learning framework), Dense vectors (word2vec model) feature extraction, data preprocessing techniques, feature selection, PCA and move on to explain how a scalable machine learning architecture for fake news detection can be built.

  • Liked Venkatraman J
    keyboard_arrow_down

    Venkatraman J - Hands on Data Science. Get hands dirty with real code!!!

    Venkatraman J
    Venkatraman J
    Sr. Software engineer
    Metapack
    schedule 11 months ago
    Sold Out!
    45 Mins
    Workshop
    Intermediate

    Data science refers to the science of extracting useful information from data. Knowledge discovery in data bases, data mining, Information extraction also closely match with data science. Supervised learning,Semi supervised learning,Un supervised learning methodologies are out of Academia and penetrated deep into the industry leading to actionable insights, dashboard driven development, data driven reasoning and so on. Data science has been the buzzword for last few years in industry with only a handful of data scientists around the world. The industry needs more and more data scientists in future to solve problems using statistical techniques. The exponential availability of unstructured data from the web has thrown huge challenges to data scientists to exploit them before driving conclusions.

    Now that's overload of information and buzzwords. It all has to start somewhere? Where and how to start? How to get hands dirty rather than just reading books and blogs. Is it really science or just code?. Let's get into code to talk data science.

    In this workshop i will show the tools required to do real data science rather than just reading by building real models using Deep neural networks and show live demo of the same. Also share some of the key data science techniques every aspiring data scientist should have to thrive in the industry.