Big Data to Big Intelligence - How I used Open source information to generate actionable insights for the Indian Army

As a data scientist i have been lucky enough to be develop to be a part of highly critical and cutting edge solutions for pristine organizations like Intel , Indian Air Force , Indian Army etc. While each of them was an amazing experience in its own right , the challenges i faced and the knowledge i gained from making an Open Source Intelligence gathering and Analytics/Prediction tool are unmatched.

By the time i implemented the tool the tool had some great capabilities like :-

  • Predict and estimate the location of an Twitter/Facebook user (who has disabled his location obviously!) through various metrics.
  • Predict occurrence of certain events (eg. Riot's) based of information gathered from various Open Sources.
  • Identify and Predict accounts of people who may be potential suspects.
  • Contextual analysis of words to derive relevant insights.

I worked on this project for over an year , and since then have been using my experience of OSINT in other sectors like Healthcare/Pharma . While working for other sectors i was appalled by the way OSINT information was being under-utilized .

I would like to share my experience of working with some great mean on a very critical project for out military by discussing the problems i faced and how they can be overcome , at the same time i hope to give you a guideline on how you can efficiently utilize the power of OSINT information in your respective field be in consumer goods , healthcare or energy. I hope that the attendees will pick up some valuable insights from my experience which will help them in projects ranging from NLP to Time-Series analytics.

 
1 favorite thumb_down thumb_up 4 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/Structure of the Experience Report

This is an overview of how we will be walking through the session , for the first part i will be explaining what exactly is OSINT and how this project was obtained and how its scope was defined . Next we will go over various modules of the project and the challenges they presented and end with some of my key takeaways :-

Intro: How this project came to be and overview of OSINT/Social Media Analytics

  • How i was able to acquire this amazing project.
  • What i thought of OSINT/ Social media analytics before starting the project
  • OSINT - The huge free data store which you never use
  • Large Data vs Relevant Data
  • How good are the social media tools that are currently availabe in the market

Module 1:- Location Estimation

  • Why it is very difficult and main challenges faced.
  • Social Media Data + OSINT = Your Location
  • People Lie on the Internet [How i dealt with faulty data]
  • Using Networks to refine location accuracy.

Module 2:- Contextual Analysis

  • Things are rarely as they seem [Problem Statement Overview]
  • The Regional Barrier & Why techniques like transfer learning didnt work too well.
  • Why this proved to be one of the most important and challenging modules.
  • Different Languages , Different Problems , Same Solution.

Module 3 :- Identifying Potential Suspects

  • Why i found this impossible at first.
  • Your tweets betray you [People reveal more than they think , how i could exploit that]
  • How i was able to implement machine learning on such a hapazard data set.

Module 3 :- Identifying Potential Suspects

  • Why i found this impossible at first.
  • Your tweets betray you [People reveal more than they think , how i could exploit that]
  • How i was able to implement machine learning on such a hapazard data set.

Module 3 :- Identifying Potential Suspects

  • Why i found this impossible at first.
  • Your tweets betray you [People reveal more than they think , how i could exploit that]
  • How i was able to implement machine learning on such a hapazard data set.

Closing Remarks :-

  • Good Data vs Bad Data and how i learnt to tell the difference.
  • How what i learnt here is applicable in other feilds.
  • How you can use the power of Machine Learning and OSINT for a variety of tasks.
  • Small demo of Tool

Learning Outcome

I hope that the key takeaways would be the following :-

- Develop an understanding over the power of OSINT data.

- Learn the common challenges faced while dealing with it.

- Understand how to use OSINT in your own projects.

Target Audience

AI Enthusiasts,Data Scientists,Managers,People working in NLP, People interested in exploring Open Source data

Prerequisites for Attendees

Basic knowledge of data wrangling techniques and machine learning concepts might help you absorb more from the talk :)

schedule Submitted 2 weeks ago

Public Feedback

comment Suggest improvements to the Speaker
  • Kuldeep Jiwani
    By Kuldeep Jiwani  ~  3 days ago
    reply Reply

    Hi Yash,

    You have chosen an interesting topic for your proposal, good to know that you wish to share the importance of OSINT feed to public.

    2 quick questions:

    1. Are you going to focus on some aspect of OSINT feeds like internet (social media) or academic data or public government data ?
      • Or this would be a general talk focusing on all major sources
    2. As for the learning outcome to the audience
      • Clearly they will learn about handling various imp information in OSINT feeds
      • Will you also sharing some of your experiences on some Data mining / ML techniques you applied on top of raw OSINT data?
    • Yash Deo
      By Yash Deo  ~  2 days ago
      reply Reply

      Hey Kuldeep ,

      1. I would be talking about most of the major sources of OSINT feeds but will be focusing more heavily on the Social Media/Internet aspect as i believe is relevant in most use cases.

      2. Of course i will sharing my experience of Data mining/ML over the OSINT data as data mining is over this data is a major road block most people new to this field face.

      Feel free to let me know if i should include something else or take a particular approach which would suit the audience better.

      Thanks!

      • Kuldeep Jiwani
        By Kuldeep Jiwani  ~  1 day ago
        reply Reply

        Thanks for the clarification

  • Ashay Tamhane
    By Ashay Tamhane  ~  1 week ago
    reply Reply

    Thanks Yash for your proposal. It is indeed very interesting. Couple of points:

    1. Since the topic is sensitive, could you clarify if you have obtained the required clearance from relevant authorities for this talk?

    2. It will be helpful if you could post a short video on same/similar topic.