Visualizing complex data has always been a major problem in the field of data science , as complex data sets often lack comprehensibility , which make them extremely difficult to interpret for people who aren’t data scientists . Training deep learning models on these datasets greatly aggravates this problem as the models tend to become very opaque and hard to interpret.

This problems is especially acute in medical datasets , where the black box nature of deep learning algorithms is a cause for concern and doctors have difficulty believing the predictions of the model. It is more easy to interpret the models and build trust if they are coupled with visualizations which could be easily explained to the doctors.

Visualizing complex datasets though , is a very arduous task because of the huge number of features in the set and the sparsity of the datasets. This problem can be tackled by using variations of deep autoencoders/VAE's/VAE-GANs' etc. which can figure out the latent structure in complex data sets and map it to a 3-D plane which can be readily understood.

As a data scientist i have been lucky enough to be a part of highly critical and cutting edge solutions for pristine organizations like Intel , Indian Army etc. While each of them was an amazing experience in its own right , the challenges i faced and the knowledge i gained from making an Open Source Intelligence gathering and Analytics/Prediction tool for the Indian Army are unmatched . This experience showed me how powerful Open source data can be if it is used correctly .

An OSINT tool can have some powerful capabalities like :-

• Predict and estimate the location of an Twitter/Facebook user (who has disabled his location obviously!) through various metrics.
• Predict occurrence of certain events (eg. Riot's) based of information gathered from various Open Sources.
• Identify and Predict accounts of people who may be potential suspects (Security use Case) or potential Influencers (Commercial use case).
• Contextual analysis of words to derive relevant insights.

Open Source Data is however very challenging to work with for a vast array of reasons . This is the issue i aim to tackle with this talk , I will be going over 3 exciting projects which have been made using open source data through which i shall demonstrate various techniques to find / modify / model / and use machine learning techniques on the data. While going over the projects i shall also try to draw parallels so as to how you can use similar techniques in your own endeavours.