The Impact of Behavioral Biases to Real-World Data Science Projects: Pitfalls and Guidance

schedule Sep 1st 03:30 - 04:15 PM place Grand Ball Room 1 people 57 Interested

Data science projects, unlike their software counterparts tend to be uncertain and rarely fit into standardized approach. Each organization has it’s unique processes, tools, culture, data and in-efficiencies and a templatized approach, more common for software implementation projects rarely fits.

In a typical data science project, a data science team is attempting to build a decision support system that will either automate human decision making or assist a human in decision making. The dramatic rise in interest in data sciences means the typical data science project has a large proportion of relatively inexperienced members whose learnings draw heavily from academics, data science competitions and general IT/software projects.

These data scientists learn over time that the real world however is very different from the world of data science competitions. In the real-word problems are ill-defined, data may not exist to start with and it’s not just model accuracy, complexity and performance that matters but also the ease of infusing domain knowledge, interpretability/ability to provide explanations, the level of skill needed to build and maintain it, the stability and robustness of the learning, ease of integration with enterprise systems and ROI.

Human factors play a key role in the success of such projects. Managers making the transition from IT/software delivery to data science frequently do not allow for sufficient uncertainty in outcomes when planning projects. Senior leaders and sponsors, are under pressure to deliver outcomes but are unable to make a realistic assessment of payoffs and risks and set investment and expectations accordingly. This makes the journey and outcome sensitive to various behavioural biases of project stakeholders. Knowing what the typical behavioural biases and pitfalls makes it easier to identify those upfront and take corrective actions.

The speaker brings his nearly two decades of experience working at startups, in R&D and in consulting to lay forth these recurring behavioural biases and pitfalls.

Many of the biases covered are grounded in the speakers first-hand experience. The talk will provide examples of these biases and suggestions on how to identify and overcome or correct for them.

 
 

Outline/Structure of the Talk

- The roadmap of a typical data science initiative. (5 mins)

- Common biases and pitfalls, and how to handle them. (10 pitfalls x 2-3 min each) - 25 mins

- Case study (one case study that will provide many of the pitfalls) - 10 mins

- Recap & concluding remarks - 3 mins

Learning Outcome

  • Improve ability to assess risk in data science project proposals and plans
  • Improve ability to manage data science projects and data scientists.
  • Improve at setting expectations with stakeholders.

Target Audience

Data Science Team Leads/Managers

Prerequisites for Attendees

Elementary knowledge of data sciences. Past experience with analytics and data science projects in Industry.

schedule Submitted 1 year ago

Public Feedback

comment Suggest improvements to the Speaker
  • Sarah Masud
    By Sarah Masud  ~  1 year ago
    reply Reply

    Hello Dr Rohit,

    This is a great proposal, well explained. It would be great it would you could mention against the outline, how many pitfalls and case studies you plan to cover, also if you add the time estimates for the 3 points in the structure of the Session.

    • Dr. Rohit M. Lotlikar
      By Dr. Rohit M. Lotlikar  ~  1 year ago
      reply Reply

      Sarah, Thank you for your comment. I have updated the outline section with those details. 

  • Vishal Gokhale
    By Vishal Gokhale  ~  1 year ago
    reply Reply

    Hello Dr Lotlikar,

    Sorry for the confusion about the previous comment. 
    It is visible again. (Got marked as private, by mistake)

    Please feel free to revert, if it is still not visible.

    Thanks,
    Vishal.

    • Vishal Gokhale
      By Vishal Gokhale  ~  1 year ago
      reply Reply

      Thanks for the proposal, Rohit!

      This is an interesting topic.
      Can you please share links (slides and videos) to any of your previous talks on this topic?



    • Liked Dr. Ananth Sankar
      keyboard_arrow_down

      Dr. Ananth Sankar - The Deep Learning Revolution in Automatic Speech Recognition

      45 Mins
      Keynote
      Beginner

      In the last decade, deep neural networks have created a major paradigm shift in speech recognition. This has resulted in dramatic and previously unseen reductions in word error rate across a range of tasks. These improvements have fueled products such as voice search and voice assistants like Amazon Alexa and Google Home.

      The main components of a speech recognition system are the acoustic model, lexicon, and language model. In recent years, the acoustic model has evolved from using Gaussian mixture models to deep neural networks, resulting in significant reductions in word error rate. Recurrent neural network language models have also given improvements over the traditional statistical n-gram language models. More recently sequence to sequence recurrent neural network models have subsumed the acoustic model, lexicon, and language model into one system, resulting in a far simpler model that gives comparable accuracy to the traditional systems. This talk will outline this evolution of speech recognition technology, and close with some key challenges and interesting new areas to apply this technology.

    • Liked Naresh Jain
      keyboard_arrow_down

      Naresh Jain / Dr. Arun Verma / Dr. Denis Bauer / Favio Vázquez / Sheamus McGovern / Drs. Tarry Singh / Dr. Tom Starke / Dr. Veena Mendiratta - Unanswered Questions - Ask the Experts!

    • Liked Dr. Ravi Mehrotra
      keyboard_arrow_down

      Dr. Ravi Mehrotra - Seeking Order amidst Chaos and Uncertainty

      45 Mins
      Keynote
      Beginner

      Applying analytics to determine an optimal answer to business decision problems is relatively easy when the future can be predicted accurately. When the business environment is very complex and the future cannot be predicted, the business problem can become intractable using traditional modeling and problem-solving techniques. How do we solve such complex and intractable business problems to find globally optimal answers in highly uncertain business environments? The talk will discuss modeling and solution techniques that allow us to find optimal solutions in highly uncertain business environments without ignoring or underestimating uncertainty for revenue management and dynamic price optimization problems that arise in the airline and hospitality industry.

    • Liked Sheamus McGovern
      keyboard_arrow_down

      Sheamus McGovern / Naresh Jain - Welcome Address

      20 Mins
      Keynote
      Beginner

      This talk will help you understand the vision behind ODSC Conference and how it has grown over the years.

    • Liked Dr. Dakshinamurthy V Kolluru
      keyboard_arrow_down

      Dr. Dakshinamurthy V Kolluru - ML and DL in Production: Differences and Similarities

      45 Mins
      Talk
      Beginner

      While architecting a data-based solution, one needs to approach the problem differently depending on the specific strategy being adopted. In traditional machine learning, the focus is mostly on feature engineering. In DL, the emphasis is shifting to tagging larger volumes of data with less focus on feature development. Similarly, synthetic data is a lot more useful in DL than ML. So, the data strategies can be significantly different. Both approaches require very similar approaches to the analysis of errors. But, in most development processes, those approaches are not followed leading to substantial delay in production times. Hyper parameter tuning for performance improvement requires different strategies between ML and DL solutions due to the longer training times of DL systems. Transfer learning is a very important aspect to evaluate in building any state of the art system whether ML or DL. The last but not the least is understanding the biases that the system is learning. Deeply non-linear models require special attention in this aspect as they can learn highly undesirable features.

      In our presentation, we will focus on all the above aspects with suitable examples and provide a framework for practitioners for building ML/DL applications.

    • Naoya Takahashi
      Naoya Takahashi
      Sr. researcher
      Sony
      schedule 1 year ago
      Sold Out!
      45 Mins
      Demonstration
      Intermediate

      In evolutionary history, the evolution of sensory organs and brain plays very important role for species to survive and prosper. Extending human’s abilities to achieve a better life, efficient and sustainable world is a goal of artificial intelligence. Although recent advances in machine learning enable machines to perform as good as, or even better than human in many intelligent tasks including automatic speech recognition, there are still many aspects to be addressed to bridge the semantic gap and achieve seamless interaction with machines. Auditory intelligence is a key technology to enable natural man machine interaction and expanding human’s auditory ability. In this talk, I am going to address three aspects of it:

      (1) non-speech audio recognition,

      (2) video highlight detection,

      (3) one technology to surpassing human’s auditory ability, namely source separation.

    • Liked Swapan Rajdev
      keyboard_arrow_down

      Swapan Rajdev - Conversational Agents at Scale: Retrieval and Generative approaches

      Swapan Rajdev
      Swapan Rajdev
      CTO
      Haptik
      schedule 1 year ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      Conversational Agents (Chatbots) are machine learning programs that are designed to have conversation with a human to help them fulfill a particular task. In recent years people have been using chatbots to communicate with business, help get daily tasks done and many more.

      With the emergence of open source softwares and online platforms building a basic conversational agent has become easier but making them work across multiple domains and handle millions of requests is still a challenge.

      In this talk I am going to talk about the different algorithms used to build good chatbots and the challenges faced to run them at scale in production.

    • Liked Atin Ghosh
      keyboard_arrow_down

      Atin Ghosh - AR-MDN - Associative and Recurrent Mixture Density Network for e-Retail Demand Forecasting

      45 Mins
      Case Study
      Intermediate

      Accurate demand forecasts can help on-line retail organizations better plan their supply-chain processes. The chal- lenge, however, is the large number of associative factors that result in large, non-stationary shifts in demand, which traditional time series and regression approaches fail to model. In this paper, we propose a Neural Network architecture called AR-MDN, that simultaneously models associative fac- tors, time-series trends and the variance in the demand. We first identify several causal features and use a combination of feature embeddings, MLP and LSTM to represent them. We then model the output density as a learned mixture of Gaussian distributions. The AR-MDN can be trained end-to-end without the need for additional supervision. We experiment on a dataset of an year’s worth of data over tens-of-thousands of products from Flipkart. The proposed architecture yields a significant improvement in forecasting accuracy when compared with existing alternatives.

    • Liked Favio Vázquez
      keyboard_arrow_down

      Favio Vázquez - Agile Data Science Workflows with Python, Spark and Optimus

      Favio Vázquez
      Favio Vázquez
      Sr. Data Scientist
      Raken Data Group
      schedule 1 year ago
      Sold Out!
      480 Mins
      Workshop
      Intermediate

      Cleaning, Preparing , Transforming and Exploring Data is the most time-consuming and least enjoyable data science task, but one of the most important ones. With Optimus we’ve solve this problem for small or huge datasets, also improving a whole workflow for data science, making it easier for everyone. You will learn how the combination of Apache Spark and Optimus with the Python ecosystem can form a whole framework for Agile Data Science allowing people and companies to go further, and beyond their common sense and intuition to solve complex business problems.

    • Liked Sohan Maheshwar
      keyboard_arrow_down

      Sohan Maheshwar - It's All in the Data: The Machine Learning Behind Alexa's AI Systems

      Sohan Maheshwar
      Sohan Maheshwar
      Alexa Evangelist
      Amazon
      schedule 1 year ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      Amazon Alexa, the cloud-based voice service that powers Amazon Echo, provides access to thousands of skills that enable customers to voice control their world - whether it’s listening to music, controlling smart home devices, listening to the news or even ordering a pizza. Alexa developers use advanced natural language understanding that to use capabilities like built-in slot & intent training, entity resolution, and dialog management. This natural language understanding is powered by advanced machine learning algorithms that will be the focus of this talk.

      This session will tell you about the rise of voice user interfaces and will give an in-depth look into how Alexa works. The talk will delve into the natural language understanding and how utterance data is processed by our systems, and what a developer can do to improve accuracy of their skill. Also, the talk will discuss how Alexa hears and understands you and how error handling works.

    • Liked Bargava Subramanian
      keyboard_arrow_down

      Bargava Subramanian / Amit Kapoor - Deep Learning in the Browser: Explorable Explanations, Model Inference, and Rapid Prototyping

      45 Mins
      Demonstration
      Beginner

      The browser is the most common end-point consumption of deep learning models. It is also the most ubiquitous platform for programming available. The maturity of the client-side JavaScript ecosystem across the deep learning process—Data Frame support (Arrow), WebGL-accelerated learning frameworks (deeplearn.js), declarative interactive visualization (Vega-Lite), etc.—have made it easy to start playing with deep learning in the browser.

      Amit Kapoor and Bargava Subramanian lead three live demos of deep learning (DL) for explanations, inference, and training done in the browser, using the emerging client-side JavaScript libraries for DL with three different types of data: tabular, text, and image. They also explain how the ecosystem of tools for DL in the browser might emerge and evolve.

      Demonstrations include:

      1. Explorable explanations: Explaining the DL model and allowing the users to build intuition on the model helps generate insight. The explorable explanation for a loan default DL model allows the user to explore the feature space and threshold boundaries using interactive visualizations to drive decision making.
      2. Model inference: Inference is the most common use case. The browser allows you to bring your DL model to the data and also allows you test how the model works when executed on the edge. The demonstrated comments sentiment application can identify and warn users about the toxicity of your comments as you type in a text box.
      3. Rapid prototyping: Training DL models is now possible in the browser itself, if done smartly. The rapid prototyping image classification example allows the user to play with transfer learning to build a model specific for a user-generated image input.

      The demos leverage the following libraries in JavaScript:

      • Arrow for data loading and type inference
      • Facets for exploratory data analysis
      • ml.js for traditional machine learning model training and inference
      • deeplearn.js for deep learning model training and inference
      • Vega and Vega-Lite for interactive dashboards

      The working demos will be available on the web and as open source code on GitHub.

    • Liked Amit Kapoor
      keyboard_arrow_down

      Amit Kapoor / Bargava Subramanian - Architectural Decisions for Interactive Viz

      45 Mins
      Talk
      Beginner

      Visualization is an integral part of the data science process and includes exploratory data analysis to understand the shape of the data, model visualization to unbox the model algorithm, and dashboard visualization to communicate the insight. This task of visualization is increasingly shifting from a static and narrative setup to an interactive and reactive setup, which presents a new set of challenges for those designing interactive visualization applications.

      Creating visualizations for data science requires an interactive setup that works at scale. Bargava Subramanian and Amit Kapoor explore the key architectural design considerations for such a system and discuss the four key trade-offs in this design space: rendering for data scale, computation for interaction speed, adapting to data complexity, and being responsive to data velocity.

      • Rendering for data scale: Envisioning how the visualization can be displayed when data size is small is not hard. But how do you render interactive visualization when you have millions or billions of data points? Technologies and techniques include bin-summarise-smooth (e.g., Datashader and bigvis) and WebGL-based rendering (e.g., deck.gl).
      • Computation for interaction speed: Making the visualization reactive requires the user to have the ability to interact, drill down, brush, and link multiple visual views to gain insight. But how do you reduce the latency of the query at the interaction layer so that the user can interact with the visualization? Technologies and techniques include aggregation and in-memory cubes (e.g., hashcubes, InMEMS, and nanocubes), approximate query processing and sampling (e.g., VerdictDB), and GPU-based databases (e.g., MapD).
      • Adapting to data complexity: Choosing a good visualization design for a singular dataset is possible after a few experiments and iterations, but how do you ensure that the visualization will adapt to the variety, volume, and edge cases in the real data? Technologies and techniques include responsive visualization to space and data, handling high cardinality (e.g., Facet Dive), and multidimensional reduction (e.g., Embedding Projector).
      • Being responsive to data velocity: Designing for periodic query-based visualization refreshes is one thing, but streaming data adds a whole new level of challenge to interactive visualization. So how do you work decide between the trade-offs of real-time and near real-time data and their impact on refreshing visualization? Technologies and techniques include optimizing for near real-time visual refreshes and handling event- and time-based streams.
    • Liked Kavita Dwivedi
      keyboard_arrow_down

      Kavita Dwivedi - Social Network Analytics to enhance Marketing Outcomes in Telecom Sector

      Kavita Dwivedi
      Kavita Dwivedi
      Head Data Science
      ISM
      schedule 1 year ago
      Sold Out!
      20 Mins
      Experience Report
      Beginner

      This talk will focus on How SNA can help enhance the outcomes of Marketing Campaigns by using social network graphs .

      Social network analytics (SNA) is the process of investigating social structures through the use of network and graph theories. It characterizes networked structures in terms of nodes (individual actors, people, or things within the network) and the ties or edges (relationships or interactions) that connect them. This is emerging as an important tool to understand customer behavior and influencing his behavior. The talk will focus on the mathematics behind SNA and how SNA can help make marketing decisions for telecom operators.

      SNA use case will use telecom consumer data to establish networks based on their calling behavior like frequency, duration of calls, types of connections and thus establish major communities and influencers. By identifying key influencers and active communities marketing campaigns can be made more effective/viral. It helps in improving the adoption rate by targeting influencers with a large degree of followers. It will also touch upon how SNA helps retention rate and spread the impact of marketing campaigns. The tools used for use case is SAS SNA and Node XL for demonstration purpose. It will show how SNA helps in lifting the impact of campaigns.

      This use case will illustrate a project focused on building a SNA model using a combination of demographic/firmographic variables for companies variables and Call frequency details. The dimensions like the company you work with, the place you stay, your professional experience and position, Industry Type etc. helps add a lot more value to the social network graph. With the right combination of the dimensions and problem at hand, in our case, it was more of marketing analytics we can identify the right influencers within a network. The more dimensions we add, the network gets stronger and more effective for running campaigns.

      Looking forward to discussing the outcomes of this project with the audience and fellow speakers

    • Liked Mahesh Balaji
      keyboard_arrow_down

      Mahesh Balaji - Deep Learning in Medical Image Diagnostics

      Mahesh Balaji
      Mahesh Balaji
      Sr. Director
      Cognizant
      schedule 1 year ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      Convolutional Neural Networks are revolutionizing the field of Medical Imaging analysis and Computer Aided Diagnostics. Medical images from X-Rays, CT, MRI, retinal scans to digitized biopsy slides are an integral part of a patient’s EHR. Current manual analysis and diagnosis by human radiologists, pathologists are prone to undue delays, erroneous diagnosis and can therefore benefit from deep learning based AI for quantitative, standardized computer aided diagnostic tools.

      In this session, we will review the state of the art in medical imaging and diagnostics, important tasks like classification, localization, detection, segmentation and registration along with CNN architectures that enable these. Further, we will briefly cover data augmentation techniques, transfer learning and walkthrough two casestudies on Diabetic Retinopathy and Breast Cancer Diagnosis. Finally, we discuss inherent challenges from sourcing training data to model interpretability.

    • Liked Dr. Manish Gupta
      keyboard_arrow_down

      Dr. Manish Gupta / Radhakrishnan G - Driving Intelligence from Credit Card Spend Data using Deep Learning

      45 Mins
      Talk
      Beginner

      Recently, we have heard success stories on how deep learning technologies are revolutionizing many industries. Deep Learning has proven huge success in some of the problems in unstructured data domains like image recognition; speech recognitions and natural language processing. However, there are limited gain has been shown in traditional structured data domains like BFSI. This talk would cover American Express’ exciting journey to explore deep learning technique to generate next set of data innovations by deriving intelligence from the data within its global, integrated network. Learn how using credit card spend data has helped improve credit and fraud decisions elevate the payment experience of millions of Card Members across the globe.

    • Liked Joy Mustafi
      keyboard_arrow_down

      Joy Mustafi - The Artificial Intelligence Ecosystem driven by Data Science Community

      45 Mins
      Talk
      Intermediate

      Cognitive computing makes a new class of problems computable. To respond to the fluid nature of users understanding of their problems, the cognitive computing system offers a synthesis not just of information sources but of influences, contexts, and insights. These systems differ from current computing applications in that they move beyond tabulating and calculating based on pre-configured rules and programs. They can infer and even reason based on broad objectives. In this sense, cognitive computing is a new type of computing with the goal of more accurate models of how the human brain or mind senses, reasons, and responds to stimulus. It is a field of study which studies how to create computers and computer software that are capable of intelligent behavior. This field is interdisciplinary, in which a number of sciences and professions converge, including computer science, electronics, mathematics, statistics, psychology, linguistics, philosophy, neuroscience and biology. Project Features are Adaptive: They MUST learn as information changes, and as goals and requirements evolve. They MUST resolve ambiguity and tolerate unpredictability. They MUST be engineered to feed on dynamic data in real time; Interactive: They MUST interact easily with users so that those users can define their needs comfortably. They MUST interact with other processors, devices, services, as well as with people; Iterative and Stateful: They MUST aid in defining a problem by asking questions or finding additional source input if a problem statement is ambiguous or incomplete. They MUST remember previous interactions in a process and return information that is suitable for the specific application at that point in time; Contextual: They MUST understand, identify, and extract contextual elements such as meaning, syntax, time, location, appropriate domain, regulation, user profile, process, task and goal. They may draw on multiple sources of information, including both structured and unstructured digital information, as well as sensory inputs (visual, gestural, auditory, or sensor-provided). {A set of cognitive systems is implemented and demonstrated as the project J+O=Y}

    • 20 Mins
      Experience Report
      Intermediate

      Generative Models are important techniques used in computer vision. Unlike other neural networks that are used for predictions from images, generative models can generate new images for specific objectives. This session will review several applications of generative modeling such as artistic style transfer, image generation and image translation using CNNs and GANs.

    • 90 Mins
      Tutorial
      Advanced
      Advancements in Deep Learning seem almost unstoppable and research is the only way to make true improvements. Tarry and his team in deepkapha.ai is working relentlessly to write a few papers pertaining to Capsule Networks, automated swiping functions, and adaptations in optimizers and learning rates. Here in this lecture, we will briefly touch how research is transforming the field of AI and finally reveal two papers namely, Neuroscience and impact of Deep Learning and ARiA, a novel new NN activation function that has already proven its dominance over ReLU and Sigmoid.
    • Liked Akshay Bahadur
      keyboard_arrow_down

      Akshay Bahadur - Recognizing Human features using Deep Networks.

      Akshay Bahadur
      Akshay Bahadur
      SDE-I
      Symantec Softwares
      schedule 1 year ago
      Sold Out!
      20 Mins
      Demonstration
      Beginner

      This demo would be regarding some of the work that I have already done since starting my journey in Machine Learning. So, there are a lot of MOOCs out there for ML and data science but the most important thing is to apply the concepts learned during the course to solve simple real-world use cases.

      • One of the projects that I did included building state of the art Facial recognition system [VIDEO]. So for that, I referred to several research papers and the foundation was given to me in one of the courses itself, however, it took a lot of effort to connect the dots and that's the fun part.
      • In another project, I made an Emoji Classifier for humans [VIDEO] based on your hand gestures. For that, I used deep learning CNN model to achieve great accuracy. I took reference from several online resources that made me realize that the data science community is very helpful and we must make efforts to contribute back.
      • The other projects that I have done using machine learning:
        1. Handwritten digit recognition [VIDEO],
        2. Alphabet recognition [VIDEO],
        3. Apparel classification [VIDEO],
        4. Devnagiri recognition [VIDEO].

      With each project, I have tried to apply one new feature or the other to make my model a bit more efficient. Hyperparameter tuning or just cleaning the data.

      In this demonstration, I would just like to point out that knowledge never goes to waste. The small computer vision applications that I built in my college has helped me to gain deep learning computer vision task. It's always enlightening and empowering to learn new technologies.

      I recently was part of a session on ‘Solving real world applications from Machine learning’ to Microsoft Advanced Analytics User Group of Belgium as well as broadcasted across the globe (Meetup Link) [Session Recording]

    • Liked Jyotsna Khemka
      keyboard_arrow_down

      Jyotsna Khemka / Dr. Amarpal S Kapoor - Distributed Deep Learning on Spark CPU Clusters with Intel BigDL

      Jyotsna Khemka
      Jyotsna Khemka
      Sr. Technical Consulting Engineer
      Intel
      Dr. Amarpal S Kapoor
      Dr. Amarpal S Kapoor
      Software Technical Consulting Engineer
      Intel
      schedule 1 year ago
      Sold Out!
      480 Mins
      Workshop
      Beginner

      Are you or your customers running on Apache Hadoop and Spark clusters? Are you curious on how you can run Deep Learning for inference and training datasets on your existing Hadoop or Spark CPU clusters without having to migrate your data from one setup to another – experience how easy it is to run DL on your general purpose CPU. This workshop will help you - Get Started on your journey with Intel's Big Data Deep learning Library - BigDL.

      Join AI experts at Intel for a deep-dive workshop to learn about computer vision and deep learning.

      • Technical overview of BigDL architecture and learn:
        • How it fits in the Apache Spark stack
        • Key features
        • Resources, code samples, and tips
      • Build with BigDL
        • Learn how to deploy BigDL on-premise or in the cloud to build end-to-end solutions.
        • Review case studies
        • Experience real-life demos