Recommendation systems (RecSys) are the core engine for any personalized experience on eCommerce and online media websites. Most of the companies leverage RecSys to increase user interaction, to enrich shopping potential and to generate upsell & cross-sell opportunities. Amazon uses recommendations as a targeted marketing tool throughout its website that contributes 35% of its total revenue generation [1]. Netflix users watch ~75% of the recommended content and artwork [2]. Spotify employs a recommendation system to update personal playlists every week so that users won’t miss newly released music by artists they like. This has helped Spotify to increase its number of monthly users from 75 million to 100 million at a time [3]. YouTube's personalized recommendation helps users to find relevant videos quickly and easily which account for around 60% of video clicks from the homepage [4].

In general, RecSys generates recommendations based on user browsing history and preferences, past purchases and item metadata. It turns out most existing recommendation systems are based on three paradigms: collaborative filtering (CF) and its variants, content-based recommendation engines, and hybrid recommendation engines that combine content-based and CF or exploit more information about users in content-based recommendation. However, they suffer from limitations like rapidly changing user data, user preferences, static recommendations, grey sheep, cold start and malicious user.

Classical RecSys algorithm like content-based recommendation performs great on item to item similarities but will only recommend items related to one category and may not recommend anything in other categories as the user never viewed those items before. Collaborative filtering solves this problem by exploiting the user's behavior and preferences over the items in recommending items to the new users. However, collaborative filtering suffers from a few drawbacks like cold start, popularity bias, and sparsity. The classical recommendation models consider the recommendation as a static process. We can solve the static recommendation on rapidly changing user data by RL. RL based RecSys captures the user’s temporal intentions and responds promptly. However, as the user action and items matrix size increases, it becomes difficult to provide recommendations using RL. Deep RL based solutions like actor-critic and deep Q-networks overcome all the aforementioned drawbacks.

Present systems suffer from two limitations, firstly considering the recommendation as a static procedure and ignoring the dynamic interactive nature between users and the recommender systems. Also, most of the works focus on the immediate feedback of recommended items and neglecting the long-term rewards based on reinforcement learning. We propose a recommendation system that uses the Q-learning method. We use ε-greedy policy combined with Q learning, a powerful method of reinforcement learning that handles those issues proficiently and gives the customer more chance to explore new pages or new products that are not so popular. Usually while implementing Reinforcement Learning (RL) to real-world problems both the state space and the action space are very vast. Therefore, to address the aforementioned challenges, we propose the multiple/distributed Q table approaches which can deal with large state-action space and that aides in actualizing the Q learning algorithm in the recommendation and huge state-action space.

References:

  1. "https://www.mckinsey.com/industries/retail/our-insights/how-retailers-can-keep-up-with-consumers":https://www.mckinsey.com/industries/retail/our-insights/how-retailers-can-keep-up-with-consumers
  2. "https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429":https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429
  3. "https://www.bloomberg.com/news/articles/2016-09-21/spotify-is-perfecting-the-art-of-the-playlist":https://www.bloomberg.com/news/articles/2016-09-21/spotify-is-perfecting-the-art-of-the-playlist
  4. "https://dl.acm.org/citation.cfm?id=1864770":https://dl.acm.org/citation.cfm?id=1864770
  5. "Deep Reinforcement Learning based Recommendation with Explicit User-Item Interactions Modelling": https://arxiv.org/pdf/1810.12027.pdf
  6. "Deep Reinforcement Learning for Page-wise Recommendations": https://arxiv.org/pdf/1805.02343.pdf
  7. "Deep Reinforcement Learning for List-wise Recommendations": https://arxiv.org/pdf/1801.00209.pdf
  8. "Deep Reinforcement Learning Based RecSys Using Distributed Q Table": http://www.ieomsociety.org/ieom2020/papers/274.pdf
 
 

Outline/Structure of the Talk

  1. Introduction of recommendation system. [2 mins]
  2. Classical approaches and algorithms for building a recommendation system. [3 mins]
  3. Limitations of classical approaches. [ 2 mins]
  4. Introduction of reinforcement learning. [ 2 mins]
  5. Deep reinforcement learning-based algorithm for building a recommendation system. [5 mins]
  6. Traning methodology, use-case and result discussion. [5 mins]
  7. Closure / Q&A [1 min]

Learning Outcome

  1. Gain an understanding of deep reinforcement learning-based recommendation system.
  2. How to train and evaluate the RL model with distributed Q-table?
  3. Reference architecture of recommendation engines.

Target Audience

Data Scientist and Machine Learning Engineer, Data Engineers, Data Architects

Prerequisites for Attendees

The talk is a somewhat advanced talk and given it is a 20-minute session, the audience will need to be familiar with:

  1. A basic understanding of machine learning and programming.
  2. A basic understanding of reinforcement learning and recommendation system.
schedule Submitted 10 months ago

Public Feedback

comment Suggest improvements to the Speaker
  • Deepti Tomar
    By Deepti Tomar  ~  4 months ago
    reply Reply

    Hello Ravi, Abhishek & Subarna,

    Thanks for your time and efforts on the proposal! Request you to help the program committee understand the following -

    - How does the overall outline leverage the 3 presenters? 

    Considering this is a 20 minutes session, we suggest you re-look into the outline/structure and the need for 3 presenters for the session. 

    Please feel free to ask us any questions to help you further,

    Thanks,

    Deepti 

    • Ravi Ranjan
      By Ravi Ranjan  ~  4 months ago
      reply Reply

      Hi Deepti,

      Initially, we had submitted this proposal for 3 hours workshop and considered all the three speakers.

      But now as we have considered this proposal for 20 mins talk, I will be taking this session covering all the topics mentioned in the talk outline section.

      Thanks & Regards,

      Ravi.

      • Deepti Tomar
        By Deepti Tomar  ~  3 months ago
        reply Reply

        Sure, Thanks for your response, Ravi! Please update your proposal accordingly with you as the speaker. We will let you know in case if we have more questions.

        • Ravi Ranjan
          By Ravi Ranjan  ~  3 months ago
          reply Reply

          Hi Deepti,

          I have updated the proposal.

          Thanks,

          Ravi.

          • Deepti Tomar
            By Deepti Tomar  ~  3 months ago
            reply Reply

            Great, Thanks Ravi!

            • Ravi Ranjan
              By Ravi Ranjan  ~  3 months ago
              reply Reply

              Hi Deepti,

               

              We have discussed internally and come up with the final outline. Subarna will initiate the presentation with a discussion on below topics:

              1. Introduction of recommendation system. [2 mins]
              2. Classical approaches and algorithms for building a recommendation system. [3 mins]
              3. Limitations of classical approaches. [ 2 mins]

              Then I will discuss on the below topics:

              1. Introduction of reinforcement learning. [ 2 mins]
              2. Deep reinforcement learning-based algorithm for building a recommendation system. [5 mins]
              3. Traning methodology, use-case, and result discussion. [5 mins]

              We will combinedly take Q/A for the audience.

              Thanks,

              Ravi.

              • Deepti Tomar
                By Deepti Tomar  ~  3 months ago
                reply Reply

                Hello Ravi,

                Thanks for the update and details. As discussed earlier, we suggest you to update the proposal with 1 speaker for 20 mins session.

                Thanks,

                Deepti

                • Ravi Ranjan
                  By Ravi Ranjan  ~  3 months ago
                  reply Reply

                  Hi Deepti,

                  I have updated the proposal with myself as a primary speaker covering all the topics for 20 mins session.

                  Thanks,

                  Ravi.

    • Natasha Rodrigues
      By Natasha Rodrigues  ~  4 months ago
      reply Reply

      Hi Ravi/Abhishek/Subarna,

      Thanks for your proposal! Requesting you to update the Outline/Structure section of your proposal with a time-wise breakup of how you plan to use 20 mins for the topics you've highlighted? To help the program committee understand your presentation style, can you add the slides for your proposal and provide a link to your past recording or record a small 1-2 mins trailer of your talk and share the link to the same?

      Thanks,

      Natasha

      • Ravi Ranjan
        By Ravi Ranjan  ~  4 months ago
        reply Reply

        Thank you Natasha,

         

        I will update my proposal as per your suggestions.

        • Natasha Rodrigues
          By Natasha Rodrigues  ~  4 months ago
          reply Reply

          Thank you Ravi.

          • Ravi Ranjan
            By Ravi Ranjan  ~  4 months ago
            reply Reply

            Hi Natasha,

            I have updated the proposal as per the review comments to a crisp 20 mins talk. I have presented the topic at IEOM international conference in Dubai on 12 March 2020. I could not able to shoot video due to COVID 2019 lockdown. For reference, I have provided the conference URL and slides used in the conference.

            Thanks,

            Ravi.

            • Ashay Tamhane
              By Ashay Tamhane  ~  4 months ago
              reply Reply

              Thanks Ravi for an interesting proposal. Would it be possible for you / other speakers to record a brief video (mobile quality is ok) on your respective parts and share via google drive (or other)?

              • Ravi Ranjan
                By Ravi Ranjan  ~  4 months ago
                reply Reply

                Hi Ashay,

                I will try to shoot video on mobile and share it with you over the weekend. I hope that will work for you.

                Thanks,

                Ravi.

                • Ashay Tamhane
                  By Ashay Tamhane  ~  4 months ago
                  reply Reply

                  Yes, that works. Thanks!

                  • Ravi Ranjan
                    By Ravi Ranjan  ~  4 months ago
                    reply Reply

                    Hi Ashay,

                    Please find the URL below on preview video for my talk at ODSC 2020:

                    https://youtu.be/ygwOZOo5Vnw

                    Thanks,

                    Ravi.

                    • Natasha Rodrigues
                      By Natasha Rodrigues  ~  4 months ago
                      reply Reply

                      Hi Ravi, 

                      Thank you for the video, kindly update the same in the video/link section of your proposal as well.

                      Many thanks,

                      Natasha 

                      • Ravi Ranjan
                        By Ravi Ranjan  ~  4 months ago
                        reply Reply

                        Hi Natasha,

                        I have already updated the URLs in my proposal. Could you please confirm from your side?

                        Thanks,

                        Ravi.

                        • Natasha Rodrigues
                          By Natasha Rodrigues  ~  4 months ago
                          reply Reply

                          Hi Ravi,

                          Yes, it's there, thank you.

    • Sujoy Roychowdhury
      By Sujoy Roychowdhury  ~  4 months ago
      reply Reply

      I am a bit confused on the structure and content of the talk. You talk extensively of the many shortcomings of the traditional CF / content based models and their variants. You skip the Deep learning variants of it (which I am ok with) and focus on RL based models. This itself is  a vast area and there are many considerations to be taken. Why do you not just focus on RL based Rec Systems but jump into in a 20 minute talk into a hands-on into RL TF models and deploying them using KubeFlow. This goes too fast without anyone understanding the nuances of a RL based model - especially since RL is less well understood than traditional DL. Also there are to my knowledge not much difference between deploying a normal model into KubeFlow and deploying a recommendation system into KubeFlow.

      I would recommend the talk be limited to RL in Rec. Systems - the when, why and how and its limitations and challenges. Please give your thoughts.

      • Ravi Ranjan
        By Ravi Ranjan  ~  4 months ago
        reply Reply

        Hi Sujoy,

        Thank you for your feedback. We had submitted this proposal as 4 hours workshop with extensive content but will update the proposal content for 20 mins crisp talk keeping the scope to RL in RecSys.

        • Sujoy Roychowdhury
          By Sujoy Roychowdhury  ~  4 months ago
          reply Reply

          have you done the update ?

          • Ravi Ranjan
            By Ravi Ranjan  ~  4 months ago
            reply Reply

            Hi Sujoy,

            I have updated the proposal as per the review comments to a crisp 20 mins talk. I have presented the topic at IEOM international conference in Dubai on 12 March 2020. For reference, I have provided the conference URL and slides used in the conference.

            Thanks,

            Ravi.

            • Sujoy Roychowdhury
              By Sujoy Roychowdhury  ~  4 months ago
              reply Reply

              how much of this is based on a use case / POC perspective ? 

              • Ravi Ranjan
                By Ravi Ranjan  ~  4 months ago
                reply Reply

                Hi Sujoy,

                We have created this POC to solve the use-case for one of our Automobile clients.

                Thanks,

                Ravi.


    • Liked Sharmistha Chatterjee
      keyboard_arrow_down

      Sharmistha Chatterjee - Machine Learning and Data Governance in Telecom Industry

      20 Mins
      Demonstration
      Intermediate

      The key aspect in solving ML problems in telecom industry lies in continuous data collection and evaluation from different categories of customers and networks so as to track and dive into varying performance metrics. The KPIs form the basis of network monitoring helping network/telecom operators to automatically add and scale network resources. Such smart automated systems are built with the objective of increasing customer engagement through enhanced customer experience and tracking customer behavior anomaly with timely detection and correction. Further the system is designed to scale and serve current LTE, 4G and upcoming 5G networks with minimal non-effective cell site visits and quick identification of Root Cause Analysis (RCA).

      Network congestion has remained an ever-increasing problem. Operators have attempted a variety of strategies to match the network demand capacity with existing infrastructure, as the cost of deploying additional network capacities is expensive. To keep the cost under control, operators apply control measures to attempt to allocate bandwidth fairly among users and throttle the bandwidth of users that consume excessive bandwidth. This approach had limited success. Alternatively, techniques that utilize extra bandwidth for quality of experience (QOE) efficiency by over-provisioning the network has proved to be ineffective and inefficient due to lack of proper estimation.

      The evolution of 5G networks, would lead manufacturers and telecom operators to use high-data transfer rates, wide network coverage, low latency to build smart factories using automation, artificial intelligence and Internet of Things (IoT). The application of advanced data science and AI can provide better predictive insights to improve network capacity-planning accuracy. Better network provisioning would yield better network utilization for both next-generation networks based on 5G technology and current LTE and 4G networks. Further AI models can be designed to link application throughput with network performance, prompting users to plan their daily usage based on their current location and total monthly budget.

      In this talk, we will understand the current challenges in telecom industry, the need for an AIOPS platform, and the mission held by telecom operators, communication service providers across the world for designing such AI frameworks, platforms and best practices. We will see how increasing operator collaborations are helping to create, deploy and productionize AI platforms for different AI use-cases. We will study one industrial use-case (with code) based on real-time field research to predict network capacity. In this respect we will investigate how deep learning networks can be used to train large volumes of data at scale (millions of network cells), and how its use can help the upcoming 5G networks. We will also examine an end to end pipeline of hosting the scalable framework on Google Cloud with special emphasis on Data Governance and Data Mangement. As data volume is huge and data needs to be stored in highly secured systems, we build our high-performing system with extra security features that can process millions of request in an order of few mili-secs. As the session highlights parameters and metrics in creating the neural network, it also discusses the challenges and some of the key aspects involved in designing and scaling the system.

    • Liked Gunjan Dewan
      keyboard_arrow_down

      Gunjan Dewan - Developing a match-making algorithm between customers and Go-Jek products!

      Gunjan Dewan
      Gunjan Dewan
      Data Scientist
      Go-Jek
      schedule 5 months ago
      Sold Out!
      20 Mins
      Talk
      Beginner

      20+ products. Millions of active customers. Insane amount of data and complex domain. Come join me in this talk to know the journey we at Gojek took to predict which of our products a user is most likely to use next.

      A major problem we faced, as a company, was targeting our customers with promos and vouchers that were relevant to them. We developed a generalized model that takes into account the transaction history of users and gives a ranked list of our services that they are most likely to use next. From here on, we are able to determine the vouchers that we can target these customers with.

      In this talk, I will be talking about how we used recommendation engines to solve this problem, the challenges we faced during the time and the impact it had on our conversion rates. I will also be talking about the different iterations we went through and how our problem statement evolved as we were solving the problem.

    • Liked Kriti Doneria
      keyboard_arrow_down

      Kriti Doneria - Trust Building in AI systems: A critical thinking perspective

      Kriti Doneria
      Kriti Doneria
      Data Science
      Practitioner
      schedule 10 months ago
      Sold Out!
      20 Mins
      Talk
      Beginner

      How do I know when to trust AI,and when not to?

      Who goes to jail if a self driving car kills someone tomorrow?

      Do you know scientists say people will believe anything,repeated enough

      Designing AI systems is also an exercise in critical thinking because an AI is only as good as its creator.This talk is for discussions like these,and more.

      With the exponential increase in computing power available, several AI algorithms that were mere papers written decades ago have become implementable. For a data scientist, it is very tempting to use the most sophisticated algorithm available. But given that its applicability has moved beyond academia and out into the business world, are numbers alone sufficient? Putting context to AI, or XAI (explainable AI) takes the black box out of AI to enhance human-computer interaction. This talk shall revolve around the interpret-ability-complexity trade-off, challenges, drivers and caveats of the XAI paradigm, and an intuitive demo of translating inner workings of an ML algorithm into human understandable formats to achieve more business buy-ins.

      Prepare to be amused and enthralled at the same time.

    • Liked Dr. Sri Vallabha Deevi
      keyboard_arrow_down

      Dr. Sri Vallabha Deevi - How to train your dragon - Reinforcement learning from scratch

      20 Mins
      Talk
      Beginner

      Reinforcement learning helped Google's "AlphaGo" beat the world's best Go player. Have you wondered if you too can train a program to play a simple game?

      Reinforcement learning is a simple yet powerful technique that is driving many applications, from recommender systems to autonomous vehicles. It is best suited to handle situations where the behavior of the system cannot be described in simple rules. For example, a trained reinforcement learning agent can understand the scene on the road and drive the car like a human. In supply chain management, RL agents can make decisions on inventory ordering.

      In this talk, I will demonstrate how to train a RL agent to a) cross a maze and b) play a game of Tic-Tac-Toe against an intelligent opponent c) act as a warehouse manager and learn inventory ordering; with the help of plain python code. As you participate in this talk, you will master the basics of reinforcement learning and acquire the skills to train your own dragon.

    • Liked Srikanth K S
      keyboard_arrow_down

      Srikanth K S - Actionable Rules from Machine Learning Models

      20 Mins
      Demonstration
      Intermediate

      Beyond predictions, some ML models provide rules to identify actionable sub-populations in support-confidence-lift paradigm. Along with making the models interpretable, rules make it easy for stakeholders to decide on the plan of action. We discuss rule-based models in production, rule-based ensembles, anchors using R package: tidyrules.