Web Scrapping with Selenium and Data Analysis using IPython Notebook

schedule 02:30 PM - 02:50 PM place Grand Ball Room 1 people 2 Attending

Data Analysis is one of the upcoming field and as many of the data scientists says that the most of time they spend for analysis is on Data cleaning, So, In this short session we will see how one can pull the data from the web using Selenium Webdriver and will use this data further for the Data Analysis, The entire exercise will be executed on a IPython Notebook, which is a tool used to execute & save your code and perform data analysis using python data analysis libraries, it also provides a platform to massage the data and visualize it in the form of graphs and tables.

This entire exercise would be helpful for anyone who wants to understand how data can be pulled with the help of Selenium Webdriver from a website and organized using python libraries for the data analysis. During this session we would be using an open source data for analysis and see how we can draw conclusions using this data.

 
6 favorite thumb_down thumb_up 4 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

Step 1: Description of IPython Notebook

Step 2: How to execute Selenium Webdriver commands from IPython Notebook

Step 3: How to scrap data from an Open Source Data Website using Selenium Webdriver

Step 4: Convert the data(extracted in step 3) in a meaningful form for data analysis

Step 5: Data Analysis and drawing conclusion out of the data extracted using python data analysis libraries

Step 6: Summarize the Entire exercise

Learning Outcome

a) Selenium users will know how quickly we can execute the automation scripts from the IPython Notebook and everytime Eclipse or any other IDE is not required for executing the scripts.

b) Scrapping Website and Extracting Data

c) Organize the extracted data in the form of python Dictionary and List, Useful for someone who is working on the data and need to understand how  to organize the data

d) Data analysis using python libraries from the extracted data

Target Audience

Automation Testers, or Anyone who is working with Data

schedule Submitted 1 year ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Dave Haeffner
    By Dave Haeffner  ~  1 year ago
    reply Reply

    As a backup you could record a screencast of your demo in case the internet fails.

    • Vinay Babu
      By Vinay Babu  ~  1 year ago
      reply Reply

      Ok i can have the screencast in place. However I'm not sure if my proposal has been shortlisted for the presentation, No confirmation yet.

  • Anand Bagmar
    By Anand Bagmar  ~  1 year ago
    reply Reply

    hi,

    1. do you have a backup plan if internet connection is not as good as expected?

    2. will you be sharing the code (snippets) for interested folks to leverage what they learnt?

    • Vinay Babu
      By Vinay Babu  ~  1 year ago
      reply Reply

      Hi,

      1. Should be manageable without internet connection, only thing is I would not be able to demo the real time scrapping of web data with selenium

      2. Sure I would be sharing the entire IPython Notebook on my git account.

       

      Thanks

      Vinay Babu


  • Liked Oren Rubin
    keyboard_arrow_down

    Oren Rubin - Statistical Element Locator

    Oren Rubin
    Oren Rubin
    Founder and CEO of Testim
    Testim.io
    schedule 1 year ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    In this talk we will delve into one the biggest challenges that Test Automation developers face, finding elements i.e. a robust test means finding the same element with high fidelity while the Application Under Test keeps changing.

    We will categorise the methods, and show where developer fail, where machines (Record/Playback) fail, and suggest a new way for locating elements, and analyze the skill-set required to overcome those difficulties.

  • Liked Ori Bendet
    keyboard_arrow_down

    Ori Bendet - Tales from the Dark Side: The Growth, Implementation and Influence of Selenium inside Hewlett Packard Enterprise

    45 mins
    Talk
    Intermediate

    I know what you’re thinking: the creators of WinRunner, QTP/UFT are now embracing Selenium?

    Ten years after Selenium came into existence as an open source alternative to Mercury Interactive, the perception and relationship between QTP and Selenium has morphed from competition to collaboration with complementary test automation frameworks.

    Join Ori Bendet, HPE Inbound Product Manager for Functional Testing to discuss how HPE’s R&D uses Selenium and other open source tools. Understand the new roles and responsibilities of dev/test @HPE and how they fit into current team structure. Discover their lessons learned about how Selenium and open source has contributed to the success and maturity of HPE's own quality assurance and testing tools across the entire portfolio.

  • Liked Sargis Sargsyan
    keyboard_arrow_down

    Sargis Sargsyan - Better Page Object Handling with Loadable Component Pattern

    45 mins
    Talk
    Advanced

    One of the painful problems in Selenium automated testing is determining whether a HTML page has been loaded. This is especially the case when web application uses a JS  heavy framework such as the popular AngularJS.

    During this talk we will discuss how to handle Selenium Page Object pattern better with Loadable Component.

    The Loadable Component helps test case developers make sure that the page or a component of the page is loaded successfully. I will share my experience about the concept of the Loadable Component and Page Object patterns.

     

     

  • Liked Ankita Gupta
    keyboard_arrow_down

    Ankita Gupta - Web Push Notification Automation Mystery Solved!

    45 mins
    Demonstration
    Intermediate

    Push Notifications are the latest way of sending updates to our users. More and more Organisations are implementing Web Push Notifications along with emails and other notification systems.

    So the Big Question that arises is "How do we automate them?"

    We have come up with a library in various languages which can be integrated with your Automation suite and provide you everything you need about the notification triggered.

    You can easily then trigger and verify the push notification sent and ship out the product without worries :)

  • Liked Bret Pettichord
    keyboard_arrow_down

    Bret Pettichord - Checking as a Service

    Bret Pettichord
    Bret Pettichord
    Software Architect
    HomeAway
    schedule 1 year ago
    Sold Out!
    45 mins
    Keynote
    Beginner

    This talk suggests a reframe in how we understand the business value of automated testing. One shift is to see automation as "checking" rather than "testing". Another is the shift from software delivery to service delivery, including fully embracing DevOps. The resulting approach could be called Checking as a Service or CheckOps, and forces us to rethink traditional automation priorities. In this talk, Bret will explain how change in approach has affected teams he's worked with and how you can use it to improve your ability to deliver valued services.

  • Liked Michal Vanek
    keyboard_arrow_down

    Michal Vanek - Breaking down the barriers: Testing desktop apps with Selenium

    45 mins
    Talk
    Intermediate

    Selenium was born for web-application testing. But have you ever thought it could be a great tool for testing Windows desktop apps too?

    Today, more and more desktop apps use a web-like approach to implement their UI. The methods vary from basic HTMLayout environment to complex designs in CEF (Chromium Embedded Framework). However traditional GUI automation tools seem to be a step behind or ignoring the trend completely. This situation calls for finding new ways of testing.

    In our talk we shall introduce to you a new way of utilizing Selenium for automated testing of desktop applications. No matter whether the HTML UI content is completely offline or loaded and updated dynamically, Selenium is able to access and navigate it just like in a web page. We’ll also show you how to build a small framework around it and plug it into your Continuous Integration process. All of this will be demonstrated using a real-life instance of Avast Antivirus for Windows.

     

  • Liked Adam Carmi
    keyboard_arrow_down

    Adam Carmi - Advanced Automated Visual Testing With Selenium

    Adam Carmi
    Adam Carmi
    Co-Founder and VP R&D
    Applitools
    schedule 1 year ago
    Sold Out!
    45 mins
    Talk
    Beginner

    Automated visual testing is a major emerging trend in the dev / test community. In this talk you will learn what visual testing is and why it should be automated. We will take a deep dive into some of the technological challenges involved with visual test automation and show how modern tools address them. We will review available Selenium-based open-source and commercial visual testing tools, demo cutting edge technologies that enable running cross browser and cross device visual tests at large scale, and show how visual test automation fits in the development / deployment lifecycle.

    If you don’t know what visual testing is, if you think that Sikuli is a visual test automation tool, if you are already automating your visual tests and want to learn more on what else is out there, if you are on your way to implement Continuous Deployment or just interested in seeing how cool image processing algorithms can be, this talk is for you!

  • Liked Sarvesh Shrivastava
    keyboard_arrow_down

    Sarvesh Shrivastava - Functional and Security Testing - An amalgamated automation approach

    45 mins
    Demonstration
    Intermediate

    With increasing cyber threats & online attacks, an unavoidable situation for continuous security testing has emerged. Making sure all vulnerabilities are unleashed regularly is highly significant.

    The paper hence proposes a solution where automated security testing could be achieved in conjunction with functional testing carried out using selenium API.

    It introduces a framework that caters to automated security testing along with functional which could provide an integrated testing elucidation.

    The paper, takes in these two premises to offer a solution where functional automation testers can now take on security testing. I propose a framework where automated security testing could be achieved in conjunction with functional testing using existing selenium API scripts.

    The framework covers the top vulnerabilities and provides intuitive results that help a non-security tester interpret and act on the output. At the very core of this framework is the open source tool, OWASP ZAP, which is easy to use and integrates well with Selenium automation frameworks.

    I bring in hands on project experience having implemented this framework for clients, who have been able to get the value of functional and security testing using the same set of scripts – it is this experience I would like to share with the SeleniumConf2016 audience, to help groom functional testers into security testing, with minimal cost and time, also enabling security testing to be performed every time functional automation is taken up.

  • Liked Parashuram
    keyboard_arrow_down

    Parashuram - Reusing Selenium tests for catching Performance Regressions

    90 mins
    Demonstration
    Beginner

    Almost all the tests we write today are geared towards verifying the functional correctness of products. Selenium gives us a great way to ensure that our web applications and browser behave correctly and our tests usually do an excellent job running through the happy path.

    Most successful websites or hybrid mobile applications are not just functionally correct, but also have a very smooth performance and user experience. Performance, for many, is now a feature. In this talk, we will look at ways to re-use our selenium test cases to also catch any performance regressions. We will measure key performance indicators like frame rates and memory usage as the selenium scripts navigate and perform actions on the website. We will look at logging all these metrics into a dashboard, and integrating this with a continuous integration system like Jenkis or Team City. Finally, we will also look at how such a system can catch any code change that is responsible for making the website slower than a threshold we set.

     

    We will look at how we could use existing testing frameworks like Protractor or Jest to add performance metrics. We will extend our test matrix to cover desktop browsers, browsers on popular mobile platforms and even hybrid apps like Apache Cordova, Ionic or Phonegap.

     

     

    If Performance is a feature, let us test it like we test features !!

     

  • Liked Dharmesh Vaya
    keyboard_arrow_down

    Dharmesh Vaya - Selenium + SoapUI + JMeter : A complete framework

    45 mins
    Talk
    Beginner

    This talk showcases how you can develop a framework in Java with all kinds of features like WebTesting with Selenium, Service Layer testing with SoapUI and Load Testing with JMeter - all packaged as a single testing solution. Above all, make use of open source libraries and get details HTML reports as well as Summary reports. This solution allows you to seamlessly integrate all your testing requirements under a single framework.

     

  • Liked Jatin Makhija
    keyboard_arrow_down

    Jatin Makhija - Automation testing: An endangered species in Agile development

    45 mins
    Demonstration
    Beginner

    With the increased pace in adoption of agile methodology, testing is now becoming really complex in relation to the not so simple yet sweet demand of every project manager: "Why don’t you automate test cases while you test?"

    Having worked and counselled in product giants and more than 5 startups in India, we would like to throw some light on what fits the test automation use case. Is UI automation "the Only way to test efficiently?"

    The main issue with startups following Agile is “time crunch.” With multiple deploys a day and a fast paced development cycle, how does one find time to automate the product?

    In this session, we will be sharing some methods/tools/techniques which increase the overall test coverage and have worked well in the past for startup culture.

    A demo of some simple, useful and robust tools used for API, UI and Performance testing will also be given.