Exploring Automation Strategies and Frameworks What Should Your Team Be Using?

schedule Oct 16th 11:00 AM - 11:45 AM place Tiered Classroom people 21 Attending

Agile practices have done a magnificent job of speeding up the software development process. Unfortunately, simply applying agile practices to testing isn't enough to keep testers at the same pace. Test automation is necessary to support agile delivery. Max Saperstone explores popular test automation frameworks and shares the benefits of applying these frameworks, their implementation strategies, and best usage practices. Focusing on the pros and cons of each type of framework, Max discusses data-driven, keyword-driven, and action-driven approaches. Find out which framework and automation strategy are most beneficial for specific situations.

Other than using specific frameworks as examples, the presentation is framework agnostic, really focusing on capabilities of different types of framework, and how those might or might-not fit your company's/software's needs. In the talk, 5 different types of frameworks will be discussed: Record/Playback (Linear), Modular/Structured, Data Driven, Keyword Driven, Action Based, and of course the combination of these as Hybrids.

At the end of the talk, depending on questions, and how much time we have left, a slide with a long list of specific testing frameworks is displayed, and an open discussion of who has used what, and what has/hasn't worked for them is had.

If you are new to test automation or trying to optimize your current automation strategy, this session is for you.

 

 
11 favorite thumb_down thumb_up 4 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

  • Introduction
  • Test Automation Planning
    • Test Automation Objectives
    • Test Automation Scope
    • Testing Types
  • Test Automation Framework Selection
    • What is a Framework
    • Automation Framework Types
    • Test Automation Tool Selection
  • Test Automation Strategy
    • Selecting Tests for Automation
    • Determining what to Automate
    • Data Management
    • Traceability
    • Best Practices
    • Parameterized Testing
  • What Testing Framework is Right For Me?
  • Questions

Learning Outcome

At the end of this session, students will understand all of the intricacies of choosing a framework, what to look for, what capabilities exist, and be better prepared to make a selection on a test automation framework

Target Audience

Testers and QA. Automation Engineers responsible for setting up tooling

schedule Submitted 4 months ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Bob Payne
    By Bob Payne  ~  3 months ago
    reply Reply

    Max,

    Data on specific frameworks you will be reviewing would be helpful for attendees and reviewers alike.

    -bob

    • Max Saperstone
      By Max Saperstone  ~  3 months ago
      reply Reply

      So, I try to stay agnostic for specific frameworks, really focusing on capabilities of different types of framework, and how those might or might-not fit your company's/software's needs. In the talk, I discuss 5 different types of frameworks:

      • Record/Playback (Linear)
      • Modular / Structured
      • Data Driven
      • Keyword Driven
      • Action Based
      • Hybrids

      I do list some examples of each type of tool, when we go through these. All of that said, at the end of the talk, depending on questions, and how much time we have left, I usually throw up a slide with a long list of testing frameworks, and try to foster an open discussion of who has used what, and what has/hasn't worked for them.

      If you think it will help, I can update the abstract a bit more, to emphasize the tooling agnostic nature of the talk.

      • Bob Payne
        By Bob Payne  ~  3 months ago
        reply Reply

        Sounds great Max thanks for the reply.

        You could almost past the above text into the description as it greatly clarifies.

         

        -bob

        • Max Saperstone
          By Max Saperstone  ~  3 months ago
          reply Reply

          Thanks, I made some updates


  • Liked Gene Gotimer
    keyboard_arrow_down

    Gene Gotimer - Experiences Bringing Continuous Delivery to a DoD Project

    Gene Gotimer
    Gene Gotimer
    Technical Manager
    Coveros, Inc.
    schedule 2 months ago
    Sold Out!
    45 mins
    Experience Report
    Beginner

    Not every continuous delivery initiative starts with someone saying "drop everything. Let's do DevOps." Sometimes you have grow your practice incrementally. And sometimes, you don’t set out to grow a practice at all-- you are just fixing problems with your process, trying to make things better.

    I'll walk through a case study of how our team worked on an exemplar project for the Department of Defense to show that agile could work in a decidedly waterfall culture. I’ll also discuss techniques and tools we used to bring a DevOps mindset and continuous delivery practices into an environment that wasn't already Agile.

    I'll talk about how we were able to start in development, where we had the most control, with a "let's starting being Agile" initiative and working on "why is continuous integration important?" From there, we tackled one problem after another, each time making the release a little easier and a little less risky. We incrementally brought our practices through other environments until the project was confidently delivering working, QA-tested, security-tested releases that were ready for production every two weeks. I’ll discuss the journey we took and the tools we used to get to build quality into our product, our releases, and our release process.

  • Liked Gene Gotimer
    keyboard_arrow_down

    Gene Gotimer - Tests Your Pipeline Might be Missing

    Gene Gotimer
    Gene Gotimer
    Technical Manager
    Coveros, Inc.
    schedule 2 months ago
    Sold Out!
    10 mins
    Talk
    Beginner

    Developing a delivery pipeline means more than just adding automated deploys to the development cycle. To be successful, tests of all types must be incorporated throughout the process in order to be sure that problems aren’t slipping through. Most pipelines include unit tests, functional tests, and acceptance tests, but those aren’t always enough. I’ll present some types of testing you might not have considered, or at least might not have considered the importance of. Some types will address code quality, others code security, and some the health and security of the pipeline itself.

    I’ll talk about specific tools we used to supplement our pipeline testing. I won’t get into how to use each tool-- this is more of a series of teasers to encourage people to look into the tools, and even letting them know what types of tools and testing opportunities are out there.

  • Liked James Gifford
    keyboard_arrow_down

    James Gifford - 5 Metrics to Create Safety and High Performing Teams

    45 mins
    Tutorial
    Intermediate

    Description:

    I see that a lot of organizations use metrics in inappropriate ways to measure teams. At the heart of these metrics, nine times out of ten, are velocity and story points. These metrics lead to a lot of mistrust, fear, and bad technical practices. This talk will focus on shifting the focus to diagnostic metrics.

     Before shifting focus to diagnostic metrics, we need to understand what inappropriate metrics are. When questioning teams about why their velocity was lower from one sprint to another, teams are more likely to inflate their estimates to avoid questions in the future. This is one of my scenarios. We will explore this case and my other top-ten based on the 165 teams I have interacted with. Focusing on one metric does not provide a balanced view of the team.

    For balance, I promote five metrics. The combination of metrics balances each other. These five metrics are lead time, quality, happiness, agile maturity, and business value. Focusing on these five metric areas can be used as a diagnostic tool to help teams grow and support coaching. During the session, we will use my Excel-based tool and visual model to simulate this balance.

    When you push shorter lead times (how fast) on a team with a lower agile maturity, the first thing to change is quality, followed by happiness and then the delivery of value. Conversely, if a team focuses on TDD, the first thing to change is quality, followed by agile maturity, reduction in lead time, and increased delivery of value.  

    Teaching teams to harness data in a positive way will help them to flourish.

  • Liked Roland Cuellar
    keyboard_arrow_down

    Roland Cuellar - Accelerating Business Side Agility - An Enterprise Experience Report from Nationwide Insurance

    45 mins
    Talk
    Advanced

    While the IT side of this F500 financial services client has been practicing agile development for years, the business side is still, in some cases, more waterfall and traditional in its approach to requirements and releases.  In this experience report, we will hear from Associate Vice President Charlie Kennedy of Nationwide Insurance on how we have accelerated business-side agility by:

    • Re-organizing from functional silos to Product-Based Value-Stream Teams
    • Redesigning the business requirements process to achieve a continuous flow of MMPs into the teams
    • Utilization of Big-Room-Planning techniques to create alignment across a wide set of stakeholders and dependencies
    • Design of a Product-Manager-to-Product-Owner fan-out concept to achieve omni-channel digital delivery
    • Design of a portfolio-level visitation system to track the flow of MMPs across multiple investments simultaneously

     

  • Liked Rodney Bodamer
    keyboard_arrow_down

    Rodney Bodamer - Lean Delivery Learnings: Tailoring Agile for Government Programs

    45 mins
    Talk
    Intermediate

    In the last five years there has been a tremendous surge in the volume of Federal procurements calling for lean and agile solutions to complex problems.  In spite of this, many of these same Government agencies still cling to traditional waterfall models of delivery.

    How do we embrace lean and agile principles while delivering under these constraints within large-scale Government agile software delivery initiatives? 

    What agile approaches can be effectively used "out of the box" while others may need to be tailored to address legacy Government processes and operating environments? 

    In this talk I’ll walk through each of the seven Lean principles and unveil how -- on two Federal Government scaled-agile engagements -- specific lean-agile processes and approaches were tailored for program delivery success, while remaining compliant with agency mandates.   

  • Liked Phillip Manketo
    keyboard_arrow_down

    Phillip Manketo / Dave McMunn - Building Strong Foundations…. Underwriting Fannie Mae’s Agile Transformation

    45 mins
    Experience Report
    Advanced

    Over the course of the last two and one-half years, Fannie Mae has worked aggressively to transform itself from a heavily silo’d and firmly entrenched command and control culture, following a gated workflow, with long release cycles, to an Agile organization.  Today, Fannie Mae is a more dynamic value oriented organization that is responsive to stakeholders, focused on achieving greater efficiency by enabling fast-feedback loops, as well as using empirical data to optimize mature and persistent agile values and practices.  

     

    Within the larger context of the transformation to enterprise agility, this Experience Report will focus on the case for change, Fannie Mae’s journey and the corresponding challenges, benefits and key learnings realized.  Our conclusion, while it is important to build bridges with business stakeholders, mature agile teams, leverage automation and embrace the values and principles of the agile manifesto… a successful and longstanding transformation is dependent upon the unrelenting focus on changing the ecosystem supporting the organization’s change at the outset.

  • Liked Gene Gotimer
    keyboard_arrow_down

    Gene Gotimer / Ryan Kenney - Creative Solutions to Already Solved Problems

    10 mins
    Experience Report
    Beginner

    Almost everyone has to deal with bad legacy code at some point. Not just legacy code that you inherited and obviously would have been better if you had written it, but legacy code so ugly and ill-conceived that it makes you want to hunt down the person responsible just so you can scream at them (or worse). And then replace it with a one-line library function that does the same thing.

    We'll show some examples of the worst code I've seen, and we'll have a chuckle or a groan. The names, projects, and check-in comments have been changed to protect the guilty, but, unfortunately, these examples are all too real.

  • Liked Jonathan Kauffman
    keyboard_arrow_down

    Jonathan Kauffman - Current State of BDD Testing Tools

    Jonathan Kauffman
    Jonathan Kauffman
    Consultant
    Coveros, Inc.
    schedule 2 months ago
    Sold Out!
    10 mins
    Talk
    Beginner

    Have you heard about BDD and want to start using it, but don't know what BDD is and which tool you should use? In this presentation I address both of those concerns -- I start by providing an overview of BDD and then compare five tools that can be used for BDD testing. I conclude by discussing the pros/cons and popularity of these tools so that you can make an informed decision as to which tool would work best within your organization.

  • Liked Glenn Buckholz
    keyboard_arrow_down

    Glenn Buckholz - Improving Your Testing Methodology Using Docker

    Glenn Buckholz
    Glenn Buckholz
    Technical Manager
    Coveros
    schedule 2 months ago
    Sold Out!
    45 mins
    Tutorial
    Beginner

    Wonder how you can make your testing more efficient? Join Glenn Buckholz as he explores Docker, a technology that allows rapid development and deployment via containers. First, he explains exactly what composes a container, and discusses the differences between a container and an image. Once this is clear, Glenn demonstrates how Docker solves the problem of what he calls the state capture problem. When a test case produces a failure, the developer and testers often expend significant effort reproducing the issue so the developer can see the issue and fix it. Glenn demonstrates how Docker enables succinct, accurate, and quick communication between testers and developers, helping mitigate the state capture problem. In addition, testers can use Docker to load data, efficiently insert testing tools into a running system, set system state, and aid in test reproducibility. After you look at the inner workings of Docker and run through a few practical examples, you’ll find that Docker will hold an important place in your testing toolbox.

  • Liked Gene Gotimer
    keyboard_arrow_down

    Gene Gotimer - Which Development Metrics Should I Watch?

    Gene Gotimer
    Gene Gotimer
    Technical Manager
    Coveros, Inc.
    schedule 2 months ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    W. Edwards Deming noted that “people with targets and jobs dependent upon meeting them will probably meet the targets – even if they have to destroy the enterprise to do it.” While metrics can be a great tool for evaluating performance and software quality, becoming beholden to reaching metrics goals, especially the wrong ones, can be detrimental to the project. Each team needs to take care and understand what targets are appropriate for their project. They also need to consider the current and desired states of the source code and product and the capabilities and constraints of the team.

    As one of the lead architects working with a huge codebase on a government project, I often have the opportunity to influence the teams around me into watching or ignoring various metrics. I will walk through some measures that are available to most projects and discuss what they really mean, various misconceptions about their meaning, the tools that can be used to collect them, and how you can use them to help your team. I’ll discuss experiences and lessons learned (often the hard way) about using the wrong metrics and the damage they can do.

  • Liked Scott Blacker
    keyboard_arrow_down

    Scott Blacker / Darren Hoevel - Scaling Product Management in an Agile World

    45 mins
    Talk
    Beginner

    What every Product Manager needs to know about scaling Agile

     Product managers play a crucial role in a scaled Agile organization. But what does it truly mean to play this role? Product Owners sit with the scrum team and make the day-to-day decisions that fundamentally shape your products. Product Managers want to make sure the reality of the product direction is in line with what the market needs, but don’t have the time to sit with 10 scrum teams. What is the role of the Product Manager in a scaled agile organization and what role should PMs be playing in defining, executing, and commercializing an organization's product offering? How can Product Managers effectively manage and communicate the product roadmap when hundreds or thousands of stakeholders are involved?

     

  • Liked Rahul Sharma
    keyboard_arrow_down

    Rahul Sharma - Building and Testing Secure Mobile Applications

    Rahul Sharma
    Rahul Sharma
    IT Consultant
    Coveros
    schedule 2 months ago
    Sold Out!
    45 mins
    Tutorial
    Intermediate

    Mobile application development has been on the rise lately because of the convenience mobile apps have to offer. Despite the recent occurrence of security breaches on mobile devices, security testing is not as emphasized as other forms of testing such as user acceptance or functional testing. An application can consist of the greatest features but will be considered unusable if hackers can exploit it. The exponential rise in the use of mobile applications for different purposes puts mobile devices in significant danger of being hacked or compromised. In today’s world, mobile applications are used for various purposes and store Personally Identifiable Information (PII) and financial information. Due to the sensitivity of customer data, mobile applications should be built and tested with security in mind. Strategies that cover how to properly test mobile apps for security issues will be discussed.

  • Liked Marco Corona
    keyboard_arrow_down

    Marco Corona - Agile DevOps Transformation at HUD

    Marco Corona
    Marco Corona
    Consultant
    Coveros
    schedule 2 months ago
    Sold Out!
    45 mins
    Experience Report
    Beginner

    Housing and Urban Development (HUD), a federal agency committed to creating affordable homes for all Americans, has a history of systems development steeped in waterfall practices, a history of failed IT programs, and a culture that ran in direct opposition to Agile/DevOps. It often took weeks to provision a virtual machine and years for an application to get into Production.

    In a little over a year, a small team of DevOps engineers has helped modernize the agency’s legacy infrastructure in an effort to prove Agile and DevOps can work across the organization. I will present a case study that discusses how we were able to bring 10 new applications into Production in a few months time using the Cloud and DevOps. I will discuss the challenges we encountered along the way and walk through how we were able to create a culture of shared code, infrastructure and shared purpose across multiple programs and contractor teams. In addition, I will explain how to leverage Jenkins, Chef and Azure to create a repeatable, iterable DevOps pipeline that made this transformation possible.

  • Liked Thomas Stiehm
    keyboard_arrow_down

    Thomas Stiehm - Agile Testing for Embedded Software Development

    Thomas Stiehm
    Thomas Stiehm
    CTO
    Coveros, Inc.
    schedule 2 months ago
    Sold Out!
    45 mins
    Experience Report
    Intermediate

    A large part of the success of agile adoptions is due to the automated testing approach used in agile projects. Because many of these techniques were pioneered in the development of web applications it can be hard to see how these techniques can be leveraged for a project where the software being built is for an embedded application. Discover ways to leverage agile testing techniques for embedded systems. Whether you are building a medical device, embedded controller, or Internet of Things device learn how to leverage these testing practices to create fully automated tests that fit into a DevOps build pipeline and help your team create higher quality, more reliable software. Test automation is the best way to maintain and execute a comprehensive suite of regression tests that allows you to take back control of your testing process while increasing test coverage. Learn how to be in control of your test process by stepping up your test automation to the next level.

    Embedded development and Internet of Things development is often done on platforms that lack modern software development and test automation tools. The more esoteric or the smaller the target audience, the less likely tool vendors are to create products that directly support the deployment environment. This can make getting started with test automation using older tools that are not as actively supported by vendors can be a challenge that has to be overcome by a team that wants to move toward a Continuous Deployment process.

    This session is aimed at people that are trying to adopt agile and continuous delivery with embedded technology, but might be worried that it can’t work in their particular environment due to their industry, technology stack, culture, or regulatory environment.

  • Liked Jonathan Kauffman
    keyboard_arrow_down

    Jonathan Kauffman - Leveraging Zephyr and Behave for Test Case Management

    Jonathan Kauffman
    Jonathan Kauffman
    Consultant
    Coveros, Inc.
    schedule 2 months ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    Zephyr is a tool for managing manually-executed test cases, and Behave is a Python framework for writing BDD-style automated test cases. Is it possible to leverage the benefits of both Zephyr and Behave? This presentation will describe how a bio-medical device company, as part of their Agile adoption, underwent the following evolution in their test management practice:

    • Managing test cases with monolithic Word documents.
    • Managing those same test cases in Zephyr.
    • Writing those same test cases using the Behave framework.
    • Maintaining a master copy of the test cases in Zephyr while storing the implementation of each test step in version control.

    This presentation will also discuss a tool that was used to generate Behave feature files from Zephyr test cases and how that tool was integrated into both testers' workflows and the CI/CD pipeline.

  • Liked Leland Newsom
    keyboard_arrow_down

    Leland Newsom - Comparing Scaling Frameworks - LeSS and SAFe

    Leland Newsom
    Leland Newsom
    Agile Coach
    CapTech Ventures
    schedule 4 months ago
    Sold Out!
    45 mins
    Talk
    Beginner

    Scaling Agile is easily misunderstood. Scaling is the term we often hear used to describe using Agile methods with large enterprises.  Larger enterprises often deal with bigger and more complex problems than small ones. They have more employees, subcontracting companies, different business units, more processes and a strong culture that defines how things are done. At the same time, they need to be able to deliver results in an ever-changing business environment. They need to be Agile but the bigger the company, the bigger the challenges are for scaling Agile. 

     

    Scaling frameworks available in the market today are maturing quickly and provide a variety of choices. Like the Agile Manifesto, these frameworks are based on principles, and they vary widely in the specificity of the recommended approach.

     

    In this session, we will compare how two scaling frameworks, LeSS and SAFe, address the challenges of agility at scale.  We will talk about how these two frameworks align, coordinate, and manage dependencies across multiple teams to maintain consistency and agility at scale. 

     

  • Liked Ben Pick
    keyboard_arrow_down

    Ben Pick - How to test for the new OWASP Top 10 Vulnerabilities

    10 mins
    Talk
    Beginner

    The latest version of the OWASP Top 10 Vulnerabilities is about to be finalized. This talk discusses how to use these guidelines, both old and new, to perform security testing. In too many instances, security is the last phase of the SDLC. Using the OWASP Top 10 list, developers and testers can become more aware of potential vulnerabilities. This will improve their coding and testing skills, allowing them to build more robust code.

    This presentation discusses each of the latest vulnerabilities defined in the 2017 version of the OWASP Top 10. It includes testing strategies or failure scenarios which lead to exploitation. Best practices are discussed, all within the condensed time frame of a 10 minute firetalk.

  • Liked Julee Bellomo
    keyboard_arrow_down

    Julee Bellomo - Creating a Lean PMO; Empower People, Enable Flow

    45 mins
    Talk
    Intermediate

    Stop getting good at process and start getting good at business! This session is for anyone who is seeking practical, proven techniques to create a Lean Portfolio Management Office. Participants will explore the pre-requisites for a Lean PMO, the role of the PMO in agile delivery, and a high level overview of a Portfolio Kanban System that balances the speed of agile delivery with the accountability and transparency needed by organizations for complex programs.  Along the way, you will hear details of how a dynamic product company evolved to a people-centric, flow-based PMO.

  • Liked Max Saperstone
    keyboard_arrow_down

    Max Saperstone - Importance of Testing Planning for DevOps

    10 mins
    Lightning Talk
    Beginner

    In the agile world, it's important to remember that planning and documentation is still important and serves a purpose. While agile promotes “working software over comprehensive documentation,” it doesn’t say ignore documentation as much as we’d all often like to think. When developing a working DevOps pipeline, it is important to understand what your quality gates are, what thresholds you expect, and where you expect these to be. A well documented test plan should inform the pipeline, and provide information and details as to what constitutes a working code. Too often this DevOps work is done out of sync with Testers, which then either means quality definitions don't line up, or additional re-work has to be put in to align software confidence goals.

  • Liked Max Saperstone
    keyboard_arrow_down

    Max Saperstone - Testing with a Rooted Mobile Device

    45 mins
    Talk
    Advanced

    Traditional applications are tested through the GUI and through all exposed APIs. However, typical mobile app testing is only done through the front-end GUI. In addition, performance and security details are not readily available from the mobile device. Max Saperstone demonstrates some benefits of testing a native mobile application on a rooted device—one with privileged access control. Although Max does not describe how to root a device, he shares how to access back-end processes and test at this detailed level. He discusses the technical controls made available through a rooted device—together with its auditing, logging, and monitoring—and describes the gathering of additional metrics. Max demonstrates tools for penetration testing, sniffing, and network hacking; shares how to access application data directly; and shows how data security is implemented for the application. Learn how to use the admin rights associated with a rooted device to examine device performance and to simulate interrupts and system faults.