W. Edwards Deming noted that “people with targets and jobs dependent upon meeting them will probably meet the targets – even if they have to destroy the enterprise to do it.” While metrics can be a great tool for evaluating performance and software quality, becoming beholden to reaching metrics goals, especially the wrong ones, can be detrimental to the project. Each team needs to take care and understand what targets are appropriate for their project. They also need to consider the current and desired states of the source code and product and the capabilities and constraints of the team.

As one of the lead architects working with a huge codebase on a government project, I often have the opportunity to influence the teams around me into watching or ignoring various metrics. I will walk through some measures that are available to most projects and discuss what they really mean, various misconceptions about their meaning, the tools that can be used to collect them, and how you can use them to help your team. I’ll discuss experiences and lessons learned (often the hard way) about using the wrong metrics and the damage they can do.

 
 

Outline/Structure of the Talk

Rough outline

  • What makes a good metric?
  • Code complexity
  • Unit testing
  • Code coverage and mutation testing
  • Technical debt
  • Escaped defects
  • Retrospectives
  • Wrap-up

Learning Outcome

  • Metrics should be SMART- Specific, Measurable, Achievable, Relevant, Timely.
  • Set quality gates. If you can't commit to a metric, don't bother tracking it.
  • Except for 0 failing tests, trends are usually more important than absolute values.
  • Code coverage doesn't tell the whole story. Mutation testing is better.
  • Number of escaped defects is the most important metric to measure, no matter what your definition of done is.

Target Audience

This session is aimed at development leads and others that are trying to choose the right metrics to measure or trying to influence what metrics to avoid.

Prerequisites for Attendees

Attendees should have some experiences of their own with development techniques so they can see how my guidance applies to their situations. Knowledge of metrics or static analysis tools is not a prerequisite.

Slides


Video


schedule Submitted 5 years ago

  • Gene Gotimer
    keyboard_arrow_down

    Gene Gotimer - Experiences Bringing Continuous Delivery to a DoD Project

    Gene Gotimer
    Gene Gotimer
    DevSecOps Engineer
    Steampunk
    schedule 5 years ago
    Sold Out!
    45 Mins
    Experience Report
    Beginner

    Not every continuous delivery initiative starts with someone saying "drop everything. Let's do DevOps." Sometimes you have grow your practice incrementally. And sometimes, you don’t set out to grow a practice at all-- you are just fixing problems with your process, trying to make things better.

    I'll walk through a case study of how our team worked on an exemplar project for the Department of Defense to show that agile could work in a decidedly waterfall culture. I’ll also discuss techniques and tools we used to bring a DevOps mindset and continuous delivery practices into an environment that wasn't already Agile.

    I'll talk about how we were able to start in development, where we had the most control, with a "let's starting being Agile" initiative and working on "why is continuous integration important?" From there, we tackled one problem after another, each time making the release a little easier and a little less risky. We incrementally brought our practices through other environments until the project was confidently delivering working, QA-tested, security-tested releases that were ready for production every two weeks. I’ll discuss the journey we took and the tools we used to get to build quality into our product, our releases, and our release process.

  • Gene Gotimer
    keyboard_arrow_down

    Gene Gotimer - Tests Your Pipeline Might be Missing

    Gene Gotimer
    Gene Gotimer
    DevSecOps Engineer
    Steampunk
    schedule 5 years ago
    Sold Out!
    10 Mins
    Talk
    Beginner

    Developing a delivery pipeline means more than just adding automated deploys to the development cycle. To be successful, tests of all types must be incorporated throughout the process in order to be sure that problems aren’t slipping through. Most pipelines include unit tests, functional tests, and acceptance tests, but those aren’t always enough. I’ll present some types of testing you might not have considered, or at least might not have considered the importance of. Some types will address code quality, others code security, and some the health and security of the pipeline itself.

    I’ll talk about specific tools we used to supplement our pipeline testing. I won’t get into how to use each tool-- this is more of a series of teasers to encourage people to look into the tools, and even letting them know what types of tools and testing opportunities are out there.

  • Max Saperstone
    keyboard_arrow_down

    Max Saperstone - Exploring Automation Strategies and Frameworks What Should Your Team Be Using?

    45 Mins
    Talk
    Beginner

    Agile practices have done a magnificent job of speeding up the software development process. Unfortunately, simply applying agile practices to testing isn't enough to keep testers at the same pace. Test automation is necessary to support agile delivery. Max Saperstone explores popular test automation frameworks and shares the benefits of applying these frameworks, their implementation strategies, and best usage practices. Focusing on the pros and cons of each type of framework, Max discusses data-driven, keyword-driven, and action-driven approaches. Find out which framework and automation strategy are most beneficial for specific situations.

    Other than using specific frameworks as examples, the presentation is framework agnostic, really focusing on capabilities of different types of framework, and how those might or might-not fit your company's/software's needs. In the talk, 5 different types of frameworks will be discussed: Record/Playback (Linear), Modular/Structured, Data Driven, Keyword Driven, Action Based, and of course the combination of these as Hybrids.

    At the end of the talk, depending on questions, and how much time we have left, a slide with a long list of specific testing frameworks is displayed, and an open discussion of who has used what, and what has/hasn't worked for them is had.

    If you are new to test automation or trying to optimize your current automation strategy, this session is for you.

     

  • Phillip Manketo
    keyboard_arrow_down

    Phillip Manketo / Dave McMunn - Building Strong Foundations…. Underwriting Fannie Mae’s Agile Transformation

    45 Mins
    Experience Report
    Advanced

    Over the course of the last two and one-half years, Fannie Mae has worked aggressively to transform itself from a heavily silo’d and firmly entrenched command and control culture, following a gated workflow, with long release cycles, to an Agile organization.  Today, Fannie Mae is a more dynamic value oriented organization that is responsive to stakeholders, focused on achieving greater efficiency by enabling fast-feedback loops, as well as using empirical data to optimize mature and persistent agile values and practices.  

     

    Within the larger context of the transformation to enterprise agility, this Experience Report will focus on the case for change, Fannie Mae’s journey and the corresponding challenges, benefits and key learnings realized.  Our conclusion, while it is important to build bridges with business stakeholders, mature agile teams, leverage automation and embrace the values and principles of the agile manifesto… a successful and longstanding transformation is dependent upon the unrelenting focus on changing the ecosystem supporting the organization’s change at the outset.

  • Gene Gotimer
    keyboard_arrow_down

    Gene Gotimer / Ryan Kenney - Creative Solutions to Already Solved Problems

    10 Mins
    Experience Report
    Beginner

    Almost everyone has to deal with bad legacy code at some point. Not just legacy code that you inherited and obviously would have been better if you had written it, but legacy code so ugly and ill-conceived that it makes you want to hunt down the person responsible just so you can scream at them (or worse). And then replace it with a one-line library function that does the same thing.

    We'll show some examples of the worst code I've seen, and we'll have a chuckle or a groan. The names, projects, and check-in comments have been changed to protect the guilty, but, unfortunately, these examples are all too real.

  • Jonathan Kauffman
    keyboard_arrow_down

    Jonathan Kauffman - Current State of BDD Testing Tools

    Jonathan Kauffman
    Jonathan Kauffman
    Consultant
    Coveros, Inc.
    schedule 5 years ago
    Sold Out!
    10 Mins
    Talk
    Beginner

    Have you heard about BDD and want to start using it, but don't know what BDD is and which tool you should use? In this presentation I address both of those concerns -- I start by providing an overview of BDD and then compare five tools that can be used for BDD testing. I conclude by discussing the pros/cons and popularity of these tools so that you can make an informed decision as to which tool would work best within your organization.

  • Glenn Buckholz
    keyboard_arrow_down

    Glenn Buckholz - Improving Your Testing Methodology Using Docker

    Glenn Buckholz
    Glenn Buckholz
    Technical Manager
    Coveros
    schedule 5 years ago
    Sold Out!
    45 Mins
    Tutorial
    Beginner

    Wonder how you can make your testing more efficient? Join Glenn Buckholz as he explores Docker, a technology that allows rapid development and deployment via containers. First, he explains exactly what composes a container, and discusses the differences between a container and an image. Once this is clear, Glenn demonstrates how Docker solves the problem of what he calls the state capture problem. When a test case produces a failure, the developer and testers often expend significant effort reproducing the issue so the developer can see the issue and fix it. Glenn demonstrates how Docker enables succinct, accurate, and quick communication between testers and developers, helping mitigate the state capture problem. In addition, testers can use Docker to load data, efficiently insert testing tools into a running system, set system state, and aid in test reproducibility. After you look at the inner workings of Docker and run through a few practical examples, you’ll find that Docker will hold an important place in your testing toolbox.

  • Rahul Sharma
    keyboard_arrow_down

    Rahul Sharma - Building and Testing Secure Mobile Applications

    Rahul Sharma
    Rahul Sharma
    IT Consultant
    Coveros
    schedule 5 years ago
    Sold Out!
    45 Mins
    Tutorial
    Intermediate

    Mobile application development has been on the rise lately because of the convenience mobile apps have to offer. Despite the recent occurrence of security breaches on mobile devices, security testing is not as emphasized as other forms of testing such as user acceptance or functional testing. An application can consist of the greatest features but will be considered unusable if hackers can exploit it. The exponential rise in the use of mobile applications for different purposes puts mobile devices in significant danger of being hacked or compromised. In today’s world, mobile applications are used for various purposes and store Personally Identifiable Information (PII) and financial information. Due to the sensitivity of customer data, mobile applications should be built and tested with security in mind. Strategies that cover how to properly test mobile apps for security issues will be discussed.

  • Marco Corona
    Marco Corona
    Consultant
    Coveros
    schedule 5 years ago
    Sold Out!
    45 Mins
    Experience Report
    Beginner

    Housing and Urban Development (HUD), a federal agency committed to creating affordable homes for all Americans, has a history of systems development steeped in waterfall practices, a history of failed IT programs, and a culture that ran in direct opposition to Agile/DevOps. It often took weeks to provision a virtual machine and years for an application to get into Production.

    In a little over a year, a small team of DevOps engineers has helped modernize the agency’s legacy infrastructure in an effort to prove Agile and DevOps can work across the organization. I will present a case study that discusses how we were able to bring 10 new applications into Production in a few months time using the Cloud and DevOps. I will discuss the challenges we encountered along the way and walk through how we were able to create a culture of shared code, infrastructure and shared purpose across multiple programs and contractor teams. In addition, I will explain how to leverage Jenkins, Chef and Azure to create a repeatable, iterable DevOps pipeline that made this transformation possible.

  • Thomas Stiehm
    Thomas Stiehm
    CTO
    Coveros, Inc.
    schedule 5 years ago
    Sold Out!
    45 Mins
    Experience Report
    Intermediate

    A large part of the success of agile adoptions is due to the automated testing approach used in agile projects. Because many of these techniques were pioneered in the development of web applications it can be hard to see how these techniques can be leveraged for a project where the software being built is for an embedded application. Discover ways to leverage agile testing techniques for embedded systems. Whether you are building a medical device, embedded controller, or Internet of Things device learn how to leverage these testing practices to create fully automated tests that fit into a DevOps build pipeline and help your team create higher quality, more reliable software. Test automation is the best way to maintain and execute a comprehensive suite of regression tests that allows you to take back control of your testing process while increasing test coverage. Learn how to be in control of your test process by stepping up your test automation to the next level.

    Embedded development and Internet of Things development is often done on platforms that lack modern software development and test automation tools. The more esoteric or the smaller the target audience, the less likely tool vendors are to create products that directly support the deployment environment. This can make getting started with test automation using older tools that are not as actively supported by vendors can be a challenge that has to be overcome by a team that wants to move toward a Continuous Deployment process.

    This session is aimed at people that are trying to adopt agile and continuous delivery with embedded technology, but might be worried that it can’t work in their particular environment due to their industry, technology stack, culture, or regulatory environment.

  • Jonathan Kauffman
    keyboard_arrow_down

    Jonathan Kauffman - Leveraging Zephyr and Behave for Test Case Management

    Jonathan Kauffman
    Jonathan Kauffman
    Consultant
    Coveros, Inc.
    schedule 5 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Zephyr is a tool for managing manually-executed test cases, and Behave is a Python framework for writing BDD-style automated test cases. Is it possible to leverage the benefits of both Zephyr and Behave? This presentation will describe how a bio-medical device company, as part of their Agile adoption, underwent the following evolution in their test management practice:

    • Managing test cases with monolithic Word documents.
    • Managing those same test cases in Zephyr.
    • Writing those same test cases using the Behave framework.
    • Maintaining a master copy of the test cases in Zephyr while storing the implementation of each test step in version control.

    This presentation will also discuss a tool that was used to generate Behave feature files from Zephyr test cases and how that tool was integrated into both testers' workflows and the CI/CD pipeline.

  • Ben Morris
    keyboard_arrow_down

    Ben Morris - The 12 Factor App, a primer on the 'manifesto' for DevOps & cloud-native apps

    Ben Morris
    Ben Morris
    Consultant
    STSI
    schedule 5 years ago
    Sold Out!
    10 Mins
    Talk
    Beginner

    If you haven't heard of The 12 Factor App, you probably will soon. Think of it as "the agile manifesto for DevOps." This talk helps you quickly become familiar with the basics of the 12 Factors that make applications cloud ready or "cloud native."

    This talk allows you to trade 10 minutes of your time in order to get a bit smarter. Learn *just* enough to be dangerous, and use that knowledge to impress developers by spewing buzzwords like persistence, disposability, statelessness, and port binding. At least be able to push back intelligently when someone is telling you the app can't be put on the cloud. Learn what is meant by "livestock, not pets" and where to find out more if the talk sparks your imagination.

  • Jason Cusack
    keyboard_arrow_down

    Jason Cusack - Facebook Agile: From Green to Greener, plus other lies and bull$#!t

    Jason Cusack
    Jason Cusack
    Agile Coach
    White Shark Consulting
    schedule 5 years ago
    Sold Out!
    45 Mins
    Talk
    Executive

    All companies tell lies. All executives endorse lies. And all employees live lies.

    We must pretend we are amazing, and brag to our peers about how much farther ahead we are, than they are. We change the narrative to suit our lies. We have reports to prove our lies.

    This happens at EVERY company.

    When you look at Facebook feeds – all you see is pictures of vacations. Smiling kids. BBQs. Clean pools. You never see the miserable couple, the dirty laundry, or the crying kids.

    Enter Facebook Agile.

    We are conditioned to hide the bad $#!t that is going on around us. There’s no reason anyone has to know what’s really going on (even though most people know what’s going on).

    Our leaders are afraid of what other peers might think. Or worse, what other “senior management” might perceive or judge. So they choose not to be leaders, despite the obvious need for leadership.

    This session will provide an experience report on what leaders can do to embrace transparency, trust, and courage. Building a culture of continuous improvement starts with embracing the things that aren't working, instead of hiding them to avoid overhead.

  • Ben Pick
    keyboard_arrow_down

    Ben Pick - How to test for the new OWASP Top 10 Vulnerabilities

    10 Mins
    Talk
    Beginner

    The latest version of the OWASP Top 10 Vulnerabilities is about to be finalized. This talk discusses how to use these guidelines, both old and new, to perform security testing. In too many instances, security is the last phase of the SDLC. Using the OWASP Top 10 list, developers and testers can become more aware of potential vulnerabilities. This will improve their coding and testing skills, allowing them to build more robust code.

    This presentation discusses each of the latest vulnerabilities defined in the 2017 version of the OWASP Top 10. It includes testing strategies or failure scenarios which lead to exploitation. Best practices are discussed, all within the condensed time frame of a 10 minute firetalk.

  • 45 Mins
    Talk
    Intermediate

    I have interviewed 14 of the 17 Agile Manifesto authors for a special podcast project. Originally the intent was to capture the intent of authors and to chronicle the manifesto story. What emerged was much more. The story of why the event was needed, what the vision was, and what we have ruined in agile we all artifacts of the project. One beautiful outcome was the emergence of 3 themes in all 14 conversations. This talk covers the real story behind the rise and fall of Business Agile in industry and what we can do to reclaim it.

  • Manjit Singh
    keyboard_arrow_down

    Manjit Singh - What Effective Agile Contracts Look Like

    45 Mins
    Talk
    Intermediate

    The Agile Manifesto has been around for over 16 years. That seems like enough time for organizations to adapt to Agile processes and get the hang of writing Agile contracts. Yet, when it comes time for US Federal Agencies to enter into a contract about Agile work processes and deliverables, we're still seeing Waterfall language persist.

    If we want to see Agile software development contracts that are truly aligned for the best interests of all parties involved, there are a few steps that we need to take. Learn what these steps are in this presentation.

  • Alden Hinds
    keyboard_arrow_down

    Alden Hinds / Jerry Frese - Agile Transformation at the IRS: What Ken Schwaber didn’t teach us

    45 Mins
    Case Study
    Intermediate

    In the 1040 environment of strict compliance, frequent audits and heavy penalties, the IRS has been pursuing the adoption of agile practices in order to realize faster project delivery that also result in better quality products. We now have bright spots throughout the enterprise, but getting here was a journey in which we had to adapt the traditional teachings of Scrum to the IRS culture. In this presentation, we will discuss the agile transformation journey of the IRS and provide the audience with a transformation framework that accounts for our lessons learned in a bureaucratic organization. The IRS is by no means the pinnacle of Scrum, we have a lot to learn. We hope that by sharing our challenges, other agencies undergoing transformation efforts will be able to avoid our missteps and learn from our actions.

  • David Fogel
    keyboard_arrow_down

    David Fogel / Yogita dhond - TAS'mania! Successes (and failures) at TSA Agile Services

    45 Mins
    Case Study
    Intermediate

    TSA Agile Services (TAS) began on January 4th, 2017 after extensive collaboration with contracting officials, various bids, and the Agile community. Come hear the aspects that have enabled successful delivery within a government agency. TAS encompasses over 65 applications supported by a team of 80+ people. Also - because transparency is a large key to our success - we will cover the failures we have experienced and the struggles we are still working on. Lastly - it is important for contractors and government to work closely together - this presentation will be co-presented by two Agile Coaches: one is a federal employee and the other a contractor.

  • Valerio Zanini
    keyboard_arrow_down

    Valerio Zanini - Spark team building without ice-breakers... and boost creativity

    0 Mins
    Talk
    Intermediate

    When new teams come together or when managers facilitate meetings of all sorts, they often use icebreakers to create a safe comfort zone among participants and lower the barriers to share ideas. Sometimes these sessions occur with a diverse group of people, who have never worked together, and often have not even met each other before. The challenge is always to create a sense of team, to get people comfortable working together, and to minimize any impediment to sharing ideas freely.
    This session presents a set of exercises that bring together many ideas from all the team building experiments I conducted over the years, in a simple format. These exercises engage the right side of the brain, and mix visual, verbal, and kinesthetic experiences, so they works for various types of learning styles. The exercises push participants to use drawing - any kind of drawing - to share ideas with team members. The drawing process can be unnerving, and will undoubtedly push many beyond their comfort zone. This process in itself creates a bond amongst the participants, as they all confront a common challenge. The goal is sharing, not quality. The exercises become just a fun way to create an opportunity for social bonding among team members.
    These exercises can be used for teams of any size, are easy to setup, and deliver positive energy since their first use. Enjoy your Spark creativity!

  • Todd Hager
    keyboard_arrow_down

    Todd Hager / Dave Fullerton / Eric Protzman / Laird Williams - Making Agile Work: Avoiding Agile Landmines

    45 Mins
    Talk
    Intermediate

    With the continuous reduction of barriers to entry for adopting agile methodologies within federal agencies, organizations are facing bigger challenges of tailoring agile approaches and making them work for their specific needs and business practices. There is no one-size-fits-all agile dictate, rather, it is a framework that allows for a self-organizing refinement of practices and team-norms based on constant feedback and retrospectives. Given the flexible nature of agile practices, it can be challenging to identify potential hazards and anti-patterns that can derail even those teams that have a solid buy-in to adoption.

    Over time, organizations will try to pull agile teams away from the manifesto. As agencies adopt agile at various levels, in fits and starts, with or without top down support, it just happens. Examples that will be discussed in the presentation include things like the following:

    1 - Applying mature teams' norms to new teams violates the principle of the self organizing teams. Let Teams ALWAYS Self Organize no matter how mature other proximal teams may be.

    2 - Requiring status or metrics that are hard to gather or can't be automated can steal time from producing software or services. Hunger for arbitrary status can lead to unintended "documentation".

    We’ve seen and managed these types of pitfalls in long term agile projects with experience that comes from adopting agile in cross-cutting organizations that often have competing interests. Certain behaviors and practices may, on the surface, appear to be the right things to do for an organization’s specific needs, but have shown in the larger sense to be impediments to realizing the longer term productivity and responsiveness benefits that agile can bring.

help