Estimating how much a project would cost and how long it would take has always been a challenge.  These are critical business questions, and not answering them is not an option.  Estimating large-scale projects is even more difficult and complicated not only because of their large scale and distributed nature, but also due to faulty estimation methods widely used today. Large-scale agile projects consist of several teams (organized into programs and portfolios).  Teams are often distributed. If you are doing story point estimations and generating reports for large-scale agile projects in blissful ignorance of the fact that story point scales used by different teams may not be the same, you will make wrong decisions caused by wrong estimates and metrics.  

All Agile Lifecycle Management tools expect and assume that the story points entered by you in the tool are "normalized" across teams, i.e., they follow the same scale.   Story points entered into the tool without normalization (garbage-in) will generate meaningless reports and metrics (garbage-out).  

You may also be hard pressed to estimate portfolios and programs when their stories are not even defined.  This is like estimating something that is unknown!

I will present solutions to these and other estimation challenges for large-scale agile projects.

I will present the Calibrated Normalization Method (CNM) for scalable estimation, which I have developed and applied in my client engagements since 2010.   CNM promotes local, decentralized, and autonomous decision making at the team level by allowing teams to use their own story point scales, and normalizing team story points with a novel technique.   I will also contrast and compare CNM with centralized methods and the SAFe method for estimation. 

I will demonstrate the use of a normalization calculator for doing normalization math needed for bottom-up as well as top-down estimations in large-scale projects.  This calculator has been developed and refined with actual usage; it makes story point normalization calculations very quick and easy, and avoids human errors.  


Outline/Structure of the Talk

I will first present the challenges of developing estimation and metrics for large-scale projects using a concrete example of a large-scale projects with 2 programs and a total of 8 agile teams.

I will explain the need for story point normalization to be able to properly estimate large-scale agile projects.

I will then illustrate centralized, semi-distributed and fully-distributed estimation methods using the concrete example of the large-scale project of 2 programs and 8 agile teams.

I will finally demonstrate the use of a simple normalization calculator.

Learning Outcome

Understand the trade-offs and advantages of different large-scale estimation methods: centralized, semi-distributed, and fully distributed methods, and be able to choose an estimation method appropriate to your situation.    

Understand how to use CNM for both bottom-up (from teams to programs up to portfolios) and top-down (from portfolio down to programs and down to teams) estimation for both fix time/flex scope and fix scope/flex time agile projects.   

Experience scalable estimation methods through an example of a large-scale project with 2 programs and 8 agile teams.

Understand the use of story point normalization calculator.  

This normalization calculator will be provided to all attendees of the session.

Target Audience

ScrumMasters, Project Managers, Program Managers, Product Managers, Portfolio Managers, PMOs, business managers, Team Leads


schedule Submitted 6 years ago

Public Feedback

    • Matt Badgley

      Matt Badgley - Yes, Words Really Do Mean Things - Establishing a Shared Language

      60 Mins

      During this conference, within the books we read, in our day-to-day lives -- we use words as a means to negotiate, interact, express, and do. Words, whether written or spoken can play differently based on the people that exchange them. In the world we are living today, words are bantered so freely that they cause a war or unite a community or save a marriage or demoralize a team.

      As we see today, the concepts of agile are permeating the enterprise and scaling out from the team to the program management office to the executing chambers. Words are often mis-used, mis-understood, and lead to bad behaviors.

      In this session, we'll discuss the general challenges of communications and the overwhelming vocalbulary that we have embedded in our craniums.  We'll explore words -- in particular, the words we use everyday around software development. We look at how some of the basic words we use like Velocity, Sprint, and Team have clear meanings and plenty of baggage. 

      To help solidify the learning of this workshop, we'll use a couple brainstorming games -- so come prepared to get engaged. We'll wrap-up by using our collective experiences to either find better ways to explain our words or establish brand new ones. Our ultimate goal is to establish a way an organization can establish an ubiquitous language around the work they do and ultimately improve communication which will lead to better agile transformations and hopefully better solutions.

    • Itamar Goldminz

      Itamar Goldminz - Lean Scaling: From Lean Startup to Lean Enterprise

      60 Mins

      Congratulations! You've found the right product-market fit, and it's now time to scale your business. But growing your organization often means slower decision making, increased complexity, and higher chance for misalignments. How can you grow your business while staying lean? Learn five key lessons on how to use smart tooling and process to address these complicated growth challenges.

    • Dave Chesebrough

      Dave Chesebrough - Considerations for Agile Adoption at the Team, Project, and Organizational Levels

      60 Mins

      Change is hard. For any organization, team, or individual, the ability to change is difficult even when the desire for the change exists. Some studies have revealed that even when people know they need to change, even at the risk of their lives, it is still difficult to adopt new practices and behaviors.  Knowing this, what are organizations and project teams doing to make agile adoption easier and how are they supporting the teams and the individual new to this way of developing software products and systems?

      Through a roundtable discussion with representatives from industry and government, we will share with you our experiences with Agile on Federal government projects and programs, the challenges we faced, lessons learned, and different activities we performed as we went through an agile transition. The intent is that our experiences will provide you with ideas that you can take back to your organization and teams to support your agile journey.

      The panelists will share their experiences in bringing agile to their own organizations as well to their government clients.  Topics to be addressed include:

      • What makes adoption easier?
      • Challenges faced and tactics to overcome them.
      • Lessons learned from a broad spectrum of successful, and unsuccessful, adoptions of agile methods in acquisition.


      Dave Chesebrough, President, Association for Enterprise Information


      Dr. Suzette Johnson, PMP, CSP, CSC, Certified (Agile) Scrum Coach, NGIS Technical Fellow and Chair of the Northrop Grumman Agile CoP.  Suzzette leads development of agile practices across programs serving government customers, including DoD and Federal Health IT. 

      Robin Yeman, Agile Transition Lead / SME, at Lockheed Martin where she defines Agile Strategy across capability areas at IS&GS; identifies and implements metrics to ensure results of strategy and enable course correction; develops Agile SMEs to support strategic consulting for program start-up, transition for waterfall, release planning, and execution; teaches and educates all levels at LM to allow LMCO to better meet customer needs; certifies large teams in the Scaled Agile Framework; and provides support in developing Performance Measurement Baseline and Agile EVM.

      Jerome (Jerry) Frese, Program Management Analyst at the Internal Revenue Service, is the organizer of an Inter-Agency Seminar whose purpose is to bring federal SDLC practitioners together so they can establish a network, learn about and share best practices and collaborate on new and innovative ways to support projects. Through the series of nine seminars he has worked with 33 other Government agencies fostering the implementation of agile in Federal IT. In his own agency, he brings 40 years of software development experience to his job the Senior Methodologist at the IRS.    

      James Barclay, Senior Systems Engineer, NGA Architecture & Engineering Group National Geospatial-Intelligence Agency.

    • Tom Friend

      Tom Friend - Agile Methods Embedded in the United States Military War fighting Methods.

      60 Mins

      Agile DC 2014 Track Government

       Agile Methods Embedded in the United States Military War fighting Methods.

       a) Title –

       Agile Methods Embedded in the United States Military War fighting Methods. How OODA & MDMP War Fighting & Maneuver Warfare Stacks up Against Agile Software Development. Reflections of a Crew Dog / Scrum Master

       b) Summary –

      Agile = Military Decision Making Process

      SCRUM = OODA loop Observe Orient Decide Act

      Military Maneuver war theory = Lean principles

       c) Description

       This lecture walks the participants through the crossover points of AGILE SCRUM to Observe Orient Decide Act (OODA), the Military Decision Making Process (MDMP), and the lean principles of Maneuver warfare.

       The lecture provides the Agile practitioner, engaged in Federal DOD Agile organizational transformation, tools and touch points that will resonate with military decision makers. These tools and narratives are bridges to build trust and dialog. They are concrete starting points to engage in relevant conversations that lead to constructive outcomes.

      The application of the content in this lecture is for a focused audience. However; the message is a fantastic way to show how the Agile Scrum processes are used in other areas. For the non Federal Agilest the outline of OODA and MDMP will be quite novel. For example the history of the OODA loop that formed during the birth of dog fighting in the Jet Age was the beginning of iterative refinement that led to what we know as SCRUM today.


      Boyd’s OODA Loop Applied Relates human behavior

      Goal: Successful interaction with other loops

      Objective: Get inside the opposing OODA Loop

      Outcome: Destructive: Air Combat, Warfare

      Outcome: Constructive: Agile Software Engineering Process

       When you’re doing OODA “loops” right, accuracy and speed improve together; they don’t trade off. A primary function of Agile “loops” is to build an organization that gets better and better at things.

       Additionally this lecture shows numerous crossover examples of MDMP and Agile in general along with an overview of how Maneuver warfare is an adaptation of Lean principles.

       The end goal is to how that Scrum, Agile, and Lean maps to Military methods. The focus of these process is to quickly develop a flexible, tactically sound, and fully integrated synchronized plans that increases the likelihood of mission success. This is the same within IT development.

       d) Learning Objectives

       Learning Objective - Provide Federal DOD Agilests ways communicate to Military decision makers that Agile Scum is OODA MDMP only by different terms. It is nothing new just being applied differently using a new vocabulary of terms.

       Outcome - Present the similarities of Agile Scrum vs traditional proven Military Decision Making Processes.

       Outcome - Provide bridge of understanding between AGILE SCRUM and OODA & MDMP for Military and DOD contractors that are unfamiliar with the Agile methodologies.

       Outcome - Present talk tracks and narratives that demonstrate how the Agile Methodology complements MDMP.

       e. Target Audience

       The primary level of audience understanding and comprehension is Level 3. Performing – Target audience is experienced Scrum/agile practitioners (2+ years)

       This is a very focused / specialized session for those that can apply the lessons. However it is a very cool session for those that just want to sit in and see how Scrum is applied in Aerial Combat dogfights and Agile in the broader war fighting process.

      For those in the Federal DOD game the take always are to provide several narratives to leverage for agile transformations within the Federal and DOD space targeting DOD Military decision makers in order to break down transformation barriers and perceived risk.

       f) Information for Review Team – Link to Presentation:!1092&app=WordPdf

       g) Presentation History –

       This presentation was given at the Agile in Government Summit in Washington DC 2014. It has been well received within the Federal and DOD space.

      It is new thought capital and slide ware that has not been presented to a general agile audience.

       About the presenter: This lecture is presented by LtCol Tom Friend USAF Retired. A US Military Combat veteran, Pilot, Squadron Commander that has operational experience in the Navy, Air Force, and served on the ground with the Army and Marines as Forward Air Controller. He is a distinguished graduate from Air War College and has a BS in Aeronautics. On the federal side he is a graduate of The Army Logistic Management College in federal contracting. He has served as a Federal Acquisition Program manager and Acceptance test pilot at a US Military aircraft manufacturing facility. He additionally has 20+ years as a project manager and 10+ years of Agile XP SCRUM software development experience within various IT markets.

    • Cindy Shelton

      Cindy Shelton - Retrospective: An Agile Failure in Government Application of Agile

      60 Mins


      Unfortunately, much too often everyday practice deviates undesirably from ‘‘best practice’’ or what is considered optimal.  While we don't like to admit it, there ARE failures and challenges in applying the Agile philosophy to the US Government and other burecratic organizations.  This working session uses the Agile practice of a Retrospective with the attendees as the team to explore those challenges and actions to take in the next "iteration."