The trends behind such things as DevOps, Agile, and Git lead us to a place where teams desire to own everything involved in their development space including code, process, and tools. Supporting this democratization can pose challenges to centralized teams who now have to support quickly evolving, "team unique", processes and needs. This presentation will use some experiences of the CarMax CM team with the ProGet tool to highlight some of the challenges we experienced and the techniques we used to make this transition.


Outline/Structure of the Presentation

  • Introduction
    • Who I am, experience, etc.
    • What the presentation is about (CM team's journey from centralized control to 'democratization')
  • Where we started
    • Centralized 'ownership' of tools
      • ClearCase
      • TeamCity
      • Homegrown delivery scripts
    • Centralized process
      • Build/Deploy development
      • Formal requests
      • Build/Deploy performance
      • User provisioning
  • Where we ended (so far)
    • Centralized tools *support*
      • GitHub
      • TeamCity
      • ProGet
      • Subversion
      • Homegrown delivery system (integrated)
    • Teams own their processes
      • Development teams own usage patterns. (CM team steps in only when tool stability is an issue).
      • Self service build, deploy, and user management.
      • CM build/deploy is more a 'value add' service (particularly to smaller teams) rather than a requirement.
    • CM team focus is stability, mentoring, and resolving issues.
  • Benefits of this new world
    • Allows the CM team to focus on keeping tools up-to-date, stable, and research best use practices.
    • Able to support more users: people are able to fish for themselves rather than being given fish. ex: Time that was spentwriting the builds is spent helping with issues/advising on best use.
    • Allows the CM team to keep out of the weeds and 'see' better solutions to issues: such as ProGet.

  • Real life illustrations

    • ProGet for managing legacy database/unix packages
      • Replacement for a home-grown system using ClearCase
        • Scripted
        • Code grouped into packages (labels)
        • Involved a number of 'states' - recorded using ClearCase attributes.
      • Source moved into GitHub
        • No feature that could replace CC attributes. (Branches are not a good way to 'store' state information)
      • ProGet Universal Packages was an elegant solution
        • Standard directory structure (in GitHub repo)
        • User creates manifest file, checks into GitHub.
        • 'Right-Click' submit of manifest file - picked up by TeamCity job.
        • TeamCity job uses Upack add-in to package and ProGet API calls to add package to 'Ready for Test' feed.
        • Packages are consumed manually or through automation (by API calls).
        • Additional feeds - 'Ready for Implementation' and 'Implementation' were provisioned with the intent that the ProGet promotion model would be used just as the ClearCase one was.
    • ProGet for managing third party packages
      • driven by Git binary issue.
      • Git LFS is a good mitigation and deals with the symptom - but maybe not the underlying root 'need'.
      • The right solution - for us - was ProGet
        • Good price and feature set.
        • Has roots in nuget/.net, and we have substantial .net development.
        • Universal packages were attractive to solve some of our unique needs.
        • This was a case where the CM team implemented the product (ProGet) based on seeing a strategic need rather than as a result of any pain.
      • Initial implementation.
        • Turned out the first benefit was entirely unforseen: enabled usage of Maven by the automation testing group.
        • Prior to using ProGet there was a lot of difficulty getting items from the community sites due to network, firewall, proxyissues. Much suffering was removed by configuring things so ProGet could get to the 'outside' then serve the packages internally.
        • This alone ended up moving ProGet into the 'mission critical' category.
        • Caching also was a big draw as it would speed up builds.
        • Beware of the dog: WebDAV and ProGet incompatability.
      • It took us a few attempts to come up with a good usage pattern.
        • The key development in this was a shift of 'responsibility' from the package producer to the package consumer: each consumer of a package is ultimately responsible for making sure it works for them. This resolves a major CM issue: the producer of a package often times doesn't know everyone who is using the package, let alone for what and so has difficulties testing all the possible scenarios.
        • After that realization, the usage model almost wrote itself:
          • Cascading feed model: Enterprise -> Team/Product
            • The enterprise feed connects to the third party community feed(s), and also contains home grown packages that can be shared across the company.
            • Team/Product feed connects to the enterprise feed, and contains home grown packages in testing or those that are not intended for general sharing.
            • Promotion from team/product feed to enterprise feed is performed by the producing team.
  • Misc
    • Experience working with Inedo/Support
    • Features we really like
      • Access token to user rather than feed
      • Promotion model
      • Storage reporting.
    • Exiting new features
      • "package consumer" API

Learning Outcome

Get a basic understanding of how centralized teams must change to support decentralized ownership of tools and process.

Target Audience

People involved in DevOPs and those that manage them

Prerequisites for Attendees

Have some basic knowledge of CI/CD

schedule Submitted 1 year ago