DevOps lead IT Transformation story of an Investment Bank
Total cost of ownership of the Investment Bank IT portfolio is way too high. Business is also keen in getting high responsiveness from IT. There is high desirability from all stakeholders in changing the way IT work. Based on a value stream analysis, key aspects that impede speed and value are identified. Common and prominent impediments identified are silo, handovers, local optimization, manual and inefficient software engineering. Principles and practices of DevOps seems to be an excellent fit for this change. This means new capabilities for IT workforce, new operating model, new way of measuring, new way of aligning with other corporate groups like security etc. This session is a journey in progress of IT transformation using DevOps as core theme. Where to begin? What to change? How to create capabilities? How to onboard teams into this wave? How to sustain? Where are we now? Where we want to reach? Lessons learnt?
Outline/Structure of the Experience Report
Context of the portfolio – current challenges, priorities, initiatives and expectations – 5 minutes
Invest bank, highly compliance driven, huge dependency on of the shelf Products and components, large silo teams, heavy dependency on SMEs, competing priorities. Goal is to bring down total cost of ownership and bring up effectiveness & efficiencies.
How it got started ‘ The poster boy’ - 5 minutes
To drive this IT transformation and bring change in the overall outlook of IT, first thing aimed at is to create a ‘poster boy’. Identified an Asset class were there is high business desirability on change, and business is keen to collaborate. Identified top 5 impediments using VSM 1) sporadic business involvement 2) Red-tape ‘paper’ work for release management 3) indefinite wait for infrastructure 4) waterfall mindset of teams – dev- test – support etc.. 5) Lack of automation in build, test and deployment- long error prone manual interventions.
Launched 6 focused interventions - this lasted for 6-8 weeks..
1) Fenced team from bureaucratic process – Agile coach started coaching and working with other downstream, upstream teams and corporate functions
2)‘Parachuted ‘ 3 experts – Agile coach, Build and deployment automation expert, rock- star programmer – to do remediation of code/ version / build etc.. later on they got embedded into the team..
3) Introduced Simple and meaningful KPIs -measures like Release frequency, Defect Removal efficiency, meantime to detect and repair
4) Established tooling orchestration around Jira – Jenkins- GIT- Sonar cube-Nexus- Ansible - splunk for a CD pipeline
5) Created a simplified release process – DevOps release process.. Reduced multiple approvals and email based manual approvals. A consolidated Jira dashboard to understand health of release candidate build.
6) Empowering Developer workbench – capability building for Developers, testers, production support folks around TDD, test automation, refactoring – hackathones for continuous improvement
From an initial baseline of 4 release per month, this team went up to 1 release per week in 8 weeks.. ( as we speak, today they are continuously improving to meet daily release target) Conducted DevOps roadshows, posters, and newsletters along with DevOps sessions- enough PR and hype got created. By this time many other asset classes want to get into this new way of working
How to scale it up – In progress – 10 minutes
Based on the initial success, having enough hype around DevOps and special status to the ‘Poster boy’ asset class, there was enough pull.. The key is on how to scale it, & sustain so that it can impact the whole portfolio. For this we attempted following
- Leadership training and workshop on Agile and DevOps. Industry expert interaction and other case study discussion – formulated on what is banks long term view on DevOps. Strong leadership statement and message around DevOps across location
- Wave 1 Transformation candidates: Identified top 25% apps/ asset that are expensive and there is predicted double digit growth in the coming financial year
- Based on the ‘poster boy’ experience created a DevOps maturity assessment, through which identified assets can be assessed to find their current state and find key initiatives and interventions required by them for transformation
- Conducted assessment for 50 assets - great insights. Find out common pattern across assets. Classified assets into 3 different groups. Group 1 – needs lot of hand holding and spoon feeding. Group 2- fundamentals are in place, need coaching, directions and technical support. Group 3- self-starters, motivators, gentle push and removal of obstacle does the magic..
- Assessment provided clear need of key capabilities and interventions. Came up with 5X3 strategy. 5 interventions and 3 capabilities
- Build automation
- Test automation
- Deployment automation
- Agile coaching
- Tech Debt reduction
- Tools Engineering
- DevOps QA
- Change management
- Created asset specific business case for what investment is required for these interventions- presented to management and CIO asking for funding for the change initiatives
- Created a central DevOps group to build and facilitate people – practice – tools capability. Following are the team created
- Tools engineering service team: to provide required tooling infrastructure.
- DevOps cook-book team: to provide necessary practice, coaching, measures and change support.
- DevOps QA team: to build test automation capabilities and mindset.
- Production support optimization tem : To analyses current service requests, problems, incidents, and improve monitoring capabilities
- Eat your own dog food: The whole transformation program is run in agile fashion – Jira is used for creating backlogs related to transformation, specific interventions. It’s a Kanban system with clearly defined steps for each interventions, and WIP limits. Prioritization and grooming is done along with Asset class leads.
- Intervention across Plan, Build and Run. Agile & DevOps community of practice to drive culture of sharing and learning. Tools engineering services to bring effectiveness in practices across Plan, Build and Run. DevOps CookBook as guide to transformation - Onboarding, Transformation, Sustain
- To sustain the momentum created DevOps community of practice, have got 30 volunteers across groups. Meets once an in 2 weeks, learn – share platforms. Got a confluence site for collaboration. Beer & Build program once a month for demonstrating to larger group.
- To govern this transformation program – lean based governance approach is created with the participation of every asset class head
- Asset specific focus groups to drive above interventions and capabilities – want to have indigenous teams
- Launched 5 killer KPIs – Build Cycle time, Defect removal efficiency, Release frequency, Meantime to detect and remove incidents
- A ‘Tiger’ is formed to interact, align and bring change in corporate level teams like Information security, Infrastructure, central quality team, suppliers and vendors
- Tiger team is getting the dependent team to involve in big room planning for each asset teams
- CTO and Enterprise Arch team involvement: - Key bottle neck identified along with this journey is architectural and tech debt impediment. Many application assets are not service oriented. Frequent release is a dream. A future state architecture program is getting launched, focusing on enabling micro service based architecture and related principles.
- Plan for central IT4IT team to create different orchestration layer to support Plan, Build and Run phases
- From a change management perspective – location strategy has kicked in.. have right team combination and right people at right place
Q& A – 5 minutes
Since this is in progress transformation journey, I am sure I will have more interesting lessons to share. As we are taking this journey seeing lot of wow and not so good stuffs..
This session is a journey in progress of IT transformation using DevOps as core theme. Where to begin? What to change? How to create capabilities? How to onboard teams into this wave? How to sustain? Where are we now? Where we want to reach? Lessons learnt?
CIOs, Directors, Architects, DevOps Change agents, Transformation Champions
schedule Submitted 2 years ago
People who liked this proposal, also liked:
Rajith Raveendranath - From Monolith to Micro Service - A Kick StartRajith RaveendranathVPSunTec
schedule 3 years agoSold Out!
[Update based on the panel review]
Inspired by Martin Fowler's introduction to micro services (https://www.youtube.com/watch?v=wgdBVIX9ifA);
The demo will introduce the "ABC"s of the transition to micro services. We will refer to the Hadoop Distributed File System open source and demonstrate to (re)design the NameNode module as a micro service. This will introduce the three primary challenges and their possible solutions as in;
1. Componentize using Services instead of the conventional componentization using the design
2. Data segregation using an event driven framework to separate the centralized data across services
3. Accessing a micro service as a web service instead of an orchestrated access; the delegate & facade patterns will be demoed to loosely couple the interfaces
The demo will conclude with a listing of the next steps in transition, which need to be considered after the primary challenges are addressed.