As we continue to experience the multiple pandemic waves, we quickly learn to adapt and sustain to new ways of living while building on the opportunities manifested by the crisis. Today, even IT systems are expected to have a high user expectation to be ‘Always-on’, make chaos a part of Business-As-Usual (BAU) while ensuring a healthy sustenance to overcome the failures.

The shift in the user expectation demands a rigorous move towards resilience engineering practices to build anti-fragile applications. Smart balancing of proactive (shift-left) and post-production (shift-right) strategies are required to ensure the application is designed and built with resiliency first, instead of considering reactive chaos testing as a quality gate ritual for production release.

To facilitate early understanding of system recovery characteristics, we need to develop a resilience culture to experiment and fail fast, early in the life cycle and implement chaos engineering activities as a part of delivery pipeline. By provisioning a robust observability platform with intelligence (AI/ML) to trace, correlate and report the failure impact on the critical business processes, the product engineering team gets the opportunity to proactively detect, diagnose, and improve the system resiliency. 

Our approach towards low-blast radius focused early chaos testing combined with ongoing failure attacks (across application, network, and infrastructure) on production environment has helped systems to recover from real time failures while reducing MTTR by more than 50%. Resiliency further improves over time as more intensive failure experiments are carried out in production as well as test environment and self- healing automation is employed.

Managing stringent SLOs to meet the ever-increasing user demand though continuous monitoring of service level indicators, meaningfully managing error budgets & TOIL automation powered by Site Reliability Engineering (SRE) principles helps in maintaining the customer trust and preserving the promise of high availability despite a complex distributed multi-cloud ecosystem.

In this tutorial session, you will learn and experience the practical challenges faced during the culture transformation journey of a Financial Fintech. Join the session to experience how marrying chaos engineering practices with SRE practices can help in improving the Business Resilience.