Continuous Testing in Real DevOps World

Rakuten Membership team has realized continuous testing in real DevOps world through implementing Shift-Left/Right test in our CI/CD pipeline in order to keep ZERO incident and high quality for our systems.

It is extremely difficult to achieve both agility and quality, especially critical legacy system. But we have achieved Zero Production defect in 2019.

In this talk, we’d like to share our methodologies based on Behavior Driven Development (BDD) automation test and Monitoring Driven Release (MDR), which can guarantee test coverage in all environments. We have implemented 100% UI/API automation test with Selenium and Cucumber, and have 300,000 test cases in our daily regression test which has been triggered every night and has 100% success ratio. MDR is to perform the release with limited user traffic on production environment and increase the traffic to 100% gradually based on monitoring feedback.


Outline/Structure of the Talk

  • Rakuten Membership Scalable Pipeline Introduction
  • Rakuten Membership Systems Quality and Defects Analysis
  • Continues Shift-Left/Right Testing in the real DevOps world
  • Shift-Left Testing details
    • Page/API Object Model Based Core Automation Test Framework Introduction
    • Self-Service Automation Test Implementation on Cross Browsers and Devices with Saucelabs and Jenkins
    • Daily Regression Test Execution Challenge on Flaky Test Case
  • Shift-Right Testing details
    • Canary release introduction
    • Monitoring driven release approach
    • Conclusion of introducing canary release in real environment
  • Result and Future Plan on continuous testing
  • Q&A

Learning Outcome

  • Outline of Automation Test Framework
  • Current DevOps Practice Improvement Direction
  • Better understanding of monitoring-driven release
  • Canary release strategy implementation & process

Target Audience

Intermediate Level on CI/CD ; Basic Programming Skills Recommendation; Any role of a project

schedule Submitted 2 years ago

  • Yasunobu Kawaguchi

    Yasunobu Kawaguchi - DevOps パイプラインゲーム

    120 Mins

    Agile Testing Days で学んできた、DevOpsパイプラインゲームをやってみます!