How to kill test flake in Appium?
Whether you are a beginner learning the ways of writing UI automated tests or an experienced practitioner, you must have faced the below situation:
You write a test and it works perfectly in your local, happily you promote the tests to your CI environment and surely it passes initially, however after sometime the test starts failing sporadically. This phenomenon is described as Test flakiness and it’s a bummer for any engineer worth his salt. This can cause numerable side effects and doubts into the minds of software teams not trusting UI automation.
However all hope is not lost. We all know a Tool is only as good as its master.
In this talk, I will cover the measures or patterns which you can implement to reduce the flakiness in your Appium tests as much as "humanly" possible and increase their speed and reliability.
Outline/Structure of the Talk
- Introduction of test flakiness as a first class test automation problem [~2 mins]
- Common patterns for why tests are flaky and ideas to avoid them [Total: ~33 mins]
- Unsuitable Locator or selector strategy (~10 mins)
- Avoiding XPATH and look into what are the alternatives? E.g. Accessibility ids, IOS class chain, UI Selector
- Using image selector/coordinates to find elements which are hard to find.
- Unreliable wait strategies (~ 5 mins)
- Avoiding Hard coded sleeps and replacing with standard explicit waits (Expected conditions) and customization
- Long winding E2E steps that take time to arrive at screen under test and fail due to different reasons: (~10 mins)
- Will discuss use of API's to setup artifacts required for tests
- Will discuss use of android activities to arrive at exact screen or try out kitchen sinks as hooks to different app states
- Use of Deep links.
- Avoiding unstable network API calls and instead use mocking as a technique for predictable test data (~5 mins)
- Disable animations and make use of some of Appium capabilities to improve reliability and speed (cross platform, android or IOS caps) (~ 3 mins)
- Debugging and Best practices (~7 mins)
- Reading exception stack traces and error messages to separate errors in Test code vs actual Appium problems
- Logging test runs/screenshots in a persistent DB to get historical information around tests is your best friend
- Analyzing Appium server logs or device logs
- Co-ordinating with dev to build testability in the app
- Q & A (~3 mins)
By the end of this talk,
You as a test automation engineer would be able to have a broad understand of what are the different factors which cause tests to be flaky and identify approaches which you can try to improve your tests reliability
- Understand waits and locators pitfalls
- Understand cross platform or specific implementations for android/IOS to improve speed and reliability
- Basics of debugging and log analysis to arrive at root cause of problems
Automation Testers or developers who want to use Appium effectively for their test automation efforts
- Should have basic working experience in UI test automation using either Appium/selenium.
- Code examples would be in Java and Python