IoT world grows by 30% every year (up to 25 billion(!) devices till 2025) and that’s just a matter of time when some of us will have to…get back to the past. Many of the current testers very soon will start their adventure with real hardware devices. It also means more and more complicated integration with modern services and technologies (Amazon DRS, Alexa, Behavioral Recognition, vision processing, AI, ML and many more).

What are the limitations encountered in such conditions? What are the main challenges and what a modern tester should know before she or he starts their testing game? Why we should focus on safety aspects? And finally, how to find yourself in a testing environment when Time to Market statement has paramount importance in terms of building business models?

During my presentation, I’ll show how current challenges and risks will change in the near future and how to deal with them to avoid a safety headache. I’ll also present the way to easier identify and mitigate them. All of that in the context of pet cameras, snack launchers, and lasers…in other words - IoP (Internet of Pets) world. All dogs and cats are warmly welcomed!

The content of this presentation consists of 3 major parts:

• Risks, challenges and safety aspect the modern tester should be aware of
• The real-life examples of safety vulnerabilities and its impact on customers (and the scale of it!)
• What makes testing so exciting despite so much responsibility and so many challenges

Risks, challenges and safety aspect:

1) Be aware of flaky tests and its reasons

Now we live in the era of continuously delivered things. That causes even more and more test automation that we rely on. However, can we still treat every environment as safe enough? We should be still aware of false negatives (the test is marked as Passed when the functionality is not working) and false positives (the test is marked as Failed when the functionality is working). There is also something even worse - flaky tests (tests failing intermittently). And there are a lot of reasons behind that… (both, human and environmental ones)

2) Updates (OTA) as the most risky area

One of the most used “functionalities” by (aware or not) customers. When you think about SW update, usually that’s not something to be considered and safety critical. For web applications, every blocker can be fixed by fast deployment or bugfix. For mobile apps, it’s a matter of releasing a new version to store. Even for a desktop, you can force your user to reinstall it once again. But what if something goes wrong for IoT/HW devices? Is there any safety “belt” or procedure to prevent your firmware from turning into a brick?

3) Scale matters

New smart devices are very often responsible for public and private safety. To cover dedicated area (or just get your customers involved) there are a lot of them needed - sometimes thousands of them! Do you think that scale can be skipped during testing? How to ensure safe infrastructure when your network is growing up? Which parameters should be considered as important (e.g connectivity issues, configuration, geolocation, external influences, wireless signals multiplication)?

4) Testing real user behavior

Is your customer always following feature workflow intended by you? In most cases, your predictions are not fully consistent with the user’s real behavior. In that case, to achieve the high level of safety, you must rethink and redesign your test strategy (and implementation) approach. Let’s look at the example…

5) Focus on CXA/CEA (customer experience assurance)

Keep your customer safe! That’s the main principle for modern, smart devices world. Sometimes functionality working perfectly according to business requirements, but not exactly covering users’ expectations, can be much more harmful than functionality not working at all! What if light or laser strength suits all regulations but is not appropriate for young children? What if pets’ cookies are flying too fast and can frighten your cat? What if your house locks automatically and doesn’t want to let you in?

6) Limited trust in open source

We all know that statement. It relates not only to safety aspects but it also relates to security issues even more! Using open source for testing in the worst case can destroy your infrastructure or make it unmaintainable… Is there any chance to avoid such destructive impact?

7) Keeping safety during working with IoT & HW

Last but not least! Working with HW and equipment means embracing all physics rules. Rules that are sometimes ruthless… It is more than recommended to get familiar with some major principles while working with current, volts, and temperature - to minimize the number of explosions in your test lab :)

Automate everything! That’s the most suitable description of DevOps culture. The culture that quickly created job position with the same name. Position, mostly focused on broadly defined automation, leading to fast product delivery. And the division was pretty simple: DevOps = Process automation, QA = Test automation. But is it the right approach? What about (still more and more) popular “(Dev)TestOps” term?

Classical testers are now also very often responsible for set-up and maintenance of the major part of Continuous Integration or Continuous Delivery environment (especially test automation part). The main problem from the business perspective is, like always, time! E.g. many start-up and companies in a phase of early, dynamic growth cannot afford to waste too much time on test automation. How to speed up the delivery process in that cas? How to quickly generate a valuable increment?

I’ll show you how to improve and speed up testing and delivery process by clever automation in 3 steps:

1. Automation supporting manual regression testing activities
   • One Click test environment setup
   • Preparing fully readable and executable test cycles and test cases
   • Ensuring that unchanged component is not double-checked or unnecessarily tested
2. Automation maintaining project workflow, transitions, and statuses
   • Recognizing when an issue is QA ready vs. when it is dev ready
   • Avoiding (very!) common misunderstanding regarding testable issues
   • Assuring that functionalities have been released without reading documentation and changelogs
3. Automation enhancing bug catching and reporting during testing or normal application usage
   • Comprehensive incorporation of all crashes with bug reporting system
   • Immediate notification based on priority and severity threshold level
   • Automatic preparation and update of reported bugs, including necessary statistics

All of that in the context of 5 good (TestOps’) friends: Test Management Tool (Zephyr), Project Management Tool (Jira), Crash & Log Reporting System (Crashlytics), Communication & Notifications Channel (Slack) and Continuous Integration Tool (Jenkins).

It is predicted that till 2025 there will be over 75 billion devices spewing 180 zettabytes of data and generating up to 6 trillion dollars. That enormous increase force companies to introduce a continuous approach to deliver the product as fast as possible and be able to compete on the market.

The main question is how to test application for end user among so much hardware equipment and ecosystems combining HW, FW, mobile devices and complex backend architecture? Considering all factors and possible obstacles is it for companies a real “A New Hope” for companies or just simply “Mission Impossible”?

I will take the participants on a journey to the IoT world. It will be a talk about the challenges that any tester will face at some point. I will present the dangers, risks and snares but also good practices and practical approach to mobile E2E test automation for the IoT solutions in CI approach.

Technical examples will be presented using Python languages and supported by physical devices (mobile phones and IoT equipment).

It is predicted that till 2025 there will be over 75 billion devices spewing 180 zettabytes of data and generating up to 6 trillion dollars. That enormous increase force companies to introduce a continuous approach to deliver the product as fast as possible and be able to compete on the market.

The main question is how to test application for end user among so much hardware equipment and ecosystems combining HW, FW, mobile devices and complex backend architecture? Considering all factors and possible obstacles is it for companies a real “A New Hope” for companies or just simply “Mission Impossible”?

I will take the participants on a journey to the IoT world. It will be a talk about the challenges that any tester will face at some point. I will present the dangers, risks and snares but also good practices and practical approach to mobile E2E test automation for the IoT solutions in CI approach.

Technical examples will be presented using Python languages and supported by physical devices (mobile phones and IoT equipment).

It is predicted that till 2025 there will be over 75 billion devices spewing 180 zettabytes of data and generating up to 6 trillion dollars. That enormous increase force companies to introduce continuous approach to deliver product as fast as possible and be able to compete on the market.

The main question is how to test application for end user among so much hardware equipment and ecosystems combining HW, FW, mobile devices and complex backend architecture? Considering all factors and possible obstacles is it for companies a real “A New Hope” for companies or just simply “Mission Impossible”?

I will take the participants on a journey to the IoT world. It will be a talk about the challenges that any tester will face at some point. I will present the dangers, risks and snares but also good practices and practical approach to E2E test automation for the IoT solutions in CI approach.

Technical examples will be presented using Python languages and supported by physical devices.