Four testing experts recently explored testing beyond the "traditional" desktop and web, including the new battlefields of Mobile and IoT. Read on for a recap or to watch a webinar replay.
The testing domain has always included work outside the "traditional" UI applications in web and desktop—it's just never been as well publicized. That's changing rapidly with the explosion of mobile and embedded systems. In addition, the devices making up the Internet of Things, automobile systems, and similar systems all need testing too.
On June 15th we hosted an hourlong webinar where I, Jim Holmes, and three other testing experts discussed areas in technology outside "traditional" web and desktop. I was joined in the webinar by:
- Richard Bradshaw, AKA the "Friendly Tester." Richard's an independent tester, consultant, trainer and speaker.
- Daniel Knott, who is a senior tester working on Android projects at Xing.
- Iliyan Panchev, Program Manager for Test Studio at Progress.
You can find the deck hosted at SpeakerDeck.com, and a recording of the webinar is available on YouTube (or you can watch it at the bottom of this post).
You can also find related posts from Iliyan at the Telerik blogs, Richard at his blog, and Daniel at his blog.
Panelists covered topics from embedded systems in household appliances to understanding modeling in mobile applications.
Mobility Isn't New, or Is It?
Much of the discussion centered around mobility. Mobility's certainly not new, but it's definitely seeing an acceleration of interest!
One aspect that makes testing in the mobile domain so interesting is how personal users view their devices. People expect startup delays in desktop applications, and they're used to web pages taking time to load. That's far from the case with mobile apps. Users want to quickly access suspended apps and quickly interact with them.
This mindset brings lots of complexity in to the testing space: how do you interact with apps in this manner? How do you handle creating or simulating interactions between apps, or interruptions due to notifications, incoming calls, etc.?
A number of approaches for these situations were discussed, including use of third party services to trigger texts, calls, or other notifications. The importance of proxies (discussed below) was also highlighted.
Modeling for Value
System modeling is the practice of breaking apart a system into its subcomponets to identify critical responsibilities, data flows, actions, etc. Taking the time to discuss modeling with the rest of the team helps testers focus in on effective, efficient testing. Modeling ensures testers understand where to focus on API testing, where to flesh out automated unit checks, where to invest in exploratory testing, etc.
Modeling doesn't have to be complex. It's more important to get a conversation going with the proper folks on your team—and often a simple whiteboard diagram is all that's needed!
Tooling for testing mobile apps took up a fair amount of time. The webinar's audience had several questions in addition to the panelists' own thoughts on tooling.
Tooling in IoT and embedded has a long way to go to catch up with that in desktop or web. There's not much for testing frameworks, unit or otherwise, in either space, nor is there anything close to the smooth delivery pipeline tools. Embedded can be particularly difficult, especially the time and effort involved in pushing to the final device, whether it's a refrigerator or automobile!
Mobile tooling has come quite a long ways, although it's still behind desktop and web.
- Android Debugger Bridge is critical for anyone doing work on the Android platform. It's a command line tool for communicating with the simulator.
Driving the UI for automated tests is critical. The following three toolsets were mentioned in one form or another.
- Appium is an open source framework for native, hybrid, and web apps on mobile devices. It uses the WebDriver open source browser driver.
- Applitools is a commercial product that supports automated visual checks of UIs. It supports Windows 8, Android, and iOS.
- Test Studio is a commercial product that supports native, hybrid, and web automation on iOS and Android.
Proxies are a tool that sit in the middle of two communicating systems, say a mobile app and a server, and allow you to interact with traffic as it moves through. This allows you to monitor and change communication between the app and server—critical if you're troubleshooting or trying to cause errors in data or transmissions.
- Fiddler is undoubtedly the best-known proxy/web debugging tool.
- Charles is another well-known proxy for OSx.
- Browserstack is another well-used proxy available for Windows, OSx, and Linux.
Vastly Different Risks
Vastly Different Risks
All the panelists agreed "risk" is a very different notion, especially in the embedded space. Errors in embedded systems can have a much more dramatic impact. Some of the concerns discussed include:
- Difficulty of Bugfixes. Few embedded systems are designed for easy updates. Many give no path for updating firmware, ever. This means what you ship has to be solid—there's little or no chance for shipping a service pack.
- Death. Mistakes in some embedded systems (automobiles, medical devices, avionics, etc.) can outright cause a loss of life. Bugs don't get riskier than that.
Selected Audience Questions
Audience members had a great many questions during the hour-long session. A few selected questions are discussed below.
Q: I have had trouble with Android and requests not going through the proxy I set up.
A: Proxy settings for Android devices are set two ways: first, in your development environment, and secondly in the device's network settings. Unfortunately, there's no common way across environments or devices.
For Android Studio, read this article. For Visual Studio 15 RC2 check this StackOverflow post.
On-device proxy settings are generally through the device's wifi network settings. Here's one example on StackOverflow—but keep in mind this will vary between devices and OS versions...
Q: We have found that our firewall blocks the use of some proxy tools. Any advice?
A: Unfortunately you'll need to work with your organization's firewall staff. Ask about setting up a specific rule to support your development needs. This may take some time for coordination. Be sure to lay out a solid business case so they'll understand it's not just some geeky testers and developers trying to game the system—you're trying to solve concrete business needs!
Q: What are best practices for Android Testing?
A: First, there is no such thing as "Best Practices!" There are only starting points which vary for each situation. Sorry, that's a pet peeve of mine.
Testing Android is very much the same as other mobile devices: Focus on risk and business value first. Ensure you're using modeling to break the system into sensible components you can write appropriate automated checks for, then focus on the harder parts you can't automate.
Some of those areas we discussed during the session included:
- Interactions. Think notifications, calls, texts, etc.
- Power and resource usage. How does the app behave when other high-usage activities are occurring?
- Suspended mode. Does the app behave well when coming out of suspensions?
- Dropping/intermittent connectivity. Real users travel on trains, planes, elevators, and through regions where coverage might not be so great.
Q: What is more beneficial—just the mobile app automation or an end-to-end automation with backend systems?
A: Automating end-to-end checks rarely makes sense for mobile device apps. Instead, automate everything you can on the device and on the back end systems separately. That includes service testing to and from both the device and backend. That way you've got solid coverage in and out of the client and server. Then focus on great exploratory testing around the components (see modeling above) and the communications too.
Q: Are there tools to extract data from complex JSON formats to excel or csv to verify or visualize the data correctness/quality?
Yes! All platforms (.NET, Java, Ruby, etc.) have great libraries that will let you do this programatically or live. Some quick suggestions:
Q: What tools do you suggest to generate test data?
A: Each major language platform has various data generation libraries. Some commercial tools exist as well. The Faker library is available for all major languages and provides a great way to generate many types of data such as internationalized phone numbers, postal codes, company names, etc.
Watch the Webinar
Missed the webinar or just want to see it again? You can watch the entire webinar by signing up right here.
Watch the Full Webinar