Four testing experts recently explored testing beyond the "traditional" desktop and web, including the new battlefields of Mobile and IoT. Read on for a recap or to watch a webinar replay.
The testing domain has always included work outside the "traditional" UI applications in web and desktop—it's just never been as well publicized. That's changing rapidly with the explosion of mobile and embedded systems. In addition, the devices making up the Internet of Things, automobile systems, and similar systems all need testing too.
On June 15th we hosted an hourlong webinar where I, Jim Holmes, and three other testing experts discussed areas in technology outside "traditional" web and desktop. I was joined in the webinar by:
You can find the deck hosted at SpeakerDeck.com, and a recording of the webinar is available on YouTube (or you can watch it at the bottom of this post).
You can also find related posts from Iliyan at the Telerik blogs, Richard at his blog, and Daniel at his blog.
Panelists covered topics from embedded systems in household appliances to understanding modeling in mobile applications.
Much of the discussion centered around mobility. Mobility's certainly not new, but it's definitely seeing an acceleration of interest!
One aspect that makes testing in the mobile domain so interesting is how personal users view their devices. People expect startup delays in desktop applications, and they're used to web pages taking time to load. That's far from the case with mobile apps. Users want to quickly access suspended apps and quickly interact with them.
This mindset brings lots of complexity in to the testing space: how do you interact with apps in this manner? How do you handle creating or simulating interactions between apps, or interruptions due to notifications, incoming calls, etc.?
A number of approaches for these situations were discussed, including use of third party services to trigger texts, calls, or other notifications. The importance of proxies (discussed below) was also highlighted.
System modeling is the practice of breaking apart a system into its subcomponets to identify critical responsibilities, data flows, actions, etc. Taking the time to discuss modeling with the rest of the team helps testers focus in on effective, efficient testing. Modeling ensures testers understand where to focus on API testing, where to flesh out automated unit checks, where to invest in exploratory testing, etc.
Modeling doesn't have to be complex. It's more important to get a conversation going with the proper folks on your team—and often a simple whiteboard diagram is all that's needed!
Tooling for testing mobile apps took up a fair amount of time. The webinar's audience had several questions in addition to the panelists' own thoughts on tooling.
Tooling in IoT and embedded has a long way to go to catch up with that in desktop or web. There's not much for testing frameworks, unit or otherwise, in either space, nor is there anything close to the smooth delivery pipeline tools. Embedded can be particularly difficult, especially the time and effort involved in pushing to the final device, whether it's a refrigerator or automobile!
Mobile tooling has come quite a long ways, although it's still behind desktop and web.
Driving the UI for automated tests is critical. The following three toolsets were mentioned in one form or another.
Proxies are a tool that sit in the middle of two communicating systems, say a mobile app and a server, and allow you to interact with traffic as it moves through. This allows you to monitor and change communication between the app and server—critical if you're troubleshooting or trying to cause errors in data or transmissions.
Vastly Different Risks
All the panelists agreed "risk" is a very different notion, especially in the embedded space. Errors in embedded systems can have a much more dramatic impact. Some of the concerns discussed include:
Audience members had a great many questions during the hour-long session. A few selected questions are discussed below.
Q: I have had trouble with Android and requests not going through the proxy I set up.
A: Proxy settings for Android devices are set two ways: first, in your development environment, and secondly in the device's network settings. Unfortunately, there's no common way across environments or devices.
For Android Studio, read this article. For Visual Studio 15 RC2 check this StackOverflow post.
On-device proxy settings are generally through the device's wifi network settings. Here's one example on StackOverflow—but keep in mind this will vary between devices and OS versions...
Q: We have found that our firewall blocks the use of some proxy tools. Any advice?
A: Unfortunately you'll need to work with your organization's firewall staff. Ask about setting up a specific rule to support your development needs. This may take some time for coordination. Be sure to lay out a solid business case so they'll understand it's not just some geeky testers and developers trying to game the system—you're trying to solve concrete business needs!
Q: What are best practices for Android Testing?
A: First, there is no such thing as "Best Practices!" There are only starting points which vary for each situation. Sorry, that's a pet peeve of mine.
Testing Android is very much the same as other mobile devices: Focus on risk and business value first. Ensure you're using modeling to break the system into sensible components you can write appropriate automated checks for, then focus on the harder parts you can't automate.
Some of those areas we discussed during the session included:
Q: What is more beneficial—just the mobile app automation or an end-to-end automation with backend systems?
A: Automating end-to-end checks rarely makes sense for mobile device apps. Instead, automate everything you can on the device and on the back end systems separately. That includes service testing to and from both the device and backend. That way you've got solid coverage in and out of the client and server. Then focus on great exploratory testing around the components (see modeling above) and the communications too.
Q: Are there tools to extract data from complex JSON formats to excel or csv to verify or visualize the data correctness/quality?
Q: What tools do you suggest to generate test data?
A: Each major language platform has various data generation libraries. Some commercial tools exist as well. The Faker library is available for all major languages and provides a great way to generate many types of data such as internationalized phone numbers, postal codes, company names, etc.
Missed the webinar or just want to see it again? You can watch the entire webinar by signing up right here.
Watch the Full Webinar
Jim is an Executive Consultant at Pillar Technology. He is also the owner of Guidepost Systems. He has been in various corners of the IT world since joining the US Air Force in 1982. He’s spent time in LAN/WAN and server management roles in addition to many years helping teams and customers deliver great systems. Jim has worked with organizations ranging from startups to Fortune 100 companies to improve their delivery processes and ship better value to their customers. When not at work you might find Jim in the kitchen with a glass of wine, playing Xbox, hiking with his family, or banished to the garage while trying to practice his guitar.