Mobile app testing: device matrix, checklist and strategy

Designing the device matrix

A device matrix is the list of device and software configurations the app is tested on, and the mistake is to make it either a wish list of every phone on the market or the two phones the team happens to own. Build it from four dimensions and your own analytics. For each dimension the table shows the options and a rule for choosing. The result for a typical consumer app is 8 to 12 primary configurations that run every release and another 6 to 10 sampled configurations that rotate through exploratory sessions, an approach that borrows from risk-based testing.

Interruption testing

An interruption is anything that takes the app out of the foreground or steals the screen while a task is in progress. Every mobile app should survive each of the following in the middle of its three most important flows, resuming exactly where the user left off or failing with a clear message.

• Incoming call, answered and declined, during a form submission and during a payment.
• Incoming notification from another app, tapped, then return via the app switcher.
• Backgrounding for 10 seconds, 5 minutes and 30 minutes, then resume; check whether the session, the draft and the scroll position survive.
• Screen lock and unlock mid-flow, including with biometric authentication.
• Low battery warning, low storage warning and a system update prompt.
• Device rotation on every screen that allows it, with text entered in a field.
• Killing the app from the switcher during a network write, then relaunching.

Network conditions

Most mobile defects that reach production are network defects, because developers build on a fast office connection and testers often test on one. Use the platform's network conditioning tools or a proxy that throttles bandwidth and adds latency and packet loss. Test each critical flow at four states: fast (50 Mbps, 20 ms), slow (1 Mbps, 300 ms, 2 percent loss), intermittent (connection drops for 10 seconds every minute) and offline. Watch for four failure modes: a spinner that never times out, duplicate submissions when the user taps again, data loss when the connection drops mid-write, and a cached screen that shows stale data without saying so. A well-behaved app times out within 30 seconds, tells the user, keeps the draft and retries safely.

Permissions, battery and performance

Permissions have their own state machine: not yet asked, granted, denied, denied permanently and revoked while the app is running. Test every permission the app requests through all five states, and pay attention to the last one, because a user who revokes camera access from settings while the app is backgrounded will find the crash that nobody else does. Battery and performance testing on mobile means measuring on a budget device, not a flagship: cold start under 2 seconds, warm start under 1 second, no dropped frames on the main list scroll, memory stable across a 30-minute session, and battery drain under 3 percent for 10 minutes of typical use. Location and background sync are the usual culprits when drain is high.

Store submission checks

Both major app stores review submissions against published guidelines and reject a meaningful share on first attempt, typically for missing privacy disclosures, broken links to policies, crashes on launch on a reviewer's device, or requesting permissions the app cannot justify. Add a pre-submission pass to the release checklist: install the release build clean on a device the team has never used for development, launch with no account, walk every entry point including deep links and notifications, and verify the privacy declarations match the permissions actually requested. Issue 28 of the original magazine (December 2014, the final issue) carried a mobile testing theme and appears to have been cited by at least one mobile testing book; the issue 28 record preserves what is known about it, and much of its advice on device coverage still holds.

The 25-item mobile testing checklist

Grouped in five sets of five. The first three groups run every release on every primary configuration; the last two run on the release candidate.

• Install and launch: clean install; upgrade from the previous two versions with data; cold start time on the slowest primary device; launch with no network; launch from a deep link and from a notification.
• Core flows: each of the three most important user journeys end to end; the same journeys at slow network; the same journeys offline with later sync; form validation and error messages on every input; back navigation from every screen.
• Interruptions and state: incoming call mid-flow; backgrounding for 30 minutes and resume; kill and relaunch during a write; rotation with data entered; permission revoked while running.
• Device and accessibility: smallest and largest screen class; largest system font; screen reader through the core flows; dark mode on every screen; one right-to-left locale.
• Release readiness: crash-free session rate above 99.5 percent on the beta; memory and battery on a budget device; privacy declarations match requested permissions; analytics events fire as specified; store listing screenshots match the build.

Emulators, real devices and device clouds

Emulators and simulators are the right tool for development and for the fast layer of an automated suite: they are free, scriptable and start in seconds. They cannot test real network radios, thermal throttling, biometric hardware, manufacturer skins, or the interaction with other installed apps, so they are the wrong tool for the release decision. Real devices in a local lab give the most faithful results and the fastest debugging, at the cost of buying and maintaining hardware. Device cloud services rent remote real devices by the minute and cover breadth (many models, many OS versions) that a local lab cannot. A sensible split for most teams: emulators for developer checks and the unit and component layers of the pyramid, a local lab of the 8 to 12 primary devices for release testing, and a cloud service for the sampled configurations and for reproducing customer-reported device-specific defects.

This guide is part of the software testing techniques hub. It is best read alongside performance testing and software testing metrics, which cover the neighbouring questions.