Understanding Mobile App Testing Limitations – Maze Help

Before you start

Mobile app testing operates differently from desktop testing due to the unique characteristics of mobile devices and app ecosystems. Being aware of these differences will help you design more effective studies and set appropriate expectations.

Key considerations for mobile testing

App updates and bug fixes

Updates to the Maze Participate app must go through app store review processes, which typically take several days to complete.

Impact on data collection

More importantly, if an issue affects data collection during a study, the impacted responses cannot be recovered. Once a participant's session has been affected, we cannot go back to recollect that specific data. This is the nature of user research: each participant's experience and feedback happens in a specific moment that cannot be recreated.

Run comprehensive pilot tests before launching your full study to identify potential issues early and prevent data loss.

Audio and video recording considerations

Device audio switching

When participants connect Bluetooth headphones or switch to CarPlay during a session, audio routing may interrupt.

Media content interference

Websites with embedded media (YouTube videos, audio players) can interrupt audio recordings by taking over the device's media output when participants interact with them. You might not be able to collect participants' voice in the same block after the media is displayed. Voice recording typically resumes automatically for subsequent blocks.

To ensure quality audio collection, we recommend designing your tests without embedded media when possible. If media element is an essential part to your websites or apps, add clear guidance instructing participants which elements to avoid clicking. Including a context screen at the beginning of your study helps set proper expectations and prevents unintended interruptions.

Session completion and connectivity

Mobile testing can be affected by battery life constraints, memory limitations on older devices, or connectivity changes between WiFi and cellular networks.

The following devices are supported:

Android 9
iOS 16.5

If a participant’s device is not up-to-date, they will be blocked from downloading the app store/play store.

Setting up for success

Include welcome screens explaining testing requirements like battery life and stable internet. Clear communication at the beginning of your study helps participants prepare properly and reduces the likelihood of interruptions.

We recommend monitoring early completion rates for unusual patterns in your data. If you notice systematic drop-offs or unexpected behavior, please contact our support team (help@maze.design) so we can help identify and address any issues.