User testing and usability testing are often used interchangeably, but they’re actually different things.

Before you start throwing these terms around in your next production meeting, it’s a good idea to have a solid understanding of what each of means individually and how they relate to each other.

I’ll get you caught up on not only what they mean, but also how you can conduct them efficiently for your next mobile app project.

User Testing Vs Usability Testing 

So what’s the difference between user testing and usability testing? Let’s take a look.

What is User Testing?

User testing is about validating your app idea. It involves getting out and asking people questions related to your app so you can confirm a distinct market for your idea. 

Think of user testing as research on potential users. Let’s look at the ridesharing app Uber as an example.

Before rolling out the Uber app, it’s very likely that the creators of the app, Garrett Camp and Travis Kalanick, took a look first at the current system of transportation.

User testing and research here would involve asking people the following questions:

• How do you get to school/work every day?
• How often do you use a Taxi, bus, or other forms of public transportation?
• How reliable is public transportation for you?
• Can you show me what steps you take when taking a bus/taxi/train?

Remember, the idea here is to do research to find out what the current problems are with the market you’re trying to get into in order to validate your app idea.

This phase is about proving that people need your app, and it should be done as soon as you have the idea.

What is Usability Testing

Usability testing is about seeing if and how well people can physically use your mobile app. 

Is it being used as intended?

This phase involves having real users interact with a prototype of your mobile app (or some type of usable wireframe) so that you can step back and observe their behaviors. 

Like in the image you see below by Proto.io, a prototyping tool that lets you create fully-interactive prototypes that look and work as your app should without having to code anything.

Using our same Uber app example, here are a few questions you might ask someone during usability testing:

• Can you show me how to request an Uber driver?
• Can you enter your location and your destination?
• Can you navigate to the dashboard?
• Can you leave a tip and review for the driver?

Usability testing is about observing users conduct various tasks to see that they are able to use the app as it’s intended. 

You should conduct usability testing after user testing and as soon as you have some kind of prototype or wireframe that users can get their hands on.

User testing and usability testing are both incredibly important testing processes that should be conducted well before you develop your app.

User testing will validate that your app idea is in fact a marketable one, and usability testing will help ensure you build an app that is used correctly and intuitively.

User Testing 101

Now that you have a clear understanding of what user testing is and what usability testing is, let’s dive first into user testing since this process should come before anything else. 

Then we’ll move onto usability testing in the next section.

Define your goals

The first step in user testing is defining the goals you have for your app. What is the value you hope it will bring to users and what are you hoping to achieve? 

Asking yourself this question will set the stage for what comes next, like what testing methods you’ll use, what kind of environment you’ll have, and what questions to ask participants.

How to run User Testing

User testing is pretty informal and doesn’t require setup or a specific environment like usability testing, where they may need access to a computer or the prototype they’ll be testing.

With user testing, you can simply have a coffee shop discussion, a virtual Skype or Google Hangout call, host it in an office, or even send out a Type Form or Google survey.

The goal here is just to talk candidly to people and learn about their experiences to see where your app fits in.

There really isn’t much to it! 

Now let’s jump into usability testing, which is a bit more involved. It’s also the term people are typically referring to when they use the two terms interchangeably.

Usability Testing 101

Once your app idea has been validated and you’re ready to move on to prototyping and wireframing, it’s a good time to bring in users to test your app.

Keep in mind that usability testing should be iterative, meaning, you want to do it throughout the development of your app.

It’s a good way to see that your app is on the right path and that the features and designs you’re adding are being used correctly.

With that said, your first round of usability testing can be done before you even start coding your app.

Think of it as an early-stage opportunity to check on the design, functionality, and marketing strategies you have before coding the final product.

While a high-fidelity prototype can allow your testers to try out various complex tasks and interactions, you can start testing even earlier when all you have are hand-drawn paper prototypes, like what you see below.

Unlike high-fidelity prototypes that allow users to interact with your app using some kind of online prototyping tool, paper prototypes are sketches you make of the different screens of your app.

Find your users with screening questions

To conduct usability testing, you need users, but you can’t just use anyone. You have to first think about who your ideal participant is and then make sure to only invite people who fit these criteria.

For example, an app like Uber is something everyone could benefit from—young people, old people, men, women, teenagers—everyone uses Uber. But is your app for everyone?

To get the most value out of user testing, you have to first establish who your target audience is through user research and let those people be your participants. 

You can do this by asking screening questions for potential candidates. 

Here are some ideas for questions you can ask:

Question #1: What’s your age?

Knowing someone’s age is important because it ensures your participants are within the correct  (or desired) age range. 

Question #2: What’s your highest level of education?

Education will give you a greater insight into the type of work someone does and how they do it. 

Depending on a participant’s educational background, including what their degree is, you can see how they interact with your product.

For example, if someone comes from a highly technical background, they will probably have an easier time with your app, even if it’s poorly designed, than someone without a degree, which might make it harder to find flaws in your app.

But then again, someone with a technical background could offer useful feedback about what your app is lacking and what its strengths are.

Question #3: What’s your income level?

If your app offers paid services, you need to ensure your test participants can afford it.

It wouldn’t make any sense to invite someone to interact with and test your app when it’s something they wouldn’t even consider using in the future due to the price tag.

Question #4: Ask lifestyle questions

This includes questions about people’s hobbies and what their day-to-day life looks like. This is a good opportunity to allow them to speak freely about what they like and what they don’t like.

These questions will help give you greater insight into their behavior and it’ll reveal if any of their activities and hobbies relate to your app.

Question #5: How much time do you spend online?

Similar to the education background questions, this one can help you see how tech-savvy potential participants are.

Generally speaking, those who spend a lot of time online or using digital products will often be able to provide better and deeper feedback about features.

The questions you ask should help you understand the experience and demographics of your participants.

How many testers do you need?

Once you have your sample users who will be testing your app, you’re ready to get started.

According to user testing master Jakob Nielson, “The best results come from testing no more than 5 users and running as many small tests as you can afford.”

As you can see from the chart above, 0 users give 0 feedback, obviously, but as soon as you collect information from your first user, insights shoot up and you’ve already learned nearly a third of what there is to know about your app’s usability.

Testing the second user will uncover the same things as the first user, which has a little overlap.

But since people are different, they can still bring something new that wasn’t observed with the first user. 

And so on with your third, fourth, and fifth users. His idea is that as you add more users, the less you learn because you’ll keep finding out the same things, which is a waste of time.

In short, 5 is a good number. Having more testers beyond that may just be wasting your time.

Questions to ask during the test

You already have some good questions to ask to find your participants, but what should you be asking once you have them?

Question #1: Why are you using our app?

Asking this will help you understand what’s driving someone to use your specific app. It will also help you understand the goals they’re trying to achieve.

Question #2: Have you used a similar app?

This will help you identify who the competition is and what users think about them. 

Question #3: Can you do this ____?

Asking your participants to conduct various tasks within your app is going to help you observe how easy it is for them to complete it and whether or not they struggled along the way. 

This will help you remove any barriers users may be facing when interacting with your app.

Question #4: Which features are most valuable to you and why?

This will help you trim down the number of features you have planned for your app, which will save you time and money when it’s time to develop your app. While you may think some features are crucial, the reality may be different.

Question #5: Is this app easy to navigate?

Having an intuitive app that’s easy to navigate should be a top priority for any mobile app.

Depending on the insights you get from asking this question, you can adjust your app’s design and architecture to make it a more seamless experience.

Question #6: Can you give me an example of when you would do ____?

Get specific use cases from your testers so feedback is more accurate and less generic. You’ll be surprised at how many people have scenarios you would have never considered.

Question #7: What do you think about the app’s design?

Open-ended questions like this are a great way to get more specific opinions about your app and it’s overall design.

This can help you further improve big things like pages, and smaller individual elements like buttons, icons, and so on.

Question #8: Why did you do ____?

Ask your testers why they perform various actions when interacting with your app.

This will help you identify hooks that can convince users to complete certain operations in your app. It will also help you better understand the thought process behind various actions.

Question #9: What would you change about this app?

This will help you learn more about what your app is lacking and where it could use improvement. 

Question #10: What would you like to see in an app like this?

This question will compel your testers to offer their thoughts about what features or functionality they’d like to see in your app, which you could either implement during development or keep in your backlog for future versions.

Final Thoughts

When building a mobile app, user and usability testing are incredibly important processes that deserve careful consideration and time.

User testing will give you peace of mind that your app ideas are valid, and usability testing will ensure your app is on the right track and constantly improving as you make your way from start to finish.