Preparing for User Experience Testing

For the last few weeks I’ve been preparing for the user experience testing. In this post I share the final version of my usability test: the opening script, scenario tasks, pre-test and wrap-up interview.

Pre-test Interview

I plan to start each session with a brief welcoming part that explains to participant how the test is going to work. Below is the current version of the script that I’m going to read during the introductory part of the test.

You have been asked to participate in a “first experience” test for the GNOME desktop. Just like Windows is a desktop and MacOS is a desktop, GNOME is a free software desktop. If you have any further questions about GNOME I’d be happy to try and answer them.

For this test, we are interested in what people think of their first experience with GNOME. We are looking for people who haven’t used GNOME before, so we don’t expect that have used GNOME before today. We want to see what you think of GNOME when you use it for the first time.

This is entirely a test of GNOME. We are not testing you and there is no wrong answer, so please do not feel pressured by time or anything else. All we’re looking for is what you think about your first experience with GNOME.

For this first experience, I’ll ask you to login using a test account. I’ll give you some time to experiment with GNOME. Use it like you would use a new computer for the first time. To help guide you, I’ll ask you to do a few sample tasks that mimic how most people would probably use a new computer.

But before we begin, I’d like to learn a few things about how you would use a new computer:

Let’s say at work or at home you have a new computer with pre-installed operating system which is new for you.

You have booted this computer for the first time. How would you use it at first?

Scenario Tasks

Scenario tasks are the main part of the test. For each task I’ll give a participant a copy of the scenario and read it aloud. In some tasks I’ll give additional information, like login credentials, or a USB drive with sample folders and files.

First I’ll start with a short intro:

Okay, let’s have you use the computer now. Here’s a username and a password you can use to login. No one else has used this account before, and no one else will use it when you’re done here today. (I’ll delete any data you leave behind when we’re done.) To help you explore the system, here are a few tasks that we think most people would do with a new computer.

Then I’ll give a participant some context of each task:

Task 1: Managing files

You’ve booted a new computer for the first time. Let’s say this USB fob drive has files from your old computer. Please copy the files to the new computer. Put them wherever makes sense to you.

Task 2: Using a browser

After you used your new computer for a while, you want to browse the internet and open some of the sites you visit more frequently. Please open a few websites that you would normally visit, like Google or Facebook.

Task 3: Checking email

After you start up your new computer, you want to check your email. Go ahead and check your email. I’ll delete everything after we’re done today, and you are the only person who will use this account, so please access your email however you normally do it at home.

Wrap-up Interview

I have some questions prepared for the wrapping up part of the test to help me get the participants’ general impressions of the experience:

What things were really easy to figure out?

What things were harder to figure out? Why?

Can you summarize your first experience today in a single word, like an adjective? What one word describes the test today?

Jim suggested an interesting method — asking testers to summarize different parts of their experience using an emoji. It will help testers express their own feelings, but because we’ll provide a set of emojis (testers won’t make their own) it will be easier to collate results.

Think back to the start of the test today. If you had to pick one emoticon (from this list) to describe the first time you used GNOME today, what emoticon would that be? What emotion does that represent to you?

After you got settled into GNOME, and had played around with it for a while, what emoticon would you use to describe that part of the experience? What emotion does that represent to you?

emoji - Copy2

I’m excited to perform this UX test very soon, and I’m definitely open to your thoughts and feedback on any part of this test preparation!



3 thoughts on “Preparing for User Experience Testing

  1. I love the new emojis! These are a great way for people to concisely describe their reactions to using GNOME for the first time. The emojis are abstract enough for people to project their own thoughts onto them, but recognizable enough that I think there’s strong agreement to what these emojis represent. And ten emojis seems like a good number, so people can have a choice of how they respond, but not too many that we have watered down the options.

    When we do the analysis, I think we can group the first three emojis together as “very negative.” These emojis seem to represent Angry, Sad, and Sick. I think the next three emojis are related to “not sure.” They seem to represent “hmm,” “meh,” and “what?” The last four are also very related as “very positive: “Oooooh,” Happy, Very Happy, and Love.

    At a guess, I think your first experience test will probably take 30 minutes per tester. That assumes 5-7 minutes to get settled, do the intro, and pre-interview. Users may spend 5+ minutes on each task. Then another 5-7 minutes to do the wrap-up and post-interview. I recommend you schedule your testers with some free time between them, so you can prepare for the next volunteer before they arrive. If each tester takes 30 minutes, you should plan for testers to arrive AT LEAST 45 minutes apart. Maybe an hour apart, so you have some extra time in case one or two testers decide to do a “deep dive” on the tasks.

    Don’t forget to take notes during each session. If you can, do an audio recording so you can use the recording for reference. (If you do a recording, please let the tester know, and have them indicate on the recording that they are aware they are being recorded for test review purposes.)

    Good luck on your test!


  2. My main interest for this test is to get some fine-grained information on how people perceive and relate to the product. It’s not intended as a usability test: I’m therefore less interested in standardised tests or quantitative measures, and I’m not so interested in finding out how easy or difficult the tasks were.

    You might want to think about allowing each test participant to explore the system. You might also want to give them the freedom to pick some tasks, based on what they tend to use their computer for.

    Likewise, the emojis might be a useful tool to start the conversation, but the real thing I’m interested in is qualitative insights. For example: do the participants feel like it is attractive, like it is designed for them, like it is something they would like? What social and cultural signifiers do they associate with it?

    This will require that you adopt a semi-structure interview approach, where you ask questions aimed at understanding and meaning, and where you explore the answers in order to try and find out insights. The types of questions you might ask:

    Who can you imagine using this? Do you imagine men or women prefer it? Old or young? What kind of jobs do you think they might have?
    If you had the choice of choosing this system over your current one, which would you choose? Why?
    Do you think it’s attractive?

    I quite like the questions you have about asking the participant to pick words, but the focus of the question should be the product not the experience: if you had to pick a word that describes GNOME, what would it be? You might also want to think about allowing them to pick more than one word.

    And again: explore. Have some interview prompts in mind: why did you pick that word?

    You’ll need to record the interviews and you might need to transcribe them afterwards.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s