First Impressions of UX test

This week has been great, I’ve finally completed my 5 persons user experience test, hurray! In this post I’d like to share my first impressions and briefly describe what went well during the test and some difficulties I had to face.

Test had a total of 5 participants, which is not many, but enough to get information on how people perceive and relate to the product and uncover important usability problems.

There is an article by Nielsen that explains how many test users we need in a usability study, it provides a mathematical formula and a chart:

Nielsen assumes that after the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new.

Frankly, I would be happy to test more users, as the process is really exciting and fun, and the testing results are surprising, but last month I moved to a new city and even appointing five tests was a little challenging. It also took a little bit longer than I expected (people are busy in general and it’s not easy to find them available).

I had the chance to welcome volunteers from different fields, with different computer expertise and expectations. I learned that person’s background matters, and it was very interesting to observe how behavior of a thirteen-year-old differed from behavior of adults, and how a designer’s perception differed from programmer’s and manager’s.

123 - Copy1 - Copy.pngIt appeared that conducting UX tests is not as easy as it might seem, we must be really well prepared. And it’s essential to do a dry run of the text by yourself — in my case I had to do it twice before I made sure everything would work properly during the test session.

The participants didn’t hesitate to speak their mind when they were confused or something bothered them during the test, which made me happy, because they said really useful and sometimes unexpected for me things.

Unlike the usability test that Renata and Ciarrai performed, the UX test didn’t require participants to do “get down to work” kind of tasks. Rather, I allowed each participant to explore the system, and pick some tasks, based on what they tend to use their computer for.

After the testers interacted with GNOME and its applications, I asked them some questions about their experience, which you can find in my previous post. I also added more questions aimed at understanding and meaning, suggested by Alan:

Who can you imagine using this [GNOME]?
Do you imagine men or women prefer it?
Old or young?
What kind of jobs do you think they might have?
If you had the choice of choosing this system over your current one, which would you choose? Why?
Do you think it’s attractive?

In general the tests ran smoothly, and lasted for about 35 minutes each. I did audio recording during most of the test sessions.

I am pleased with how the tests went, and the volunteers seemed to be pleased and enthusiastic too! Most of them found that they liked GNOME and indicated interest in using it in the future.

In the next post I will write the analysis and get into details of the tests. I’ll also present an analysis of the tester’s engagement, using the emoji-with-counts method.

Stay tuned!


Preparing for User Experience Testing

For the last few weeks I’ve been preparing for the user experience testing. In this post I share the final version of my usability test: the opening script, scenario tasks, pre-test and wrap-up interview.

Pre-test Interview

I plan to start each session with a brief welcoming part that explains to participant how the test is going to work. Below is the current version of the script that I’m going to read during the introductory part of the test.

You have been asked to participate in a “first experience” test for the GNOME desktop. Just like Windows is a desktop and MacOS is a desktop, GNOME is a free software desktop. If you have any further questions about GNOME I’d be happy to try and answer them.

For this test, we are interested in what people think of their first experience with GNOME. We are looking for people who haven’t used GNOME before, so we don’t expect that have used GNOME before today. We want to see what you think of GNOME when you use it for the first time.

This is entirely a test of GNOME. We are not testing you and there is no wrong answer, so please do not feel pressured by time or anything else. All we’re looking for is what you think about your first experience with GNOME.

For this first experience, I’ll ask you to login using a test account. I’ll give you some time to experiment with GNOME. Use it like you would use a new computer for the first time. To help guide you, I’ll ask you to do a few sample tasks that mimic how most people would probably use a new computer.

But before we begin, I’d like to learn a few things about how you would use a new computer:

Let’s say at work or at home you have a new computer with pre-installed operating system which is new for you.

You have booted this computer for the first time. How would you use it at first?

Scenario Tasks

Scenario tasks are the main part of the test. For each task I’ll give a participant a copy of the scenario and read it aloud. In some tasks I’ll give additional information, like login credentials, or a USB drive with sample folders and files.

First I’ll start with a short intro:

Okay, let’s have you use the computer now. Here’s a username and a password you can use to login. No one else has used this account before, and no one else will use it when you’re done here today. (I’ll delete any data you leave behind when we’re done.) To help you explore the system, here are a few tasks that we think most people would do with a new computer.

Then I’ll give a participant some context of each task:

Task 1: Managing files

You’ve booted a new computer for the first time. Let’s say this USB fob drive has files from your old computer. Please copy the files to the new computer. Put them wherever makes sense to you.

Task 2: Using a browser

After you used your new computer for a while, you want to browse the internet and open some of the sites you visit more frequently. Please open a few websites that you would normally visit, like Google or Facebook.

Task 3: Checking email

After you start up your new computer, you want to check your email. Go ahead and check your email. I’ll delete everything after we’re done today, and you are the only person who will use this account, so please access your email however you normally do it at home.

Wrap-up Interview

I have some questions prepared for the wrapping up part of the test to help me get the participants’ general impressions of the experience:

What things were really easy to figure out?

What things were harder to figure out? Why?

Can you summarize your first experience today in a single word, like an adjective? What one word describes the test today?

Jim suggested an interesting method — asking testers to summarize different parts of their experience using an emoji. It will help testers express their own feelings, but because we’ll provide a set of emojis (testers won’t make their own) it will be easier to collate results.

Think back to the start of the test today. If you had to pick one emoticon (from this list) to describe the first time you used GNOME today, what emoticon would that be? What emotion does that represent to you?

After you got settled into GNOME, and had played around with it for a while, what emoticon would you use to describe that part of the experience? What emotion does that represent to you?

emoji - Copy2

I’m excited to perform this UX test very soon, and I’m definitely open to your thoughts and feedback on any part of this test preparation!


Moving to the project phase in Outreachy

A few weeks ago we officially started the usability testing phase of the internship, and in this post I’d like to share some insights of the work that is going on. It took some time to settle into a project focus, figure out who wants to do what and start building out the usability tests. Now our team is working on three usability testing projects for GNOME:

Renata will perform a traditional usability test of other ongoing work in GNOME. In previous cycles of Outreachy, our mentor Jim Hall and the interns Sanskriti and Gina uncovered design patterns that work well and others that need improvements, and this time Renata will look at those design patterns that have been improved in recent GNOME design iterations.

Ciarrai is preparing to perform a paper prototype test of the new Settings app. This would be testing a “future” Settings design — testing on changes that haven’t even been implemented in GNOME yet.

On my side, I am getting ready to examine a “user experience” of a user’s first exposure to GNOME. I will look at product identity and user experience, rather than straight usability. In this test, I will ask testers to simulate an unboxing of a new system. The tester will turn on the computer, watch it start up, and login to a fresh test account so they get first-user experience.

The tester will experiment with the system, using a few scenarios to suggest real tasks that real people might do, so they get an opportunity to poke around and launch a few applications.

My goal is to find out how people feel about their experience: whether GNOME is something that appeals to them, feels welcoming, is something they can imagine themselves using.

With help of Jim Hall and the GNOME Design team I have created the opening script, scenario tasks, and wrap-up questions for the UX test. The final version of the test is ready and I will share it with you soon!

Reflections on Usability Tasks

Before I entered the Outreachy internship, I had to present a “first contribution” — perform a small formal usability test, and perform some basic analysis using the heat map method. In this contribution my mentor Jim asked me to find ten scenario tasks used in other GNOME usability tests, and perform my own usability test with a few testers.

Instead of using pre-written scenario tasks I decided to create my own tasks, and made a lot of mistakes in them :) So today I want to reflect on my first contribution, and write about my experience and mistakes I made in these scenario tasks.


One of the applications I chose for usability testing was gedit — the default text editor of the GNOME desktop environment. It is designed as a general-purpose text editor.
First I came up with a list of some general user goals that gedit users might have:

  1. Autosave the file
  2. Change the font size
  3. Correct spelling mistakes in the text
  4. Find and replace text
  5. See the document statistics

After choosing the tasks to test, I formulated scenario tasks for usability testing:

You work as a freelance copywriter. This time you are writing a book review.
G1. Your laptop battery got broken, and if the laptop suddenly goes off you might lose recent changes in your document. So before you start, make sure that all changes you make in the document will be automatically saved every 10 minutes.
G2. Make the text bigger.
G3. Correct all spelling mistakes in the text.
G4. You realize that misspelled the last name Feinman. Replace Feynman with Feinman in the whole text at once.
G5. You need your review to be less than 200 words. Check if the size of the document is fine.

Now let’s see which of these scenario tasks were well-crafted.

G1 is a good task, as it resembles a real situation in which a user would want to do the certain task — turn on autosave. The task is specific enough, and at the same time it is short and provides just as much information as a user needs to complete it.

G4 and G5 are realistic tasks too, I think people often find themselves in similar situations when using text editors. There are no task-solving cues, and the descriptions are at the right level of detail. I asked testers to do something specific, set the context and used user’s language to make the tasks clear.

All these scenario tasks worked well during the usability test, as they were well-formulated, actionable and written clearly enough that the testers knew when they completed the task.

There also were scenario tasks that I would improve in the next usability tests.

G2. Make the text bigger.
This is a very abrupt scenario task, and a tester might be confused by the task itself. There is no context, no reason or purpose for performing the task. I wanted to keep the task short, but it’s more important to provide the participants with all the information that they need to complete it.

G3. Correct all spelling mistakes in the text.
This task doesn’t set a context too. More importantly, it provides an unintended hint to the tester by re-using keywords from gedit interface. Task scenarios that include terms used in the interface bias testers’ behavior and give less useful results, and it is essential to avoid giving such clues in usability testing.

It requires time and practice to evolve skills in writing tasks. But with help of a bright mentor and some effort — we’re almost there! Soon I’ll start working on a new usability test and hopefully will write good scenario tasks that will help to uncover usability issues effectively.


Scenario Tasks

The most effective way of understanding what works and what doesn’t in an interface is to watch people use it. This is the essence of usability testing.

In order to assess the usability of an interface, we ask testers to carry out a number of tasks using the interface. To watch people try to use what you are building, you need to give them something to do. It’s a two-step process:

  • First you choose the tasks to test.
  • Then you expand the tasks into scenario tasks — the little scripts that set the stage for the action and provide a bit of explanation and context.

The first step is to come up with a list of general user goals that visitors to your site or application may have. Ask yourself: What are the most important things that every user must be able to accomplish on the site?


As a quick example, here’s a list of tasks for youtube:
Search for a video.
Watch a video.
Create a playlist of videos.

Once you’ve figured out what the users’ goals are, you need to formulate scenario tasks that are appropriate for usability testing.

Task: Search for a video.
Scenario task: You’ve heard from your friend that Coldplay released a new music video. You want to see it. Find the video on youtube.

A scenario task has the character, the context and the necessary details that the user needs to know, but doesn’t. Scenario tasks resemble those that users would perform in a real life context. A well-crafted scenario task will help you to uncover usability issues more effectively.

Tips for writing scenario tasks

Make the scenario task realistic.
Make sure that each scenario task is realistic and typical for how people actually use the system, when they are on their own time, doing their own activities.

Avoid giving clues.
Phrase your scenario tasks without uncommon or unique words that appear on the screen. If you do, you turn the tasks into a simple game of word-finding.

Keep the scenario task concise.
Provide the participants with the information that they need to complete a task, and trim every detail that doesn’t contribute.

GNOME Sample Scenarios

In the previous post I made up a fictional GNOME user Hannah. She is 38 years old and is an average user. Now let’s consider a few scenarios in which Hannah would use GNOME:

Scenario 1
Hannah works as an instructional designer, she develops engaging face-to-face and eLearning content. She also leads online workshops on learning technology for school teachers and instructional designers. Thus, at work Hannah often has to prepare and deliver presentations.

Scenario 2
Hannah has enthusiasm for her work and is continuously working on development of her skills. She is going to attend the annual conference in London next week. After the conference she plans to stay in London for another couple of days, and needs to know the weather forecast.

Scenario 3
Hannah often communicates with her friends Amber and Zoe. Amber lives in San Francisco, and Zoe travels a lot and lives in different parts of the world. Hannah wants to know current local time and date in different cities and countries to choose time for meetings with her friends.


These are the examples of why someone might use GNOME. In order to accomplish her goals, Hannah would have to use LibreOffice applications, clock, and weather.


In the beginning of design process we build user understanding, and use personas to define who our users are. Next step is to answer the question why they use our product. Here is where scenarios appear.

Scenarios are hypothetical stories that work together with personas. They describe the context behind why the particular persona would use our product or come to our website. 


Scenarios give insight into the user’s motivations and goals while using a system, and once we know their expectations, it becomes easier to make users satisfied.

How to use scenarios

Use scenarios during design to ensure that all participants understand and agree to the design parameters, and to specify exactly what interactions the system must support. Translate scenarios into tasks for conducting walk-through activities and usability tests.