UX Test Results: Part II

In the previous post I described the test methods of user experience test. In this part I share the test results and my recommendations based on them.

One of the main goals of the UX test is to capture how people engage with GNOME when they interact with it for the first time. After conducting the test with 7 participants I noticed several patterns of users behavior that I would like to mention in the beginning.

During the warm-up period the volunteers had some time to explore the system. There were a few most popular things people did after the computer was launched:

  1. Launch a browser.
  2. Change settings to personalize the computer.
  3. Look through pre-installed applications.
  4. Launch “Software” to check out available applications and search to download browsers users normally use.

After this, participants started doing the scenario tasks. Each participant completed all three tasks: Managing files, Using a browser, and Checking email.

What was hard for users

User onboarding

I consider welcome setup to be the stage of user onboarding — that’s the first thing people see when a new computer starts. After each tester logged in to a fresh test account, they had to follow a number of steps — select language, keyboard layout, privacy settings, and connect online accounts. For most users it was difficult to make decisions in each step, they either didn’t understand clearly where it would lead, or they didn’t know the difference between the available options.

gnome_setup

gnome_setup2 So during the initial setup users skipped all the steps that could be skipped.

Using email client

The biggest difficulties users had with the Evolution, the email client. Firstly, most users had issues with the mail authentication. After submitting the user name and password, they didn’t move forward, instead the same mail authentication request window appeared again.

gnome_mail1.jpg

No notifications of the origin of the problem or recommendations occurred on the screen, and users supposed the problem was with the password they entered, so they tried different passwords. After several attempts to log in they became annoyed because were unsure if the application worked at all.

Besides, wrong password wasn’t always a reason of the issue, for a couple of testers who had Gmail account Gmail didn’t allow Evolution to access their accounts. Testers got to know it from notification that came to their email.

When finally logged in, the testers claimed that the interface of Evolution was quite outdated. After checking email and exploring Evolution most testers found it uncomfortable, some looked through settings and said it was “too much”. One of the testers created a note, but after saving it he couldn’t find it.

6e2c0a7e47

In general, the testers didn’t express desire to use Evolution. It is worth noting that all the participants normally use web email client and none of them ever had to use any desktop client (but a few tried and didn’t like it).

What was easy for users

Managing files

The participants had no problems with managing files, as the interface is intuitive and similar to the systems that they usually use. They could easily navigate through the file system, copy files from the USB drive to the computer, create and delete folders.

Using a browser

gnome_browser

All participants considered using GNOME Web very simple. The testers managed to open a few websites that they normally visit without any difficulties. However, some users commented that the browser lacks some liveliness and a colorful theme.

Follow-Up Questions

Directly after each UX test, I asked the participants the following questions. Not every participant had a response to every questions, and some overlapped, but here are the general responses:

Who can you imagine using GNOME?

IT specialists that use Unix systems.

Young people, as they tend to learn new things faster.

Anyone, as it can fit the needs of people of various occupations, age and interests.

If you had the choice of choosing this system over your current one, which would you choose? Why?

I am happy with my current system, and now I don’t have a reason to switch to this one. It doesn’t have any obvious advantages for me.

As a person who doesn’t like adopting and getting used to new systems, I wouldn’t like to use GNOME.

If I had to switch to this system (for example at work), I wouldn’t be upset.

Do you think it’s attractive?

The system is attractive enough, but it’s too monochrome and lacks colors.

I would like the design style to be more modern. It feels out of date and clumsy for me.

I use Unix, so Fedora with GNOME solves the tasks I normally do. The interface design is not very different, and the differences are not critical for me.

The design is understandable and looks good enough.

Recommendations

There are a few recommendations for GNOME based on what I observed in the test:

Design style

The most often testers commented on the design style. As Windows and Mac users, they are used to more colorful and comfortable user interface, so the UI of GNOME seemed dull and outdated to them. The users claimed the main reason why they wouldn’t choose GNOME over their current system was the design style, so I would recommend to modernize it. Differentiation in design can give GNOME more competitive advantages over other systems and help acquire users for which design is one of the top priorities.

The first launch of a computer

I recommend to reduce number of steps in welcome wizard. Language settings are must have, but privacy settings and connecting online account makes people hesitate and don’t give much value on this stage of engagement. Users do want to personalize the computer, but they are happier when they are not forced to do it.

Passwords

There are two scenarios connected with passwords that make users confused:

  1. When setting a new password, the system doesn’t accept “low-security” passwords, and only replies with “Sorry, that didn’t work. Please try again”.
  2. When a user enters wrong password, a form reloads without any notifications and disclaimers.

In both situations it is hard to understand what’s the problem and how to fix it. My recommendation is to clarify requirements for a password.

Keyboard input language

For people who communicate in several languages it is important to see the selected input language and to switch it easily. Windows and Mac users are used to seeing the language indicator in the taskbar, and they find it complicated to follow a number of steps in settings to do such a simple thing.
Place the input language settings where people would normally look for it.

UX Test Results: Part I

This is the first part of analysis of the recently performed GNOME user experience test. The test was conducted by myself (Diana) with the help of volunteer testers with the purpose to examine user experience of a person’s first exposure to GNOME. In this post I will describe the test methods, and in the next will share the results.

Demographics

Seven volunteer testers participated in the test. The participants were between 13 and 35 years of age. Every participant claimed to use the internet daily.

demography (2).png

Two out of seven participants claimed being familiar to GNOME. Both testers used GNOME at the computer lab when studying in the university — one of the testers used GNOME 2.x between 2004-2008, and another one used GNOME 3.x between 2010-2014. The testers didn’t have much experience with GNOME, and the exact versions that were used and dates are not known. 

Methods

Testers were provided a laptop running GNOME 3.20 on Fedora 24 operating system, without any modifications that could possibly affect the participants overall experience during the test. They executed the test separately, using identical settings. Each participant used a separate guest account and a pre-loaded USB fob drive with sample files needed to complete the tasks — a bunch of music files, photos, and documents in different formats. The test was conducting using the think aloud protocol. The responses were recorded for note taking purposes.

Each session started with an introduction:

You have been asked to participate in a “first experience” test for the GNOME desktop. Just like Windows is a desktop and MacOS is a desktop, GNOME is a free software desktop. If you have any further questions about GNOME I’d be happy to try and answer them.

For this test, we are interested in what people think of their first experience with GNOME. We are looking for people who haven’t used GNOME before, so we don’t expect that have used GNOME before today. We want to see what you think of GNOME when you use it for the first time.

This is entirely a test of GNOME. We are not testing you and there is no wrong answer, so please do not feel pressured by time or anything else. All we’re looking for is what you think about your first experience with GNOME.

For this first experience, I’ll ask you to login using a test account. I’ll give you some time to experiment with GNOME. Use it like you would use a new computer for the first time. To help guide you, I’ll ask you to do a few sample tasks that mimic how most people would probably use a new computer.

After the introduction the volunteers were asked how they would likely use a computer for the first time:

Before we begin, I’d like to learn a few things about how you would use a new computer:

Let’s say at work or at home you have a new computer with pre-installed operating system which is new for you.

You have booted this computer for the first time. How would you use it at first?

This interview was meant to do two things: get the tester into the mind of someone who is booting a new computer for the first time, and help to get an idea for what the tester would do with a new computer to guide them during the test.

After the introduction each tester had an opportunity to poke around and explore the system. The participants changed settings, desktop wallpaper, and looked through the pre-installed applications.

Then, to help the testers explore the system, they got three tasks based on how people tend to use their computer for the first time:

Task 1: Managing files

You’ve booted a new computer for the first time. Let’s say this USB fob drive has files from your old computer. Please copy the files to the new computer. Put them wherever makes sense to you.

Task 2: Using a browser

After you used your new computer for a while, you want to browse the internet and open some of the sites you visit more frequently. Please open a few websites that you would normally visit, like Google or Facebook.

Task 3: Checking email

After you start up your new computer, you want to check your email. Go ahead and check your email. I’ll delete everything after we’re done today, and you are the only person who will use this account, so please access your email however you normally do it at home.

Participants accomplished the tasks without difficulties, and all them completed the tasks in less than 40 minutes.

Wrap-up Interview

Directly after the test tasks, I asked the participants the following questions:

What things were really easy to figure out?

What things were harder to figure out? Why?

Can you summarize your first experience today in a single word, like an adjective? What one word describes the test today?

Who can you imagine using GNOME?
Do you imagine men or women prefer it?
Old or young?
What kind of jobs do you think they might have?

If you had the choice of choosing this system over your current one, which would you choose? Why?

Do you think it’s attractive?

The testers were also asked to describe their reactions to using GNOME for the first time with an emoji:

Think back to the start of the test today. If you had to pick one emoticon (from this list) to describe the first time you used GNOME today, what emoticon would that be? What emotion does that represent to you?

After you got settled into GNOME, and had played around with it for a while, what emoticon would you use to describe that part of the experience? What emotion does that represent to you?

emoji - Copy2

In the end of this post I would like to show you how the testers responded to the last two questions about their first experience with GNOME.

emojis_new.png

If we look into the shift in user experience to GNOME for those testers who used GNOME for the first time in this test, we will see that:

#1 went from “curious” to “happy”;
#2 went from “curious” to “hmm”;
#5 went from “surprised” to “meh”;
#6 went from “meh” to “sad”;
#7 was “meh” in the start and didn’t change. 

Interestingly, when I asked testers what the last chosen emoji represented to them, tester #2 said that it was “thinking, hesitating”, and tester #5 said “difficulty to shift from the system I am used to”. Tester #1 said it was “easy and interesting”.

Testers who previously had some experience with GNOME, were optimistic about their experience:

#3 went from “hmm” to “happy”;
#4 went from “hmm” to “happy”.

In the next post I will share results of the UX test in more detail and suggest a few recommendations for GNOME based on what I observed in the test.

 

First Impressions of UX test

This week has been great, I’ve finally completed my 7 persons user experience test, hurray! In this post I’d like to share my first impressions and briefly describe what went well during the test and some difficulties I had to face.

Test had a total of 7 participants, which is not many, but enough to get information on how people perceive and relate to the product and uncover important usability problems.

There is an article by Nielsen that explains how many test users we need in a usability study, it provides a mathematical formula and a chart:

20000319-user-testing-diminshing-returns-curve
Nielsen assumes that after the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new.

Frankly, I would be happy to test more users, as the process is really exciting and fun, and the testing results are surprising, but last month I moved to a new city and even appointing seven tests was a little challenging. It also took a little bit longer than I expected (people are busy in general and it’s not easy to find them available).

I had the chance to welcome volunteers from different fields, with different computer expertise and expectations. I learned that person’s background matters, and it was very interesting to observe how behavior of a thirteen-year-old differed from behavior of adults, and how a designer’s perception differed from programmer’s and manager’s.

It appeared that conducting UX tests is not as easy as it might seem, we must be really well prepared. And it’s essential to do a dry run of the text by yourself — in my case I had to do it twice before I made sure everything would work properly during the test session.

The participants didn’t hesitate to speak their mind when they were confused or something bothered them during the test, which made me happy, because they said really useful and sometimes unexpected for me things.

Unlike the usability test that Renata and Ciarrai performed, the UX test didn’t require participants to do “get down to work” kind of tasks. Rather, I allowed each participant to explore the system, and pick some tasks, based on what they tend to use their computer for.

After the testers interacted with GNOME and its applications, I asked them some questions about their experience, which you can find in my previous post. I also added more questions aimed at understanding and meaning, suggested by Alan:

Who can you imagine using this [GNOME]?
Do you imagine men or women prefer it?
Old or young?
What kind of jobs do you think they might have?
If you had the choice of choosing this system over your current one, which would you choose? Why?
Do you think it’s attractive?

In general the tests ran smoothly, and lasted for about 35 minutes each. I did audio recording during most of the test sessions.

I am pleased with how the tests went, and the volunteers seemed to be pleased and enthusiastic too! Most of them found that they liked GNOME and indicated interest in using it in the future.

In the next post I will write the analysis and get into details of the tests. I’ll also present an analysis of the tester’s engagement, using the emoji-with-counts method.

Stay tuned!

 

Preparing for User Experience Testing

For the last few weeks I’ve been preparing for the user experience testing. In this post I share the final version of my usability test: the opening script, scenario tasks, pre-test and wrap-up interview.

Pre-test Interview

I plan to start each session with a brief welcoming part that explains to participant how the test is going to work. Below is the current version of the script that I’m going to read during the introductory part of the test.

You have been asked to participate in a “first experience” test for the GNOME desktop. Just like Windows is a desktop and MacOS is a desktop, GNOME is a free software desktop. If you have any further questions about GNOME I’d be happy to try and answer them.

For this test, we are interested in what people think of their first experience with GNOME. We are looking for people who haven’t used GNOME before, so we don’t expect that have used GNOME before today. We want to see what you think of GNOME when you use it for the first time.

This is entirely a test of GNOME. We are not testing you and there is no wrong answer, so please do not feel pressured by time or anything else. All we’re looking for is what you think about your first experience with GNOME.

For this first experience, I’ll ask you to login using a test account. I’ll give you some time to experiment with GNOME. Use it like you would use a new computer for the first time. To help guide you, I’ll ask you to do a few sample tasks that mimic how most people would probably use a new computer.

But before we begin, I’d like to learn a few things about how you would use a new computer:

Let’s say at work or at home you have a new computer with pre-installed operating system which is new for you.

You have booted this computer for the first time. How would you use it at first?

Scenario Tasks

Scenario tasks are the main part of the test. For each task I’ll give a participant a copy of the scenario and read it aloud. In some tasks I’ll give additional information, like login credentials, or a USB drive with sample folders and files.

First I’ll start with a short intro:

Okay, let’s have you use the computer now. Here’s a username and a password you can use to login. No one else has used this account before, and no one else will use it when you’re done here today. (I’ll delete any data you leave behind when we’re done.) To help you explore the system, here are a few tasks that we think most people would do with a new computer.

Then I’ll give a participant some context of each task:

Task 1: Managing files

You’ve booted a new computer for the first time. Let’s say this USB fob drive has files from your old computer. Please copy the files to the new computer. Put them wherever makes sense to you.

Task 2: Using a browser

After you used your new computer for a while, you want to browse the internet and open some of the sites you visit more frequently. Please open a few websites that you would normally visit, like Google or Facebook.

Task 3: Checking email

After you start up your new computer, you want to check your email. Go ahead and check your email. I’ll delete everything after we’re done today, and you are the only person who will use this account, so please access your email however you normally do it at home.

Wrap-up Interview

I have some questions prepared for the wrapping up part of the test to help me get the participants’ general impressions of the experience:

What things were really easy to figure out?

What things were harder to figure out? Why?

Can you summarize your first experience today in a single word, like an adjective? What one word describes the test today?

Jim suggested an interesting method — asking testers to summarize different parts of their experience using an emoji. It will help testers express their own feelings, but because we’ll provide a set of emojis (testers won’t make their own) it will be easier to collate results.

Think back to the start of the test today. If you had to pick one emoticon (from this list) to describe the first time you used GNOME today, what emoticon would that be? What emotion does that represent to you?

After you got settled into GNOME, and had played around with it for a while, what emoticon would you use to describe that part of the experience? What emotion does that represent to you?

emoji - Copy2

I’m excited to perform this UX test very soon, and I’m definitely open to your thoughts and feedback on any part of this test preparation!

 

Moving to the project phase in Outreachy

A few weeks ago we officially started the usability testing phase of the internship, and in this post I’d like to share some insights of the work that is going on. It took some time to settle into a project focus, figure out who wants to do what and start building out the usability tests. Now our team is working on three usability testing projects for GNOME:

Renata will perform a traditional usability test of other ongoing work in GNOME. In previous cycles of Outreachy, our mentor Jim Hall and the interns Sanskriti and Gina uncovered design patterns that work well and others that need improvements, and this time Renata will look at those design patterns that have been improved in recent GNOME design iterations.

Ciarrai is preparing to perform a paper prototype test of the new Settings app. This would be testing a “future” Settings design — testing on changes that haven’t even been implemented in GNOME yet.

On my side, I am getting ready to examine a “user experience” of a user’s first exposure to GNOME. I will look at product identity and user experience, rather than straight usability. In this test, I will ask testers to simulate an unboxing of a new system. The tester will turn on the computer, watch it start up, and login to a fresh test account so they get first-user experience.

The tester will experiment with the system, using a few scenarios to suggest real tasks that real people might do, so they get an opportunity to poke around and launch a few applications.

My goal is to find out how people feel about their experience: whether GNOME is something that appeals to them, feels welcoming, is something they can imagine themselves using.

With help of Jim Hall and the GNOME Design team I have created the opening script, scenario tasks, and wrap-up questions for the UX test. The final version of the test is ready and I will share it with you soon!

Reflections on Usability Tasks

Before I entered the Outreachy internship, I had to present a “first contribution” — perform a small formal usability test, and perform some basic analysis using the heat map method. In this contribution my mentor Jim asked me to find ten scenario tasks used in other GNOME usability tests, and perform my own usability test with a few testers.

Instead of using pre-written scenario tasks I decided to create my own tasks, and made a lot of mistakes in them :) So today I want to reflect on my first contribution, and write about my experience and mistakes I made in these scenario tasks.

200px-Gedit-logo-clean

One of the applications I chose for usability testing was gedit — the default text editor of the GNOME desktop environment. It is designed as a general-purpose text editor.
First I came up with a list of some general user goals that gedit users might have:

  1. Autosave the file
  2. Change the font size
  3. Correct spelling mistakes in the text
  4. Find and replace text
  5. See the document statistics

After choosing the tasks to test, I formulated scenario tasks for usability testing:

You work as a freelance copywriter. This time you are writing a book review.
G1. Your laptop battery got broken, and if the laptop suddenly goes off you might lose recent changes in your document. So before you start, make sure that all changes you make in the document will be automatically saved every 10 minutes.
G2. Make the text bigger.
G3. Correct all spelling mistakes in the text.
G4. You realize that misspelled the last name Feinman. Replace Feynman with Feinman in the whole text at once.
G5. You need your review to be less than 200 words. Check if the size of the document is fine.

Now let’s see which of these scenario tasks were well-crafted.

G1 is a good task, as it resembles a real situation in which a user would want to do the certain task — turn on autosave. The task is specific enough, and at the same time it is short and provides just as much information as a user needs to complete it.

G4 and G5 are realistic tasks too, I think people often find themselves in similar situations when using text editors. There are no task-solving cues, and the descriptions are at the right level of detail. I asked testers to do something specific, set the context and used user’s language to make the tasks clear.

All these scenario tasks worked well during the usability test, as they were well-formulated, actionable and written clearly enough that the testers knew when they completed the task.

There also were scenario tasks that I would improve in the next usability tests.

G2. Make the text bigger.
This is a very abrupt scenario task, and a tester might be confused by the task itself. There is no context, no reason or purpose for performing the task. I wanted to keep the task short, but it’s more important to provide the participants with all the information that they need to complete it.

G3. Correct all spelling mistakes in the text.
This task doesn’t set a context too. More importantly, it provides an unintended hint to the tester by re-using keywords from gedit interface. Task scenarios that include terms used in the interface bias testers’ behavior and give less useful results, and it is essential to avoid giving such clues in usability testing.

It requires time and practice to evolve skills in writing tasks. But with help of a bright mentor and some effort — we’re almost there! Soon I’ll start working on a new usability test and hopefully will write good scenario tasks that will help to uncover usability issues effectively.

 

Scenario Tasks

The most effective way of understanding what works and what doesn’t in an interface is to watch people use it. This is the essence of usability testing.

In order to assess the usability of an interface, we ask testers to carry out a number of tasks using the interface. To watch people try to use what you are building, you need to give them something to do. It’s a two-step process:

  • First you choose the tasks to test.
  • Then you expand the tasks into scenario tasks — the little scripts that set the stage for the action and provide a bit of explanation and context.

The first step is to come up with a list of general user goals that visitors to your site or application may have. Ask yourself: What are the most important things that every user must be able to accomplish on the site?

50682fe3fdc95f9784307c031f29141f

As a quick example, here’s a list of tasks for youtube:
Search for a video.
Watch a video.
Create a playlist of videos.

Once you’ve figured out what the users’ goals are, you need to formulate scenario tasks that are appropriate for usability testing.

Task: Search for a video.
Scenario task: You’ve heard from your friend that Coldplay released a new music video. You want to see it. Find the video on youtube.

A scenario task has the character, the context and the necessary details that the user needs to know, but doesn’t. Scenario tasks resemble those that users would perform in a real life context. A well-crafted scenario task will help you to uncover usability issues more effectively.

Tips for writing scenario tasks

Make the scenario task realistic.
Make sure that each scenario task is realistic and typical for how people actually use the system, when they are on their own time, doing their own activities.

Avoid giving clues.
Phrase your scenario tasks without uncommon or unique words that appear on the screen. If you do, you turn the tasks into a simple game of word-finding.

Keep the scenario task concise.
Provide the participants with the information that they need to complete a task, and trim every detail that doesn’t contribute.