UX Test Results: Part I

This is the first part of analysis of the recently performed GNOME user experience test. The test was conducted by myself (Diana) with the help of volunteer testers with the purpose to examine user experience of a person’s first exposure to GNOME. In this post I will describe the test methods, and in the next will share the results.

Demographics

Seven volunteer testers participated in the test. The participants were between 13 and 35 years of age. Every participant claimed to use the internet daily.

demography (2).png

Two out of seven participants claimed being familiar to GNOME. Both testers used GNOME at the computer lab when studying in the university — one of the testers used GNOME 2.x between 2004-2008, and another one used GNOME 3.x between 2010-2014. The testers didn’t have much experience with GNOME, and the exact versions that were used and dates are not known. 

Methods

Testers were provided a laptop running GNOME 3.20 on Fedora 24 operating system, without any modifications that could possibly affect the participants overall experience during the test. They executed the test separately, using identical settings. Each participant used a separate guest account and a pre-loaded USB fob drive with sample files needed to complete the tasks — a bunch of music files, photos, and documents in different formats. The test was conducting using the think aloud protocol. The responses were recorded for note taking purposes.

Each session started with an introduction:

You have been asked to participate in a “first experience” test for the GNOME desktop. Just like Windows is a desktop and MacOS is a desktop, GNOME is a free software desktop. If you have any further questions about GNOME I’d be happy to try and answer them.

For this test, we are interested in what people think of their first experience with GNOME. We are looking for people who haven’t used GNOME before, so we don’t expect that have used GNOME before today. We want to see what you think of GNOME when you use it for the first time.

This is entirely a test of GNOME. We are not testing you and there is no wrong answer, so please do not feel pressured by time or anything else. All we’re looking for is what you think about your first experience with GNOME.

For this first experience, I’ll ask you to login using a test account. I’ll give you some time to experiment with GNOME. Use it like you would use a new computer for the first time. To help guide you, I’ll ask you to do a few sample tasks that mimic how most people would probably use a new computer.

After the introduction the volunteers were asked how they would likely use a computer for the first time:

Before we begin, I’d like to learn a few things about how you would use a new computer:

Let’s say at work or at home you have a new computer with pre-installed operating system which is new for you.

You have booted this computer for the first time. How would you use it at first?

This interview was meant to do two things: get the tester into the mind of someone who is booting a new computer for the first time, and help to get an idea for what the tester would do with a new computer to guide them during the test.

After the introduction each tester had an opportunity to poke around and explore the system. The participants changed settings, desktop wallpaper, and looked through the pre-installed applications.

Then, to help the testers explore the system, they got three tasks based on how people tend to use their computer for the first time:

Task 1: Managing files

You’ve booted a new computer for the first time. Let’s say this USB fob drive has files from your old computer. Please copy the files to the new computer. Put them wherever makes sense to you.

Task 2: Using a browser

After you used your new computer for a while, you want to browse the internet and open some of the sites you visit more frequently. Please open a few websites that you would normally visit, like Google or Facebook.

Task 3: Checking email

After you start up your new computer, you want to check your email. Go ahead and check your email. I’ll delete everything after we’re done today, and you are the only person who will use this account, so please access your email however you normally do it at home.

Participants accomplished the tasks without difficulties, and all them completed the tasks in less than 40 minutes.

Wrap-up Interview

Directly after the test tasks, I asked the participants the following questions:

What things were really easy to figure out?

What things were harder to figure out? Why?

Can you summarize your first experience today in a single word, like an adjective? What one word describes the test today?

Who can you imagine using GNOME?
Do you imagine men or women prefer it?
Old or young?
What kind of jobs do you think they might have?

If you had the choice of choosing this system over your current one, which would you choose? Why?

Do you think it’s attractive?

The testers were also asked to describe their reactions to using GNOME for the first time with an emoji:

Think back to the start of the test today. If you had to pick one emoticon (from this list) to describe the first time you used GNOME today, what emoticon would that be? What emotion does that represent to you?

After you got settled into GNOME, and had played around with it for a while, what emoticon would you use to describe that part of the experience? What emotion does that represent to you?

emoji - Copy2

In the end of this post I would like to show you how the testers responded to the last two questions about their first experience with GNOME.

emojis_new.png

If we look into the shift in user experience to GNOME for those testers who used GNOME for the first time in this test, we will see that:

#1 went from “curious” to “happy”;
#2 went from “curious” to “hmm”;
#5 went from “surprised” to “meh”;
#6 went from “meh” to “sad”;
#7 was “meh” in the start and didn’t change. 

Interestingly, when I asked testers what the last chosen emoji represented to them, tester #2 said that it was “thinking, hesitating”, and tester #5 said “difficulty to shift from the system I am used to”. Tester #1 said it was “easy and interesting”.

Testers who previously had some experience with GNOME, were optimistic about their experience:

#3 went from “hmm” to “happy”;
#4 went from “hmm” to “happy”.

In the next post I will share results of the UX test in more detail and suggest a few recommendations for GNOME based on what I observed in the test.

 

Advertisements

One thought on “UX Test Results: Part I

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s