Measuring User Experience

This is the first article in a series of what I learned from a 5 day NN/G Conference in Washington D.C. - NN/G is one of the premier resources for UX designers and does a great job with their content and research. I have repurposed or reimagined much of the content below from the slides and knowledge I gained by attending the conference. I wanted to create these articles to help myself commit the information to memory, be a resource for those around me, and to encourage anyone considering attending the NN/G Conference to do so!

Topics Covered:

• Formative vs. Summative
• Qualitative vs. Quantitative
• Usability Testing
• Tasks
• Metric Scales

---

Formative vs. Summative

	Formative	Summative
What	Iterative Make incremental changes and recommendations	Evaluate against a set of criteria Compare with other designs
When	Early in the design process	When the design is finalized
Method	Qualitative user testing Expert reviews Not quantitative (too expensive and formal)	Quantitative user testing Can be qualitative
Possible Outcome	Improve the design	Funding for a new redesign project

Formative: User research informs how the design will evolve, during the design process

Summative: User research describes how a complete design performs

Formative and Summative are evaluation methods while quantitative and qualitative are research methods

 

 

---

Qualitative vs. Quantitative

Quantitative	Qualitative
Answers: How many? How much?	Answers: Why? How can we fix it?
Metrics based	Observation based
Differences between conditions are statistically meaningful	Differences between conditions are subjective
Requires larger samples (30-70 or more)	Can use smaller samples (5-15 or so)
Usually more expensive	Usually less expensive

Qualitative research methods: Focus groups and interviews

Quantitative research methods: A/B or multivariate testing and tree tests.

Quantitative is effective when comparing a redesigned product to an earlier version. We can gather data from a baseline test during the initial summative phase of a design cycle and then run a similar or identical test at the closing summative phase of a design cycle and compare the success/failure metrics. Quantitative has an advantage over potential disadvantages from qualitative methods such as:

• Stakeholder objections to small sample sizes
• Evaluator's Effect - Evaluators who use the same usabilty evaluation method detect different problems
• Observer-Expectancy Effect – Evaluators’ beliefs and expectations can bias participants in an experiment – due to more interaction between participants and facilitators

---

Usability Testing

There are two groups of usability tests, remote and in-person. Within remote testing there is another split, moderated and unmoderated. In-person and remote moderated tests help testers acquire more data for their research. Especially during in-person tests the facilitator (that's you) can ask better follow-up questions, remind participants to think aloud, and encourage the continuation of open ended comments from the user. Why do we even do remote unmoderated testing? It's cheaper and allows us to collect data from larger sample sizes faster. It is important to alway pilot a remote unmoderated testing to make sure your questions and tasks make sense.

Tools I use for testing and usability analysis:

• Get Site Control
• Optimal Workshop
• Morae
• Google Analytics

---

Tasks

When writing tasks it can be helpful to build an affinity map to find the best tasks for what you are trying to accomplish with your test. Brainstorm to identify all possible tasks for the question you are trying to answer. Write all your ideas on sticky notes, cluster similar ideas, define the cluster, and dot vote for the best tasks.