A general overview of what you observed from both informal and formal usability testing in this round.
Objective Given to Both Usability Testers Verbally
When I say start, lift the screen. Your objective is to find the price of Folger’s Coffee and tell it to us.
Extra Context Given Only to Experimental User Verbally
Our product is intended to let shoppers compare items in a store; part of that experience is finding the price of an item. If you want to see more details about any product, you can click on the product icon or name.
Control Notes (Time to Objective: 54.86s)
The user was moving the mouse around the red and green buttons. She then clicked one of the buttons. That made the coffee listing disappear (the price was now visible). She appeared to look for where to click next, because she was wandering the cursor around.. She moved the mouse to the bottom left corner and was attempting to click on the shopping list item. She then clicked on the background screen to see which options were available to click on. She then clicked on the delete button to remove the item from the shopping cart and make the listing reappear. She then noticed the price.
Experimental Notes (Time to Objective: 103.24s)
The user quickly identified the Folger’s Coffee in the corner. He proceeded to move the cursor there, but rather than clicking on the gray part, he clicked on the green icon, and clicked it again to confirm. This made the Folger’s Coffee suggestion disappear and move to the shopping cart. He then was clicking around to see how to make the Folger’s Coffee reappear (observer insight). He spent some time exploring other menus, like adding the milk to the cart.
Some thoughts this user had after the experience was that his eyes were automatically drawn to the red and green buttons, not the whole Folger’s Coffee widget. He also viewed the background image as a map and wondered if he would click on it to move around the location, like Google Maps. He suggested that it would be nice to see the prices in the cart when the item was added. After finding out how to view the price, he said it was very easy to do it again.
A specific report of the response variables on your formal usability pilot, i.e., the thing you said you would be measuring.
For the control user who received no extra “onboarding” instructions, the time to complete the above objective was 54.86s.
For the experimental user who received the extra “onboarding” instructions, the time to complete the above objective was 103.24s.
An interpretation of those response variables: if those results held with a larger user population, what would it mean for your system?
Our initial hypothesis, which was including an explanation of what a feature does will decrease the time it takes for the user to figure out how to use its function on their own, was false. The team thought it was interesting how in the control experiment the user was able to accomplish the objective faster than the experiment whereby the user had clear instructions given to them. Therefore, it would be helpful to have a larger user population to test if there were users who could accomplish the task faster with the instructions than others and assess the time difference between the fastest user and the slowest user. Furthermore, based on how the control users perform, we would be in a better position to come to a conclusion on whether having feature explanations would assist users to accomplish tasks faster or not.
Additionally, we think that the method of how we provide the extra context to the experimental users could have an impact on how quickly the users absorb the information. In future iterations of this study, we would try providing the extra onboarding instructions visually as opposed to verbally, with the expectation that a visual explanation is more effective than a verbal explanation.
An explanation of what you intend to do before the finals week session / demo day (or what else you would do if you had more time) in response to what you've seen so far. This is the most important part of the write-up, because you need to think about how you would fix your system as a result of what you observed.
Our experimental user gave us information that the yes and no buttons adjacent to the product weren’t helpful to the objective they were trying to solve. The objective we gave the user was to find the price of Folger’s Coffee and the buttons distracted him from clicking on the product description and finding the price. Our group has decided to remove the buttons and add the price of the product on the description button so a user doesn’t have to click. In addition, we are going to add tags describing the product, following this pixel format.