Explain the Hypothesis that your Formal Study would Allow you to Test
We are measuring the extent to which onboarding instructions lessen the learning curve for new users of our interface. For each feature that is present in our interface, we intend to test whether a one-sentence explanation of that feature lessens the time it takes for the user to make use of its functionality. Our hypothesis is that including an explanation of what a feature does will decrease the time it takes for the user to figure out how to use its function on their own.
Describe the Factor(s) (Independent Variable(s)) that you will Vary
Eventually, we will vary the method of how we deliver the explanation of the feature (writing, diagram) with different control groups to see which is more effective, but for this iteration, we will test the effectiveness of a spoken explanation.
We will give one user a one-sentence explanation of how to open more details about a product and how to add to the shopping list from the explore panel, and we will give the other user no onboarding to that feature. Both users will be given the objective to find the price of Folgers Coffee. Some control variables include the screens users will move through, the explanation they receive (if given), and the interface device (laptop).
The condition that we will test is the time it takes for each user to complete the objective we give them.
What Dependent (Response) Variable(s) will you Observe?
To evaluate the efficacy of an onboarding explanation/instructions, we will measure the time, t, it takes each user to accomplish a simple shopping task. For our first study, we will measure the time it takes first-time users to find the price of Folgers Classic Roast Coffee using our design.
We will then compare the time it took each group (one that received explanation vs. no explanation) to complete the task. If the time difference is minimal, then the explanation/onboarding has little impact on the user experience, and might not be worth including. However, if the explanation test group demonstrates a short amount of time required to complete a task, then we know the necessity of an explanation/onboarding process. If the explanation group takes even longer, then we know that our explanations are even worse than no-explanation at all.
This information will help us quantifiability evaluate the quality and necessity of explanations in the usability of our design.
If your hypothesis is correct, what pattern would you expect to observe in the response variables?
If our hypothesis is correct, we expect to see a reduction in the time it takes for the user(s) to complete their task. We may provide the user(s) with multiple tasks to complete to get a better interpretation of the difficulty of the tasks compared to the competency of the users. For our first study, if we provide the user with some guidance (a scripted explanation) on how to get more details, they should find the price of the Folgers Classic Roast Coffee more quickly.
Also explain how many subjects you would ideally test, how many iterations each would participate in, and if each participant tries multiple conditions, how these conditions would be ordered and distributed over participants. Note: if your team has no prior experience in the design of controlled experiments, please indicate that in this section.
Additionally, we would test additional “tasks” with accompanying instructions. For example, putting Folger’s coffee in the shopping list using the explore menu or putting Fairlife Whole Milk in the shopping list using the search menu.
As stated briefly above, for each given task, we will eventually vary the method of how we deliver the explanation of the feature (writing, diagram) with different control groups to see which is more effective. Our results will eventually inform how we design our onboarding to convey the most important information in the most effective way.
Ideally, we would want a representative sample size to ensure that we are gathering good-quality data. We would like to test 50 people per instruction method. No participant will try multiple conditions; however, a participant will be able to test many features/objectives with the same style of instruction. Participants would be assigned to their control groups randomly.
Our team has no prior experience in the design of controlled experiments.