Interface Prototyping

We developed high-fidelity prototypes of our designs to evaluate their usability and usefulness. The prototypes were check our designs' support for those tasks that are most commonly performed by teachers and students in Algebra 1 classrooms (based on our teacher interviews and textbook reviews) as well as to give us feedback on design decisions we were unsure about. For example, we wanted to know if our users would press an Options button on a calculator to find more operations, were one available. We added an Options button to our graphing prototype to see if users interpreted it the way we expected them to.

We then implemented the functionality necessary to support the tasks and some user exploration. In cases where the user moved off the path supported by the prototype, we used a combination of the screen prototype and paper sticky notes (to simulate the unimplemented portions of the calculator screen).

Most of the prototypes we developed were interactive screen prototypes implemented in Macromedia Flash or Microsoft Visual Studio. We used paper prototypes during the initial stages and when the implementation cost didn't justify what we expected to gain from a higher-fidelity test.

User Testing Methodology

To identify problems with our designs and obtain detailed user feedback, we conducted usability studies with our prototypes. The prototypes their associated tasks are available off of each feature in the features section.

We tested our prototypes with students and teachers we recruited by cold-calling, posting up flyers, and through asking people we knew. One or two test adminstrators ran the test with a single user. On occasions where there were more users than we could handle one-on-one, we tested users in pairs. The primary duties of the test administrator were to keep notes, keep the user on task, keep the user thinking aloud, and compensate for non-functional parts of the prototype (usually by putting up stickies). If there were two test administrators, one administrator would run the test while the other took notes.

The studies themselves followed a standard think-aloud protocol, described below.

We asked the user to review and sign our statement of informed consent. In the case of minors, we asked a parent or legal guardian to sign the statement and obtained the minor’s verbal assent.
We gave the user a quick training session on thinking aloud. First, the test administrator demonstrated the protocol by adding two numbers in his or her head while thinking aloud. Next, the administrator asked the user to count the number of windows in his or her house while thinking aloud. If the user appeared to understand how to implement the protocol, we continued.
We began the actual test by allowing the user to read over the task and ask clarification questions. We observed the user as he or she worked on the task and recorded any critical incidents that occurred during the session. When possible, we recorded the screen and user’s voice using Camtasia. We did not assist the user with the task unless they had tried two to three things and were still clearly stuck on the task. In cases like this, we noted that the user could not complete the task step as a critical incident for later analysis. We defined a “critical incident” as any event that fit the following criteria:
1. The user explicitly gave up on the task.
2. The user failed to complete the task after trying two or three different approaches.
3. The user expressed anger or frustration.
4. The user did not want to continue with the task.
5. The user did not appear to be making any progress towards the task within a reasonable amount of time.
When the think-aloud was complete, we asked the user for qualitative feedback on the interface design. We asked teachers how they might integrate the design into their lesson plans and whether they would feel comfortable using the design to teach and test students. We asked students to give their opinions on the design, and occasionally asked them to describe the concept the interface was attempting to teach to see if they were able to verbalize it.
Finally, we thanked the users for their time and paid them the agreed-upon amount.

Iterative Design

After we tested a prototype with users, our team's two primary user test administrators would review the notes, videos and transcripts recorded during the user test to analyze the data. During analysis, they noted both critical incidents and places where the design succeeded, with particular attention to the hypotheses we wanted to test with the task. We had two team members review this data together so they could compare their interpretations of the problems uncovered in the user tests.

The test administrators also used the critical incidents they uncovered to identify specific problems with the interface and make design recommendations. An administrator would then discuss the data with another team member to brainstorm alternate design possibilities. Once they agreed on a solution, an implementer would revise the prototype to meet the new design specifications. See the iterations pages on each of the pages in the features section to view the design changes and the rationale behind them.

Form Prototyping

To prototype and test our form designs, we conducted 3 rounds of form studies. We began with Velcro modeling studies with 30 participants reviewing 13 forms. Our second round continued with 4 higher-fidelity forms, which we reviewed with 23 students. Finally, we developed high-fidelity wood models of 2 final concepts and obtained feedback from 13 students to help guide future improvements.

Velcro Modeling

We constructed 13 widely varying forms. We based these forms on products such as game controllers, cellular phones, and current calculators that our interviews and mall visits indicted were already used and/or desired by high school students. Each model was made of blue foam and felt, allowing participants to Velcro provided screens and buttons to the surface.

We asked participants to view and handle each form while thinking aloud about their impressions. Next they chose the three forms they liked the best and the two they liked the least and explained why they formed these impressions. Finally, they chose their favorite form and placed buttons and screens on the felt in whatever arrangement they preferred while thinking aloud.

We included both teachers and students in our studies to understand both what students wanted to own and carry with them and what teachers felt was appropriate in their classrooms.

Form Associations

We analyzed the comments participants made about our top 5 models from the Velcro modeling stage to produce 4 revised forms. We created these in pink high-density foam with the buttons pre-arranged and glued in place.

Once again, we asked the student participants to view and handle the forms while thinking aloud. Next, the students ranked the forms in order from least to most favorite and explained their reasoning for each choice.

Finally, we gave the students one series of square cards with images of people, objects, actions, and animals and one with color swatches. The images were purposefully generic enough to take on multiple meanings. We asked the students to associate the cards with the forms in whatever way they wished while thinking aloud. The color cards helped them express color preferences for the forms. The image cards helped them think about the forms in different ways so that they could express their gut reactions more descriptively.

Final Form Impressions

We constructed our final models after considering the comments from the associations study. Although we were unable to iterate further, we asked 13 students to record their impressions of the two forms after viewing and handling them by selecting their favorite and then describing the aspects of each form that they liked and disliked. We hope our conclusions will prove useful in guiding TI's future industrial design decisions.