Carnegie Learning Assistment Interface

Design Document - teacher intents

Step 1 Step 2 Step 3 Step 4 Step 5

Step 4: What are these skills? Who is missing these skills?

Teacher Goals 1. Understand what the student is struggling with
2. See progress over time to assign grade
3. See how student could improve
4. Identify who is struggling and figure out where they are having trouble.

click to expand 4.1 Skills are better understood in the context of a question. Teachers support which is seeing the actual questions in order to see which skills exactly students are having trouble.
Helps communicating skills when presented in the context of questions / presented through suggestions for remedy. Teachers need to know students' performance on individual mathematical skills		 [must have]
Design Ideas:
1. Present the actual questions that the students are having trouble with in main view/skill report
2. Provide link to identify for teachers what skills/steps students are failing on as part of a problem --> learning trajectory view
3. Provide view of how students step through each scaffold within problem -- we realize this in our prototype when teachers click on a particular question you can view the steps through it.		 Supporting User Data:
U1 & Inference from data in Oakland user studies: Some teachers didn't find skills identified helpful for remediation "Doesn't tell us the real story" unless seeing how they do in problems teachers still find it really helpful to watch kids do the problems. They rely on that interaction to inform remediation.
Inference from data: show me problems in each skill so I can use them or combine them to create my warm ups, show me how well they know terminology, show me/give me a way to find out what the students steps are through a problem.
U3:Tell me that students are struggling with XYZ, then give me 3 sample problems that will prove that.
U10: Do the Carnegie learning skills group into the “big ideas”? --> need to present skills in context of standards and questions
U14: want % of students that failed the question than just individual skill [artifact4]
U20: (seeing the skills visualized with the question) Helpful but seeing the actual problem helps better [artifact4]
U20: (looking at other related skills within question) These are great because this breaks down where the students are having trouble in the question. [artifact4]
U16: I like that I can see the problems given to the students. [artifact4]
U14 this seems to be telling me the problem, what steps they should follow, what they get suck on and what to practice.
U14 Sometimes I assume the kids know all the steps, I would use this to remind me what all the steps are. This breaks it down and this is what the kids need to do.
U15 visual is good rather than a bunch of #s
U15 I really like how it shows each skill
U15 this definitely tells me what skills are working and what aren't
U20 I like the setup, if anyone breaks it down into how students should solve it I do want to know the route; would be good to have one of these for each section (then I would assign a worksheet on the skill)
[Teachers look at how individual students step through problems, and values personal interaction in gauging performance, but this takes way too much time]
U3: Teachers have no means or time to gather results about what skills students are missing on standards practice. It's like a separate full time job to take cognitive tutor reports and translate these to the kids --->need to put the skills in context.
U9: Wants to see how students are stepping through the problem (show problem steps... ) [ 2.1, 4.2]
U2: Uses "Speak method" to teach: where students verbalize "in this problem I know... and I will... then I will.. " step by step
U6: I look at their homework sheet to see how they do in actual problems
U9: Watches students do the steps on the board[U9 2.1]
U3: Teachers value personal interaction with student in assessing progress
U14: (Looking at the Sometimes I assume the kids know all the steps, I would use this to remind me what all the steps are. This breaks it down and this is what the kids need to do.
U15: Looking at the learning trajectory prototype: I really like how it shows each skill
U24: In learning trajectory view, it took teacher 21seconds to quickly & accurately identify what the most difficult steps was in the context of the scaffolding (25:09 through 25:30) 21 seconds (Success!!!)
U26: Love the fact that I can see scaffolding questions for items
U30: talking about the step within the problem - they mess up 1 thing. If you can find the 1 thing and fix it. talking about skills ONLY within the actually problem not at the abstract level
U32: will give them a similar problem to asses where they are getting it wrong (9:43)
U32: "it definitely gives you a hint as to what stage you lost most of the kids, and that's what I'd be looking for as a teacher" (17:40)
U32: recognizes skills as broken down parts of questions and says that's the level she wants to know how students are doing at (0:46:10)
U32: really liked question breakdown with how many got each scaffold wrong (0:51:40)
U33: "I like the scaffolding question so you can see where in the process the students are having difficulties or how far in they need to go with the cues or pointers" (0:17:50)
U1: teachers still find it really helpful to watch kids do the problems. They rely on that interaction to inform remediation. [4.2, 4.3.1]
U9: have students up on the board and solve questions, and watch how they do it.
U33: question broken down is wonderful cuz its what they already do one on one, but now is for all the class (0:57:30)
click to expand 4.2 Teachers need to identify which students would fail, and further see individual information on the poor performers [must have]
Design Ideas:
see requirement 4.3		 Supporting User Data:
U3: Uses reports to determine who is falling behind in lessons
U3: If you can identify what skill student struggles with, doing more math isn’t the problem—they need to pull them out and give individual attention. Some of the students problems arise because they have learned how to avoid intervention/slip through the cracks
U10: (teacher as researcher) watches kids over their shoulder to figure out their understanding of the big ideas — wishes he had more time to do this with students
U10: Teacher feels that class as a whole is doing ok, so he tries to spend more time on the kids who are struggling, i.e. wants to see that individual information on poor performers
click to expand 4.3 Teachers need to see how ALL the students are doing *per skill* and sort/group them by best-to-worst or worst-to-best, mid-to-best, mid-to-worst, and by different "common denominators" - by standards, skills, overall performance, etc. This is needed for remedy at class level and planning lessons/warm-ups. [must have]
Design Ideas:
1. show plot of students' performance per skill. (see final prototype)
2. show plot of students' performance per standard. (see final prototype)
3. In addition to showing numbers, visually show which category the students fall under - advanced/proficient/needs improvement/failing
4. enable teachers to SORT by performance- mid to low, low to high, high to low, etc. so they can remedy by pairing up students or grouping students for group projects (see future recommendations)		 Supporting User Data:
U26: "If I can group students by their deficiencies or strengths & deficiencies
U24 - 24:00 looking at grouping, higher failing, and lower failing as a way to group students
U26: I would use this by sorting by ability on skill for group work to pair strong & weak students - 11:25
U27: On grouping – I would filter/sort by characteristic like second language learner
U27: Clustering of students, giving students who are advanced more advanced work
U28: Group well-performing kids & give them a role in helping teaching (leadership roles) -01:19:54
U30 -students learn when work in groups, to learn the skills from a practical application problem
U30: remediation time outside of class is unreliable. So In class remediation is factored in to schedule, with computer time.
[View which category students fall in]
U3: Uses reports to determine who is falling behind in lessons
U26: wants graph of each student's performance to also be rated in the 4 categories "proficient" etc - 17:55
U27: what range does the student fall in: warning, needs improvement - 1:03:05 State goal: to get everyone proficient - what percent do they need to be proficient is different every year, percentage of questions changes every year 1:03:05
U33: sees that only a few students are having trouble on the standard (via the students performance graph) and would need individual help (0:16:24)
U33: "right before the exam, I'd be happy to see the number of topics that are covered as advanced or proficient, and I would definitely go into the 'kill-and-drill' on the needs improvement or marginal, and do a lot of directed practice towards those, which is actually this year what we did, we spent about the week before the assessments working on open-ended questions trying to improve the scoring, teaching them the techniques, to write them out, because we had done poorly on those" (0:44:8)
U33: question broken down is wonderful cuz its what they already do one on one, but now is for all the class (0:57:30)
click to expand 4.4 Teachers need to see how EACH individual student is doing in *all skills/all standards* This is needed for individual diagnosis of/interaction with student. [must have]
Design Ideas:
1. Provide individual view with performance on all standards/all skills and highlight the ones they are under the proficiency category.
2. show previous year MCAS scores to support teachers in assessing if student has the basic concepts but is struggling with recent learned ones, or has fallen behind years ago		 Supporting User Data:
Competitive Analysis: Highlight assumed student mathematical background, and show if the student actually has these skills by referencing previous assistments results for that student [ENC CA]
U3: Poor performers slip through the cracks: Some of the students problems arise because they have learned how to avoid intervention/slip through the cracks
U10: Teacher feels that class as a whole is doing ok, so he tries to spend more time on the kids who are struggling, i.e. wants to see that individual information on poor performers
U30: in votech school, keeps portfolio on each student, go look at the file absentees, exam reports, examples of work. Let students keep some examples. Keep records of students who may leave come back. 25:40
U32: "if I had a more comprehensive view of… breaking it down per kid" - choose each one's worst 10 standards and practice those (0:32:35)
U32: "but I think that having - clicking on a kid and having the kid's overall would be like, a very nice option, so that I won't have to click there, click there… I get some basic information about individual kids from one screen" (0:46:45)
U32: "I usually look at two main things: the overall, the mean, the general spread of the class, and then I also look at individual kids" (0:48:6)
U33: individual breakdown lacking, esp. scaffolding for only one student (0:59:5)
click to expand 4.5 Teachers need to see special groups of students separately [must have]
Design Ideas:
[not fully implemented. in future recommendations]
1. Allow teachers to sort by different things
2. Teacher can select students from a roster and drag their names into a screen, then name that group as something.
3. All the presentation of data will be OF the group of students
4. Then they could view that group later on & view their progress over time		 Supporting User Data:
U27: On grouping - she filters/sorts by characteristic like second language learner
U27: District & comparing classes
click to expand 4.6 System needs to provide explanation of where kids are failing and support ways for teachers to understand through targeted interaction. Support teachers' mental model of personal interaction with student [must have]
Design Ideas:
1. Provide task analysis or give teachers a way to see how students step through problem from student observation
2. Assistments should have a way to show the questions & what answer students chose so teachers can get an idea of what students may have done wrong
3. Identify for teachers what what skills/steps students are failing on as part of a problem [U1 4.3.2] 		Supporting User Data:
U1: Teachers didn't find skills identified helpful for remediation [S, U1 4.2, 4.3.1] "doesn't tell us the real story" teachers still find it really helpful to watch kids do the problems. They rely on that interaction to inform remediation.
U3: Credibility/trust of computerized system is important: "Make me think this is something I've done before or that it's something the computer can do better" [U3]
U32: disagreeing with scaffolding - different teachers have different ideas (0:19:45)
U33: use whether or not the student asked for a hint or definition to further pinpoint which skill they lacked that made them get the scaffold wrong (0:33:15)
About Us | HCI Masters Project 2004