Carnegie Learning Assistment Interface

Step 3: What skills do I need to teach in order to pass in these standards?

3.1	Skills are an effective language to communicate with teachers, and teachers DO need support in understanding the performance on skills. (to Ken's and Steve's question on whether or not skills should be visible to teachers as a means of communication)	[must have]
	Design Ideas: 1. Overall Skill Performance: Class and student performance in relation to state standards" 2. Provide context for skills/problems, provide structure for targeted teacher-student interaction --> task analysis is a good way cause teachers think of skills as steps within problem [U 1, 6] -see requirement 3.2 3. Show skills & outline common "mistakes" on the skills [U6] 4. Identify for teachers what what skills/steps students are failing on [U1 4.3.2] as part of a problem --> highlight them in learning trajectory and in requirement 3.3 5. Provide skill description – maybe just ones which weren’t passed 6. Provide task analysis and measurements/metrics of improvement for comparing classes between curriculums to aid in lesson study [U1 6.3] 7. Need to differentiate between skills that should be mastered versus skills that only require exposure --see requirement 3.4 Supporting User Data: [Teacher understands skills. Need to see the skills to improve teaching] U1, 3, 10: Need to see individual skills to improve teaching U1: Lesson study teachers perform task analysis and measurements/metrics of improvement for comparing classes between curriculums to aid in lesson study [U1 6.3] U20: Want a note about what the problem is about and where in the problem the students are having trouble with. (on the page with the actual problems) [artifact4] U3: If you can identify what skill student struggles with, doing more math isn’t the problem—they need to pull them out and give individual attention. U10: Watches kids over their shoulder to figure out their understanding of the big ideas—wishes he had more time to do this with students U3: Teacher is able to see general patterns of skills that students are struggling with (teacher as researcher) [U3] U3: Students can ace the assessment, but struggle with the same skills/concepts in class, teacher feels that this will bite on ass in the PSSA U16: I like the skill table better than standards because its broken down more U1: Skills as steps within a question: Teachers didn't find skills identified helpful for remediation "doesn't tell us the real story" teachers still find it really helpful to watch kids do the problems. They rely on that interaction to inform remediation. [U1 4.2, 4.3.1] U30: talking about the step within the problem - they mess up 1 thing. If you can find the 1 thing and fix it. talking about skills ONLY within the actually problem not at the abstract level U24:46:50 Teacher demonstrates that she understands what the skill means, by explaining it in context/example [Teachers have own process but need support in analysis] U3: Teachers have no means or time to gather results about what skills students are missing on standards practice. It's like a separate full time job to take cognitive tutor reports and translate these to the kids --->need to put the skills in context. U32: they are bad at the core concepts, the thinking is what they are good at (15:55) U32: likes skills though not sure how useful, finds them overwhelming (0:50:20) U33: doesn't believe problem is with simple subtraction skill, as showed by scaffold data, but in vocabulary (0:20:55) U33: "a lot of times you'll find that if there's several sections that they're doing poorly on, those may tie into a common theme. So rather than looking at the individual standards, you might be looking at what skills they are doing poorly on" (0:40:24) U33: "we don't tend to teach on the standards, we tend to teach on the topics [groups of skills?] cuz one of the things we did for the assessment was going from the traditional pre-algebra, algebra 1, geometry, to integrated mathematics where you're actually developing the skills more than necessarily a particular topic" (0:49:12)
3.2	Provide both the zoom in/zoom out view - Provide an overview as well as a detailed view in skills	[must have]
	Design Ideas: 1. see sequence 4 for detailed view of skills. 2. Zoom out view: main view of final prototype shows skills by performance worst to best, summary in skill view 3. Color coding to show trend Supporting User Data: U29: used color coding to quickly identify critical skills U14: This is a v very practical report. Like this the best. This tells you which still needs to be reviewed. This is what I do. (e.g. 20 of the class missed #5) Because tells me the big question: "what should I address next". and this does the job for me. Like it because don't have have to figure out stuff or go find references (U15) U15, U20 would be nice to see (percentages) for each skill
3.3	Skills need to be sorted/prioritized by different goals in different times of the year. (see the requirements 3.4 and 3.5 listed below)	[must have]
	Design Ideas: (see design ideas for requirements 3.4 and 3.5 listed below) future design: find method to calculate difficulty and factor in on skill prioritization. (U14 could tell students are doing bad by large number of hints) Supporting User Data: U20: doesn't draw me to what needs to be worked on first U28: I probably would pick 4; I can't do *a lot*- 23:54 U28: ..and group skills I can teach together or ordered what I did worst; may pick the middle, something I can improve on U28: (I like seeing skills within questions, BUT rather than me analyzing the learning trajectory and seeing which skills they missed most,) I want you guys to do the work for me…look at the steps in the questions for me, do stats for me & find out 5 things where they did the worst- 28:59 I want it done by the machine: I want it ranked for me; I don't want to have to do that U29: WOW all the little, little, little skills 45:00 - so many skills, it doesn't tell her what to do or how to help U29: Where do I start, is there any hope? Too many skills mapped – intimidating U32: "this thing is just a little too… overwhelming" (talking bout the list of skills) (0:33:21)
3.4	Support for goals at different times of year: At the beginning/ middle of the year when there is still time to teach, the prioritized list of skills should reflect the class performance in each scaffold within the questions (items)	[must have]
	Design Ideas: 1. sort by student performance in each scaffold within question 2. as the slider filtered out the standards (Kawana proto) mechanism for teachers to specify which range they want to see - be it the 4 skills that the students are doing most poorly on, the 5 skills that the students are most close to becoming proficient, or the 5 skills that the students are close to moving from failing to needs improvement. Supporting User Data: U28: (I like seeing skills within questions, BUT rather than me analyzing the learning trajectory and seeing which skills they missed most,) I want you guys to do the work for me…look at the steps in the questions for me, do stats for me & find out 5 things where they did the worst- 28:59 I want it done by the machine: I want it ranked for me; I don't want to have to do that U28: You go through these skills and rank them for me- 01:15:00 U28: philosophically, cramming doesn't make sense to me 8:00; concerned about getting my kids to proficient, etc 9:00' U30: weight different skills based on "what’s important for the test" and the students' ability level 6:30
3.5	Support for goals at different times of year: When the MCAS is close at hand, teachers need to be able to sort/view skills by how beneficial it is to address that particular skill	[must have]
	Design Ideas: 1. sort by factors "time to teach" and "importance" . However, "time to teach" estimate idea receives some resistance. Get seed data from teachers and enable other teachers to submit their own ideas on how long it will take to teach --> evolve over time. "time to teach" should be very different depending on whether it is taught the first time or as review. (Steve). "importance" should take into account ethical considerations when teachers want to see the skills that are important for the test. Supporting User Data: U28: Little pieces- just push them; where are my kids almost passing- 24:35 U28: I would pick the ones that they were getting wrong but could be learned quickly- 25:48 (less Time to teach) U28: If had 2 weeks till the MCAS: pick out those topics that are crammable; find the crammable ones; last assistments should be full of those types U29: some skills are considered "easy to learn skills” even if they are failing miserably as something good to remediate U30: weight different skills based on "what’s important for the test" and the students' ability level 6:30 U28 - in 2 weeks, I really want to know 2-3 areas that they need help in.
3.6	Differentiate between skills by different levels of expectation for performance (see also requirement 2.3)	[should have]
	Design Ideas:   Supporting User Data: U2: Need to differentiate between skills that should be mastered versus skills that only require exposure U3: Teachers should be able to modify the Assistments to test a specific subset of skills U9: Allow teachers to highlight particular skills for more information to be gathered by system (give students more problems/practice) by weighting them heavier and having them show up more often in problem line up (diagnostic, not remedy). [ 2.1] U9 phone: Makes distinction in expected performance based on level of exposure/time spent on skill in class U15: this must be what is expected? but how do they know I've covered it? U22: is this the date the mastered it by? (confusion of concept/could be just wording) U16 were they introduced to these? U31: Distrust: when seeing the skills weighed- maybe I haven’t taught it, or maybe the test isn't right U26: wanted to adjust expected mastery line in detailed view, extension of Kawana proto "because realistic expectations" U27: Sees expected level of familiarity to be tied to district expectations (protoannotations-tabby) [present with caution- make explicit] U28: She's misread expected familiarity as actual measure. (If the categories are actual measures, give me numbers to back them up.) -- present these with caution. (tabby prototype) U27 - Understood expected level of familiarity, missed the dates 1:07: U28 - expected level of familiarity… encountering it, when are they familiar? 14:00 U29 - Difficult to understand the familiarity by date, what's the connection? – DI: change to "has this been covered"; more advanced/less advanced
3.7	Order of skills need to be flexible to encompass teacher’s curriculum, skills that they’ll group to teach together, etc. Need to give teachers the control to reorder/group things around. Since curriculums are all different and situations are different having a generic ordering will not be useful.	[should have]
	Design Ideas: 1. In main view of prototype where skills are currently prioritized by performance - lowest to highest, enable sorting by chronological order --tradeoff: close to impossible to input all the different curriculum, and too much work for teacher to have them input their entire lesson plan for the year. 2. Enable teachers to move skills around or group them, so they see by chronological order aligned to their own curriculum [inspired by U33 comments] -- requires work for teachers but less work than entering their entire curriculum and gives them control. 2.1-- this could serve as a mini-planning tool, though without the timeline and not detailed [U15 at beginning of year would use to plan topics, in middle of year decide which to go back to, .U33 post testing questionnaire] Supporting User Data: U33: "I'd like to rearrange [the skills] so that I could break them down chapter by chapter, in the order that we would be teaching them" (0:10:15) U30: (looking at list of skills in main view) This is nice, but I want to sort them and rearrange them to how I want it. U30: (When seeing skills grouped by day in the lesson planner view in paper extension of prototype) I would group this this and this together, cause these skills go together.. U?: some skills build on top of each other so they can be taught together. U24: skills are often grouped for remediation/teaching sections (52:22) Solution: possibly group for remediation U26: shape lesson by going over basic skills; warm-up gives him a feel for deficiencies- 27:20 - needs to know basic skills missing first U24: “Is there a way that I specify what skills I want them to practice?” + Links to scheduling, curriculum, give teacher more control over what assistments is teaching/addressing U30: does filter out what she can teach/reteach U33: "if you're doing this weekly or monthly, to be able to select or generate smaller reports, ones on a particular topic… because this is also a flood of information, only some of which may be useful" (1:13:41)
3.8	Use explicit wording for measure when presenting performance on skills, e.g. "n out of m (x%)" This applies to standards and questions as well, for different teachers think of different measures.	[must have]
	Design Ideas:   Supporting User Data: U16: Is the number doing well or poorly? U14, U15, U20, 22: what does performance mean? also, is this 20% passing, or 20% failing? U28: is 75% good or 75% bad? U28: what is this? Percentages right or wrong? U25: strands statistics not helping; stats have nothing helpful (51:50) – DI: give 8th grade scores & show when tripped up & on what (where left behind) U29: list of reporting categories - thought they were the% of questions that fell into categories, or student performance on each standard U27: Is percentage - students got 45% of the questions right as a class. - label, make it proficiency U27: Mastery could be percent correct, number completed U29: student mastery - 12/32, is that the number of item they needed to get or how they did U32: steps correct percentage in standards view -> assistments applying is confusing (0:36:30) U32: likes the "steps correct" in standard view -> assistment now that she understands it (0:49:40) U33: wants a percentage instead of 18 out of 34, or a color coding by percent (0:33:32)