MS 100 College Algebra item analysis

The following table reports an item analysis for MS College Algebra final examinations from three instructors as aggregated by student learning outcomes on the outline. The first column references the outline. "1_" refers to final exam items that met course level outcome 1 but were not easily assigned to a specific learning outcome (1a, 1b, etc.). The fac columns represent averages for a given faculty member, the names were stripped out to keep the data anonymous as per Dr. Mary Allen's advice in her books. The z column is not a true z but rather represents standard errors away from the mean. Cond refers to whether the average might be considered low, high, or ordinary - not unusually high or low. The distribution of averages was uniform, not at all a normal distribution, hence the large number of items that were either high or low. Performance simply varied widely on items.

The upshot is that we now know where students tend to be weaker and where they tend to be stronger. We also have a value of 48% that can be reported, on aggregate, for the mathematics program learning outcome.

That 48% reminds me of the following chart, presented at the November 25 mathematics meeting:

The chart shows results for students passing (D or better) or being eligible for the next course (C or better) for all students spring 2004 to fall 2005. Note the pass rate for MS 100 over this period is around 52%. Half of the students (48%) are able to correctly answer final examination questions (on aggregate) and half of the students (52%) have passed the course on a historical basis. Qualified faculty implementing courses based on student learning outcomes produce trusted grades. Grades have meaning. Where qualified instructors implement courses based on student learning outcome, grades can be used as a valid assessment of student learning - on aggregate.

This is not to say we can dispense with formative assessment - the data above provides each of us who teaches MS 100 useful information on where students tend to succeed or fail. This is valuable. On aggregate, however, grades also have a learning meaning.

The following is a table where each row is a question on a final. The table includes the instructor's original description and the outline item I assigned to that item. For the benefit of those who might wish to quibble with my choices, the table below could be copied to a spreadsheet, changes could be made, and the result could be re-pivoted to produce a variation of the above.