JIT assessment

Recalling one faculty member at the national site bogging down last fall under the weight of some 50,000 data points in a learning outcome assessment effort, I am continuing my own efforts to find a way to utilize item analysis to provide ongoing, continuous status of learning information. My goal is not a five year cycle, annual cycle, or biannual cycle, but rather a weekly refresh of information at the microlevel to provide just-in-time assessment.

I am now playing with a spreadsheet structure that keeps the workload to something more reasonable, although I do have to make a "double pass" through my tests. On the first pass I mark my papers, on the second I use hash marks to tabulate an item analysis. Then I build a table where I note the actual topic.
Source Description
P1a Evaluate x
P1b Subtract rational terms
P2 Simplify rational term
P3 Multiply binomial by quadranomial
P4 Factor quadratic
P5 Simplify rational expression
P6a Plot coordinates
P6b Plot coordinates
P6c Plot coordinates
P6d Plot coordinates
P7 Graph linear equation
P8 Solve quadratic

The above is based on the prequiz which can be seen at:
http://www.comfsm.fm/~dleeling/math/algebra/m62/html/prequiz.html

Working from the outline, I enter an outline reference code which indicates what the quiz item best matches on the outline.
Source Description Outref
P1a Evaluate x 2b
P1b Subtract rational terms 2d
P2 Simplify rational term 1_
P3 Multiply binomial by quadranomial Pre
P4 Factor quadratic 1_
P5 Simplify rational expression 1_
P6a Plot coordinates Pre
P6b Plot coordinates Pre
P6c Plot coordinates Pre
P6d Plot coordinates Pre
P7 Graph linear equation 1a
P8 Solve quadratic 1b

This is not too difficult and does not take much time. Those codes are then referenced by a VLOOKUP function, =VLOOKUP(C2;outline.A$2:B$21;2), to automatically fill-in the student learning outcomes in the fourth column:
Source Description Outref SLO Corr Corr%
P1a Evaluate x 2b Evaluate functions and find their domains. 5 63%
P1b Subtract rational terms 2d Find arithmetic combinations and compositions of functions. 7 88%
P2 Simplify rational term 1_ Graph and solve linear and quadratic equations and inequalities including those with complex roots. 6 75%
P3 Multiply binomial by quadranomial Pre [Preliminary skill] 5 63%
P4 Factor quadratic 1_ Graph and solve linear and quadratic equations and inequalities including those with complex roots. 6 75%
P5 Simplify rational expression 1_ Graph and solve linear and quadratic equations and inequalities including those with complex roots. 2 25%
P6a Plot coordinates Pre [Preliminary skill] 7 88%
P6b Plot coordinates Pre [Preliminary skill] 6 75%
P6c Plot coordinates Pre [Preliminary skill] 5 63%
P6d Plot coordinates Pre [Preliminary skill] 7 88%
P7 Graph linear equation 1a Sketch the graph of an equation 4 50%
P8 Solve quadratic 1b Solve linear, quadratic, polynomial, and radical equations. 4 50%

The fifth and sixth columns above contain the results of the item analysis itself. The vlookup function pulls its information from another tab in the workbook, the outline tab, which contains the following table:
Outref Students will be able to: Sum Count Avg
Pre [Preliminary skill] 3.75 5 0.75
1_ Graph and solve linear and quadratic equations and inequalities including those with complex roots. 1.75 3 0.58
1a Sketch the graph of an equation 0.5 1 0.5
1b Solve linear, quadratic, polynomial, and radical equations. 0.5 1 0.5
1c Perform operations with complex numbers. 0 0 0
1d Solve linear, quadratic, polynomial, and radical inequalities. 0 0 0
2_ Evaluate and analyze functions and their graphs including combinations and compositions of functions. 0 0 0
2a Find and use slopes of lines to write and graph linear equations in two variables. 0 0 0
2b Evaluate functions and find their domains. 0.63 1 0.63
2c Analyze the graphs of functions. 0 0 0
2d Find arithmetic combinations and compositions of functions. 0.88 1 0.88
2e Identify inverse functions graphically and find inverse functions algebraically. 0 0 0
3_ Sketch and analyze graphs of polynomial functions and mathematical models of variation. 0 0 0
3a Sketch and analyze graphs of polynomial functions 0 0 0
3b Use long division to divide polynomials 0 0 0
3c Write mathematical models for direct, inverse, and joint variation. 0 0 0
4_ Determine the domains of rational functions, find asymtotes, and sketch the graphs of rational functions. 0 0 0
4a Find the domains of rational functions. 0 0 0
4b Find the horizontal and vertical asymptotes for graphs of rational functions. 0 0 0
4c Recognize graphs of circles, ellipses, parabolas, and hyperbolas. 0 0 0
PSLO define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas. define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas.

0.16

Thus I only have to type in the outline once, tests will use vlookup to pull the SLO over to the item analysis tab of my workbook. Columns three to five of the outline tab then automatically update with information from the items analyses using sumif and countif functions. The sum column contains, in the first data row:
=SUMIF(analysis.C$2:C$13;A2;analysis.F$2:F$13)
The count column function is:
=COUNTIF(analysis.C$2:C$13;A2)=COUNTIF(analysis.C$2:C$13;A2)
Then I divide the sumif by the countif to get the averageif, or the average in plain old English.  At the bottom I take the average of the averages, and "voila!" I am 16% of the way home to accomplishing the program student learning outcome for these students (I disincluded the prerequisite materials percentage).

A couple notes: my formulas are OpenOffice formulas, for Excel change each semi-colon to a comma if you are trying to replicate the above. Better yet, let me know if you want to play with the spreadsheet I have built and I will email it to you in the format of your choice - Excel, OpenOffice, or StarCalc.

In case one is thinking that "just-in-time" assessment is some form of insanity, I have noted that education always seems to want to imitate the best-business-practice of the previous decade. Performance based budgets are another way of referring to return-on-investment which was so popular in the 80s and the 90s as a be-all metric for making business decisions.

What is in vogue now is warehouseless supply chains with just-in-time delivery of goods to retail outlets direct from factories, ordering being determined by demand as measured instantaneously by what customers are buying at computerized check-out lanes. Education will go this way too.


Dana Lee Ling wrote:
My syllabus is available at:
http://www.comfsm.fm/~dleeling/math/algebra/m62/calendar.html

My prequiz can be viewed at:
http://www.comfsm.fm/~dleeling/math/algebra/m62/html/prequiz.html

With the original OpenOffice 2.0 document at:
http://www.comfsm.fm/~dleeling/math/algebra/m62/prequiz.odt