Update on MS 100 item-analysis ongoing continuous assessment effort

Recalling one faculty member at the national site bogging down last fall under the weight of some 50,000 data points in a learning outcome assessment effort, I am continuing my own efforts to find a way to utilize item analysis to provide ongoing, continuous status of learning information. My goal is not a five year cycle, annual cycle, or biannual cycle, but rather a weekly refresh of information at the microlevel to provide just-in-time assessment.

I am now playing with a spreadsheet structure that keeps the workload to something more reasonable, although I do have to make a "double pass" through my tests. On the first pass I mark my papers, on the second I use hash marks to tabulate an item analysis. Then I build a table where I note the actual topic.

Source	Description
P1a	Evaluate x
P1b	Subtract rational terms
P2	Simplify rational term
P3	Multiply binomial by quadranomial
P4	Factor quadratic
P5	Simplify rational expression
P6a	Plot coordinates
P6b	Plot coordinates
P6c	Plot coordinates
P6d	Plot coordinates
P7	Graph linear equation
P8	Solve quadratic

The above is based on the prequiz which can be seen at:
http://www.comfsm.fm/~dleeling/math/algebra/m62/html/prequiz.html

Working from the outline, I enter an outline reference code which indicates what the quiz item best matches on the outline.

Source	Description	Outref
P1a	Evaluate x	2b
P1b	Subtract rational terms	2d
P2	Simplify rational term	1_
P3	Multiply binomial by quadranomial	Pre
P4	Factor quadratic	1_
P5	Simplify rational expression	1_
P6a	Plot coordinates	Pre
P6b	Plot coordinates	Pre
P6c	Plot coordinates	Pre
P6d	Plot coordinates	Pre
P7	Graph linear equation	1a
P8	Solve quadratic	1b

This is not too difficult and does not take much time. Those codes are then referenced by a VLOOKUP function, =VLOOKUP(C2;outline.A$2:B$21;2), to automatically fill-in the student learning outcomes in the fourth column:

Source	Description	Outref	SLO	Corr	Corr%
P1a	Evaluate x	2b	Evaluate functions and find their domains.	5	63%
P1b	Subtract rational terms	2d	Find arithmetic combinations and compositions of functions.	7	88%
P2	Simplify rational term	1_	Graph and solve linear and quadratic equations and inequalities including those with complex roots.	6	75%
P3	Multiply binomial by quadranomial	Pre	[Preliminary skill]	5	63%
P4	Factor quadratic	1_	Graph and solve linear and quadratic equations and inequalities including those with complex roots.	6	75%
P5	Simplify rational expression	1_	Graph and solve linear and quadratic equations and inequalities including those with complex roots.	2	25%
P6a	Plot coordinates	Pre	[Preliminary skill]	7	88%
P6b	Plot coordinates	Pre	[Preliminary skill]	6	75%
P6c	Plot coordinates	Pre	[Preliminary skill]	5	63%
P6d	Plot coordinates	Pre	[Preliminary skill]	7	88%
P7	Graph linear equation	1a	Sketch the graph of an equation	4	50%
P8	Solve quadratic	1b	Solve linear, quadratic, polynomial, and radical equations.	4	50%

The fifth and sixth columns above contain the results of the item analysis itself. The vlookup function pulls its information from another tab in the workbook, the outline tab, which contains the following table:

Outref	Students will be able to:	Sum	Count	Avg
Pre	[Preliminary skill]	3.75	5	0.75
1_	Graph and solve linear and quadratic equations and inequalities including those with complex roots.	1.75	3	0.58
1a	Sketch the graph of an equation	0.5	1	0.5
1b	Solve linear, quadratic, polynomial, and radical equations.	0.5	1	0.5
1c	Perform operations with complex numbers.	0	0	0
1d	Solve linear, quadratic, polynomial, and radical inequalities.	0	0	0
2_	Evaluate and analyze functions and their graphs including combinations and compositions of functions.	0	0	0
2a	Find and use slopes of lines to write and graph linear equations in two variables.	0	0	0
2b	Evaluate functions and find their domains.	0.63	1	0.63
2c	Analyze the graphs of functions.	0	0	0
2d	Find arithmetic combinations and compositions of functions.	0.88	1	0.88
2e	Identify inverse functions graphically and find inverse functions algebraically.	0	0	0
3_	Sketch and analyze graphs of polynomial functions and mathematical models of variation.	0	0	0
3a	Sketch and analyze graphs of polynomial functions	0	0	0
3b	Use long division to divide polynomials	0	0	0
3c	Write mathematical models for direct, inverse, and joint variation.	0	0	0
4_	Determine the domains of rational functions, find asymtotes, and sketch the graphs of rational functions.	0	0	0
4a	Find the domains of rational functions.	0	0	0
4b	Find the horizontal and vertical asymptotes for graphs of rational functions.	0	0	0
4c	Recognize graphs of circles, ellipses, parabolas, and hyperbolas.	0	0	0
PSLO	define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas. define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas.			0.16

Thus I only have to type in the outline once, tests will use vlookup to pull the SLO over to the item analysis tab of my workbook. Columns three to five of the outline tab then automatically update with information from the items analyses using sumif and countif functions. The sum column contains, in the first data row:
=SUMIF(analysis.C$2:C$13;A2;analysis.F$2:F$13)
The count column function is:
=COUNTIF(analysis.C$2:C$13;A2)=COUNTIF(analysis.C$2:C$13;A2)
Then I divide the sumif by the countif to get the averageif, or the average in plain old English.  At the bottom I take the average of the averages, and "voila!" I am 16% of the way home to accomplishing the program student learning outcome for these students (I disincluded the prerequisite materials percentage).

A couple notes: my formulas are OpenOffice formulas, for Excel change each semi-colon to a comma if you are trying to replicate the above. Better yet, let me know if you want to play with the spreadsheet I have built and I will email it to you in the format of your choice - Excel, OpenOffice, or StarCalc.

In case one is thinking that "just-in-time" assessment is some form of insanity, I have noted that education always seems to want to imitate the best-business-practice of the previous decade. Performance based budgets are another way of referring to return-on-investment which was so popular in the 80s and the 90s as a be-all metric for making business decisions.

What is in vogue now is warehouseless supply chains with just-in-time delivery of goods to retail outlets direct from factories, ordering being determined by demand as measured instantaneously by what customers are buying at computerized check-out lanes. Education will go this way too.


Dana Lee Ling wrote:

My syllabus is available at:
http://www.comfsm.fm/~dleeling/math/algebra/m62/calendar.html

My prequiz can be viewed at:
http://www.comfsm.fm/~dleeling/math/algebra/m62/html/prequiz.html

With the original OpenOffice 2.0 document at:
http://www.comfsm.fm/~dleeling/math/algebra/m62/prequiz.odt

--