The value of indicators in MS 150 Statistics
09 Sep 2007
Although item analysis helps determine what
the students did and did not learn, and thus provide feedback for
adjusting the course, basic statistical indicators remain valuable.
Among the various statistics that my grade book tracks somewhat
automatically is the coefficient of variation on each quiz and test
instrument. The coefficient of variation is the standard deviation on
the instrument divided by the mean on the instrument. Experience has
indicated that for instruments with at least ten questions marked at a
point a piece, the coefficient of variation will be between 25% and
45%.
MS 150 Statistics test one this term came in at 47%. While only a hair
over the expected range, the number suggested a possible bimodal schism
developing in the course. Prompted by this single indicator, I ran a
frequency analysis of scores on test one:
The frequency distribution confirmed the split on the test. There is
precedent for this split at test one, although the sharpness of the
split is stronger than I've seen before. I have come to conceive of
this split as being indicative of two groups in the class. One group
has been putting in a concerted effort from day one, the other group
has yet to get started.
The homework completion rate in the course is at 65% (another
indicator) and the correlation between homework completion and test one
was 45%. While the later is only moderate correlation, the homework
rate is based on four assignments. Both lend credence to the theory
that some students are putting in an effort, while others have been
laying back.
I track a number of possible "schism" points in my grade book. One term
the average in two sections split by a statistically significant
amount. I now track the 95% confidence interval for the mean by section
as seen in the following snapshot at test one:
The differential between the sections is not statistically significant.
In addition, there is usually a difference between the sections, with
the earlier section more often outperforming the later section, despite
the earlier section being plagued by more late arrival and tardy
students.
The gender differential is also not
statistically significant with females holding a 64% average and males
holding a 66% average. The rank order of these averages is not
consistent. With no statistically significant difference, one would not
expect a consistent rank order.
Indicators are broad looks at groups of students. The assessment drive
has been about pushing assessment down to the level of the one - the
individual student. What can Johnnyleen do, what does she know, or
value? Neither indicators nor item analysis provide this level of
granularity, nor do grades. Grades are summative. Knowing "who has
learned what and when" sometimes seems like the holy grail.
I actually did this in my ESS 101j Joggling course, tracking roughly
twenty students on sixteen outcomes. That required a grid to track 320
events, with the date of accomplishment recorded in the grid. Once
gained, skills were retained by the students. No student ever "forgot"
knowing how to juggle after having learned to juggle.
As of test one this term 51 students have completed 49 "events" for two
thousand four hundred and ninety-nine events to track by the end of the
third full week of class. Simply not possible.
Even determining who is improving and who is already on a downward
trend is difficult. The number of measurements, two quizzes and a test,
are simply not a statistically significant sample size to generate
meaningful trend data. I am running an experimental stab at this using
z transformations of linear trends on percentage performance on test
instruments. A small chunk of this table can be seen below. 1, 2, and 3
are the percent correct on the first two quizzes and test one.
First |
trend |
z |
lin reg |
1 |
2 |
3 |
Lolla K. |
down |
-1.26 |
-0.28 |
0.92 |
0.36 |
0.36 |
Jefflyn |
down |
-1.53 |
-0.32 |
1.00 |
0.14 |
0.36 |
Kimberly |
up |
1.01 |
0.06 |
0.83 |
1.00 |
0.96 |
Mickleen |
stable |
-0.3 |
-0.13 |
0.67 |
0.07 |
0.40 |
Samuel |
stable |
0.61 |
0 |
0.83 |
0.71 |
0.84 |
Only one student exceeded the ordinary
z-value/extraordinary z-value boundary of two, a student who missed
test one:
Tom |
Kilarise |
strong down |
-2.45 |
-0.46 |
0.92 |
0.07 |
0 |
As a technical note, one of the reasons I
prefer spreadsheets to commercial grade books is that once I think of a
new indicator to explore, I can usually get the spreadsheet to produce
the desired indicator. One such as the above, however, requires all
twenty-six of my years of experience using spreadsheets, a grasp of
linear regressions and z-scores to put together. This is not for
everyone, nor is it something I could teach others to do.
Although I cannot track 2499 events, what I can do is use the mail
merge capabilities of modern office suites to produce a complete "data
sheet" for each student on their performance to date and let them make
their own determination on the state of their own learning. I will also
report to them their current trend. Ultimately I am not responsible for
their learning - learning is an internal change in a student that only
they can make. Only the student can choose to learn. My role is to
facilitate that process, provide a support environment that makes
learning possible.