Linear regression trend line attendance (not absences) versus performance in course
Review of performance: MS 150 fall 2009. 77 students were enrolled in course. Submitted by Dana Lee Ling.
| n | SLO | Program SLO | I, D, M | Reflection/comment | |
|---|---|---|---|---|---|
| 1 | Identify levels of measurement and appropriate statistical measures for a given level | define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas. | M | 74 | of 77 students were successful on this SLO based on an item analysis of the comprehensive final examination |
| 2 | Determine frequencies, relative frequencies, creating histograms and identifying their shape visually | M | 61 | ||
| 3 | Calculate basic statistical measures of the middle, spread, and relative standing | M | 71 | ||
| 4 | Perform linear regressions finding the slope, intercept, and correlation; generate predicted values based on the regression | M | 52 | ||
| 5 | Calculate simple probabilities for equally likely outcomes | M | 45 | ||
| 6 | Determine the mean of a distribution | M | 40 | ||
| 7 | Calculate probabilities using the normal distribution | M | 34 | ||
| 8 | Calculate the standard error of the mean | M | 70 | ||
| 9 | Find confidence intervals for the mean | M | 56 | ||
| 10 | Perform hypothesis tests against a known population mean using both confidence intervals and formal hypothesis testing | M | 42 | ||
| 11 | Perform t-tests for paired and independent samples using both confidence intervals and p-values | M | 42 | ||
In the table above, n is the outline outcome number. As noted below, outline outcome items five, six, and seven were not directly tested on the final examination. The data for those outcomes above is based on in-class testing data.
Performance in MS 150 Statistics was measured by quizzes and tests throughout the term. A comprehensive final examination consisting of fifty-two fill-in-the-blank questions was administered. Forty-six questions mapped back to a course level student learning outcome. Performance against the outline proposed in 2008 has been measured for the past three terms and provided a basis for comparison across the terms. The following table indicates the percentage of students who correctly answered questions for each outcome on the outline over four terms. The data below derives from an item analysis of the final examination, except for outcomes five, six, and seven. These three outcomes are foundation material for outcomes eight, nine, ten, and eleven. These three outcomes are tested during the term. In the table below quiz data from the term is used to report on success in outcomes five, six, and seven.
| n | Students will be able to... | Sp 08 | Fa 08 | Sp 09 | Fa 09 |
|---|---|---|---|---|---|
| 1 | Identify levels of measurement and appropriate statistical measures for a given level | 0.78 | 0.82 | 0.94 | 0.96 |
| 2 | Determine frequencies, relative frequencies, creating histograms and identifying their shape visually | 0.83 | 0.84 | 0.75 | 0.79 |
| 3 | Calculate basic statistical measures of the middle, spread, and relative standing | 0.9 | 0.8 | 0.86 | 0.93 |
| 4 | Perform linear regressions finding the slope, intercept, and correlation; generate predicted values based on the regression | 0.8 | 0.79 | 0.63 | 0.69 |
| 5 | Calculate simple probabilities for equally likely outcomes | 0.67 | NA | 0.82 | 0.59 |
| 6 | Determine the mean of a distribution | NA | NA | 0.77 | 0.52 |
| 7 | Calculate probabilities using the normal distribution | 0.71 | NA | 0.61 | 0.45 |
| 8 | Calculate the standard error of the mean | 0.84 | 0.84 | 0.87 | 0.91 |
| 9 | Find confidence intervals for the mean | 0.66 | 0.85 | 0.67 | 0.73 |
| 10 | Perform hypothesis tests against a known population mean using both confidence intervals and formal hypothesis testing | 0.36 | 0.56 | 0.55 | 0.55 |
| 11 | Perform t-tests for paired and independent samples using both confidence intervals and p-values | 0.55 | 0.63 | 0.43 | 0.56 |
| 12 | Go beyond outline | NA | 0.33 | 0.16 | 0.37 |
| PSLO | define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas. | 0.72 | 0.76 | 0.69 | 0.71 |
Sp refers to spring terms, Fa to fall terms. The digits that follow are the last two digits of the calendar year. Performance marked NA indicates that the material was not directly performed on the final examination and alternative assessment data was not recorded in that term.
Overall aggregate performance has remained stable between 69% and 75%. This term saw a small but not significant uptick from a 69% success rate spring 2009 to a 71% success rate fall 2009. This stability is fairly typical for a mature course that is in its ninth year and twentieth running.
On basic statistical calculations such as the mean, median, mode, standard deviation, slope, and intercept, success rates on the final examination were 99% to 100%. Students well mastered the basic concepts. On more complex calculations, such as finding confidence interval, success rates were lower. 75% of the students were able to correctly calculate a 95% confidence interval for the mean from raw data. Some of the most complex calculations, such as finding a t-statistic had some of the lowest rates of success. Finding a t-statistic was successfully done by only 35% of the students. There are many opportunities to make an error in a chain of dozen calculations while operating under the time pressure of a final examination.
The final six questions on the final examination are from material beyond the outline. The students are unaware that this material will appear, the material is not covered in the textbook, and the material is deleted from the posted finals that students use to study for the examination. This material is a complete surprise to the students. The point of this material is noted on the final examination itself, "One intention of any course is that a student should be able to learn and employ new concepts in the field even after the course is over."
The 37% success rate of students on this material is double that of the previous term. This success rate, however, was seen a year ago in the fall of 2008 on the same material. This value is probably highly variable from term-to-term.
Questions that required an inference, an interpretation of a result, had only a 55% success rate this term, up from a 40% success rate last term. Last term I identified inferences as an area of weakness that I sought to see improvement. This increase may simply be random, but I did focus more attention on drawing conclusions, inferences, from data. Whether this emphasis resulted in the increased performance, or whether this group of students was simply better at making inferences than prior classes remains to be seen.
For a fourth term in a row students were asked to engage in putting together a basic statistical research project. The number of projects completed improved with the addition of an earlier (fourth week of class) initial commitment to a project idea. Writing on the projects also showed improvement, but this was not specifically analyzed this term.
A separate report on mean absenteeism across multiple terms has been published. The statistics on absences for this term is reported in the following table.
| statistic | absences |
|---|---|
| sample size n | 79 |
| min | 0 |
| max | 20.9 |
| range | 20.9 |
| midrange | 10.45 |
| mode | 1 |
| median | 4.6 |
| mean | 5.99 |
| standard deviation | 4.8 |
| correlation | -0.76 |
The above table represents 77 students who range from perfect attendance (0 absences) to 20.9 absences with a mean of 5.99 absences. Students could accumulate 20 absences if, for example, they did not withdraw from the course prior to the last day to withdraw with a "W".
In my courses a late is a third of an absence. Since I run attendance as a point per day, a third of an absence would be "0.6666..." Thus I actually enter "0.7" making a late technically 30% of an absence. Extremely late (arriving half way through the period) causes a reduction to 50% of a point, hence the fractional mode and median.
The correlation is the correlation between a students course percentage and the number of absences. Thus an increasing number of absences is negatively correlated with performance. The higher the numer of absences, the lower the student's percentage (points) in the course.
The chart below depicts the attendance (not absences) versus performance in the course. The slope suggests a gain of 2% in course performance per day of attendance.
Linear regression trend line attendance (not absences) versus performance in course
The final examination score distribution provides another view of student comprehension of material in statistics. The fifty-two question final ranges from basic statistics to making inferences based on p-values for two samples. The following chart shows the distribution of the student papers as a function of the student's score. There were 59 points possible on 52 questions, no student attained more than 57 points. Bear in mind that the last six questions, equivalent to the final six points, was on material "beyond the outline." Thus a score of 53 is potentially equivalent to a perfect score.
Cumulative raw score distribution for 77 students on the final examination.
A score of 37 or higher represented a success rate of 70%+. 80% of the students had a score of 37 or higher.
Although not a direct assessment of learning, grade distributions do have meaning when coupled with in-course assessment that is aligned with the student learning outcomes on the outline. Questions on in-course quizzes and tests consist of questions that can each be mapped back to a student learning outcome on the outline. The bulk of the questions are valued at one point, with certain multiple-entry tables and charts being worth up to five points. Grades are then calculated from the points attained. While this is not matrix-level assessment which permits one to say, "Johnny can do x," I would argue the result is grades that have underlying meaning in terms of student learning. The course grade distribution is shown in the following chart.
Grade distribution