MS 150 Statistics underwent an evolution this term. I think of the evolution as a "drift" towards a distance education model. I had always emphasized that the memorization of formulas was not important in the course. The correct usage of the appropriate statistical measure and knowing the limitations of that measure are important. I had always provided formula sheets for the students to refer to during quizzes and tests.
Just prior to midterm a student was absent due to sickness. The missed quiz was given on a Friday, and the class next met at 8:00 on Monday morning. The student had Internet access and asked for the chance to take the quiz from home. As with all of my statistics quizzes, the quiz was posted on line, unlinked until Monday. I sent the student the URL of the quiz and later that day they submitted their solution.
The next week a student asked in class if they could look up a formula in the text that they believed they needed to solve a problem. That formula was not actually necessary, but the request was reasonable. No one takes away all my reference books and then asks me to solve a statistical problem. I permitted the text access.
From then on I permitted reference to the text and course notes during quizzes and tests, making evaluation effectively "open book." This would be the situation if the course were a distance education course with students operating from unmonitored remote locations. Hence my considering the shift a "drift" towards a distance education model.
As a course that was designed to reflect introduction to statistics courses as are typically taught in community colleges elsewhere, the course has internally defined specific learning outcomes. These have been measured in a traditional manner via class quizzes and tests. Questions on quizzes and tests measure outcomes on the outline. Measurement of learning has been by item analysis of the individual questions and then aggregating the results against the outline.
The following table notes student success during the spring 2008 and fall 2007 terms as aggregated against the 2005 approved outline.
| Spring 2008 | Fall 2007 | ||||||
| 2005 | Students will be able to: | Sum | Count | Avg | Sum | Count | Avg |
| 1 | Calculate basic statistics | 37.08 | 45 | 82% | 82.36 | 117 | 70% |
| 2 | Represent data sets using charts and histograms | 8.33 | 12 | 69% | 8.83 | 17 | 52% |
| 3 | Solve problems using normal curve and t-statistic distributions including confidence intervals for means and hypothesis testing | 9.81 | 20 | 49% | 32.88 | 53 | 62% |
| 4 | Determine and interpret p-values | 0.73 | 1 | 73% | 7.39 | 11 | 67% |
| 5 | Perform a linear regression and make inferences based on the results | 12.09 | 16 | 76% | 25.36 | 43 | 59% |
| PSLO | define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas. | 68.04 | 94 | 72% | 156.81 | 241 | 65% |
Overall performance improved, although the improvement is at the margin of statistical significance. Improved performance on basic statistics was significant with the only decline seen in confidence interval calculations.
| Final only | Spring 2008 | Fall 2007 | |||||
| 2005 | Students will be able to: | Sum | Count | Avg | Sum | Count | Avg |
| 1 | Calculate basic statistics | 16.3 | 18 | 91% | 14.79 | 17 | 87% |
| 2 | Represent data sets using charts and histograms | 3.32 | 4 | 83% | 2.38 | 4 | 60% |
| 3 | Solve problems using normal curve and t-statistic distributions including confidence intervals for means and hypothesis testing | 9.81 | 20 | 49% | 8.33 | 13 | 64% |
| 4 | Determine and interpret p-values | 0.73 | 1 | 73% | 3.14 | 4 | 79% |
| 5 | Perform a linear regression and make inferences based on the results | 5.59 | 7 | 80% | 6.55 | 9 | 73% |
| PSLO | define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas. | 35.76 | 50 | 72% | 35.19 | 47 | 75% |
Performance on the final examination, however, was statistically unchanged. Small but not significant improvements were seen in basic statistics. The strongest gains were seen for charts and histograms, with a drop in performance on confidence intervals. Bear in mind that spring 2008 was an "open book" final while fall 2007 was done without reference to notes or the text. The 3% drop in performance was not statistically significant. That performance remained effectively the same term-to-term was a surprise. The ability of the students to use their notes and text did not confer an advantage to the students.
Based only on earlier quizzes and conversations, it appears that students studied less diligently, counting on the ability to look up material they did not know. In the past students typically completed the final examination in under two hours, this term the plurality of the students used all two hours. Looking up material slowed the students, suggesting far less intense studying occurred than on earlier final examinations.
The spring 2008 final included a new twist. Five questions at the end of the fifty question examination involved running a hypothesis test on the linear correlation coefficient. The students had never done this before, nor does the current text cover this type of statistical analysis. The final provided the necessary variations of formulas the students had used on other material including the t-statistic formula and the adjustment to the degrees of freedom. Performance on this material was weaker than overall performance with an aggregate mean success rate of 38%. The intent of this material was to push the students beyond the knowledge set in the course, to try to gain some insight into whether the students "learned to learn" about statistics. The success rate, while low, is still promising. Some students were able to go beyond the material in the course within the time constraints of a final examination. I would argue that some students have learned how to learn in the field of statistics.
Performance against the proposed 2008 outline for the course is given in the following table.
| 2008 | Students will be able to: | Sum | Count | Avg |
| 1 | Identify levels of measurement and appropriate statistical measures for a given level | 2.15 | 3 | 72% |
| 2 | Determine frequencies, relative frequencies, creating histograms and identifying their shape visually | 8.33 | 12 | 69% |
| 3 | Calculate basic statistical measures of the middle, spread, and relative standing | 32.48 | 38 | 85% |
| 4 | Perform linear regressions finding the slope, intercept, and correlation; generate predicted values based on the regression | 12.09 | 16 | 76% |
| 5 | Calculate simple probabilities for equally likely outcomes | 0.5 | 2 | 25% |
| 6 | Determine the mean of a distribution | N/A | N/A | N/A |
| 7 | Calculate probabilities using the normal distribution | N/A | N/A | N/A |
| 8 | Calculate the standard error of the mean | 0.84 | 1 | 84% |
| 9 | Find confidence intervals for the mean | 2.65 | 4 | 66% |
| 10 | Perform hypothesis tests against a known population mean using both confidence intervals and formal hypothesis testing | 1.81 | 5 | 36% |
| 11 | Perform t-tests for paired and independent samples using both confidence intervals and p-values | 7.19 | 13 | 55% |
| PSLO | define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas. | 68.04 | 94 | 72% |
The material marked N/A was covered on tests or quizzes which were not item analyzed this term. Material that covered probabilities in more detail is also not reflected in the above data. The data for outcome number five above is not statistically significant due to the small samples size (only two questions on this material were item analyzed during the term).
The proposed outline has an advantage over the existing outline in that it separates out confidence intervals from hypothesis testing. Where the extant outline showed a 49% success rate on confidence intervals and hypothesis testing - a broad swath of material - the proposed outline points more clearly at weakness in hypothesis testing material.
The proposed outline also indicates strength in basic statistics calculations with 85% success on aggregate. On the final this rose to a 90% success rate.
As presently designed, the course is weak in measuring student ability to design a statistical study and carry out research independently. Coupled with this are weaknesses in coverage and therefore learning of sampling methods and limitations. In an attempt to tackle learning in these areas the course introduced the submission of a lightweight statistical research project.
This term has been an experimental pilot. The projects varied widely in both statistical quality and in the quality of the overall report. Arising out of the pilot this term were clearer expectations for the project. The project as currently designed does not necessarily lead to testing of a hypothesis. The project does not focus on writing skills and remains primarily a data gathering and analysis exercise.
I recently saw a memo calling for 360° views into learning from all aspects internal and external. The data that is "out in the parking lot" that I would like to have and cannot is whether graduates use basic statistics beyond graduation. Are there situations which might call on alumni to employ statistical measures and do they then do so, and do so in an appropriate and correct manner?