Over the course of the fall 2007 term I developed and adjusted the laboratory marking rubric. This is an ongoing process, however the rubric is fairly stable with only minor changes now being made. Each laboratory has a rubric designed for that laboratory based on a common baseline generic rubric. With some sense that the rubric has some stability and appears to produce appropriate distinctions in quality, I performed an analysis of performance against the rubric for laboratory three.
To understand the assessment data, it helps to have seen the rubric. The rubric was built up from the entrance essay rubric and ideas borrowed from other science lab rubrics seen on line:
| 2. [d] Data (often in a table of results) | |
| +1 | five drop times |
| +1 | mean drop time |
| +1 | squared times |
| 3. [t] Data tables format (both tables) | |
| 3 | Clear, concise, well thought out, informative, labels and units in the head |
| 2 | Missing borders or other minor format inconsistencies, or missing units in the head |
| 1 | Incomplete, runs off edges of page, lacks minimal margins, or missing two or more elements such as borders and units |
| 4. [g] Data display: Quadratic Graph | |
| 3 | Correct graph type, correct axis labels |
| 2 | Missing a label or other element, or reversed x and y axes on an xy scattergraph, or separate graphs for the fast and slow ball |
| 1 | wrong graph type(s), or plotting the wrong data, or wrong result on graph, or otherwise incorrect |
| 4. [g] Data display: Linear Graph | |
| 3 | Correct graph type, correct axis labels |
| 2 | Missing a label or other element, or reversed x and y axes on an xy scattergraph, or separate graphs for the fast and slow ball |
| 1 | wrong graph type(s), or plotting the wrong data, or wrong result on graph, or otherwise incorrect |
| 5. [a] Analysis | |
| +1 | slope calculation correctly done |
| +1 | calculation of a value for g |
| 6. [c] Conclusions – Content | |
| Each student must write their own individual conclusion. I want to assess each individual student's reasoning and writing ability. Where a lab team writes and submits a single common conclusion, the conclusion points are split between the team members. | |
| 5 | Thoughtfully put together, well-reasoned, logical, sensible. Fully complete and thorough summary of the findings of the laboratory. Correct usage of vocabulary, appropriate use of scientific concepts. Discusses potential sources of error and how these were controlled. Includes background research on the laboratory subject. Cites appropriate text book information related to laboratory. |
| 4 | Moderately well reasoned. Relevant and adequate answer to the task set with only a single gap or missing task item. |
| 3 | For the most part answers the task set, though there may be gaps or redundant information, or the conclusion is essentially tangential to the experiments, or based on misconceptions, or incorrect conclusion, muddled |
| 2 | Conclusion of little relevance to the laboratory, major gaps, or overly short such as to be incomplete, Confusing, or highly incomplete, or illogical, or made confusing by serious grammar problems, or merely restated the procedure, or a variant of a non-specific and vague "I learned a lot in this laboratory" |
| 1 | Bears little relation to the task set, unclear, very confusing, not well reasoned, extremely tangential, or extraordinarily weak. Almost incomprehensible, or a single (one to two) sentence conclusion |
| 0 | Omitted |
| 7. [f] Format | |
| +1 | Double spaced |
| [G] Grammar and Syntax [-2 if conclusion too short to judge grammar properly] | |
| 5 | No errors of grammar or word order. |
| 4 | Some errors of grammar or word order but communication not impaired. |
| 3 | Fairly frequent errors of grammar or word order; occasional re-reading necessary for full comprehension. |
| 2 | Frequent errors of grammar or word order; efforts of interpretation sometimes required on reader's part. |
| 1 | Very frequent errors of grammar or word orde; reader often has to rely on own interpretation. |
| 0 | Errors of grammar or word order so severe as to make comprehension virtually impossible. |
| [V] Vocabulary [-2 if conclusion too short – taken as evidence of vocabulary limitations] | |
| 5 | Appropriate terms used consistently, clear command of vocabulary with a focus on correct usage of physical science vocabulary, no misspelled words. |
| 4 | Occasionally uses inappropriate terms or relies on circumlocution; expression of ideas not impaired; or a few misspelled words. |
| 3 | Uses wrong or inappropriate words fairly frequently; expression of ideas may be limited because of inadequate vocabulary, or many misspelled words. |
| 2 | Limited vocabulary and frequent errors clearly hinder expression of ideas. |
| 1 | Vocabulary so limited and so frequently misused that reader must often rely on own interpretation. |
| 0 | Vocabulary limitations so extreme as to make comprehension virtually impossible. |
| [O] Organization | |
| 5 | All sections present in the proper order. Material exceptionally well organized. Conclusion well structured with introductory and concluding phrases. |
| 4 | One section out of sequence or omitted. Material well organized; structure could occasionally be clearer but communication not impaired. |
| 3 | Multiple sections out of sequence, some lack of organization; re-reading required for clarification of ideas. For example, tables and graphs printed from a spreadsheet and then stapled to the back of a lab write-up printed from a word processing program. |
| 2 | Multiple sections omitted. Little or no attempt at connectivity, though reader can deduce some organization. |
| 1 | Individual ideas may be clear, but very difficult to deduce connection between them. |
| 0 | Lack of organization so severe that communication is seriously impaired. |
| [C] Cohesion [0 if conclusion too short to judge cohesion] | |
| 5 | Consistent choices in cohesive structures. Ideas flow logically. Conclusion remains on topic. Connector words assist the reader. |
| 4 | Occasional lack of consistency in choice of cohesive structures and vocabulary but overall ease of communication not impaired. |
| 3 | Patchy, with some cohesive structures or vocabulary items noticeably inappropriate to general style. Ideas tend to be disconnected from each other. Reads more like an outline than a coherent essay, or written as a list of answers to questions without connector words and phrases generating a choppy, disjoint style |
| 2 | Cohesive structures or vocabulary items sometimes not only inappropriate but also misused; little sense of ease of communication. Connector words and phrases confuse and mislead the reader, but sense can be made of the conclusion. |
| 1 | Communication often impaired by completely inappropriate or misused cohesive structures or vocabulary items making it difficult to make scientific sense of the conclusion. |
| 0 | A 'hodgepodge' of half-learned misused cohesive structures and vocabulary items rendering communication almost impossible. |
Note that the organization sub-section is substantially re-written to reflect whether the student followed the prescribed laboratory organization. In addition, spelling is counted in the vocabulary section. Students use spreadsheet and word processing software to crank out a printed lab each and every week.
The data for 23 students is presented in the following table, the letter codes in the column headers are from the rubric above.
| Lab 03 | d | t | g | g | a | c | f | G | V | O | C | Sum |
| Student 1 | 3 | 2 | 3 | 3 | 1 | 5 | 1 | 5 | 5 | 5 | 5 | 38 |
| Student 2 | 3 | 3 | 3 | 3 | 1 | 4 | 1 | 5 | 5 | 5 | 5 | 38 |
| Student 3 | 3 | 2 | 3 | 3 | 1 | 4 | 1 | 5 | 5 | 5 | 5 | 37 |
| Student 4 | 3 | 3 | 3 | 3 | 0 | 4 | 1 | 4 | 5 | 5 | 5 | 36 |
| Student 5 | 3 | 2 | 3 | 3 | 0 | 4 | 1 | 5 | 5 | 5 | 5 | 36 |
| Student 6 | 3 | 3 | 3 | 3 | 0 | 3 | 1 | 5 | 5 | 5 | 5 | 36 |
| Student 7 | 3 | 2 | 3 | 3 | 0 | 4 | 1 | 4 | 5 | 5 | 5 | 35 |
| Student 8 | 3 | 2 | 2 | 3 | 0 | 4 | 1 | 5 | 5 | 5 | 5 | 35 |
| Student 9 | 3 | 3 | 3 | 3 | 1 | 2 | 1 | 4 | 5 | 5 | 4 | 34 |
| Student 10 | 3 | 3 | 2 | 2 | 0 | 3 | 1 | 5 | 5 | 5 | 5 | 34 |
| Student 11 | 3 | 3 | 2 | 2 | 0 | 2 | 1 | 5 | 5 | 5 | 5 | 33 |
| Student 12 | 3 | 3 | 3 | 2 | 0 | 2 | 0 | 5 | 5 | 5 | 5 | 33 |
| Student 13 | 3 | 1 | 3 | 3 | 1 | 3 | 1 | 5 | 4 | 5 | 4 | 33 |
| Student 14 | 3 | 3 | 2 | 2 | 0 | 3 | 1 | 4 | 5 | 5 | 5 | 33 |
| Student 15 | 3 | 3 | 3 | 3 | 0 | 2 | 1 | 4 | 5 | 5 | 4 | 33 |
| Student 16 | 3 | 3 | 3 | 3 | 1 | 2 | 1 | 4 | 3 | 5 | 4 | 32 |
| Student 17 | 3 | 3 | 3 | 2 | 0 | 2 | 1 | 4 | 4 | 5 | 4 | 31 |
| Student 18 | 3 | 2 | 2 | 2 | 0 | 2 | 1 | 4 | 5 | 5 | 5 | 31 |
| Student 19 | 3 | 2 | 3 | 3 | 0 | 2 | 0 | 4 | 4 | 5 | 4 | 30 |
| Student 20 | 3 | 2 | 2 | 1 | 0 | 2 | 1 | 4 | 5 | 5 | 4 | 29 |
| Student 21 | 3 | 2 | 0 | 3 | 1 | 3 | 1 | 3 | 4 | 4 | 4 | 28 |
| Student 22 | 3 | 2 | 2 | 2 | 0 | 2 | 0 | 3 | 4 | 4 | 3 | 25 |
| Student 23 | 3 | 3 | 3 | 2 | 0 | 2 | 1 | 3 | 4 | 3 | 3 | 27 |
| Mode | 3 | 3 | 3 | 3 | 0 | 2 | 1 | 4 | 5 | 5 | 5 | 33 |
| Median | 3 | 3 | 3 | 3 | 0 | 3 | 1 | 4 | 5 | 5 | 5 | 33 |
| Means | 3 | 2.48 | 2.57 | 2.57 | 0.3 | 2.87 | 0.87 | 4.3 | 4.65 | 4.83 | 4.48 | 32.91 |
The upshot is that students are quite capable at producing tables and graphs. Performing mathematical analysis [a] has been more difficult. In this laboratory it was clear from the laboratories that student's did not understand that they should go ahead and get the slope of the linear regression for the second graph and use that to calculate g. I am rewriting the laboratory to make this more clear, and I will be changing my in class introduction next fall to clarify this area of confusion.
Conclusion scores suffered because of the lack of results (analysis) to discuss. That said, the grammar, vocabulary, organization, and cohesion scores were fairly strong. Grammar remains the areas of greatest weakness, especially where grammar "meets" science. When students attempt to wrestle their grammar around an explanation of a phenomenon, their difficulties with choosing the correct prepositions is often problematic and obscures clarity of meaning.
Anecdotally, I am seeing fewer of the overly short laboratories that I saw fall term. I appear to have better communicated to the students what I expect in laboratory reports. Rather than being overly brief, some were slightly overly detailed with up to six pages of work submitted.
A missing statistic above is that only 23 students turned in laboratories on time of 32 students in the course. Students have one week to complete the lab, it is due at the next lab period. Lab is on a Thursday, papers turned in Friday incur a five point penalty. Students who fail to turn in a lab have an additional week to turn in the lab with a ten point penalty. Some labs will come in during the coming week. Currently the turn-in rate to date is 81% (78 of 96 possible lab reports have been turned in).
As I knew when I began this effort last August, the marking load is ferocious. Even with 23 lab reports, this batch required something on the order of eight hours spread over three days to mark. This is in part because I make notes on the papers both on the science and the writing issues in the report. Since no lab is resubmitted, there are no first, second, and final drafts, the intent is that the information will provide input to the next laboratory. Marking is an every weekend endeavor with the lab-a-week pace. While this sort of load is the norm for language and literature instructors, the marking of writing mechanics would be new to most science instructors.
Despite the load, seeing the students wrestle with science in their own words is nothing short of exciting. One student noted, "I found out that when a ball falls from a higher distance it is much faster than a ball falling from a lower distance. The graph made a curve. Our hypothesis that we wanted to prove was true." One student reflecting on the difficulty of timing the falling ball by hand noted, "It's hard to find the exact time, because the changing of the velocity." That is a core concept underneath acceleration: the velocity is changing as the ball falls.
Science as the result of the concurrence of a community of scientists was captured in a note by one student, "Each group seemed to had [sic] fun measuring the time and distance of the falling object. Not only did each student stuck [sic] to its group, they went around the room comparing their results with other groups. Asking questions such as why both data did not seem [sic] to relate to each [other]. The questions led to going through the experiment again."
Although the timing of the falling ball was done by hand, in a meta-study of the lab data the overall average fall times generated a parabola. Squaring the time led to a line with a correlation r of 0.9982 and a g value of 920 cm/s², only 6% below the accepted value. Not bad for students hand timing the fall of a ball from heights as low as 50 cm.