SC 130 Physical Science assessment report and grid

11 May 2009

College mandated assessment report format

Review of performance: SC 130 Physical science spring 2009. 34 students enrolled in course. Submitted by Dana Lee Ling

For the first three outcomes, student counts based on number who passed the class. For subsequent outcomes, student counts based on submission of laboratory reports for that outcome.

SC 130 Physical science assessment

Some files linked from this assessment grid use XHTML+ MathMl + SVG and require the use of browsers such as FireFox which can render these technologies. Sample laboratory reports are in Adobe Acrobat (.pdf) format.

Program Learning Outcomes

The relevant program learning outcomes are from the general education core.

• Define and explain the concepts, principles, and theories of a field of science.
• Perform experiments that gather scientific information and to utilize, interpret, and explain the results of experiments and field work in a field of science

Course Learning Outcomes

Learning themes cut across all activities and explorations in the course. These learning themes are the first three items on the course outline. The first theme centers the course on exploration and science as a process as expressed by the scientific method. The second them places mathematics at the core of the course. Many of the laboratories lead to linear relationships, linear regressions, with the attendant concepts of slope and intercept. The third theme brings writing across the curriculum into the course. In addition to being marked for content, the laboratories are marked for grammar, syntax, vocabulary, spelling, format, cohesion, and organization.

Students will be able to...

Specific learning is centered on the laboratory experiences. Laboratory reports cited above are considered a primary measure of student performance in the course. Through the weekly laboratory reports, the course integrates writing into the core of the course curriculum.

Documenting actual activity during the laboratories is difficult. The laboratories are intended and designed to engender cooperative learning in scientific teams of exploration. One of the design intents is that the acquisition of scientific knowledge is a journey, not a destination. Science is not about a set of accumulated and memorized facts. Science is a process of discovery, careful thought and analysis. Science is about finding and testing explanations for systems, in physical science those explanations are typically mathematical models.

Tests can document acquired facts. Documenting the journey, as opposed to the acquired facts, is difficult. The table further below uses links to photo documentation as indirect evidence of science as an exploration.

The final includes single numeric problems typifying that area of study. Each final utilized a spreadsheet to randomly generate data values. A mail merge was used to produce individually unique final examinations for each and every student. The merge also generated an answer sheet for each student final, these were separated from the final and used to mark each unique exam paper.

The results of an item analysis for the spring 2008, fall 2008, and spring 2009 final are in the table further below. Twenty-nine students took the spring 2008 final, twenty-eight students took the fall 2008 final, and thirty-one students took the spring 2009 final. The item analysis is the percent of students answering the question in that area correctly.

Comments and notes spring 2009

The core of the course are the activities and laboratories. The laboratories involve a write-up using spreadsheet and word processing software. The laboratories are marked using a rubric. The course focuses on physical science as a process and method, an exploration in search of mathematical models of system behavior. The final examination is not a well aligned measure of process, method, and exploration. The final examination is a set of fifteen questions, one per laboratory, usually centered on the central mathematical relationship of the corresponding laboratory. As such, the final is an exercise both in remembered knowledge and calculations.

The laboratories are each typically worth 40 to 50 points. With quizzes, tests, homework, and attendance, the course generates over 900 points per term. The final is typically worth 30 to 40 points, no more than 3% to 4% of the overall mark. While the final could be weighted, the true focus of the course are the laboratories. Laboratory 14 is a better measure of the achievement of outcomes than the final examination. The students are aware that the final has little impact on their grade, and this may be reflected in the performance seen in the table above.

As an additional confounding variable, end of term review was briefer than in previous terms due to a new cosmology unit being added to the end of the term. The cosmology unit received a favorable response from the students and may be expanded upon in future terms.

Overall performance, term-to-term, has remained fairly consistent at around 55% of the questions attempted by the students being answered correctly. Individual items rise and fall without an apparent pattern. The questions are not entirely identical from term-to-term, refer to the finals in the links provided earlier to see the differences.

The item analysis success rate is related to the complexity of the problem. Questions involving multiple calculation steps had lower rates of success. The table includes success rates for the spring 2008, fall 2008, and spring 2009 terms on the final examination based on question difficulty. The sample size for this data is 29 students for spring 2008, 28 students for fall 2008, and 31 students spring 2009.

^a Based on in-term item analysis of inference questions on quizzes and tests.
^b Based on in-term item analysis of inference questions on quizzes and tests.
^c Based on question six of the final examination.

If the course has de-emphasized the value of memorized factoids, then the loss in success on fact level questions over the past three terms is evidence that this message is being internalized by the students. That said, the trend indicates a potential risk of over-de-emphasizing facts. Science is not a fact-free void by any means. The trend suggests a need to rebalance the emphasis in the course so as not to lose the importance of facts in the pursuit of process and method.

The ability to calculate results involving a remembered formula and a single calculation has remained remarkably stable term-on-term at 62%. Of interest is that the student's must produce the correct formula from memory, thus this question includes the lower level of memorized fact.

Complex calculations, while difficult for 59% of the students, have shown a marked improvement in percentage of success rate. Some caution should be taken, however, in over-interpreting this result. Laboratory nine was a low outlier on this question in past terms, helping to pull the average down significantly. A simplification in the laboratory design spring 2009 led to an easier to comprehend model and simpler calculation both in the laboratory and on the final examination.

The ability to examine data or a graph and draw a conclusion based on that information, to make an inference, is an area in which students in another course, MS 150 Statistics, have difficulty. Success rates in SC 130 Physical science are higher. I would like to make the leap that the later course involves physical systems, concrete realities, that the students have experienced while the former course involves more abstract problems, but this is too large a leap to be supported by the existing data. At best this provides food for thought.

Comment on linear regressions

Spring 2008 the laboratories were rewritten to focus on mathematical relationships, specifically linear relationships. Students are introduced to using spreadsheet software in laboratories one and two to assist them in making xy scatter graphs and in finding the slope and intercept. The course requires access to a computer laboratory for the second half of the first two laboratories during a term in order to present this material to the students.

An instructor on another campus noted that the specific reference to "linear regressions" had to be removed from the outline. The students on that campus were deemed unable and unready to handle linear regressions. One might think that given the placement of linear relations at the core of the course, the course should have a math pre-requisite. The lack of a math pre-requisite is intentional. The course does not presume the student has anything more than high school level contact with algebra one material. The course not only undertakes to teach students to run linear regressions, the instructor presumes that using a computer to find slopes and intercepts is wholly new material for the students. The course intentionally seeks to introduce algebra to the students through the vehicle of physical science. This instructor sees the course as a pre-requisite to college algebra, not the other way around.

Observational comments

The students continue to have difficulty distinguishing when a set of coordinate points forms a pattern that can be mathematically modeled and when the points are merely random. A contributing factor may be that no laboratories are designed to generate a random relationship. In terms of content, unrelated variables are not usually studied in physical science. Where there are two variables being studied there is usually a relationship.

The fall 2008 term introduced a sharper focus on physical dimensionality early in the term: time, space, matter and the fundamental units used to measure these quantities. The sharper focus reflected observations the prior year that students were unfamiliar with the philosophical framework by which physical scientists view the world. This led to difficulties in reading graphs, remembering to report units of measurement in laboratories, and including units in answers on tests. Spring 2009 this focus was emphasized from day one and the new cosmology unit at the end of the course returned the course back to day one with a look at the origins of time, space, and matter.

Post-script: Final examination item analysis details

Details on the final examination item analysis spring 2009. Number of students answering a particular question correctly. Fact refers to the question being a recall of a fact, memorized knowledge. Single refers to a single calculation required to determine the correct answer. Complex refers to a chain of two or more calculations required to obtain a correct answer. Inference means the student must make an inferential conclusion based on the data presented. Note that this inference is not a statistical hypothesis test, simply a conclusion drawn from a table, graph, or other data provided by the question. This is the data that was mapped to the student learning outcomes on the outline.

Physci spring 2008 • Physci • Assessment • Courses • COMFSM