Science Program Learning Outcomes

Pilot Evaluation Report 04 May 2004

Executive Summary

In order to understand the potential usefulness of a science program learning outcomes evaluation instrument, a pilot test was conducted on a non-random sample of students.

This run of the instrument was purely an experimental trial run. The instrument is unusual in its nature and structure, the instrument is nothing related to a standard multiple choice fact and content oriented test. On the contrary the instrument seeks to explore the students thinking about phenomenon and asks the students to engage in the process of science. The first part asks the students to observe a demonstration and explain why an event occurred. The second part has the students performing a simple experiment.

The result was that students have difficulty using the concepts of science to explain events, but are fairly competent at taking data, filling in tables, and making graphs. The students are evenly split on making a simply calculation, more advanced calculations that require a chain of calculations and understanding completely elude the students.

Detail Report

Information on the student sample

The 21 students were members of the SC/SS 115 Ethnobotany class spring 2004. The students are a mix of freshmen and sophomores from a broad variety of majors. The students were permitted as much time as they needed to perform the experiment in part two, but the whole instrument took less than thirty minutes for the students to complete.

Due to equipment limitations, students worked in pairs with one group working at a triple. Thus the instrument says nothing about individual abilities.

The concept of having students work in pairs should not necessarily be automatically discarded if more equipment were made available. Twenty-first century science is done in teams. The team results still inform the division as to what students, in the plural, take away from our programs. Science laboratories are done in pairs, surgeons work as a member of a team, scientists send papers for review to trusted colleagues, researchers function as members of inter-institutional teams. Collaboration, cooperation, and communication are crucial to successful science.

Program learning outcomes for science

Students will be able to...

  1. define fundamental concepts, principles, and theories of science.
  2. gather scientific information through experiments, field work, and research.
  3. perform experiments that support the development of scientific theory.
  4. utilize appropriate laboratory and field work procedures.
  5. interpret and express the results of experiments.
  6. explain observations of new phenomenon, systems, and entities, using the theories of science.

There was no design intent to evaluate each and every outcome, nor to comprehensively evaluate all the facets of each outcome.

Instrument

The evaluation instrument consisted of two parts. The evaluation instrument can be viewed at http://www.comfsm.fm/~dleeling/physics/progoutscience.html Understanding these comments virtually requiers having this document in hand as this report is read.

In part one a drinking glass was filled with water and a piece cut from a file folder was placed on top of the full glass. The class was asked to predict what would happen when the instructor held the paper, turned the glass over, and then let go of the paper. Students answered this section as individuals.

The event was designed to act as a discrepant event for the students. The system behaves in a manner that surprises those who have not seen this demonstration. The intent was to elicit from the students why this system behaves as it does. Would any student hit upon the correct explanation? What various incorrect concepts would the students fall back upon in this novel situation?

This section was intended to have a look at program learning outcome six above: would the students explain observations of new phenomenon using the theories of science?

In part two the students were asked to perform a very quick and simple experiment. The intent was to have students perform an experiment (3), gather data (2), utilize appropriate procedures (4), express results of the experiment and make predictions from their data (5), and then show a knowledge of some of the fundamental concepts of science (1). A single experiment of limited scope can only provide a glimpse into the students scientific capabilities, but a worthy glimpse.

Analysis

Part one

Of 21 students, only five predicted that the water would remain in the glass. Sixteen predicted the water would fall out. This is not unusual, the event is generally considered counter-intuitive, a discrepant event. Tik K. Liem has authored a series of science laboratory guides that build curriculum around discrepant events. These events challenge a students world view and often force students away from stock, memorized answers and solutions.

Of 21 students, no student hit upon the correct answer. The following table lists in abbreviated form the students answers:

Why
because of the kind of paper
card sticky when wet
force and pressure
force of water sucks up paper
I don't know
no air in water
no air in water
no air pressure in cup
no air pressure in cup
no energy in the cup
paper is getting wet
paper is strong w/ a force
paper sealed by moisture bond
paper sticky
pressure in cup suck
pressure in the cup hold cardboard
pressure in the cuphold cardboard
pressure inside suck
water absorb paper
water pressure pull on paper
water stick paper

A number of students, at least six, relied on a "suction" theory. Unfortunately "suction," like "centrifugal force" is a phantom force. Despite all of our experience with vacuum cleaners, nothing actually "sucks." The energy and force are, on the contrary, on the other side: an area of high pressure attempts to move towards or press towards an area of lower pressure. A vacuum cleaner motor removes air from a canister, reducing the air pressure. The outside air then seeks to find a way into the canister. The force, the pressure, is from the outside in, a push. Not a suctional pull.

Four students believed that lack of air in the glass was critical. Actually, the experiment is rather tolerant of air in the glass. In a future run of the demonstration there might be information profit in running the demonstration again with air in the glass and going for a second round of predictions.

Some will interpret the failure of the students to hit upon the correct explanation as a call for teaching the concepts surrounding pressure better. On the contrary, look past the specifics of this demonstration. The students relied on previously held concepts or concepts made up at the moment of observation. The key result is that no student fell back upon learned knowledge. Not all students had been in physical science or chemistry, but some students had been in each.

The meta-failure is that our students do not think scientifically or analytically. This is not unusual, studies in the United States have found the same result. I have seen students with degrees in engineering who fall back on non-scientific explanations when faced with new and puzzling phenomenon. This is has been linked by some to the willingness of Americans educated in science in the schools to also believe in astrology and pseudo-science.

Part Two

Students showed a consistent ability to gather data in a simple experiment, plot that data, and make basic inferences from their data. All 21 students gathered essentially correct data. Seventeen students charted the points and put a line through their data. Fourteen students were able to correctly look up a y-value (bounce height) from a given x-value (drop height) using their line. Ten students were able to perform the inverse operation: correctly look up an x-value (drop height given a y-value (bounce height).

This skill, of reading a graph and interpreting the result, is a basic inferential skill. With success rates of 67% and 48%, student success on this basic skill is similar to overall performance rates in mathematics – an underlying skill to graph reading. Overall success rates for mathematics program learning outcomes hover around 55%.

Twelve of 21 students were able to calculate the coefficient of rebound. This shows a slightly better than half capacity at following a set of directions that ask to derive a value from experimental data. It should be noted that stability of performance on such a measure year-to-year may be complicated by the small sample size and the student sample. The class is overrepresented in marine science and HCOP students with respect to the general student population. This may skew the year-to-year results were the class to see a more representative student sample.

When asked to use the coefficient of rebound to try to determine the bounce height for a drop of 678 centimeters only a single student succeeded. The student was not, as might be anticipated, a HCOP, marine science, or business major (who showed high rates of success in mathematics). The single student was a 54 year old female of unstated major who notes that she has been out of school for years.

In the final section the students were asked what type of energy the ball possessed when held aloft just prior to being dropped, what type of energy the ball possessed just prior to hitting the floor, the name of the theory that suggests the energy aloft should be equal to energy with which the ball hits the floor, and why doesn't the ball bounce as high as it is dropped. The answers in raw form are instructive.

sqenergy aloftenergy bottom theory namewhy not as high?
1 kinetic gravitational energy Newton's first law amount of fource that was put on the ball equals to the amount it was dropped.
2 kinetic gravitational energy Newton's first law amount of fource that was put on the ball equals to the amount it was dropped.
3 kinetic gravitational energy
4 kinetic gravitational force Newton's theory
5 kinetic gravitational force energy between height and drop
6 kinetic gravitational force
7 kinetic potential Newton's theory
8 kinetic potential gravity
9 kinetic potential gravity
10 magnetic electromagnetic plastic not have any space in it gravity push it down
11 mass velocity relativity In every change of energy some is lost
12 potential kinetic accelerated gravity
13 potential kinetic accelerated gravity
14 potential kinetic force kinetic and potential energy
it has more height
gravity

Nine students mistakenly thought the energy aloft was kinetic energy, only three correctly identified that the ball possessed potential energy when aloft. Only the same three correctly noted that the potential energy had become kinetic energy a the bottom of the ball fall. Note that none of these three then went on to correctly explain why the ball does not return to its drop height. Only number eleven above correctly reasoned the loss of energy at interchange points – energy is conserved, but that conservation include energy converted to heat and sound.

More disturbing is that not a single students could name the theory of the conservation of energy, one of the most fundamental guiding theories of the universe and everything. Conservation of energy, mass, linear momentum, and angular momentum are bedrocks of Newtonian physics. Dozens of problems are solved only with the understanding of these principles.

In general our students have very poorly developed understandings of the concepts of science. The vocabulary and language of science does not come easily to our students, and they are certainly confused about the difference between kinetic and potential energy.

Conclusion

As a first look at the pilot instrument the trial has been successful. The instrument did yield insight into what students can do and what they know even in the relatively unstructured environment of an open answer evaluation. The lack of multiple choice answers is necessary in order to not constrain the students thinking with limited answers or to prompt memories of encountered ideas. The open answer instrument provides a better idea of what students can do or what they know as a part of who they are as they walk through life. No student studied for this evaluation, the evaluation was a surprise to the students.

The students solutions will be retained for future reference. The recommendation would be to run the same instrument spring 2005, although year-to-year comparisons should be meaningless given the non-random nature of the sample.

The students undoubtedly leave our institution with basic charting and chart reading skills, plus some basic computational skills. Obviously these students would make highly confused and uncertain science teachers, but then few of them are heading directly into a classroom.

NexusDNSMWorkCOMFSM