[assessment] Update on assessment in MS 150 and SC 130

MS 150 Statistics

The mapping and aggregation of item analysis results continues to drive MS 150 assessment. This is the first term that I've tried tackling item analysis in MS 150 for all instruments. The work load is considerable. Coupled with physical science, grading and analysis typically requires working Friday evening and throughout the day on Saturday. Data entry usually occurs on Sunday morning. I have not been logging the time consumed, but it is something like twelve hours of work each weekend.

That said, the results do provide insights I can use in the classroom. One result was that I spent an extra day on linear regressions and added them to an additional quiz, with very positive results.

Q Description l Outref SLO Corr Corr%
1 linear, non-linear, or random 5 5 Perform a linear regression and make inferences based on the results 48 100%
2 slope 5 5 Perform a linear regression and make inferences based on the results 45 94%
3 intercept 5 5 Perform a linear regression and make inferences based on the results 43 90%
4 correlation r 5 5 Perform a linear regression and make inferences based on the results 45 94%
5 positive or negative relation 5 5 Perform a linear regression and make inferences based on the results 48 100%
6 strength of relationship 5 5 Perform a linear regression and make inferences based on the results 45 94%
7 coefficient of determination 5 5 Perform a linear regression and make inferences based on the results 42 88%
8 perc var x that explains var y 5 5 Perform a linear regression and make inferences based on the results 31 65%
9 predict y given x 5 5 Perform a linear regression and make inferences based on the results 26 54%
10 predict x given y 5 5 Perform a linear regression and make inferences based on the results 22 46%
11 probability 1 1 Calculate basic statistics 38 79%
12 probability 1 1 Calculate basic statistics 9 19%
14 three ways to deter. prob. 1 1 Calculate basic statistics 6 13%

Student success rates on the linear material exceeded 90% on many questions, with all students answering questions one and five correctly. These are strong rates of success on his material. Historically these numbers are generally on par with performance or, in a number of instances, better.
Fall 2005 fx Fall 2006 fx Fall 2007 q04 sSLO
95% 29% 94% Calculate the slope of the least squares line. (Calculate)
95% 92% 90% Calculate the intercept of the least squares line. (Calculate)
65% 92% 100% Identify the sign of a least squares line: positive, negative, or zero. (Define)
53% 90% 54% Solve for a y value given an x value and the slope and intercept of a least squares line. (Solve)
45% 48% 46% Solve for a x value given an y value and the slope and intercept of a least squares line. (Solve)
91% 37% 94% Calculate the correlation coefficient r. (Calculate)
82% 92% 94% Use a correlation coefficient r to render a judgment as to whether a correlation is perfect, high, moderate, low, or none. (Interpret)
98% 42% 88% Calculate the coefficient of determination r². (Calculate)
73% 94% 65% Calculate amount of variation in dependent variable explained by variation in independent variable. (Calculate)
76% 63% 100% Based on analysis, infer whether a linear regression is appropriate? (Interpret)

There has been fall off in the ability of students to use the slope and intercept to predict a y value when given an x value against fall 2006, but the performance is on par with fall 2005. This concerns me only in that this is a fundamental skill that mathematic instructors likely believe their students can do at the end of MS 096, but putting the pieces together from real data is more elusive than the simple "Given x = 9.45, determine y from the equation y = 0.15x + 0.08" In MS 150 one is given multi-point raw data requiring a linear regression to find the slope and intercept, and then one has to go on to use these values. Little wonder the new Mathematics Association of America Curriculum Renewal Across the First Two Years project's College Algebra Guideline endorsed on 31 January 2007 called for "Data analysis including collecting data, applying algebraic transformations to linearize data for analysis, fitting an appropriate curve to a scatter plot, and use the resulting function for prediction and analysis."  This is not meant at the "add-on" in section 3.5 or whatever of Hostetler, this is the third major leg of the course!  A Google search for the CRAFTY College Algebra Guidelines should turn up the guidelines.

Aggregating all student performance to date yields the success rate on the proposed outline outcomes:
Outref Students will be able to: Sum Count Avg
1 Calculate basic statistics 21.62 40 54%
2 Represent data sets using charts and histograms 2.96 8 37%
3 Solve problems using normal curve and t-statistic distributions including confidence intervals for means and hypothesis testing 0 0 0%
4 Determine and interpret p-values 0 0 0%
5 Perform a linear regression and make inferences based on the results 16.3 25 65%
PSLO define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas.

31.26%

Aggregate accomplishment of the program learning outcome continues to rise.

Program learning accomplishment is climbing up

Bear in mind that 100% accomplishment is not possible. The aggregate performance includes all performance to date. Poor performances early in the term will "weigh down" the average throughout the term. To remedy this, I run a separate aggregation analysis on the final examination alone.

SC 130 Physical Science

Performance on quizzes remains abysmal, but those students who are completing laboratories (74% on the most recent lab, 81% overall term to date) are producing laboratories that show improved reasoning and writing skills. The most recent quiz suggests students are approaching the course in the usual manner - studying and learning only this week's material.
Q Description Corr Corr%
1 calculate pace 10 32%
1 units 9 29%
1 significant digits correct 10 32%
2 calculate velocity 3 10%
2 units 3 10%
2 significant digits correct 2 6%
3 calculate momentum 0 0%
3 units 0 0%
3 significant digits correct 0 0%
4 calculate kinetic energy given formula 3 10%
4 units 2 6%
4 significant digits correct 1 3%
5 temperature of boiling water? 30 97%
6 temperature of healthy human body? 21 68%
7 what happens at zero degrees celsius? 23 74%
8 best heat conductor in lab? 22 71%
9 best heat insulator in lab (terminology issue) 10 32%
10 list three ways to "move" heat 26 84%
11 explain the three ways heat is "moved" 2 6%

Questions one to four came from material in prior weeks. Question one was a drop dead simple divide the two numbers and write the answer. Only a third of the students answered this correct. Questions two to four were dependent in that getting number two wrong would lead to answering three and four incorrectly. Two was difficult - students had to convert from minutes to seconds and kilometers to meters to get a final answer in meters per second. Only ten percent could answer this correctly, and these three students also knew to include the units, but only two were able to determine the correct significant digits.

The momentum result is especially disappointing given that this was the focus of a laboratory, but then only three students even had a shot at answering correctly. It did appear that some students used the correct formula, but because their velocity was wrong it was a case of bad numbers in, bad numbers out. That lab has already been rewritten to focus more clearly on momentum.  Question four included the formula for kinetic energy, making it slightly easier and the three who answered two correctly went on to get four correct.

The higher rates of success on the end questions was this week's lab and lecture material. Students did well on simple single word recall and on the "list three" question. When asked to explain concepts, as in number eleven, only two students explained all three concepts correctly. Other students were able to explain one or two forms of heat transport, with radiation being the most apprehensible for the students.

I have not been item analyzing the laboratories, although there is potentially useful information to be gathered from an item analysis. The students this week were instructed to design their own data tables, data charts, and data analysis sections. Because of the variety of choices made by students, I altered the rubric by collapsing these three sections into a single data section. The latest iteration of the rubric is given in the following table:
1. [d] Data tables, charts, graph, display, analysis
3 Very well laid out, communicated well
2 Minor issues or somewhat incomplet or missing an appropriate element
1 incomplete or unclear or confusing or diagrams without axis labels or that otherwise do not make sense
0 Omitted
2. [c] Conclusions
4 Strong reasoning based on the data, logically reasoned, complete and thoughtful
3 Moderately well-reasoned, but tangential to the data or based on misconceptions
2 Weak reasoning or overly brief or compromised and impaired by grammar and syntax errors
1 Unclear, not well reasoned, or highly incomplete, or unusually weak
0 Omitted
[G] Grammar and Syntax
3 No errors of grammar or word order.
2 Some errors of grammar or word order but communication not impaired.
1 Frequent errors
0 Errors of grammar or word order so severe as to make comprehension virtually impossible.
[V] Vocabulary
3 Appropriate terms used consistently, clear command of vocabulary with a focus on correct usage of physical science vocabulary, no misspelled words.
2 Occasionally uses inappropriate terms or relies on circumlocution; expression of ideas not impaired; or a few misspelled words.
1 Frequent errors in vocabulary or spelling, ability to communicate limited by vocabulary
0 Vocabulary limitations so extreme as to make comprehension virtually impossible.
[O] Organization
3 All sections present in the right order
2 One section missing or out of order
1 Multiple sections out of order, or turned in as separate spreadsheet and word processing docs
0 Only one section or no sections apparent.
[C] Cohesion
3 Ideas flow logically. Connector words assist the reader.
2 Ideas are disconnected and the conclusion reads more like an outline or answers to a list of questions without connector words. There is a choppy and disjoint sense to the writing style.
1 Communication impaired by a hodgepodge of inappropriate and misused cohesive structures
0 Incomprehensible collection of disconnected ideas and words, or only one sentence.

Although only 23 of 31 students turned in the laboratory, the average was a 15 out of 19 possible points with a median of 16 and a mode of 17 points. In general students are performing well against the rubric. One might note that this rubric, in collapsing three content sections into one content section, placed a heavier relative weight on grammar, vocabulary, organization, and cohesion. This is not inappropriate to the design of the course in which writing is intended to have a strong role. At present 44% of the overall points are in the labs, 48% are in quizzes and tests. Attendance and homework constitute the remaining eight percent of the points.

I have the perception that some students are accustomed to slipping into lab a little late or sliding out a tad early, I am tracking this by taking attendance at the start, middle, and end of labs. Long trips to the bathroom around 1:15 may mask a quick trip to the cafeteria.

That said, this past week I was pleased with the spontaneous development in both lab sections of class discussions on the report format. In the second laboratory in particular, a lengthy class discussion occurred as the students wrestled with how to best present their data. I would note that this was not lecture discussion - I did not play a role in either discussion. In the 11:00 section the discussion was going so well I even opted not to answer a couple questions asked of me - it would have shut down discussion and provided a "right answer." The students were engaged in the process of science - debating how to put together tables, what data was truly relevant. As I noted, my goal is always to build a constructivist learning environment which I can only hope will maximize the SLOs: the significant learning opportunities. I look to create significant learning opportunities. If all goes well, then student learning is the outcome of these significant learning opportunities.

Physicists, astronomers, and their ilk are nearly infamous for their prodigious coffee consumption. I'll have to look it up, but the world's first web cam was a digital camera rigged up by graduate students in a physics lab in Cambridge or some other such university. The students, working on a another floor, wanted to know when the pot was full so they could go grab a cup of coffee. So they mashed up some hardware and wrote code to sent a series of images of the coffee maker across the building's ethernet network. Now web cams are common. In view of this history, I have made coffee the past two Thursdays for the 8:00 class only. This past Thursday I also brought a bag of donuts. This was a big hit with the students, most of whom come to the 8:00 class without having had breakfast. This simple act really seemed to change the spirit and atmosphere of the laboratory. Their only complaint was that my own penchant for black coffee means I did not bring in sugar or creamer. Maybe next week.

I would note that the physics laboratory is usually a "food and drink safe" environment. In chemistry there are too many dangerous chemicals around to risk drinking in the laboratory. Biology has its bacteria and preservatives. The physicists, meteorologists, geologists, and astronomers all tend to inhabit less toxic laboratories. Of the physical sciences only chemistry itself regularly includes highly toxic compounds. With chemistry already offered at the college, I use physical science to focus on the other physical sciences. We will touch on chemistry late in the course, but I already have determined that few if anyone have learned to "read" the table of elements let alone done anything beyond this in chemistry.

As always I share my explorations in assessment in the hope that my efforts may provide ideas to others on how to get at formative assessment information in their own classrooms. I also hope that others will share their own assessment explorations.