Unified Assessment Plan: Flowing assessment up the plan
> ...how soon can we have [a]
clear picture of whether our PLO's, CLO's and SLO's are viable or need
further assessment...To be honest our PLO's are clear,... - BM
Bear in mind that the "unified assessment plan," or whatever one wishes
to call it (someone suggested Fairly Obvious and Objectively Logical
plan), takes those "clear PLO's" and provides a framework from which to
generate inferential information on achievement of program and
institutional level assessments.
In the world of mathematics we cannot prove that the square root of two
is irrational. We can only prove that it is not rational. The only
other choice left is that the root is irrational. Inference works in
this manner. I may never be able to show that the communication
program learning outcome is directly achieved by measuring learning at
the program level. I can, however, show that the program learning
outcome can never be achieved if I am able to show that no course in
the program has an outline with a communication student learning
outcome. The matrices in that spreadsheet show distributions of
institutional effort: where we are putting our emphasis.
We have fine PLOs, no need to alter these.
Obviously the holy grail, that which everyone desires, is to know what
a graduate knows, can do, and values as they walk down the graduation
aisle. Like the holy grail, this may prove ethereally elusive. And in
the end it will turn out not to be a gold chalice but a simple
carpenter's cup.
In pursuit of knowing what a graduate knows, one's mind is likely to
leap to exit tests. Yet if I gave the entrance test to the faculty, my
guess is that many would place into developmental mathematics if not MS
090 - the lowest developmental level at this time.
I suspect that ultimately we will look at the achievement of course
level student learning outcomes and use those to infer accomplishment
of program learning outcomes, which can in turn be used to infer
accomplishment of institutional learning outcomes. I have no idea if
this is acceptable, but it simplifies things in my small mind. One
measures student learning outcomes at the course level. The higher
levels are inferred. Faculty would only have to focus on what students
are mastering and not mastering in their own courses and then reporting
this information.
Permit me to digress in order to come back to my point.
I was asked today
whether we would eventually be able to generate numbers such as "70% of
the students communicate effectively through writing, speaking,
performing, exhibiting, or other forms of expression." Or to use such a
statement as an institutional objective, "70% of the students will communicate effectively
through writing, speaking, performing, exhibiting, or other forms of
expression."
Before I share
my response, think for a moment what one would have to do to be able to
say this. One would have to do as one instructor in my division
attempted to do this term: to measure every outcome for each and every
student individually. It looked achievable in August and we discussed a
possible presentation on the system in December. The faculty member has
asked to defer the presentation until after the spring term, I gather
he has found a need to modify the system. My understanding is that the
data entry load was ferocious. Bear in mind each outcome was to be
assessed ten times during the course of the term for each and every
student to ensure reliability of the assessment.
I have long argued
that one's ability to handle the data entry and tracking load varies
with the sheer number of outcomes and students. If the outcomes are few
and the class sizes small, this is a feasible approach. Learning can be
proven and documented on a student-by-student basis.
In larger classes
with large batteries of outcomes record keeping and management become
problematic. Proving learning becomes difficult.
Gödel proved the
unprovability of "sufficiently complex systems" once and for all in
mathematics - the home court of quantification. Any sufficiently
complex system will have theorems that can neither be proven true nor
shown to be false. I have long asserted that proving achievement of
student learning, especially that which the student knows, can do, and
values at graduation, will have Gödelian limitations.
One thing I
suspect we can do is the internal inferential model such as that
proposed by the so-called unified assessment plan and other inferential
models such as those used at Mesa College. At a recent workshop the
president was given the Mesa model. I noted that their "program
assessment model" was, at its core, a set of three surveys. The
students, the community, and employers. The surveys sought to determine
the value of a particular program to each of those stakeholders.
Again, inferential.
My response to the question of whether we would
eventually be able to say, "70% of the students communicate effectively
through writing, speaking, performing, exhibiting, or other forms of
expression," was
that I thought this is something the faculty should probably decide.
That is, whether or not we would want to attempt to generate such data
or set some such goal.
Before I digressed, I said that I wanted to
infer the higher levels. So how might this work, beyond the analysis
work I've done in the "unified assessment plan" or "grand unified
theory"1 as I dubbed it in my first emails? Although I do
not know, what follows is one possibility. The following presumes one
is familiar with the unified assessment spreadsheets. The spreadsheets
are available at:
http://www.comfsm.fm/~dleeling/assessment/islos.xls
Take a small piece of the p versus c HCOP spreadsheet tab in
the islos.xls spreadsheet I sent to the accreditation working group and
the curriculum committee on the 11th of December:
Students
will be able to: |
analyze |
foundations |
human
body |
problem
solve |
Define
the concept of total fitness and the essential characteristic and
consideration of a physical fitness program. |
|
1 |
|
|
Describe
the process of conception as well as the influence of heredity and
environment to the developing human organism. |
|
1 |
|
|
Describe
what it means to be healthy. |
|
1 |
|
|
Discuss
the effect of drugs to the human body, human behavior and its
consequence to family and society. |
|
|
|
1 |
Enumerate
the causes and prevention of communicable diseases. |
|
1 |
|
|
Explain
how the environment can affect health. |
|
1 |
|
|
Explain
the body functions and the relation of each system to one another. |
|
|
1 |
|
Explain
the effect of the electrical signals and chemical messengers to human
physiology and behavior. |
|
|
1 |
|
Suppose I choose this year to focus on the
foundations program learning outcome. Having ruled out the direct
approach above, I would sit down with the faculty teaching each of the
courses and "survey" them on each course level outcome that "matrixed"
against foundations. The results for the small piece above might look
like:
Students
will be able to: |
analyze:
2005 |
foundations:
2006 |
human
body: 2008 |
problem
solve: 2009 |
Define
the concept of total fitness and the essential characteristic and
consideration of a physical fitness program. |
|
Students
have a good understanding of the concept based on tests and in class
discussions, according to the instructor. Yet few exhibit behaviors
that show they actually value fitness. |
|
|
Describe
the process of conception as well as the influence of heredity and
environment to the developing human organism. |
|
Students
have a good understanding of this concept. |
|
|
Describe
what it means to be healthy. |
|
The
students are able to respond correctly when asked this question. |
|
|
Discuss
the effect of drugs to the human body, human behavior and its
consequence to family and society. |
|
The
students are very capable at expressing themselves on these effects,
whether orally or in writing. |
|
1 |
Enumerate
the causes and prevention of communicable diseases. |
|
Students
are able to enumerate the causes and prevention. |
|
|
Explain
how the environment can affect health. |
|
The
students exhibit a number of misunderstandings and misconceptions
concerning this topic. |
|
|
Explain
the body functions and the relation of each system to one another. |
|
|
1 |
|
Explain
the effect of the electrical signals and chemical messengers to human
physiology and behavior. |
|
|
1 |
|
Is this direct assessment? No. In
"Benchmarking to Improve Departmental Systems for Managing Quality and
Standards," James Tannock and Norman Jackson include the following
quote from their research, "A major theme in deciding on the type of
assessment to be used the amount of marking time involved. Staff will
tend to resist more student-centred [sic] formative assessment methods
when these involve more marking time." In other words, getting the
actual data of how many students mastered each of the above is likely
to fail, especially when we have some faculty in open opposition to the
whole "student learning outcomes" approach.
The proposed hybrid returns the workload to the chair to work with
faculty and generate information that can then be aggregated and
reported. After surveying all of the outcomes in all of the courses
that impact the targeted program learning outcome, the chair or
instructional coordinator could then summarize the findings in the
cross-matrix between the institutional student learning outcomes and
the program student learning outcomes.
Thus the i versus p matrix sheet at row 63:
Prgrm |
program
Student Learning Outcome: students will be able to: |
iSLO:
analysis |
iSLO:
communication |
iSLO:
creativity |
iSLO:
intellect |
iSLO:
knowledge |
iSLO:
skills |
HCOP |
Demonstrate a
foundation in basic biology, chemistry, microbiology, anatomy,
nutrition, health, and physiology. |
|
|
|
|
1 |
|
Becomes:
Prgrm |
program
Student Learning Outcome: students will be able to: |
iSLO:
analysis |
iSLO:
communication |
iSLO:
creativity |
iSLO:
intellect |
iSLO:
knowledge |
iSLO:
skills |
HCOP |
Demonstrate a
foundation in basic biology, chemistry, microbiology, anatomy,
nutrition, health, and physiology. |
|
|
|
|
Students show solid
foundations in these fields, with the exception of nutrition. In
general, students tend to be capable of reciting the facts but show
evidence they value the information as evidenced by a lack of changes
in their behaviors. |
|
Here my reach is exceeding my grasp, I am
uncertain that terse summaries are possible.2 If they are,
then the i versus p matrix would become, after five to seven
years, a "single sheet" overview of how well the institution is meeting
program and institutional learning outcomes.
The above approach seems to me to be
achievable and to work with some of the practical realities. Work loads
on higher level assessment remain with program coordinators (chairs and
instructional coordinators) rather than on faculty.
General education would require each "area coordinator" - each chair
and instructional coordinator - "chipping" in on their area, or
something like that, contributing to the i versus p matrix for
their area.
My apologies if I have overstepped my bounds or am simply too far off
the beaten track with these ideas. This is definitely a model that I am
developing as I go. Each new piece leads me on around a bend to another
concept, but at some point I may indeed be "around the bend." The
unified assessment plan laid out in islos.xls, however, feels useful
because the structure itself creates options and possibilities for
assessment of the structure.
Dana
1 Borrowed from high-energy physics, the acronym also
connoted that the theory makes "gut sense" to a physicist. Occam's
razor simply demands that this is how the universe works, at least
according to the proponents of the theory. I cannot remember if super
symmetric theory is predecessor to, a part of, or successor to GUT.
2 I am reminded of the story I first heard told by Pete
Seeger about the three wise men. The king had an heir and wanted the
heir to learn all that there is to learn in the world prior to
ascending to the throne. The king set his three wise men out into the
world to compile all the knowledge in the world into a single book.
Three years later they return with a massive tome and the king presents
the volume to the royal tutors and the heir.
The king, having nothing better for the three wise men to do, said,
"You did such a fine job on the book, go back out into the world and
boil down all the knowledge of the world into a single sentence." Ten
years later the wise men returned, looking increasingly aged and worn.
"Sire," reported the wise men, we have done as you asked. Then the wise
men said, "Seek and you shall find, knock and the door will be opened,
ask and you shall receive."
The king was thoroughly impressed. So impressed he asked, "Could you
boil all the world's wisdom down to a single word?" The three wise men
committed to try.
The years past and no one heard from the wise men, the king thought
perhaps age had caught up with them and they would not be returning. As
the king lay in bed, his health fatally failing, the equally weakened
wise men returned. Ancient beyond reason, the three wise men collapsed
by the king's bed. The king, his voice reduced to a whisper, said
"Well?" The wise men said, "Maybe."