Ladera Institutional Assessment Institute

21 May 1998 to 23 May 1998
Ladera Tower, Guam
By: Dana Lee Ling Title III Math Science Specialist at COM-FSM.

This is a transcriptional report of my stay on Guam and the Institutional Assessment conference. We arrived on 20 May 1998, the conference started on 21 May 1998. Four digit numbers that appear periodically are Guam local time in 24 hour format.

During the conference, comments in [square] brackets are my own opinions, thoughts, comments, daydreams, and mental digressions. I am not a court reporter, I do not even type very fast. This transcript is surely faulty, a missing "not" may turn a whole sentence upside down. I apologize in advance for any errors. Please let me know if you are aware of errors by emailing me at dana@mail.fm

Unfortunately some sections of this report make the most sense when you have the conference notebook to refer to. Transcript sections that make no sense in and of themselves often make perfect sense when you refer to the proper notebook section.

James did the bulk of the speaking, and most unlabelled paragraphs belong to him. I did not learn names so I frequently refer to unknown speakers as "audience."

0834 Joe, John, Eric and I arrive at the conference at Ladera Tower and sign in. We are registered but there is no receipt for registration. A projection unit displays what appears to be a PowerPoint presentation.

GCC spent a year examining models. We adopted a model and piloted in tourism and... we found that varying degrees of utility for the models. In last ten years every time we look at a model one of the first things that would pop up there would be an instrument. Somehow my mind could not adopt the idea of an instrument that looked at data as a starting place and then focused on data. This Institute was the first one I saw that did start with an instrument.

In the light of competing financial need areas we needed a model that would provide output information to the community. Also wished to assess institutional effectiveness.

Introduction: Dr. James Ratcliff Director National Center Post-Secondary Teaching and Assessment. Blonde hair, tall, blonde mustache. Conducting assessment institutes since 1992. Works on custom crafted assessment models as opposed to out of the box assessment solutions. We will explore ways in which we can use the assessment process to better understand issues the students confront and that you face. How can we structure ourselves so we can learn more about the teaching and learning process. We will not look at measurements. Tomorrow how to gather information. Third day how to use information to effect change in the teaching and learning process.

The notebook includes items that we feel are important to assessment. We are on page one of section A in the notebook. This is a PowerPoint presentation. He covers the PowerPoint screen. How does the institution affect the lives of students after they have left the institution? If you have focused on student learning you have captured most of what WASC wants to be captured.

Institute helps seek new ways to frame assessment and invent ways... see screen 4. This is a messy business. The things that can be measured are very limited relative to the number of things that relate to teaching. There are a lot of things we have spent time reflecting upon that are not reflected upon by people outside of our areas. You may for years be able to listen if you are a music teacher and tell the difference between a different levels of skill. But can you explain that to the non-musician. Assessment asks us to think about the process in a very clear and overt way. We seek to understand the conditions under which the student understands.

Can the students be made partners in the process of learning? Think about what are the essential elements of an assessment plan. What are the factors and conditions that affect your assessment plan. How do you make the assessment plan yours and not just an add-on demanded by other outside agencies. You often find institutions with a pile of tests and when you ask what the test tell the College they will say " we don't know but we were required to do these tests."

0913 How do goals and adjectives differ. Outcome is some manifestation of the goal? What do you think of when you hear the word "assessment?" "Tests" "Finals" "Faculty evaluations." Why do you think of grades and assessment? That is how students see assessment. A nicer softer word than test. We try to show you there are fundamental differences between test and grades. What about faculty evaluation? "Universities and colleges want a way to convince themselves that faculty is doing the job you are supposed to be doing."

Student evaluations are often confused with assessment. Usually done at the end of the year. While they are useful tools, we conducted a 4000 student study and found no [correlation between student evaluations of faculty and student achievement.] Two factors were found to be significant:

Unique here: student assessments. Only the survivors evaluate the instructor as many students have already dropped out. Student evaluations are a different enterprise. Institutional assessment is whether students learned anything.

Where did students start and how much did they improve.
Did they attain the level of skill required?
Did they acquire new information?

In the spirit of assessment maybe it would be good to look at some of the questions that are directed at your own campus and your own institution.

1. What elements in the external environment that are impacting our institution? Obviously the terrible things that happened in Asia have had an impact. Quality of primary and secondary schools has a bearing on quality of students. Are there major issues in regards to certain vocational programs at your school that are short on students but long staff or short on staff but long on students? What kind of information would be useful to focusing your mission?

2. Think about the history of your institution. Background in teacher education or vocational or professional area like a school of nursing. Has each evolved to a general community college?

3. What are the values and beliefs that surround assessment? It plays out against societies and cultures. How is that played out? What is the critical information? What are the concerns surrounding the information? Perry Mason never let data speak for him. He let the witness make his case. So it is with assessment information. It is incumbent on you to interpret what the assessment information means.

GCC (Cel Babauta) is impacted by government primarily. Also by primary and secondary education.
UOG is impacted by articulation.
Student preparedness impacts UOG.
Regional articulation impacts GCC. Articulation with other colleges will attract more students regionally.

Negative: whenever government feels like it money gets pulled.
Lack of recognition of importance of GCC to Guam from government.
Positive impact: GCC's relationship with Dept. Of Education.
Occupational programs at GCC are unique to GCC.
A guiding principle: Go along to get along.
GCC: Closer ties with business and industry.

GCC impactors
Extended family
Political funding: in a good year it is good. In a bad year it is bad.
Legislature changes every two years

Demographic: migration and how to adapt when people come from the outer islands. That is a challenge. Downsizing military personnel. Within Asia we are a hub. We are working on distance learning. Industry partnerships. Educational elements. We have competition and external and articulation with our neighbors across the street and the lack of articulation. Duplication in programs. UOG competition. Private industry pseudo competition like the Institute of Aviation but they also represent a possibility for cooperation. We used to be GCC Votech. We are satelliting out our HS programs and focusing more on our collegiate role. Vocational role is a defining element. We have good programs. Customs, GFD, police, tourism marketing and all the other good stuff that we have.

John Haglelgam: Our group here consists of PCC and COM-FSM. We share a common problem. We found that the one external element that impacts our ability to teach was the fact that our secondary educational system always gives us students who are not ready to enter college. We have a lot of remedial courses. We require they take remedial courses until they pass the TOEFL. We run into a problem. A student will keep on repeating one course semester after semester and will use up their eligibility for Pell grant. Once used up they simply withdraw from school.

We find that the problem is only confined to the students who come from the public high schools, the private high schools pass the TOEFL.

Another external element is political. Funding. We find that sometimes to be difficult and uncertain. In FSM we is very the island are very dispersed over a wide area and covers many cultures. Getting them to come to Pohnpei. The school pays their way to come to the national campus. Another cultural problem:

Students from Yap etc., come from five different cultures. Or six. The cultural problem is a problem. Differences between faculty and student cultures. Faculty is from USA, Africa, India. We are truly an international institution. And

Eric: Cultural context problems. Not only language but and not only with faculty, the bulk of our faculty are not from Micronesia.

Woman: We are a vocational institution. Many are familiar with GCC. We feel local students and businesses have had positive experience. We have a good balance of markets... talented staff, young overall, adapt quickly.

Different person: some cultures differentially value education, some want immediate gratification, many look towards government Guam for employment. Employment instabilities. There is a negative impact of constant stress. A supporting and caring environment. We do not have fiefdoms (a term used at UOG).

James Ratcliff (James from here on): Educational content has changed over the past twenty years. I saw a ditto machine three weeks ago. I went up to it, I just had to smell it. How have adjusted to these things? We try to cram these things in. And then we have declining resources. High stress high caffeine diet. Financially stressed institutions have often been the leaders in assessment.

1025 Portland State developed innovative program in working through curricular changes with local community colleges. They developed this in response to mandate to reduce budget by 5% each year for five years.

There is a big gap between the understanding of what a college level education is and the preparation. An agenda item for assessment might be working on assessing cooperative efforts between the college and lower levels of education.

Assessment is a way of describing student learning to identifiable audiences for some good reasons. We generated a variety of issues that might drive the assessment process. Assessment as a process [If assessment is a process] then you need to think about information for improvement. It is sometimes an annual event or driven on whatever schedule is appropriate. There is an information loop that assessment creates for improving systems. Some definitions. Measurement: the gathering and quantifying information. Assessment occurs when measurements are analyzed. Evaluations are when you take courses of action based on the assessment.

Think of a family photo album. If you want to show the development of a child you would not show just one photo but rather a time sequence of photos. The assessment endeavor is the same. You do not want to take a picture everyday. Just at critical times, important times. You want information at key points in the educational career of the student. At intake. Annually. At diploma time. After graduation.

Driving this process are feedback principles. The information must be useful and timely. Assessment benchmarks student learning, can reduce bureaucratic regulation wherein the college becomes responsible for its own performance, clarify and give direction to higher education...

Question man: There are some students who come to school not to learn but to be with friends or come because they desire some specific certification. We are not going to focus on that?

We would like all students to be motivated for academic reasons. There are two baskets we can put students into... [three?] Those who are really there to learn journalism. Or they really want to learn law enforcement. The second are those who want to learn what college is all about and they have mythical vision of college. But they are not there for a specific reason. The second group is a group that is there for social reasons. Social reasons are defined differently in different colleges. A private Baptist JC in SC a lot of kids came from different Baptist congregations. They come for the religious environment. That is a different motivation that students going to Penn state for football or to be a Greek.

1106 That social factor may be the greatest predictor of success and yet we focus on finding students who are there for purely academic reasons. The third: the pragmatic student. A course offered at the right time of day. Courses taken to lead to a specific certificate. Students who mix part-time employment with education sometimes find the part-time work environment more rewarding and then leave education even though the choice limits future opportunities.

Comment: One pragmatic student we have is the student who comes to school on Guam with an eye on being able to go to stateside schools in order to have an "off-island" experience.

James: SACS accredits in the Southeast and Caribbean including the College of the Bahamas. CB has a similar role to GCC in some ways.

SACS example. Southern Association of Colleges and Schools. See example for details. The one credit hour capstone speech requirement to meet oral skills requirement. Faculty judged the presentation. Might not have any background in the speech subject area. So they did a faculty development in speech judging given by their speech arts department. Unsatisfactory performance led to referral to a speech department coach. Then they would try the speech again.

WASC: Each institution develops a broad based institutional planning and assessment and how the information will be used for institutional improvement. The later is often missing. [I gather the SACS example is guilty of not closing the loop. The assessment does not lead to institutional changes.] Do you seriously take a look at all information gathered and is it useful to the institutional improvement process. Do the procedures you implement improve the teaching process.

Russ: They are us. They are chosen from other institutions. Chosen often from the same area. They look at the self-study we started preparing and they come and review the self-study and validate the findings of the self-study and reaffirm the accreditation.

David: A process of mutually assured comparable standards. Ensures some degree of uniformity and consistency.

James: This is a unique process. [Coverage of processes in Australia and elsewhere]

1132 Three kinds of information are used: background information on student, on culture, what are their attitudes to college? [We are now on the jigsaw puzzle slide B-10] [This is sources of assessment information. Lee Ling is spacing out a tad.] There are some baskets. 1) Learning in content areas 2) Critical thinking improvement, leadership learning, 3) attitude towards to learning, a commitment to continuous improvement [Page B-12 top now] in knowledge, 4) efficiency variables: progress rates, consistency in program, persistence of students in the program.

Are there gaps between goals and where we are now? Other important areas are out of class experiences [out of box, out of class, there are a lot of out of's in this stuff and plenty of baskets. I do know that out of body experiences tend to lead not to baskets but caskets...]

Organizing assessments... out of notebook slide experience. Turn to worksheets now. [We are jumping to the end of the section] Ten-step plan [good, not a twelve-step plan] We will walk you through this.

First think a little bit about the educational assessment program. What assessment would help deal with the key areas you identified earlier? [Speakers light up with static in the room] This concretely about assessing a program. Think about when an assessment should occur. Suppose thinking about transition experience for outer island students. What are you going to assess to measure the transition experience?

What and when. When would best be able to utilize the information in the college. You need time to develop a transition program. When do you need that information? Targeting to a specific audience is critical to determining:

What is the fix? Close the loop.
Identify the goals you are seeking?
Identify the levels of performance that make sense. [Step six, page C-1, which is in section A after page B-16. Go figure.] Whatever you plan will need fine tuning in whatever you do. What are the out of class learning experiences that support the educational environment? What reinforces learning and what competes with learning? Think about the first step: why are you doing this stuff? What is your rationale? Why? What for? That is different form the purpose which tells who is being assessed, what is being assessed, and when it is being assessed.

That group with whom you plan to communicate the assessment, what is that communication going to say? How is that going to be delivered? By a written document? Going to be select group of portfolios? This is fuzzy and reflective business. There is no absolute answer.

Now we are running a little bit longer than anticipated. We will work in groups for 40 minutes. Then lunch.

James: the Santa Ana group, could they share what they did prior to lunch? Carmelita, Eric, Lisa, Karen? Can you move between groups?

Carmelita (Lita): we have worked on a draft and we have sample handouts. We were looking at an institutional perspective. Here you will looking at a program level. It is a draft. A working document. We are looking to refine that draft. These are just samples for you.

James: Who is succeeding? Who is not? What are the reasons? Academic, social, economic? Do not go for a yes or no up or down the IEP is better than remedial. Go for what are the skills to be learned? Communications, listening, reading? Or another way? Try to for the purposes of developing a skeleton this is just a draft start to se if this makes sense. What are the four or five different areas a person would study? What might they be? Let me show you in the context... this is set of writing inventories J2-17 areas. A number of people were involved in this study. Student learning outcomes. Goals and evaluating their importance. Whether the five or six areas that a person spends time learning when they enter the ESL program because you want to not just say is it working or is it not but rather what areas do they build skills easily and what areas do they not build skills easily. [This way you can see what areas to target.] You want six or eight areas.

Rationale: Not all students are up to college level in English when they leave secondary school. An Intensive English Program has been implemented to bridge students up to College level. It is having mixed results. Perhaps the teachers involved in this program, in an effort to appear successful, are not being accurate in their assessment of their student's skills. As a result, we have been misplacing students, and...

Purpose: Bring students up to college level English within the span of two terms.

Students should be able to write a sentence.
Students should be able to speak clearly.
Students should be able to express themselves in a written format.
Students will be able to recognize a thesis statement.
Students should be able to speak and write.
Students should be able to give oral reports.
Students should be able to recognize the parts of speech.
Students should be able recognize an argument.
Students should be able to write an argument.
Students should be able to write longer research enriched arguments.

1422 Eric reports on what we did.... [It must be naptime. Harmon Loop blew a water main this morning. The team is operating shower free. During our discussions I was interested to see the use of student learning outcomes in the setting of assessable goals. I have pushed for the use of student learning outcomes unsuccessfully in the past. I did become rather over excited at seeing the use of student learning outcomes as this reaffirmed my conviction that modern assessment is built around these. Our current system does not use student learning outcomes. We use behavioral objectives. Behavioral objectives are a student learning outcome with a measurement component and a specified level of mastery along with a specified length of treatment time. Assessing programs by behavioral objectives fell out of favor in the 1980's as far as I know. Seeing the use again here of student learning outcomes suggests that these remain a core to assessment systems and hence to curriculum development and program outlines. In my excitement I did note that the curriculum committee has rejected outlines I have done in student learning outcome format. Bear in mind the student learning outcomes banks have a strongly different appearance from a behavioral objective. A course can have a couple hundred student learning outcomes, and not all outcomes will be measurable or quantifiable. An example of a non-measured outcome would be "Students will feel safe from physical harm." This might arise in a chemistry lab outline. There is no measurement, unless one considers a Likert type study. Pages J2-6 and forward detail the process of building these banks, or inventories of student learning outcomes.]

1435 ...this iterates the difference between qualitative and quantitative analysis... see D2.

It is premised on the notion that you know what questions you want to ask. You find only fifteen percent say they want to attend NMC. Shoot the messenger: you asked the wrong question, you are creating a mountain out of a molehill. Crisis response: oh my goodness this is a disaster the end is near. What you would want to do actually is to form focus groups and then find our why students are not opting to go to NMC. What do students in the community see as the pluses and minuses of NMC (that is, future students). You want to go after the reasons for the results.

Students who have gone from our college to the mainland think students think it is better to go off island to school. I've had students from Chuuk say "Oh, nothing is good in Chuuk." Is there a perceptual problem or a real problem? We have the same problem: students see going off island as going to better schools.

James: A rural community college was not attracting local students. They did a survey and found they were not offering what the local students wanted. They were votech focused and students wanted general education. Students were willing to go if they could transfer on to other schools. I think your question is an important on. The answer is to get into the decision making college choice processes. Then go into an interview process with both future students and on campus students. You are trying to do a gap analysis. Identify the perceptions and the reality of the campus. Either you have problems or there is a perceptual issue that you have to overcome. It is not as good as going overseas... why?

Audience: CNMI students get $12000 to go off-island to school but only $2000 if they stay on-island.

James: That is another reason. Some states have high tuition and high aid. Some, like CA, have low aid and low tuition.

J: How is that campus climate issue? We can't tell our students to stay at GCC when their courses do not articulate even to UOG. That is a serious problem but not a campus climate problem.

Audience: Culture of Micronesian students is one of the problems hat was presented by them, the local students, why would they leave Micronesia and...

J: The students from Micronesia are not getting involved in campus activities at a level that would be beneficial to their growth.

James: Lack of involvement leads to a lack of student participation and growth and impacts....

They came to Guam because they were told education is better. Intent was to become better educated and return but while here their government should have provided more comfort [and support].

Audience: But these students are Pell grant CWS recipients as far as that issue I just think that is a money issue and do not think the college should be held hostage to that I do not buy into that is an issue we have well qualified teachers but we are stretched to the maximum and we are stretched to the point of the rubber band that is about to snap. We cannot meet external business demand. We do not have buckaroos to pay for the adjuncts so we have to increase the class limits to generate the money.

James: Many community colleges have struggled with the mission to meet their mission with limited resources. Some states ration the resources to remedial program (Georgia, Government. Miller, Alabama wants to eliminate funding to remedial education). [If your mission is remedial then one can get cash crunched...]

1456 The way that was approached in California was to identify at what level of remediation or deficit were people able to succeed and move into college courses. Two programs were created: a fast track for students [who moved quickly into college level and a tightly constrained track for students who had... ]

Often many courses reteach skills. Students tend to forget material from past course. Reteaching is costly. Heavy resources can wind up in reteaching. Assessment can lead to determining what level of resources go to reteaching.

Highly selective: most nursing programs. Costs can be contained by constraining the size of the program.

Q: Should we be looking at how we assimilate them and prepare them to succeed in the local culture?

Is the student willing to change to the indigenous culture of the college or is the college going to change to adapt to the student? I would use focus groups to look at this sensitive question. Focus groups are ideal to getting a better handle on this, both students and staff. If a student does not have the support mechanisms necessary to make it in college, then something is wrong. It may be communication problem or perceptual problem. Some things you have no control over. Break. And then briefly share we will ways to goals. 1503

1525 During the break I discovered a couple of Micronesian women down on the ground floor in the fitness center working weights and pedaling on the stationary bicycle.

How many colleges here have a general education degree? A liberal arts degree. Could you open to your catalog or your self-study? Who has a general education requirement? Why do you have a general education core?

At NMC we sought comparability when we set up our core requirements. We looked at the death rate and decided we needed a health course. We thought we were affirming the value of life by adding that to the core requirements.

Students do so much better when they know why they are doing what they do. Often we have trouble articulating questions like "Why do we do general education requirements?"

General education is not limited to the first two years of college. How many of you have a statement as to why students must do general education requirements?

GCC does: Recognizing the diversity of its students general education stimulates the student awareness of the world and ... and..... reads very fast interpret communicate knowledge of ancient states and modern nations, precise confident basis for ethical and aesthetic judgments, familiarity with ideas and general knowledge...

1538 There was a goal in mathematics and quantitative reasoning. How do you go about assessing those goals?

Frank: We looked at what does a receiving institution want from our students. Very specific, like logarithms and functions. Now I need to contact employers and see what they need. I need to look at other two-year colleges. Once I compile the data I have an overall picture. Now I look at our program and look and implementation, adjustment, and deletion of irrelevant material from our programs.

Well put and an excellent example because it does not take what is important to mathematicians but takes what is needed to transfer and what the community needs.

Eric reads our general education requirement. Ours is more focused on school internals rather than the outward orientation of Guam's. Guam's tends to give external reasons.

NMC Education program is designed to expose student to broad domains of higher education. College has implied general education philosophy.

James: We are working on assessment and not curriculum. Eliminate the word course from these things. Say instead, 'this is what the student will learn.'

Audience: All three requirements would be simpler to assessment if they were not so verbose. From my business background these things sound so verbose.

Some things go in not as a course but across the curriculum: critical thinking, information technologies. If you teach a course in sociology you are also teaching a writing course.

Audience: We are trying to move English into all the other courses across the curriculum and not have specific English courses in the general education core.

James: Take John's area of work, "understanding the cultures and history of Micronesia", this could be far more than 3 hours. If you look for a way to integrate those skills into existing courses without adding on courses then we get away from creating 3 credit hour goals. That is a curricular challenge and not assessment.

[Ah! We are still in the program, but we have leaped up to page E1-1 forward. See those slides! Now on E1-3. He is recommending a goal-oriented list as opposed to course laundry list. I think. Thus "Students will understand and apply logical and ethical principles to personal and social situations." " Students will understand scientific terms, concepts, and theories, and will formulate empirically-testable hypotheses." "Students will understand and interpret major world events." Student learning outcomes every one of them. Brings back fond memories of graduate courses taken...]

What if you have assessable goals, but no specific curriculum to support them? This is the case where our general education requirements is not a laundry lit of courses but rather a broad set of conceptual requirements that are met in the normal course of the student's time in school. In essence, where there is no preset laundry list of courses. See E1-5 for what to do in this situation...

1406 Page E1-6 [my back hurts from typing on a too high table sitting in a too low chair] Sample planning grid.

College BASE
Academic profiles
Collegiate Assessment of Academic proficiency
College outcomes measures program

[I'se lost the train here. Jonesy! I can't make head or tails out of the percents. Oh good, Vince doesn't get it either. ] Faculty have put two exams on the table. Ask the faculty to go through the tests and ask to what extent does the test impart the shown skill. You get x% for each area. You really have to be looking at E1-7 to understand what I am typing. Now if the COMP exam covers 25% then the other 75% would be covered elsewhere... [huh? I thought... that the percent referred to the test being 25% spoken English but I guess it is that the 25% is that the test in question covers 25% of what we want to test in terms of Spoken English. You gotta be looking at the top of E1-7. ] So if the test covers only 25% of what you want covered then you have toss other instruments at the students to assessment the other 75%. NMC we did TOEFL and CA assessment test.

I blurt in on the limits of TOEFL tests and learn in response that Northern Arizona found that while the SAT was predictive of College success for Anglo students it was NOT predictive for the success of Native American students. Tests are designed for particular populations and purposes. [But then I knew that and if I didn't I'd better pack up and go back to Peoria now]

Recognizing the diversity of its students, GCC strives to develop and maintain a general education curriculum which stimulates the student's awareness of the relationship between vocational skill and the world; assist the students in exercising the privilege and the responsibilities of citizenship, in developing values, in learning to think critically, in understanding the traditions of the Western Pacific, and in respecting diverse cultures and opinions.

General education at GCC aims to help people to:
think, write, and speak clearly
understand the natural world of which man is a part
develop the ability to understand, interpret, communicate, and apply quantitative information, perceive the social, political, historical, economic foundations of ancient states and modern nations, construct reasoned estimates of the self and of human behavior in general build precise, confident, and informed bases to support ethical, aesthetic, and cultural opinions, use free time more beneficially and more enjoyably
prepare for a satisfying life and career, grounded upon a familiarity with ideas, principles, and skills derived from mastery of general knowledge and particular disciplines.

Content:
Develop a cultural identity
Develop national identity
Develop understanding of history of Micronesia
To understand the contemporary Micronesian political relationships in the Micronesian regions.
To understand the political forces ("what do you use to measure this," asks John)

Skills:
Students will have the ability to be an informed and capable political leader (important in our situation)
Students will be able to get up in front and speak clearly and coherently
Students will be able to write in an effective manner
Students will be able to conduct research

Values
Develop values to be a good citizen and an honest politician
Students need to develop cultural pride and a sense of national identity
Respect for their culture, their elders, and tradition (the Ohwa boys attempt to start problems at PICS and the impact of local royalty)

What level of learning is expected
Criteria:
Tests of content knowledge
Write papers
Students will give presentations

Standard of achievement
Successfully attain political office
Successfully carry out traditional obligations

What the context is in which the learning takes place
In classroom, field trips (e.g. Congress, museums), guest speakers

0834 Coupon number 7028452. Session is a few minutes underway. James is at the helm.

We had performance checklist that is put together to assessment coop, a school to work transition. We have a special project we fit into. We are creating our own special projects, like an internship, but pay is not at the same level as the coop experience pay level. Can we study coop versus internship?

James: No. [Do not compare programs. Look at what is good and bad in a program and use that improve a program. ]

James: What you have to get out of your head and think about is the student and quit thinking about what you do, ideally what I want the student to do is to write a ticket and to read the big fat travel guide book. I do want them to just read and write but I want on the job training for that student and by doing that and by doing that I revised your goal to include the coop experiences. So we have a goal and a social experience. We have a goal of being able to write an airline ticket and do it within a work setting. An actual work setting. An airline or a travel agent. You have now a goal. Stated by someone who knows nothing about this field. We have put the coop experience together with practical experience and it is integrated for the student's perspective. Rather than having separate goals for coop and tourism

Everything your have listed is on the performance checklist. They do not have to create one everyday to assessment our students.

James: I would go through, as each college develops an assessment program, I would add the coop component to the program. What happens if you ask what works best, coop or internship? Then you come back and ask do certain students benefit from a coop situation, do other students benefit from internship. Find out how our students are learning, find out who is benefiting? Then ask what did they have that got them there?

[Just as COM-FSM where I look at TSP, IEP, and HCOP, and ask: what is the common thread that causes those programs to shine? A sense of group, a sense of community, a physical place they can exist as a community. Just as some have a men's house, so is having a gathering place that is uniquely yours is important. That is the success formula. So how do you replicate it? You group the freshmen day one thirty in a batch, assign an instructor to be their advisor. Advisor meets with them three times a week for an hour during freshmen year. Advisor stays with them for two years, picking some new freshmen year two. The students are level selected so they move in synch through classes. They take as many classes together as possible. They are given an identifying totem. The class is expected to work together in classes and to be a team in volleyball, softball, and other athletic events. The group is heterogeneous. Instilled in them is to not let the group down, to excel together, and for the more capable students to help life the less capable students...]

James: Please turn to page G1-12 and look at the competencies listed. This college had defined the skill areas this college should identify in their first year. As you turn the page, [so turns the days of our lives?] no, don't turn the page, if you look down here you see the framework... This second column represents the in-class and out of class experience that is part of that experience. Then you would have certain classes tat help accomplish the aim., You might have additional out of class experiences that would... in the middle you have the goals and the third column is the assessment criteria. [ah, my back hurts, eh?]

Let's take a look at some topics... he's says take a look at it and finds he's lost his glasses [James is talking in the third person] What is that on the handout, a goal or criteria.

[they are looking at the student learning outcomes... I jump in and explain that these are not assessable goals but rather the beginning of an inventory of student learning outcomes that will fulfill a goal such as a general education goal that students will be write, read, and speak clearly. The list is not a set of goals. ]

James: G1-14 The difference between the goal and criteria is that there may be differences between the goal and the criteria. [huh?] What is the difference between the one and two year program? Besides length? The difference is the general education goals. This would be articulated in the example. The students are logically going to choose the program that will finish earlier.

Audience: Not necessarily. In the second year you have vertical progression and more competencies are required and you get a higher passing rate on the ANSI test. It is heavier on technical education. The criteria for the second year will be higher than the first year.

0913 You'll notice in the second year you will notice more global tasks to be informed. Student integrates and synthesizes certain types of information.

The automotive area is first year a lot of comprehensive information. A lot higher level of thinking.

James: So you have those competencies already spelled out in the program. You take those competencies and spell them out.

James: What is the difference in the one year and two and the drop out rate relative to placement was the question I had been asking.

There is some significance in the placement. The second year students have a much more in depth knowledge. Their work placement. The one year student gets put in parts, the two year student gets put on the floor. If you are paid y the number of cars you fix the two year student makes more money. [Placement is not entry placement but job placement!]

Societal needs - instructional delivery- instructional goals- societal needs triangle.

Welding at GCC. Degree program says we are not producing anyone. These people are program specific. The cover three types of welding. Students do not complete all, admin says shut the program down, no one ever completes it. Yet they are actually serving students, they just leave after they learn the type of welding they need for their specific job. Industry hires them out once they have one course.

James: What criteria is a measure of the quality of that welding program? Completion rates are not appropriate. Find another measure.

James: Do a differential study: those who complete a given course to those who have completed the full degree program to those who only had a one week course you would find the non-completers with one course to perform the one week wonders. Success is attained after only one course, but the drop out rate is near on 100%

0926 Side discussion with Joe about agriculture program. Does a student who has only completed a year in the agriculture program, or three terms, and then does not finish, outperform peers who never came to the college? Is there success short of a degree? And thus do you measure success by attaining that degree?

If you have a unit serving another unit then you want to measure how that unit contributes to the success of the unit that receives the services.

0931 A few more minutes and then a break. J2-1 back one page. What policy and business leaders see as important outcomes of undergraduate study.

Higher order applied problem solving abilities
Enthusiasm for leaning on continuous bases
Sense of responsibility
Ability to bridge cultural and linguistic barriers
Sense of professionalism

James: We sent a goals inventories to faculty, business people, and to political leaders. 600 surveys, six on each campus. See J2-14. This is the one for writing. This is a Delphi study. J2-16. Sets of student learning outcomes. SLOs in my mind. All respondents ranked the statements. Looked for areas of agreement. About 2/3 were commonly agreed upon. Little disagreement business and faculty. Examples: faculty believe students should revise their work until it is acceptable. Business people say, "We don't have time for that in the workplace."

J2-25 has employer/faculty agreement areas and disagreement areas. [no it doesn't]

J2-7 Speech framework is looked at. [I do not have meta-cognitive sense of why we are looking at this, where we are, and where we are going. Maybe I missed something somewhere. It is like I want a map of where I am and where we are going. What are we doing now? SLOs? Assessment? Goals? Evaluation of SLOs? He keeps saying we will see information on where faculty and business people agree and disagree... I don't see it... Ah J2-9 and J2-8]

J2-8 What faculty thought was important but employers did not. J2-9 What employers think is important but faculty do not:

Build and maintain constructive relations with superiors, peers, and subordinates

J2-10 When you begin teaching writing you have to use what the students have already. Once on the job their motivation is to please the employee. My concern is that the business perspective might suggest we put the student in false situations.

0952 You need to work with the employer to achieve some alignment with a balance. Teach what the employers need but do not neglect general writing skills that might go beyond the workplace.

What level of learning is expected
Criteria:
Tests of content knowledge
Write papers
Students will give presentations

How much prehistory of Micronesia the student knows.
Students will write an essay

Admission standard: admission to the college, the successful completion of EN 089 Reading II and EN 120b Expository Writing II. Essay will not be used to deny entrance to the course, the essay is tool for the professor to asses the student's prior knowledge, or, lack of knowledge.

Exit assessment consists of a research-based presentation evaluated by a three person board consisting of experts in the field of Micronesian history and a comprehensive final examination.

Content standard on the exit is completing 70% of the questions on the comprehensive examination.

Content standard on presentation will be two of the three board members must feel that the person demonstrated a mastery of the knowledge of the topic area they researched. In the interest of presenting a less verbose version, we settled on: [Two say they did good.]

1259 Lunch drifts apart and people wander off to stretch prior to the afternoon haul. I wander downstairs to enjoy a quiet moment in the lobby of the Ladera Tower. A Palauan women empties the ash tray and then heads off to other cleaning duties. 1311

Assessment planning calendar. Find out what kind of information is useful to your identified audiences. See S-1 and then turn back to M-1 forward for slides below.

1334 [Tuned out for a time... just let the words wash over me.] …once you have the program up and going you can simply do it. Assess completing students, assess midpoint students. You want to evaluate the logistics and procedures at some point. You want to give yourself permission to update your assessment program... [Doh!]

Page M-4 Let's talk about drawing a student sample. Who are you assessing? Some or all of your students? A sample or a population?

[What is the cost versus the value of the information gained? Are you searching for a needle in a haystack? A few students in the whole population? Or is what you seek to measure likely to be broadly found and hence a sample will give you a good enough to number to use in assessment... and this what this discussion is rolling through. ]

Letter from top administration explaining the purpose, tell why participation is important, tell what benefits to expect, tell how the results will be used.

Other publicity to the student paper, student radio station, faculty meeting, ask faculty to urge participation.

Protect student rights: Describe the data collection plan, describe how the record will be used, how records be kept, describe how student consent will be obtained, describe how rights will be protected. Submit your plan to the human subjects committee.

Page M-5 Administering the assessment. I'd appropriate rooms, train proctors and admin, cover food, drink, cheating, [we are in Section F page M-5, see that rather than I retype that here.]

1358 Page M-6 Selecting assessment measures. What do I the student walking away from the program to know? Through projects, through observation of work quality, some through testing.

Maricopa Community College web site also has outcomes. Do a web search to hit this. Directory of Community Colleges. By field or program. [I am sleepy.]

1404 Try your process out. See how it goes. Go back and put your results side by side before you start deciding and do that rating process and determine which tell you the most about your students. [See page M-6, my notes just don't make sense out of the context of that page and those slides. Write, pilot, revise, run, evaluate, revise, rerun at next cycle... this is not obvious?]

Web site with testing information. Burroughs measurement yearbook: has all the test and descriptions of what the tests cover/measure. There is also an ERIC clearinghouse on testing and measurement.

1408 Set both process and content standards for achievement. Decide how much is good enough, distinguish between level of performance and mastery of key concepts, terms, ideas, and knowledge. Set entry, midpoint and exit standards for success for each goal to be assessed. Pilot with real samples of student work.

1411 Key decisions in choosing assessment measures, page M-7. You find a high correlation between reading level and critical thinking skills. There are overlaps in some criteria. That is, can you gather this information more expeditiously. Does one well crafted question tackle a number of areas. [Maybe. Seek and you shall find, knock and the door will opened.]

I miss the rest of the session due to an afternoon meeting at UOG that 1went until 1630. Conference was done by that time. Notes will have to be transcribed later, but those notes are scant due my being a major participant in the conversation.

0932 Key involvement questions. Building administrative support. Section G. Page O-1 forward. Picture taking time.

Thoroughly research existing exemplary programs. They are your best practices models. [Like I said: HCOP, IEP, and TSP. Programs that work, that pump up the students and instill a sense of academic and personal pride. They work. Remedial math and English do not. So implement your known best practices. Cohort group structuring. Radical? Yes. Some faculty members will not be able to adjust. But the College is about educating students, not about what the faculty needs. Sound cold? Yes. Other colleges have the luxury of pampering faculty. The FSM does not presently have that luxury. If you need pampering, apply to Harvard. I'm in a mood this morning.]

James: Evidence of Administrative Commitment... assessment becomes an institutional priority. Range of appropriate administrative units become involved. And atmosphere of trust and mutual respective is established.

Can I go back to this trust thing. Considering our diminishing resources and we inject the institute into our work and where does this lead. Given the climate, how will this process be used? Not by us who went through it, but by the rest of my campus. Bottom line, what is going to be expected of us after we get that certificate?

I want to be involved because this will add a tool to my toolbox, a way to pursue quality. It is the checking part that I need. This process we've gone through these three days is exactly what I need. But will everyone else think this is what we need? That we need to assess ourselves?

I think we need to focus on this as an ongoing process. We to that whole program study thing and we kind of through it out. It didn't give anything was useful. The results did not help us. By looking at different processes then we can know what to work on. Anyone who is here from GCC came on their own. We've committed ourselves individually to take what we have learned today and to take this further.

0946 I've got a Nebraska story, remember I came from Iowa. You may regard this as corn, they have more wheat so small college 78 faculty they were two years ago went through self-study and received letter from North Central your assessment in inadequate. Your assessment of students is inadequate. You are going to get another visit to see what you have accomplished. We did a session. The college had a large presence at the conference. Some depts. said this is great set of tools for me but I am not sure what this means for the college. We looked at the north Central letter. North said:

1. You need evidence what student learning in the program
2. Resource allocations must be made based on what you learn from one.

If you can do number one and two in your own department, then when the visit comes the hot seat will not be your seat.

If we are not in the hot seat, then there is the possibility we will need to teach everyone else how we did what we did!

There is another possibility: if your group is "not on the hot seat" because you have implemented the recommendation of the accreditation commission and the other units in the College have not, then it may generate friction.

The tall nail gets pounded down. The crab climbing up to escape the crab barrel is pulled back down by the ones on the bottom.

The key is to share whatever acclaim you get, share the benefits. Mehnwai, haoles here, tend to celebrate their wins. Success is something personally won that goes on your resume for mehnwai. Instead, share the success, the acclaim, the benefits. Build up everyone else.

James: Use teams, not committees. Provide multiple forms of faculty information and development. Provide multiple forms of faculty involvement. Use multiple strategies for change. Work to reduce single course evaluations in favor of assessment of learning gains.

Assessment is not a tool to get rid of dead wood faculty and non-operational programs. If you already know they are dead wood or non-operational, why do you need any process to identify them? You know they are dead wood. Assessment will only generate something to be attacked. Assessment is not a shield for hard decisions.

A committee:
Gets assignment from others
Is involved in study and formulation
Works in isolation
Leaves leadership to the chair
Ignores personal needs of the member so the committee
Relies on member opinion, just conversation around the table
Acts as a loose collection of vested interests
Tries to give solutions
Base on this, tries to give solutions to problems

A team:
Takes part in setting own objectives
Involved in all stage of problem solving
Collaborates with others
Shares leadership, no single head, NO CHAIR!
Attends to personal needs of its members rather than using some voting process to drive down the interests of its members
Searches out new information and truly act as a team
Serves as a catalyst, linker, facilitator
A team leader tends to become a chair and teams tend to evolve into committees. The natural weight of a team is to tend to devolve into committees.

Many people confuse teams with committees. Teamittees if you will. After the last accreditation visit we shifted from committees to teams, but many still think the teams are committees.

It is more than just calling it a team. A team can be a team in name only, if it acts like a committee then it is a committee.

Team that worked: small core, visiting members. Different visiting members each time. Core carries the memory. This worked well. Good attendance because one only had to attend once in a while. A rotating majority. You only attend a meeting here and there as a visitor. The core is always a minority so they cannot control the team. They carry memory only.

Defragmentation of the curriculum
Realistic investment of time and resources in curriculum development
Faculty development in the new learning environment

1016 Break. The coffee ran out before I made it to the pot. Surely death will ensue. Who says coffee is not a highly addictive drug?

Is PICS sending anyone to the PVIP project? Albert Augustine (Pohnpei Dept. Of Ed.) votech director for PVIP project. Elwel Joseph principle. Solphisio Salvador. Is PICs sending anyone this summer. Contact on this is Heromiano S. Delos Santos, Ed.D Director Continuing Education Academy for Continual Education Project Director Pacific Region Vocational Teacher Education Academy.

[Two different people have come up to me and said, "You're that guy with the web site..."]

1057 I sit in with a GCC group that is working on setting up assessment for a program that is annually renewed. The program, adult high school, is a result of an annually renewed program agreement between GCC and the Industry Advisory Council. Working with them I realized that this conference has been very beneficial to them. They have a worksheet plan laying out who their audience is, what their goals are, the rationale, some details on the sorts of outcomes they will want to assess, and now they are working on a calendar. The conference appears to have energized and focused them on their task.

I realize quite suddenly that many of the GCC participants are here with a specific program that they want to assess. Many came with an instrument approach: they were looking for an instrument solution or had sought one without success. What James has provided is a different approach. An approach where a framework is built upon which instruments will later be hung.

1104 Certificate ceremony. We are being given our workshop certificates. Our team has made it every day to every session. The non GCC participants receive a mug and pen along with their certificates.

Terry: We haven't been in any kind of process like this but we desperately need it. We will be starting on this next week. Tentative report date of December 98.

Vince from NMC: We are fortunate, we have our institutional planner and researcher here. We are already done with steps one through eight. A survey went out today at NMC during graduation practice. So the assessment is occurring even now. The completed surveys will be evidence of completion of the assessment task. In June and July we anticipate will be a down time. Data will be reviewed. Planning office will sit down and evaluate the material. A draft of findings will be published. August and September the planning council will submit reports to department chairs. In Sept. The planning committee will look at revising methods, goals, measures. We also anticipate revision as things are found out. So the reports and revisions will be issued. We will develop a report (at a time it is not usually scheduled to come out) due to an interim visit from WASC for their perusal. Steps seven to twelve are ongoing and continuous.

GCC: We went back to step seven. What we did was first we started with how we want to report how we want to report the data we are collecting on our students in underdevelopment. We have reading and writing area component. Reports are due 30 September 98, the end of the year. Too close for comfort. 30 Sept 99 chosen. Start in August. By Sept. Move into the course guides: are they aligned. Attempt to measure goals and select multiple measures. We do not need to look at a a pilot group. We would like to spend some time to develop a survey to give to students the first day of classes. We wanted an entry assessment of entering students. What population succeeds, what population does not succeed. We want to know if there is a relationship between student success and tutorial services. These are things we need to know as instructors. We looked at how assessment is administered as well as looking as final compositions do we need to admin more placement exams we then move into data evaluation and then present it to knot only us but to link in the other programs and tell them, look, we've identified that this group is going to be able to succeed. We looked at all of the positives, we looked at the data we are already collecting. We've been doing a good job, but we can do better. Actually looking at how much time this will take, from our work loads and how can we implement it into this plan and to really work on it.

Surveys done on the first day of class do not work. Do it the first day after the last day to add a class.

I do not usually buy into things. I went to Santa Ana just to accompany my friends. Upon my return we went into meetings. I kept talking about Santa Ana and what I learned there. As I asked to do things I was concerned about what opposition I might encounter or favors I might owe, but I kept saying, we can use this. And others began to see the use of this.

It lets someone know, this is what I do well. Or, this is what I could do better with support. This is really something that provides the information to say this is where we are strong. Its flexible. You do not have to take on everything that you see. You can focus your efforts. Take on this thing this year and another thing another year.

We are talking about the differences between what will make it work and what won't, and when we were struggling to understand at the institute we didn't know what would work. But we committed to work as a team, we communicated with people, we supported each other through email. This workshop is a result of the pros and cons we have considered over time and we decided that this is the way we want to approach it. We struggled with this. Many of you think there are things that you do not understand. That's OK. As a team we've made the commitment, there are departments that want to work on this. We can communicated by email, maybe from the one we provided to Dana, to support each other.

It certainly would be nice to reconvene this group a year from now. I'm not sure how this was funded.

We need to look at bringing this group back together again. We will need to look for resources.

I have a question. What is the commitment of the team? What is the program that we will look at? [What happens if the information is used wrongly?]

Eric Chong: We started institute wide and then worked our way down to programs. What I like about this is that it is not a one time thing. And that it is a grass roots thing. We can effect this from where we are. Not from the President down. [We can evaluate our own programs]

Utah three years ago, a task force to assessment general education and was to meet once and walk away. We worked out that assessment plan. They now refer to themselves as the R task force. It was not re-appointed or reaffirmed. The chancellor did not tell them to go on. It just keeps on meeting. They got a grant (on their own) and they are focusing on the programs that they want to work on.

+ Informative, especially helpful for those new to assessment and who have specific program to assessment.

D Make the groups heterogeneous instead of college by college homogeneous. We so rarely get off our little islands that conferences become precious opportunities to share conversation and ideas with other schools.

I win a prize today in the daily raffle. A Guam Hilton T-shirt. Running shirt! Every time we have a run on Pohnpei we receive a T-shirt. Over time my T-shirts have become a record of places I've been and things I've done. Maybe a book organized by the T-shirts I've received.

I am deeply grateful to Title III Coordinator Jean Thoulag for loaning this laptop to me, without it the details I have been able to include would never reach me.

First	Last	School	Title	Email
Eric	Chong	WSU	Chair Tourism Department	ejchong@gocougs.wsu.edu
James	Ratcliff	Penn State	National Center for Postsecondary Teaching, Learning, and Assessment	jlr7@psu.edu
Sherman	Daniel	PCC	School of Engineering Associate Dean	sherman@belau.org
Kasio	Oiterong	PCC	Math Department	kasioo@belau.org
Alvina R.	Timarong	PCC	Chair Committee on Programming & Curricula	alvinat@belau.org
Mike	Jenck	NMC	math instructor/other	mikej@nmcnet.edu
Barbara	Moir	NMC	Vice-President for Instruction	BarbaraM@nmcnet.edu
Caryn	Plinske	NMC	Research & Evaluation	carynp@nmcnet.edu
Dirk	Sharei	NMC	Institutional Planning	dirks@nmcnet.edu
Carol	Andersen	GCC	Asst. Professor at GCC	carolj@netpci.com
Sandy R.	Balbin	GCC	Business Instructor	sbalbin@gcc.guam.net
Lisa	Baza-Cruz	GCC	Developmental Ed	bazacruz@gcc.guam.net
Suzanne	Billings	GCC	Dean Business, Social Science, Tourist	suzanneb@gcc.guam.net
Carmelita	Connelly	GCC	President's Office	cconnel@gcc.guam.net
Herominiano S.	Delos Santos	GCC	Director of Continuing Education	hdsantos@gcc.guam.net
Olivia A.	Grande	GCC	Library/Dept Chair	ogrande@gcc.guam.net
Charlotte	Hepler	GCC	Developmental Ed	chepler@gcc.guam.net
Cathy	Leon Guerro	GCC	Coop Coordinator	clguerre@gcc.guam.net
Norman	Limtiaco	GCC	Counselor	normal@gcc9.guam.net
Jamie	Mason	GCC	Dean Student Development	jmason@gcc.guam.net
Doris C.U.	Perez	GCC	Planning and Development Office	duperez@gcc.guam.net
Ginger	Porter	GCC	Dean, THS@GCC	gporter@gcc.guam.net
Tara O.	Reyes	GCC	Prog. Specialist, Adult Ed.	Treyes@gcc.guam.net
Karen	Sablan	GCC	Asst. Prof Assessment and counseling	sablank@gcc9.guam.net
James J.	Taylor	GCC	UP Business & Finance	jtaylor@gcc.guam.net
Joe	Habuchmai	COM-FSM	Director of Academic Affairs	jhabuchmai@mail.fm
John	Haglelgam	COM-FSM	Chair Social Science Dept.	sscience@mail.fm
Dana	Lee Ling	COM-FSM	Title III Math Science Specialist	dana@mail.fm
Eric	Mecklenburg	COM-FSM	Assistant Professor English COM-FSM	langlitb@mail.fm

Criteria to consider	Qualitative measures	Quantitative measures
Validity	High	Low
Reliability	Low	High
Feasibility	Low: labor intensive like a focus group session	High
Generalizability	low	High
Ease of administration	low to high	High
Ease of analysis	low	High
Ease of interpretation	low	Low to high
Time to develop	high	Low to high
Time to score	high	Low
Time to report	low to high	low

Assessment	Writing	Math	History	Science
Admission	Essay	SAT	Course embedded exam	Locally developed test
Junior matriculation	Essay	Commercial tests	Course embedded exam	Locally developed test
Graduation	Essay	Commercial tests	Essay	Locally developed test
Alumni	Survey	Survey	Survey	Survey

Transcriptional Report from the Conference on Institutional Assessment