Math Entrance Test Meander

14 March 2005

Consider the following to be a meander through the math entrance test results led by a fool who is attracted to the next pretty number or statistic. This document presumes familiarity with the math entrance test.

In short the test is four columns of ten questions in each column. Originally the design intent was to use a score of seven or more in a single column to place a student in the next course. This was found to underplace students, so the following year the cut-off was dropped to six. A study of standard deviations showed that this was statistically significantly separated from random.

The following year unusual score sequences such as "2-2-6-2" began appearing from one state. An additional rule was crafted that the highest column with a score of six or better places the student provided no prior column is less than 5. The resulting OpenOffice.org Calc placement function is:

=IF(AND(F2>4;G2>4;H2>4;I2>5);101;IF(AND(F2>4;G2>4;H2>5);100;IF(AND(F2>4;G2>5);98;IF(F2>5;95;90))))

Basic statistics by column

Bearing in mind that two is random, the results slide towards randomness as one progresses through the columns. Note that column one, the MS 090 level material, had one problem (number ten) that had no answer. Thus a score of nine was the maximum possible in that column.

Distributions by state

The following table shows raw distribution of students into courses by state.

The four MS 101 placements from Chuuk are three from Xavier and one from Saramen Chuuk Academy. Placements by high school are further below.

Bearing in mind that many of the MS 090 placements are actually students in certificate or possibly the proposed new non-admitted category, determining the distribution of students at the national campus is usually best ascertained by looking at the ratios with the MS 090 placements removed.

The above results suggest - only suggest - that we might expect slightly higher numbers of students headed into MS 095 than in the past.

Distribution of students by high school

The distribution of students by high school is always of interest, it helps explain the distribution by state table above.

The performance by KHS stands out. As one looks over these numbers, bear in mind that this group includes the Gear Up cohort, many of whom also became Upward Bound students. These students have had programmatic intervention with summer courses and after-school/evening tutoring since sixth or seventh grade. Although the change in principal ships was only this current year, this too may have had an impact.

The category "other" refers to three students listed as having a high termed "other".

High schools in rank order

A rank order can be generated by calculating the average placement for each high school. Xavier tops the list of all schools. Of note is that Berea edged out Saramen in Chuuk, this represents a change from past performance rankings for these two schools. KHS tops the list of public high schools.

A similar order can be obtained by calculating the percent who have placed into college level mathematics

PATS is often perceived as a technical school, but on an academic math examination half of the students are performing at a level that has them ready for college algebra. This is tremendous and renders more poignant the loss of this superb school.

Among the public schools the performance of KHS is notable, placing almost six times as many students into college level mathematics as PICS, the next public school on the list.

The overall result is that 91% of the students place into developmental mathematics. That is part of the big picture: the vast bulk of the students are placing into our lowest level developmental mathematics class.

95% Confidence intervals for the high school averages

I suppose this section is more of a statistical lark because the sum of the four columns does not have a placement meaning. The sum does represent the number correct out of 40 and as such should reveal relative levels of performance. Striking averages by high school and then calculating the 95% confidence interval for that average yields the following chart:

In the above chart, where a vertical bar for one high school overlaps the average for another, the means are not statistically significant in their separation. For example, the KHS mean is not statistically significantly different than the Pohnpei SDA mean. The XHS mean, however, is statistically significantly different than the KHS mean.

Bearing in mind that random is eight, Chuuk state campus, Mizpah, Ohwa, Pentecostal Lighthouse Academy, and Weno High school are statistically indistguishable from random. As I have said in previous years: release those math faculty, cancel all math classes, teach the students to color in one oval per question. Put any salary money saved towards improving English language skills. Maybe that is harsh. And Ohwa could argue that their small sample size led to a large confidence interval, but an average that is only two more correct than random is still abyssmal. And Weno high school has an average that is random.

Data for the above table:

Kosrae

Kosrae High School ran a math program for "Advanced Students". If those students are looked at separately, they perform statistically indistinguishably from Xavier students:

The Advanced Students at KHS are arguably the top math students in the FSM this year.

Count 	
Adv     KHS
N       164
Y        29
Total  	193

Average	
Adv   KHS
N     16.12
Y     25.41
Total 17.52
	
StDev 	
Adv   KHS
N     6.22
Y     5.30
Total 6.93

Subsequent to the above work I obtained lists of Upward Bound, Gear Up, and a list of all 31 advanced class students. The following chart presents the 95% confidence intervals for these various groups on Kosrae.

The simple upshot is that too many of the advanced students are also Upward Bound students (16 of 31 advanced students) to ever determine any differential effect. The two averages are hopelessly entangled together.

Bear in mind that this group of 16 were also Gear Up students prior to tenth grade, so Gear Up is also a contributing factor to their success. Of those 16, 14 placed into MS 100 or MS 101. This is an extraordinarily strong performance. Of the 24 UB students, 19 placed into college level mathematics. Of 31 advanced class students, 18 placed into college level mathematics. Of 147 Gear Up students, 25 placed into college level mathematics.

The one result above that might seem odd is that the Gear Up average is numerically lower than the overall high school average, although the difference is not statistically significant. This might seem puzzling, but I show only one student as being in both programs. Bear in mind Gear Up consists of 147 students out of the 193 who took the math test. Basically, Gear Up is everyone not in UB. The UB students performed above average, so when you pull them out, the resulting group has to have an average that is below the overall average.

Ultimately I hold to my metaphor. A canoe called the Kosrae seniors did well on the mathematics entrance/placement test. In the canoe were many paddlers including Gear Up, Upward Bound, the advanced class, the teachers, the parents, the state leadership, and the community. All have contributed to the effort that led to these results. As the Governor noted, the key phrase might be that we should strive to continue to improve.

Yap

I had a look at the relative performance of the UB students from the Yap. The students come from three schools:

High School   UB   Non-UB Sum
NICHS Woleai   4   37     41
OIHS           5   26     31
YHS            9   110   119
Total Result  18   173   191

The sample sizes for the UB in the individual schools are too small to compare UB groups to each other. I chose to look at overall high school averages, which include the UB students in the rightmost column with the UB and non-UB students in the bottommost row.

High School   UB     Non-UB  Total Result
NICHS Woleai  16.25  9.97    10.59
OIHS          11.6   9.92    10.19
YHS           19.44  12.45   12.98
Total Result  16.56  11.54   12.02

A look at the 95% confidence intervals results in the following chart:

Note that the 95% confidence interval for the UB students does not overlap any of the high school averages. The Yap UB students, as a whole, statistically significantly outperformed the averages at the high schools in Yap.

Of related interest is that Yap high school statistically significantly outperformed NICHS Woleai and OIHS on the math portion of the entrance test.

All errors are mine, and there are some. There are names on some lists which do not appear in the entrance test list, there are entrance test results for names that appear on no list in my possession.

Upward Bound Study

With the names from the spreadsheet provided by Jonathan, I was able to put together a three-state look at Upward Bound performance on the mathematics test. The following chart depicts the 95% confidence interval for the mean sum of the number correct on the mathematics entrance/placement instrument. While the sum has no particular meaning in terms of placement, it is an indicator of overall relative mathematical strength.

The chart depicts both the mean for the UB students and the mean for their high school. Comparison to the high school is necessary because a program such as UB can only lift students relative to the overall performance capability of the high school. The Yap UB numbers, when broken out by high school, create small sample sizes that are statistically problematic. Thus the Yap UB group is considered as a whole.

The "balls" are located at the mean, the vertical lines depict the extent of the 95% confidence interval. While there is a maximum possible of 40, ten of those 40 represent MS 100 level material. In some sense, an expected maximum score might be 30 and a score of 21 or better is a reasonably strong performance.

PICS produced the largest "lift" in rank order of 9.14 points, although this is statistically indistinguishable from the 8.73 point difference seen between the KHS UB students and the KHS mean. The Yap students are more complex, but the net boost above the YHS average was 3.57 points.

The separation of the UB students from their HS averages is statistically significant.

Of some passing interest is the note that the KHS overall average is statistically indistinguishable from a Yap UB student. It may be that the elevated overall KHS average is in part creditable to Gear Up.

I remain deeply interested in looking at the A1 section at PICS and running a comparison against the advanced class in Kosrae, but I have not been able to obtain a class list and the pressures of time, work, and space may prevent me from obtaining such a list.

5/4/5 I combined some of the data above into a single chart for purposes of comparison. Note the overall Chuuk state 95% confidence interval calculation excludes Saramen and Xavier as no UB students came from these schools and they are statistically too different from the rest of the schools in Chuuk. This chart depicts the depth of the mathematical weakness of the other schools in Chuuk.

Note that Chuuk UB performs indistinguishably from the general population at PICS or YHS.

The following tables contains data that underlay some of the above charts:

Here is data on other subgroups not included above:

If there are questions, please do not hesitate to ask!

DNSM