ILLINOIS LOOP ANALYSIS OF 2006 ACT TEST RESULTS

The following piece is brought to you by Kevin Killion of the IllinoisLoop.org.

ILLINOIS LOOP ANALYSIS OF 2006 ACT TEST RESULTS

WOW -- Look at what a good PR effort can produce! Here's how the this year's release of ACT scores was headlines this morning:

Tribune: "Illinois ACT Scores Up Across Board"

Sun-Times: "City, State ACT Scores jump, Still Lag"

Daily Herald: "A Tough ACT to Follow / Test Scores Rise Across State ..."

Where the "good" news came from

Yesterday, the ISBE, our state's education bureaucracy, sent out a carefully worded press release selecting specific numbers from the ACT results, which they got to see in advance of the official release. As all good bureaucrats do, the news was spun to highlight the best stories possible.

Here are the key "good" stories that the ISBE managed to extricate from the ACT:

-- Illinois' "composite" score was higher than the previous year,
going from 20.3 to 20.5

-- Illinois' Reading score is 20.6 (which is up from 20.3 the previous year)

Now, the REAL story

But today the results are available to everyone, and the story about Illinois is less impressive than those headlines suggest.

It turns out that the ACT scores are up nationally, not just in Illinois.

The new composite ACT score is 21.1, up from 20.9, an "increase" of 0.2. So, Illinois' "increase" by the same 0.2 (from 20.3 to 20.5) turns out to be merely keeping pace with the national score!

So, can Illinois at least claim that it's improving along with the rest of the nation?

Well, we'd like to think that the whole country is improving. But we have to wonder: has the country improved, or has the test gotten a little easier? A rising tide raises all boats. If you see boats going up, that tells you more about the tide than about boats.


So, I looked at results for the 50 states plus the District of Columbia. It turns out that 45 -- 45!-- of them had ACT composite scores that were either improved (37) or unchanged (8). Only 6 states had drops!

Wow -- that means that just about EVERY state had a headline to crow about!
What about specific tests?

Illinois educrats will (no doubt) be cheering about results in each of the areas of the ACT. Illinois scores are up in English, Math and Reading. Only in Science were results flat.

But again, this "rising tide" MUST be compared to the national changes:

Composite English Math Reading Science
National +0.2 +0.2 +0.1 +0.1 0
Illinois +0.2 +0.3 +0.1 +0.3 0

The modest Illinois "math" improvement turns out to be the same as the national change.

Ah, but what about those English and Reading scores, that ARE better than the national changes?

For that, we really need to look at longer trends.

In Reading, the Illinois "improvement" this year (+0.3) is largely a recovery from a drop (-0.2) the year before. The English score is a more consistent increase. Here's the trend:

--Reading-- --English--
Rpt year Ill Natl Ill Natl
2002 20.3 21.1 19.4 20.2
2003 20.4 21.2 19.6 20.3
2004 20.5 21.3 19.7 20.4
2005 20.3 21.3 19.9 20.4
2006 20.6 21.4 20.2 20.6

So, the trends ARE up, and Illinois is at its best score in the five-year series. But once again, notice the "tide" effect: the national scores are up as well.

Finding Good News

The ACT results are a playground for eager public relations people looking for "good" news to crow about. With four subject areas plus a composite score, and then a "rising tide" of scores nationally in the ACT, wow, just about ANY STATE can find something good to announce!

I explored this a little further ...

As reported early only 6 states had declines in their composite scores, so everyone else gets to announce "Overall gains" or at least "Keeping pace".

When we break it out by subject, the spinnable good news (largely thanks the to national score increases). We find:

States that get to announce
Subject "improvements" or "keeping pace"
--------- --------------------------------
Composite 45
English 43
Math 46
Reading 44
Science 41


In another way of looking at the data, I found that 44 states would be able to announce "increases" in one or more of the test results! Only TWO states (Alaska and Rhode Island) were unable to announce even a "keeping pace" in one of the results.


Finding the Pony in the Bad News: The Woebegone Effect

A good PR effort doesn't stop even when it's hard to find the good news.

I wondered about that in looking at the results for our neighbors in Indiana. Poor Indiana! Down in English (despite national increases), flat in Reading and in the Composite (despite national increases), and math and science track at the national rate. Yikes, what can the Hoosier educrats crow about?

Well, I searched, and found that Indianapolis did indeed pull a rabbit from an empty-looking hat:

"Indiana ACT Scores Top National Average for 16th Year"
(See their whole PR release at http://www.doe.state.in.us/reed/newsr/2006/08-August/ACT081606.html)

Yup, when all else fails, shout, "We're better than average!"

So, who IS better than average in the ACT? I found that Lake Woebegone is thriving at the ACT! We have 32 states that have composites better than the national average. Moreover, 36 states get to claim they are at the national average or better in Reading!


FOR ALL THE DETAILS

I have included all of the details behind these conclusions on an Excel workbook that is available for download on the Illinois Loop website: IllinoisLoop.org.

Kevin Killion