Eleventh-grade students and teachers are spending extraordinary amounts of class time preparing for ACT, but the intense focus on test strategies and item practice is hurting, not helping, performance on this high-skills accountability exam, according to a study released Tuesday by the Consortium on Chicago School Research at the University of Chicago.
"Students are training for the ACT in a last-minute sprint focused on test practice, when the ACT requires years of hard work developing college-level skills," said Elaine Allensworth, a co-director at the Consortium and the lead author of the study, From High School to the Future: ACT Preparation-Too Much, Too Late. "Students are very motivated to do well on the ACT, so they put a lot of effort into test prep thinking it will raise their scores. Teachers need to channel this energy into what really matters for the ACT-students doing high quality work in their courses."
The research revealed that Chicago teachers commonly spend about one month of instructional time on ACT practice during eleventh grade core classes-driven by pressure to improve scores and despite misgivings about how this practice disrupts lessons. But this time is not paying off in higher scores.
ACT scores were slightly lower in schools where eleventh-grade teachers reported spending at least 40 percent of their time on test prep, compared to those schools where teachers devoted less than 20 percent of their class time to test prep. This analysis controlled for multiple factors-from student's low-income status and incoming test scores to teacher qualifications and school composition. Thus, schools are compared to others just like them.
The focus on test prep means students are not making a connection between the work they do in their classes and their ACT scores. About 83 percent of Chicago juniors believe that ACT scores are primarily determined by test-taking skills-a misconception widely shared by their teachers. Only a third of English and science teachers believe the ACT is a good measure of learning in high school.
In fact, high grades in eleventh-grade core courses are the strongest predictor of improved ACT scores. Students who started their junior year with a PLAN score of 17 (a little above the district's average) and earned a B in their English class that year scored one point higher on average on the ACT reading test six months later. Students who flunked English actually lost ground and scored below 17 on the ACT.
This study relies on qualitative and quantitative data for a cohort of students who were CPS juniors in 2005. This includes test scores from eighth to eleventh grade, student transcripts, CCSR surveys, and multiple interviews of students and teachers at three Chicago high schools. The report also relied on 2007 data on CPS juniors and teacher surveys.
Low ACT scores also reflect poor alignment of performance standards from K-8 to high school and from high school to college. Students appear to be prepared for high school when they enter ninth grade-64 percent of students who took the ACT in 2005 had met the ISAT eighth-grade standards in reading three years earlier. Yet of these students who met state standards, only 30 percent met the ACT reading benchmark three years later. Only those students who exceeded standards in eighth grade were highly likely to meet the ACT reading benchmark, but in Chicago that represents only 855 students.
Moreover, among students still enrolled as juniors, their ninth-grade EXPLORE scores suggest Chicago's incoming freshman are similar to their peers nationwide. About 42 percent of Chicago freshmen meet the EXPLORE benchmarks in reading at the beginning of ninth grade, and 41 percent do nationally, but Chicago students lose ground in high school. Only 26 percent of CPS juniors meet the ACT benchmarks for college readiness in reading, while 53 percent of students nationwide hit that reading benchmark score of 21.
The new research was announced at Kelly High, a Brighton Park neighborhood school where teachers report spending minimal class time on test prep but saw dramatic improvements on student performance from freshmen to junior year on a battery of ACT-designed tests called EPAS. Based on 2007 ACT scores, the average Kelly student who took the EXPLORE test as an entering freshman in 2004 gained nearly four points when tested on the ACT as a junior in 2007. (The expected improvement would be three points, based on national averages). Kelly had the best EXPLORE-to-ACT improvements among all neighborhood high schools last year and ranked fifth citywide, behind Northside College Prep, Whitney Young, Lincoln Park and Jones College Prep. Kelly's average ACT composite score also has climbed steadily for the past five years, from 15.4 in 2003 to 16.7 in 2007.
Kelly's effort reflects what the Consortium's research highlights as best practice in high schools, even those schools facing enormous pressure to raise low scores. Kelly leaders said they revamped the curriculum and course assessments to reflect deeper emphasis on the analytic skills tested on the ACT, starting in freshman year. Also, they don't expect eleventh-grade teachers to spend class time on practicing test strategies. Rather Kelly offered optional ACT prep classes after school and on Saturdays, attended by about 400 of the school's 560 juniors in 2008, officials said.
The ACT has been taken by all Illinois juniors since 2001, when it became part of the state's accountability exam. State officials predicted that a test of high-level skills with real-world consequences would be an effective strategy for high school reform. But average ACT composite scores statewide have barely budged over the past five years. State ACT scores are flat — 20.3 in 2007 from 20.0 in 2003 — while Chicago's ACT scores inched up to 17.6 last year from 16.8 in 2003.
Requiring the ACT does not lead to more college-oriented instruction because its structure is not an easy one to teach to, particularly given the traditional structure of high school courses, according to the report. The ACT is designed to measure college readiness more than learning in particular high school courses. While it incorporates skills taught in high school classes, it is more of a test of thinking and problem-solving skills than a test of specific content knowledge. It is not particularly aligned with any eleventh-grade course, although that is where most test preparation happens.
Students and teachers reported in surveys and in interviews that they spent a tremendous amount of time in eleventh grade working on test prep-boring work they endured because they were so motivated to succeed.
"When I first started, the ACT test was given outside of class. Now, you put your whole entire curriculum on hold. You just throw it out the window and you do nothing but test preparation the whole entire period for five, six, seven, eight weeks," one eleventh-grade teacher said in interviews quoted in the report.
Echoed one eleventh-grade student: "These teachers really want us to do well on the ACT, so every single class that we have, it was every single teacher, was making us do ACT work at least twice a week."
The research also takes a closer look at ACT scores by race and ethnicity, analyzing why the gaps remain "disturbingly large" even when comparing students with similar grades in high school. These gaps exist in CPS because African American and Latino students are starting high school with far lower eighth-grade test scores, but enrollment in different high schools and curricular tracks widen this gap.
This research is the third in a series of reports that has tracked the experiences of successive cohorts of CPS graduates and examined the relationship among high school preparation, support, college choice, and postsecondary outcomes. The goal of this research is to help CPS, other urban districts and national policy makers understand what it takes to improve outcomes for urban and other at-risk students who now overwhelmingly aspire to college.
While the report does not offer specific recommendations, it does raise several key questions for educators:
- How can schools help students better understand the connection between their scores on the ACT and the work they do in their courses?
- Is there a way to preserve class time for challenging academic work, while still making sure students are familiar with the ACT?
- Are students getting a true sense of the test when they prepare for it?
- To what extent are teachers in grades prior to eleventh incorporating the skills that students will eventually need on the ACT?
- How well are courses structured to align with the skills students will need in college and for the ACT?
"In the end, raising ACT scores requires the same strategies as improving graduation rates and better preparing students for college-a focus on the quality of students' work in their classes, clearly tied to their preparation for the future," the authors conclude. "There is no quick fix when students lack college-ready skills."
Founded in 1990, the Consortium on Chicago School Research at the University of Chicago conducts research of high technical quality that influences policy and practice in Chicago and nationwide.
—Prof. Jacqueline Stewart
