Brookings: Experts - Tom Loveless

The Banality of Deeper Learning

Tom Loveless — Wed, 29 May 2013 11:00:00 -0400

Deeper Learning is the current term for an old idea. The notion is that schools spend too much time focused on the acquisition of knowledge, especially knowing facts. In the past century, several alternatives have arisen to dethrone the prominent role of knowledge in schools: project-based learning, inquiry and discovery learning, higher-level thinking, critical thinking, outcome based education, and 21^st Century Skills. Now it is deeper learning.

These ideas represent a variety of approaches to curriculum and pedagogy. They are not all the same, but they share one characteristic. All are advertised as transcending, and therefore superior to, academic content organized within traditional intellectual disciplines. It is not enough for students to know the major events of U.S. history, for example, but to be able to critically analyze the histories, any history, that one studies. Knowing about science is inferior to doing science. It is less important to learn the algorithms and articulated procedures of mathematics than to apply them in real world contexts while solving real world problems.

My hope is that readers of this Chalkboard post will be skeptical when encountering deeper learning in the future. I will describe two examples of deeper learning that readers should find troubling. I will not offer a thorough critique of deeper learning or its philosophical kin. For that, I urge you to read E.D. Hirsch’s The Schools We Need: And Why We Don’t Have Them. Published in 1996, the book pre-dates today’s deeper learning fad but convincingly rebuts its twentieth century ancestors, showing not only that these anti-knowledge movements lack anything resembling evidentiary support for their claims, but that they also, in disparaging academic content taught in public schools, exacerbate social inequality. The premise is simple. If public schools don’t teach algebra or chemistry or history or great literature or how to write well—the old-fashioned learning that has been around for centuries and remains high status knowledge in most cultures—rich kids will get it somewhere else. Poor kids won’t.

So let’s turn to two examples of deeper learning.

First Grade Math

A blog post by James V. Shuls in January 2013 tells the story of parents trying to understand the objectives of their son’s first grade math program. The first graders were being taught addition of two-digit and one-digit numbers using word problems. The school offered an example to parents:

Stuart has fifteen pencils. Trae gives him five more. How many pencils does Stuart have altogether?

Three acceptable strategies were shown for solving the problem. All involved “chunking” the numbers into groups (in the case of fifteen, three groups of five or one group of ten and another group of five), with each group presented in a circle or box. The groups are then added together. The addition must be shown graphically.

One unacceptable strategy was also shown:

Please do not have your child stack numbers and add them like this.

The Shulses were alarmed that the standard algorithm for addition was being discouraged, let alone not being taught as the simplest, most efficient method for solving addition problems. After several tutoring sessions with their son that turned adversarial (“my teacher doesn’t do it that way”), the parents met with the teacher and school principal. The educators explained that the district had adopted a math program from the 1990s, Cognitively Guided Instruction (CGI), to implement the Common Core State Standards for mathematics in first grade. The program emphasizes deep understanding.

A compromise was struck at the meeting. Solutions using standard algorithms would not be marked wrong, the parents were told, but their son would be required to “illustrate his understanding” through graphical representations of math problems. In his blog, Shuls includes a worksheet that shows the laborious process his son went through, drawing with the uncertain strokes and still developing fine motor skills of a first grader, to prove that three tens make a total of thirty and three fifteens make a total of forty-five.

The Shulses observed several classrooms using CGI for math instruction at the school. They eventually withdrew their children and sent them to private school.

PISA’s Reading Literacy

The Programme for International Student Assessment (PISA) is an international assessment of reading, math, and scientific literacy given to 15 year olds every three years. The 2012 PISA results will be released this coming December and will surely receive widespread attention. According to the Alliance for Excellent Education, “PISA exams are some of the best available and most widely used tests of deeper learning.” Released items from PISA illustrate what deeper learning entails in terms of reading literacy (available here and here).

Most U.S. students who take the PISA exam are in the first semester of their sophomore year of high school, tenth grade. These students’ English Language Arts courses are where literacy is taught, primarily by reading selections of fiction and non-fiction prose, poetry, and plays, discussing them in class, and writing about them in essays or short assignments.

PISA is an international assessment, so it must understandably avoid using item prompts that would favor one culture over another. That is not as difficult as one might think because PISA philosophically views literacy as the reading skills needed for adult life. That encompasses many kinds of text, not just good literature. Text encountered during the conduct of everyday life is more universal than any particular nation’s literary heritage.

Among the PISA released items, you will see reading that involves interpreting graphs, charts, and tables, a health bulletin, internet posts, a tree diagram, a schedule, a magazine article, a warranty for a consumer product, an editorial, a comic strip, a consumer warning, an email, the results of an internet search, and some maps. You will see almost no great literature. As one who majored in English as an undergraduate, I am disheartened by that. The readings of PISA are flat and lifeless. Learning how to read great literature, a gift from both my high school and college teachers, electrified me as a teenager and has provided intellectual nourishment for a lifetime, even as I moved on to a profession centered on education research.

The prose that is included in PISA items frequently seems out of place. There is a baffling paragraph from One Hundred Years of Solitude, not enough to appreciate Gabriel García Márquez’s novel (see pp. 174-180). Questions about the paragraph focus on separating fiction from reality, an important theme of magical realism in general and this novel in particular, but the superficial treatment here, although understandable, is glaring. In the 2006 PISA, students read a short excerpt from a play, Amanda and the Duchess but, disappointingly, most of the questions are devoted to staging the play, not to plot or theme (see pp. 74-83). A sample item in the 2009 framework asks about a metaphor (in the prompt, the night sky is compared to the sea) but the word “metaphor” is not used. As explained in the scoring guide, the word is purposely avoided because “such metalinguistic knowledge is not part of PISA’s description of reading literacy (see p. 219).”^[1]

Skimming over big, sophisticated themes, posing questions about a play and focusing mostly on its staging, asking about a metaphor without using the word “metaphor”—these may be examples of the deeper learning valued by PISA’s admirers, but whether this type of literacy is valuable in the wider world is questionable. And to bury questions on a few excerpts from literature in an avalanche of text (pardon the metaphor) drawn from everyday life calls PISA’s fundamental approach into question.

Conclusion

Let me conclude with a couple of points. I am not disputing that some tasks are more cognitively demanding than others and some learning is more complex than other learning. Educators have known that for a long time. Bloom’s Taxonomy of Educational Objectives: Cognitive Domain (1956) laid out a hierarchy of skills: knowledge, comprehension, application, analysis, synthesis, and evaluation. It is difficult to identify a more powerful influence on the American school curriculum, and perhaps curricula worldwide, than Bloom’s Taxonomy. The first two layers, knowledge and comprehension, are synonyms for remembering and understanding what one has learned. Although the hierarchical structure of Bloom’s Taxonomy has been challenged, no serious model has emerged that eradicates the prerequisite roles of knowledge and comprehension. It is difficult to think deeply about Shakespeare without actually having read his work, remembering it, and grasping at least a good part of what he was saying.

Deeper learning, like its intellectual ancestors, tries to turn all of this on its head and upend the pre-eminence of knowledge. CGI and PISA both exist in opposition to an element of knowledge in the traditional school curriculum. The first grade math curriculum of CGI wishes to move beyond algorithms, to dig beneath them, really, so as to uncover what is happening when two numbers, as in the example above, are added together.

I appreciate that aim, but it doesn’t work. Yes, I know, algorithms can be horribly boring, especially when taught as rote procedure; however, the algorithms of arithmetic are elegant procedures jam-packed with mathematics. Long division, perhaps the most notorious of the algorithms learned in elementary school—notorious for being taught in a boring manner--carries within it four separate operations of whole numbers (addition, subtraction, multiplication, and division), ironclad rules about place-value (if you put numbers in the wrong place, you get the wrong answer), and precursors for understanding fractions (remainders can be expressed as fractions, and all division problems can be expressed as a fraction). That’s deeper knowledge that can be missed with deeper learning.

PISA involves the high school curriculum, and it’s upending of traditional approaches is more subtle. I have critiqued PISA’s approach to math and science elsewhere (see here and here). In high school reading instruction, PISA’s main project seems to be to take literature down a notch in the language arts curriculum. In doing so, it elevates the trivial to the level of the sublime. Bus schedules and recipes are as valuable as Twain and Faulkner in the United States, as Flaubert and Proust in France, and as Dostoyevsky and Tolstoy in Russia. Worldwide, it seems, PISA wants to redefine what it means to be a literate person.

In the days ahead, you will be hearing a lot about deeper learning. Please be on guard. This virtuous sounding term means much more than its two words imply.

[1] Oddly, students are required to know the word “metaphor” in another PISA item (see “Beatrice” on page 186.

Authors

Tom Loveless

Image Source: © Jim Young / Reuters

Ability Grouping, Tracking, and How Schools Work

Tom Loveless — Wed, 03 Apr 2013 11:00:00 -0400

The 2013 Brown Center Report on American Education was released two weeks ago. One of the studies is on ability grouping. A key finding is that elementary teachers are using ability grouping again. Ability grouping is the practice of dividing classes into small instructional groups, especially for teaching reading. According to data collected by the National Assessment of Educational Progress (NAEP), the frequency of ability grouping’s use in fourth grade reading instruction rose about two and a half times, from 28 percent in 1998 to 71 percent in 2009.

This year marks the 30^th anniversary of the publication of How Schools Work by Rebecca Barr and Robert Dreeben, a book in which ability grouping plays an important role. I became aware of the book at the University of Chicago in 1988 as a Ph.D. student. Robert Dreeben was my program advisor and dissertation chair.

Ability grouping is one method by which educators differentiate instruction. The term “differentiation” refers to the many ways that schools try to tailor different learning experiences to children’s varying levels of performance. In the 1980s, I earned a masters degree in special education and taught both learning handicapped and gifted students. Differentiation was in my blood when I arrived at Chicago.

Differentiation was also under fire. Ability grouping and tracking were becoming taboo. The popular research at that time, which was predominantly qualitative and impressionistic, condemned tracking and ability grouping for harming black, Hispanic, and economically disadvantaged students. This literature often depicted teachers as stupid or evil: stupid by robotically following tradition and unwittingly imposing harmful practices on students; evil by harboring race- or class-based prejudices that manifested in low expectations for many kids.

That is what made How Schools Work so refreshing. The book honors teachers in a profound way, not in a “you are all saints and we love you” way, but in a manner much more meaningful—by studying teachers’ work. Barr and Dreeben followed a group of Chicago first grade teachers as they taught reading. A wealth of data was collected so that hypotheses could be tested empirically. In How Schools Work, readers discover that first grade reading groups operate within a grand organizational scheme: groups nested in classrooms, classrooms housed within schools, schools situated within a big urban district. Seemingly routine tasks of teaching are transformed into thoughtful, important activities. Teachers do not appear to be stupid or evil. They appear to be professionals engaged in purposeful activities.

In 1988, “The Formation and Instruction of Ability Groups,” was published in the American Journal of Education. Adam Gamoran, a Chicago graduate student at the time, worked on the project producing this paper. Dreeben and Barr describe as “technological” the ways in which teachers form groups and then instruct them; not technological in the sense of using computers or electronic media but in the sense of applying craft knowledge in the pursuit of an occupational end, in this case, the goal of organizing a classroom full of first graders so that they can be taught how to read.

The notion that teaching is primarily intuitive (“teachers are born not made”) was directly refuted. When they teach reading, teachers must juggle four inputs, each with its own constraints --student aptitude, the difficulty of reading materials, time devoted to instruction, and coverage of curriculum. The combination of these four inputs must be expertly managed to optimize learning. Sure, sometimes teachers have to fly by the seat of their pants while teaching, but for most of time, they employ craft knowledge to attain just the right mix. Kids do in fact learn how to read, and first grade, more than any other grade, is where that wonderful accomplishment can be observed while it happens.

Teachers aren’t perfect. They can make mistakes. They can form groups that are too large, too small, or too unwieldy in composition; move groups too fast or too slow; teach from a curriculum that is too demanding or too easy; or fail to provide enough time for instruction. They can also be unfair – even bigoted – but that’s not the norm.

It is heartening to note that as the use of ability grouping is increasing a new generation of researchers is bringing sophisticated statistical techniques (and open minds) to bear on questions involving both ability grouping and tracking. Tracking, the middle and high school practice of grouping students into separate classes as opposed to grouping students within a class, has always drawn the most scholarly attention. And the most opprobrium.

In a recent NBER working paper, Courtney A. Collins and Li Gan classify Dallas schools as sorted or non-sorted based on the heterogeneity of classes in math or reading achievement. The study also considers heterogeneity in the dispersion of students identified as gifted and talented, limited English speaking, or special education. Sorting is found to produce significantly positive effects in both reading and math -- and for both high and low achievers. The researchers conclude:

This study has valuable policy implications because unlike many school policy variables, the composition of classes can often be changed with little need for increased funds. A school with a fixed number of classrooms and teachers can increase efficiency by rearranging students in the most effective way possible. This study suggests that creating classes with lower levels of dispersion of score or ability level may improve the achievement outcomes for students across the score distribution (Collins and Gan, 2013, page 20).

The study joins a long line of research dating back to at least the 1920s. The overriding concerns have been to determine whether tracking and ability grouping are good or bad (whether they produce positive effects) and whether they are equitable (even if some students benefit, is it at the expense of others). The evidence on these questions is mixed. To adequately summarize the literature would require a series of posts, and I will return to this topic in the future. The main point I would like to make in concluding this post pertains to the renewed popularity of tracking and ability grouping, not to whether either practice is warranted by research.

In the late 1980s and into the 1990s, powerful groups condemned ability grouping and tracking, among them, the National Governors Association, the NAACP Legal Defense Fund, and the Children’s Defense Fund. The use of ability grouping dropped significantly in the 1990s. Tracking in middle schools declined in all subjects but math. According to the NAEP data reported in the Brown Center Report, ability grouping has made a strong comeback in the past decade.

The resurgence of ability grouping accentuates the need for new research questions. If educators are going to use ability grouping again, how should they employ this tool so as to maximize potential benefits and minimize potential harms? How large should groups be? How many groups should a teacher create, and how much time should be spent with each one? Do low achieving groups require more direct instruction than high achieving groups? How often should students be assessed and regrouped? Are different curricula more effective with different groups? Notice the thrust of these inquiries. Such questions are directed towards producing new knowledge on the craft of teaching and to guide teachers in improving their practice, not towards the policy question of whether to group or not to group.

A fine example of this kind of study is provided by Carol McDonald Connor and colleagues at Florida State University. The researchers conducted a randomized field trial of software that organizes first grade reading instruction. The algorithm employed by the software considers each child’s entering skill level and progress made during the school year to recommend several dimensions of instruction, including assignment to small, homogenous ability groups, the amount of time spent on code- versus meaning-focused literacy, and teacher/child versus child-managed delivery. The targets for these recommendations are dynamic; that is, they change in response to periodic assessment of children’s progress. Children in the experimental classrooms gained about two months in reading achievement over those in the control group.

I hope the new generation of researchers will take up more questions like those in the FSU study. The debate over tracking and ability grouping has gone on for nearly a century. Research has not answered the key questions in dispute, at least not to the protagonists’ satisfaction. It’s time for some different questions. How should researchers proceed? A good place to start is reading How Schools Work. It’s just as fresh and illuminating today as when it was published thirty years ago.

Authors

Tom Loveless

Image Source: © Jim Young / Reuters

Is Tracking and Ability Grouping Making a Comeback?

Tom Loveless — Wed, 20 Mar 2013 15:00:00 -0400

On Monday, we released the annual 2013 Brown Center Report on American Education. The report contains three individual studies: one on the latest in international testing progress, one on tracking and ability grouping, and one on advanced math in eighth grade.

Below is a first look at a video we put together in which I discuss the details of Part II, on the resurgence of ability grouping and tracking.

Download Media

The Resurgence of Ability Grouping and Tracking: A Return to Controversial Practices?

Feel free to submit your thoughts and comments below.

Video

The Resurgence of Ability Grouping and Tracking: A Return to Controversial Practices?

Authors

Tom Loveless

2013 Brown Center Report on American Education: How Well Are American Students Learning?

Tom Loveless — Mon, 18 Mar 2013 00:00:00 -0400

Editors' Note: The introduction to the 2013 Brown Center Report on American Education appears below. Use the Table of Contents to navigate through the report online or download a PDF of the full report.

Table of Contents

PART I: The Latest TIMSS and PIRLS Scores
PART II: The Resurgence of Ability Grouping and Persistence of Tracking
PART III: Advanced Math in Eighth Grade

INTRODUCTION

This is the twelfth edition of the Brown Center Report. The structure of the report remains the same from year to year. Part I examines the latest data from state, national, or international assessments. This year the focus is on the latest results from the Progress in International Reading Literacy Study (PIRLS) and Trends in International Math and Science Study (TIMSS) released in December, 2012. The U.S. did relatively well, posting gains in reading, math, and science. Finland made headlines by registering declines from the last time it took the TIMSS math tests. At both fourth and eighth grades, the scores of Finland and the U.S. are now statistically indistinguishable in math. Part I also looks at the so-called “A+ countries,” named that because they were the top nations on the first TIMSS, given in 1995. Part I offers “A Progress Report on the A+ Countries,” and finds that, surprisingly, three of the six have registered statistically significant declines since 1995. Despite that, most of the A+ countries still score among the world’s leaders. The exception is the Czech Republic, which scored at approximately the international average the last time it took TIMSS in 2007.

Part II explores a perennial theme in education studies—the topics that never seem to go away in terms of research and debate. This year it’s on the controversial topics of tracking and ability grouping. An analysis of data from the National Assessment of Educational Progress (NAEP) documents a resurgence of ability grouping in fourth grade reading and mathematics. Tracking remains persistent in eighth-grade math, with about three-fourths of students in tracked classes. As readers are surely aware, both practices have been attacked for decades as inequitable, and many school analysts thought their use had diminished. Ability grouping was dominant for a long time in the elementary grades. Reading groups were the norm through most of the twentieth century and then declined dramatically in the 1990s. They are now coming back—and back strongly.

Part III is on a prominent policy or program. This year’s analysis is on the national push for eighth graders to take algebra and other high school math courses. Algebra is now the single most popular math course in eighth grade. The study in Part III uses state variation in enrollment rates to ask the question: what has happened to the NAEP scores of states that boosted their eighth-grade advanced-math enrollments? The study uncovers no relationship between change in state NAEP scores and change in enrollments. States boosting advanced math taking are no more likely to show NAEP gains than other states.

A second analysis uncovers some evidence consistent with the idea that advanced math courses are being “watered down,” that the mean achievement levels of advanced courses fall as enrollments go up. Again, change in NAEP score is the outcome of interest. The study shows that states that are more selective in math placements—not aggressively accelerating eighth graders into advanced courses—are more likely to show achievement gains in those courses.

There is one intriguing divergence from this finding: eighth-grade geometry classes. Geometry sits at the peak of the hierarchy of eighth-grade math courses, enrolling the nation’s best math students (about 5%). Presumably, these are students who took algebra in seventh grade. Increases in eighth-grade geometry enrollments evidence no association with changes in mean achievement for the course, not what one would expect if unprepared students were being accelerated into the course. This suggests that schools are implementing two different types of acceleration, one based on the age or grade of students, the other based on students’ preparation and readiness for advanced work. The analyses in the study are only correlational and cannot confirm or reject causality. Part III concludes with a discussion of hypotheses for future study to improve both strategies.

Download Media

Part I: The Latest TIMSS and PIRLS Scores »

Downloads

2013 Brown Center Report on American Education

Video

Authors

Tom Loveless

Advanced Math in Eighth Grade

Tom Loveless — Mon, 18 Mar 2013 00:00:00 -0400

As recently as 1990, taking algebra in eighth grade was unique. That has changed dramatically in recent years, and now more eighth graders take algebra than any other math class. Enrollment in eighth-grade algebra—and in other advanced math classes—varies by state. This section of the Brown Center Report exploits that variation to study the relationship of states’ enrollment in advanced math classes and scores on NAEP. The research question is whether a relationship exists between changes in advanced math enrollments and changes in 8th grade NAEP scores. Do states that boost advanced enrollments experience a concurrent increase in achievement? A second analysis uses the same technique to look at the potential that advanced courses are being “watered down.” Are rising enrollments associated with lower mean achievement in advanced classes?

Background

In 1982 Robert Moses was awarded a MacArthur Fellowship. He used the money to start The Algebra Project, a community-based effort to bring algebra to historically underserved middle school students—primarily, children from low income households and students of color. Moses called algebra “the new civil right,” an invocation of equity that cast course taking in a new light.³² The Clinton Administration tied the equity theme to international competitiveness and pushed for more students to take algebra before high school. “Around the world, middle students are learning algebra and geometry,” President Clinton observed. “Here at home just a quarter of all students take algebra before high school.”³³

Algebra soon came to be known as a “gatekeeper” course, a class standing like a sentry at the gateway to college. Take it and pass it and your odds of attending college were good. Take it and fail it and at least you had been exposed to challenging mathematics. Don’t take it at all and your chances of attending college were near zero. Algebra’s place in the typical high school math sequence enhanced its importance. Assume that college-going students should get some calculus under their belts in the senior year. In most high schools, a student who takes Algebra I in ninth grade has three remaining years to take Algebra II, Geometry, Pre-Calc/Trigonometry, and then Calculus. That’s four courses. Something has to give. Many schools change the order of the courses, and some mix in statistics with one of the year’s offerings, but the fact remains: if taking Calculus as a senior in high school is the goal, taking Algebra I in ninth grade means there are four courses to complete in three years. Taking algebra in eighth grade opens up an additional year for advanced math.

Equity, international competiveness, and practical concerns about course sequences converged in the mid 2000s to boost the campaign for eighth-grade algebra. An “algebra for all” movement emerged that pushed universal, mandatory eighth-grade algebra. Minnesota established a new high school graduation requirement that, beginning with the class of 2015, all students must complete an Algebra I credit by the end of eighth grade. California used its school accountability formula to promote eighth-grade algebra, offering a choice of two eighth-grade math assessments (algebra and general eighth-grade math) but then, in the formula for calculating Academic Performance Index (API), discounting the performance level of students taking the general math test (for example, downgrading to “basic” those students who took the test and scored “proficient”). That incentive motivated schools to dramatically increase eighth-grade algebra enrollments, and although the AYP rule was later tossed out by the courts, California ranks as the top state in the nation for eighth-grade algebra and advanced math enrollments.³⁴

NAEP Data on Advanced Math Enrollment

Table 3-1 illustrates the steady increase of U.S. eighth-grade enrollment in advanced mathematics courses. The data are taken from the NAEP eighth-grade math assessment. Students are asked: “what mathematics class are you taking this year?” The category “advanced mathematics” combines several responses, including Algebra I, courses that stretch Algebra I content over two years (whether it’s the first or second year of such a course), and courses that typically are more advanced than Algebra I, including Algebra II and Geometry. This amalgamated response is noisy and receives further discussion below.

In 1990, only 16% enrolled in an algebra course, compared to 20% in pre-algebra and 61% in 8th grade math. In this paper, the latter two courses are referred to as “basic.”
By 2011, nearly half (47%) of all eighth graders took algebra or a more advanced course. Only 48% were in a basic math course, down from 81% in 1990. The advanced math percentage may be understated in Table 3-1 for the years prior to 2000 as that was the first time geometry, advanced algebra, and algebra stretch classes were response categories in the NAEP questionnaire for eighth graders.³⁵ Moreover, some students—both then and now—may mistakenly believe they are in an algebra or geometry class when in fact they are not. Notwithstanding these data limitations, advanced math enrollments clearly rose substantially from 1990 to 2011.³⁶

More and more students are taking advanced math classes earlier and earlier. Is this a good idea?

Research on the Efficacy of Eighth-grade Algebra

The National Educational Longitudinal Study of 1988 (NELS) offers researchers a trove of information collected from a randomized sample of students. Several studies have used NELS data to investigate what happens when students take advanced math early in an academic career, whether eighth or ninth grade.³⁷ Researchers found gains for students taking algebra earlier rather than later, including—and this is important for the equity goal—low performing students. A recent meta-analysis of research on the topic (by Mary K. Stein and colleagues) reaffirmed that positive finding, with the caveat that “achievement gains occurred in settings where policies were accompanied by strong supports for struggling students, particularly more time for algebra instruction. We do not have strong evidence that universal algebra policies lead to achievement gains minus those strong supports.”³⁸

More recent evaluations of policies expanding algebra enrollment have raised cautionary flags. Chicago mandated that all ninth graders take what had been regarded as college preparatory classes, including algebra. Evaluators followed students for several years and concluded, “Although more students completed ninth grade with credits in algebra and English I, failure rates increased, grades slightly declined, test scores did not improve, and students were no more likely to enter college.”³⁹ Studies of California’s algebra policies found a trade-off: rising enrollments but also a rising number of failures. In North Carolina, researchers from Duke uncovered negative results after studying a Charlotte-Mecklenburg initiative to expand algebra in eighth grade: lower scores on the Algebra I test and then lower pass rates in Geometry and Algebra II in subsequent years.

Why have the more recent studies produced bleaker findings than suggested by the earlier work? The Duke researchers believe selection bias skewed the earlier findings. Stronger math students take algebra in eighth grade, and although they indeed may benefit academically from the course, that does not mean that weaker students will also benefit from taking algebra earlier. “Once this selection bias is eliminated, the remaining causal effect of accelerating the conventional first course of algebra into earlier grades, in the absence of other changes in the math curriculum, is for most students decidedly harmful.”⁴⁰

The Stein et al. meta-analysis and the Duke team’s policy recommendations, although different in emphasis, do share a small patch of common ground. Stein et al. say that without “strong supports” achievement gains cannot be expected. And the Duke researchers foresee harmful effects “in the absence of other changes in the math curriculum.” One is contingently positive, the other contingently negative. The common ground that they share is in forecasting the potential for a neutral effect.

Let’s return to NAEP and see what its data have to say about state efforts to encourage enrollment in advanced math courses in eighth grade.

Analytical Method

Are eighth-grade enrollments in advanced math related to states’ math scores on NAEP? To answer this question, an obvious first step is simply to examine the list of states, their NAEP scores, and the percentage of each state’s students taking algebra, geometry, and other advanced math courses in eighth grade. There is no clear relationship. In 2011, the correlation between states’ advanced math enrollments and NAEP achievement is 0.07, indistinguishable from 0.00. States with more eighth graders taking advanced math classes are no more likely to register a higher NAEP score in math than states with lower enrollments in those classes.

This kind of cross-sectional analysis is a reasonable place to start, but it’s limited to revealing correlations between variables at a single point in time. That can be misleading. A study in the 2007 Brown Center Report, for example, showed how the number of instructional minutes that nations devote to math instruction is unrelated, on a cross-sectional basis, to national math achievement. In 1995, the correlation was 0.05. In 2003, the correlation was -0.20. Neither statistic is significantly different from 0.00. But when nations are examined longitudinally, and data from the two cross-sections are modeled as change variables, the question under scrutiny is shifted to whether national changes in instructional minutes from 1995 to 2003 are related to changes in test scores over the same time period. The correlation for that relationship is 0.42, which is statistically significant. Countries that increased the amount of time devoted to math instruction tended to experience a rise in TIMSS math scores; those countries that decreased the time devoted to math instruction tended to see their scores fall.

Why is the analysis of change variables beneficial? Two reasons. The first is that the technique helps to control for bias introduced by omitted variables (including selection), a shortfall plaguing cross-sectional analyses of achievement. In the case of instructional minutes, for example, school systems might strategically decide to place low achieving students in longer classes to help them catch up. That would make it appear that more instruction is associated with lower achievement. Assuming that omitted variable bias is present at both the beginning and end points of the time interval under study—and the relationship to the dependent variable (the outcome of interest) remains consistent over the interval—such bias washes out in the calculation of change (see Gustaffson, 2007, for further explanation and applications to other educational questions).⁴¹

The second benefit of this approach is that it poses a question paramount to policy analysis. Considering whether to adopt policy X leads to the question: if we adopt policy X, what is the expected change in outcome Y? What will happen? The cross-sectional question is this: what is the relationship of policy X to outcome Y at one point in time? One often hears of cross-sectional analyses showing something along the lines of “a one-standard deviation change in X would result in the following change in Y,” but the prediction is only inferred, there being no observations of change (or data from different time periods) in the data set.

Analysis of Change Using NAEP Scores

The relationship between change in policy and change in outcome is the subject of the analysis below. The time period examined is 2005 to 2011. Be aware, notwithstanding the improvement over cross-sectional analysis, that the analysis is still only correlational and thus confined to generating plausible hypotheses for more rigorous research designs. No causality is asserted here.

Table 3-2 shows the tail end of the long term trend sketched in Table 3-1—enrollment gains in advanced math classes and declines in basic classes. The slow, steady national trend masks considerable variation among the states. In 2005-2011, the average state increase in advanced math enrollments (as a proportion of eighth graders) was 5.5%, with a standard deviation of 8.4%. The top four states that boosted advanced enrollments were Minnesota (35%), and Pennsylvania, Virginia, and Washington (all with 17%). In contrast, two states stand out for going against the national trend with shrinking advanced math enrollments: Nevada (-22%) and Georgia (-17%).

In terms of specific courses, forty-five states boosted enrollments in Algebra I, while only three states shrank enrollments and three stayed the same (in this discussion of NAEP scores, the District of Columbia is considered a state). Twenty-eight states decreased enrollments in general math, twenty increased, and three stayed the same. In general, course enrollments behave like a tube of toothpaste—squeeze on one end and the other end bulges. States with rising advanced math enrollments experienced shrinking enrollments in basic courses. And vice versa. The two states singled out for declining enrollments in advanced math courses illustrate the point. Their basic math enrollments rose. Nevada’s pre-algebra enrollments jumped 27%. Georgia’s percentage of students in general math rose 33%.

Is there a relationship between states’ change in course enrollments and change in NAEP scores? Did states experience gains on NAEP concurrent with increases in eighth graders taking advanced math? A series of correlation coefficients were computed to investigate these questions (see Table 3-3). The first model examines the relationship of advanced math enrollments and NAEP composite scores. The correlation coefficient (r = -0.01) is statistically indistinguishable from 0.00.

The NAEP composite score may assess mathematics too broadly to pick up the effects of emphasizing advanced math, which primarily involves boosting algebra. Fortunately, NAEP reports scores on specific content areas assessed within the test (called “strands”), including algebra and geometry. So the second model uses the NAEP subscore for the algebra strand as the achievement variable, which should be more sensitive to increased knowledge of algebra. Again, no significant relationship is found.

The third and fourth models use change in Algebra I enrollments as the course variable, instead of advanced math, in case aggregating several courses into the “advanced” category has muddied the waters. The change in composite NAEP score serves as the achievement variable in the third model and the change in the algebra strand score as the achievement variable in the fourth model. Neither correlation attains statistical significance.

Models five and six repeat the same treatment with geometry. Change in geometry course taking in eighth grade is used as the course variable—and the models calculate whether it is correlated with change in the NAEP composite in model five and change in geometry score in model six. Neither correlation is statistically significant.

In addition to the correlations reported here, multivariate regressions were run with three covariates controlled (also variables representing change)—change in state rates of child poverty, English language learners, and black and Hispanic students—demographic characteristics that are known correlates of state NAEP scores. The Great Recession unfolded during the time period under study, and some states, for example, witnessed growing rates of child poverty more than other states. If states experienced demographic changes, that could skew the results. It turned out not to be the case. None of the regression models were statistically significant.

In sum, no evidence was found in NAEP scores of a relationship between states raising enrollment in advanced math courses and raising achievement. States that increased the percentage of students taking algebra or geometry in eighth grade were no more likely to post NAEP gains than states with decreased enrollments in those two courses.

Do Rising Enrollments Water Down Advanced Math Courses?

Whether advanced math courses are watered down because of increasing enrollments is an important question. The notion is that filling advanced classes with academically weaker students than in the past could diminish the amount of learning that the courses are able to impart. That could help to explain the neutral correlations reported above. It could also help to explain the neutral—or even negative effects—revealed by recent evaluations of policies promoting universal algebra in eighth and ninth grades. NAEP data can only go so far in indicating whether watering down is taking place, but they do offer interesting insights into how course-shifting and achievement may be related.

Table 3-4 reports correlations between enrollment change and change in the mean achievement of students taking each course. Data from four courses are displayed. Again, the percentage of a state’s eighth graders taking each course serves as the enrollment variable. The courses are arranged hierarchically. Geometry is typically offered for the most advanced students and general math for the weakest ones. Three correlations are statistically significant.

Is there evidence of watering down? Yes, but not in all advanced courses. Let’s start with the results supporting the watering down hypothesis. Increases in Algebra I enrollments are negatively associated with achievement gains (r = -o.34, p < .05). Let’s be clear what that means. The average state registered a 5.6 NAEP scale score gain among its Algebra I students. The NAEP scores for students in Algebra I classes did not go up as much in states that raised enrollments in Algebra I (+5.2) as in states that either held enrollments constant or decreased them (+9.2). For Pre-Algebra, rising enrollments are also negatively associated with test scores (r = -0.34, p < .05). Both correlations are consistent with the watering down hypothesis if students who would otherwise be placed in lower courses are migrating upward to higher courses. We cannot tell whether that is happening using NAEP data. And, to issue an important warning once again, correlations do not prove causality.

The strongest correlation involves General Math (r = 0.47, p < .01). The positive association is also consistent with the watering down hypothesis. If the overall trend is to move students into upper-level courses—and schools are selective in the students they accelerate—General Math courses, as they shrink, should be increasingly dominated by the students who struggle the most at math. These courses presumably would have lost their best students. Falling enrollments would therefore be associated with falling scores. General Math classes that manage to keep the students who are being accelerated elsewhere would, comparatively, register higher scores.

Geometry complicates matters. Its correlation coefficient (0.27) is inconsistent with the watering down story. Geometry sits at the top of the course hierarchy. Any indiscriminate acceleration of students upward (an inextricable assumption of the watering down argument) should ultimately result in a negative association of enrollment gains and achievement scores in the course at the top. And yet, Geometry’s correlation coefficient has a positive sign and approaches statistical significance. Although statistically indistinguishable from 0.00 (p = .11), that could be due in part to the reduced number of states with data. Only thirty-six states have sufficient numbers of eighth-grade geometry students to produce a NAEP score.

Another possibility involves the noisy NAEP course variables. Perhaps more “real” geometry students are included in the NAEP course category for geometry in 2011 than in 2005—in other words, a larger proportion who are actually in a geometry class and not mistaken about their math course. As indicated in Table 3-2 above, only 5% of eighth graders were enrolled in Geometry in 2011, an increase from 4% in 2005. The mean NAEP score for geometry students was 290 in 2005 and 308 in 2011, a sharp increase of 18 points. The one-percentage-point gain in students seems to have packed a punch in terms of NAEP scores. The “real” geometry students probably took Algebra I in seventh grade. Much like algebra for eighth graders three or four decades ago, geometry is reserved for today’s very best math students.

Discussion

This study analyzed variation in state enrollment patterns to test whether rising enrollments in advanced eighth-grade math courses are correlated with achievement gains on NAEP. No evidence was found that they are. States with rising percentages of eighth graders taking Algebra I, Geometry, and other advanced math classes were no more likely to raise their NAEP scores from 2005-2011 than states with declining percentages of eighth graders in those courses.

A second analysis, again looking at changes in policy and test scores over time, investigated whether boosting the percentage of students in higher level courses is associated with decreases in the mean scores of those courses—suggesting a watering down effect. The evidence is consistent with watering down in all but one course. Negative correlations were found for Algebra I and Pre-Algebra. In those courses, mean achievement gains declined as enrollments increased. Achievement gains in general math courses were positively associated with enrollment changes. All three of these correlations are statistically significant and supportive of the watering down hypothesis.

Geometry diverges from the other courses. A positive association was found that, although statistically indistinguishable from 0.00, suggests at least a neutral relationship between rising enrollment and changes in NAEP scores. If schools were indiscriminately accelerating students into eighth-grade geometry, one would expect a negative correlation.

None of these findings can confirm or reject causality, but they are useful in generating hypotheses for future study. They also shed light on the findings from previous research. For example, a key finding from evaluations of California’s algebra policy is that universal algebra produces trade-offs. Many students benefit from the extra challenge. Rates of algebra enrollment for historically under-enrolled populations (in particular, low SES students) have increased. The raw number of students passing end of course exams has also increased. But the downside is that the number of students failing algebra goes up as well; and the failing students, too, are disproportionately low SES students.⁴² One study from California suggests that many of the failing students would have been better off spending an additional year preparing for algebra instead of taking it.⁴³ These kinds of trade-offs, when aggregated to the state level, could produce a neutral net effect.

The analysis of whether accelerating students into advanced classes is watering down achievement points to two different types of acceleration. One is selective and decided on an individual basis. Each student’s math skills are evaluated and a determination is made whether a more advanced math course is appropriate or not. That kind of acceleration appears to be occurring in eighth-grade geometry—and presumably in seventh-grade algebra. Students who would benefit from a more rigorous course are promoted. Mean test scores for eighth-grade geometry rise, or at least stay the same, despite rising enrollments.

The second type of acceleration is non-selective and group based. Students are advanced based on a characteristic independent of prior achievement or preparedness (e.g., grade level or age). Future research should compare these two types of acceleration and investigate who, when it comes to selective acceleration, should be accelerated and when. With age- or grade-based acceleration, a set of early indicators is needed (the universal algebra approach) that would identify students needing support and the type of support most beneficial for them. If the trade-offs of group acceleration are indeed real, then the policy goal should be to minimize negative effects and maximize benefits.

A final note on the Common Core. No one knows how gifted students’ needs will be met in the Common Core Era. Taking algebra in eighth grade is the new normal, and taking algebra in the seventh grade is rapidly becoming the new normal for gifted math students. In California, 8.1% of seventh graders (nearly 38,000 students) took the algebra end of course exam in 2012. If Common Core means the same curriculum for all, a time will surely come when exceptional math students need an uncommon curriculum that is appropriate for them.

« Part II: The Resurgence of Ability and Persistence of Tracking

Part III Notes

32. Background information on the Algebra Project is available at www.algebra.org.

33. Remarks by President Clinton, Education Roundtable, Springbrook High School, Silver Spring, MD, March 16, 1998. Available at http://www.gpo.gov/fdsys/pkg/WCPD-1998-03-23/pdf/WCPD-1998-03-23.pdf.

34. History of California’s algebra policy can be found in: Algebra Policy in California: Great Expectations and Serious Challenges (Oakland: EdSource, May 2009). Also see Tom Loveless, The Misplaced Math Student: Lost in Eighth-Grade Algebra (Washington, DC: The Brookings Institution, 2008).

35. The category “other” received about a 3% response rate before 2000 so the number of students taking more advanced classes was probably very small.

36. Jill Walston and Jill Carlivati McCarroll, Eighth-Grade Algebra: Findings from the Eighth-Grade Round of the Early Childhood Longitudinal Study, Kindergarten Class of 1998-1999 (ECLS-K) (Washington, DC: National Center for Education Statistics, October 2010).

37. See David Stevenson, Kathryn S. Schiller, and Barbara Schneider, “Sequences of Opportunities for Learning,” Sociology of Education 67, no. 3 (1994): 184-198; Adam Gamoran and Eileen C. Hannigan, “Algebra for Everyone? Benefits of College-Preparatory Mathematics for Students with Diverse Abilities in Early Secondary School,” Educational Evaluation and Policy Analysis 22, no. 3 (2000): 241-254; Julia Smith, “Does an Extra Year Make Any Difference? The Impact of Early Access to Algebra on Longterm Gains in Mathematics Achievement,” Educational Evaluation and Policy Analysis 18 (1996): 141-153.

38. See Mary Stein, Julia Kaufman, Milan Sherman, and Amy Hillen, “Algebra: A Challenge at the Crossroads of Policy and Practice,” Review of Educational Research 81, no. 4 (2011): 453-492.

39. Elaine Allensworth, Takako Nomi, Nicholas Montgomery, and Valerie E. Lee, “College Preparatory Curriculum for All: Academic Consequences of Requiring Algebra and English I for Ninth Graders in Chicago,” Educational Evaluation and Policy Analysis 31, no. 4 (2009): 367-391.

40. Charles T. Clotfelter, Helen F. Ladd, and Jacob L. Vigdor, The Aftermath of Accelerating Algebra Evidence from a District Policy Initiative (Washington, DC: National Center for Analysis of Longitudinal Data in Education Research, American Institutes for Research, 2012).

41. Jan-Eric Gustafsson, “Understanding Causal Influences on Educational Achievement through Analysis of Differences over Time within Countries,” in Lessons Learned: What International Assessments Tell Us about Math Achievement, ed. Tom Loveless (Washington: Brookings Institution Press, 2007).

42. Trish Williams, Edward Haertel, and Michael W. Kirst, Improving Middle Grades Math Performance: A Closer Look at District and School Policies and Practices, Course Placements, and Student Outcomes in California. Follow-Up Analysis (Mountain View: EdSource, 2011).

43. Jian-Hua Liang, Paul Heckman, and Jamal Abedi, “What Do the California Standards Test Results Reveal About the Movement Towards Eighth-Grade Algebra for All?” Educational Evaluation and Policy Analysis 34, no. 3 (2012): 328-343.

The Latest TIMSS and PIRLS Scores

Tom Loveless — Mon, 18 Mar 2013 00:00:00 -0400

In December 2012, the latest international test scores were released. The Trends in International Math and Science Study (TIMSS) is given every four years, and the Progress in International Reading Literacy Study (PIRLS) is given every five years. The latest results came from the 2011 administration of both tests, a unique event. Because of their asynchronous schedules, the two tests share the same year only once every twenty years. Forty-nine nations and nine benchmarking participants took part in PIRLS, which is given at fourth grade, and 63 nations and 14 benchmarking participants took part in TIMSS, which is given at both fourth and eighth grades.¹

U.S. National Achievement

The U.S. did reasonably well in all three subjects—reading, math, and science. In reading, the U.S. scored 556 on the international scale. All of the tests discussed in this section have a mean of 500 and a standard deviation of 100. Only four countries scored statistically significantly higher on the reading test. (In the discussion below, the term “significant” is used as shorthand for statistical significance at p<.05). Hong Kong led the world at 571, followed by the Russian Federation (568), Finland (568), and Singapore (567). The U.S. score for 2011 represented a 14-point gain since 2001 (significant).

In math, U.S. fourth graders scored 541, near the middle of second-tier countries on TIMSS. The top-tier countries were five Asian nations: Singapore (606), Korea (605), Hong Kong (602), Chinese Taipei (591), and Japan (585). The U.S. fourth-graders’ score represents a 23-point gain since 1995 (significant). Eighth graders in the U.S. scored 509, which is significantly higher than the 500 international average—but just barely. The 509 score is a 17-point improvement over the 1995 U.S. score (a significant gain).

In science, U.S. fourth graders scored 544, with six countries scoring at significantly higher levels. The fourth-grade gain of 2 points since 1995 is not statistically significant. Eighth graders scored 525, significantly above the international average and significantly below students from eight other nations. The 12-point gain since 1995 is statistically significant.

To sum up, the latest international scores are mostly positive for the U.S. American students scored above the international average on all five assessments of grade-subject pairings. For four out of the five tests, the gains since 1995 are statistically significant. Despite these encouraging results, there is much room for improvement. Over the past decade, countries joining TIMSS have been economically developing nations or, in the case of the Middle East, nations possessing abundant national wealth but lacking a tradition of public schooling. Such compositional changes can make international averages easier to surpass. Leading the world in reading, math, or science remains a challenge for the U.S.

State Achievement on TIMSS

Nine states took part in the TIMSS assessment (see Table 1-1). Let’s focus on eighth-grade mathematics as that is the only test on which all nine participated. As points of reference, be reminded that the international average for the test was 500, the U.S. national score was 509, and the top scoring nation was Korea at 613.

Massachusetts led the pack with a 561 score, followed by Minnesota (545) and North Carolina (537). Five of the states had taken TIMSS before, and three registered statistically significant gains from the first time they participated. As indicated in Table 1-2, the TIMSS scores map reasonably well onto NAEP scores. Because NAEP was also given in 2011, the National Center for Education Statistics was able to conduct a NAEP-TIMSS linking study.² Items from TIMSS and NAEP were embedded in the same booklets so that items from both tests were taken by the same student at the same time. Results of the study will be released later in 2013. The hope is that future analysts will be able to calculate, with reasonable precision, projected state TIMSS scores based on NAEP scores, allowing local leaders to place state performance in an international context.³

Finland

Finland generated headlines from TIMSS. The “Finnish Miracle” story was called into question. In recent years, the popular press has been filled with stories about Finland’s wonderful education system. Educational tourism took many observers to Finland to see schools firsthand. Tales abounded of no homework, no high stakes tests, no tardy bells, a short school day, and the national belief that requiring children to start school before age seven violates “children’s right to be children.”⁴ Visitors marveled at the relaxed, home-like atmosphere—fireplaces in lounges, kids going shoeless, teachers called by their first names.⁵ The current worldwide angst (especially evident in the U.S. and Great Britain) over achievement, productivity, and rising test scores pursued through reforms such as school choice and accountability furnishes such a stark contrast that it has even drawn a derogatory acronym—GERM—from a Finnish scholar. That stands for Global Educational Reform Movement.⁶

One problem. Finland’s reputation is based largely on its performance on PISA, a very different test from TIMSS. The gap between the U.S. and Finland on PISA is statistically significant in mathematics literacy. On the 2011 TIMSS, however, Finland and the U.S. had statistically indistinguishable scores in both fourth and eighth-grade mathematics.

Look again at Table 1-1. Finland’s score of 514 in eighth-grade mathematics places it near the middle of the list of states. The scores of Alabama and California are the only two states scoring statistically significantly below Finland; the scores for Colorado, Connecticut, and Florida are about the same as Finland; and four states—Massachusetts, Minnesota, North Carolina, and Indiana—scored significantly higher than Finland. If Finland had been a U.S. state in 2011, it probably would have scored in the middle of the pack on NAEP. More troubling for the Finns, their TIMSS scores have declined significantly. Finland’s seventh graders took the test in 1999, scoring 520, and again in 2011, scoring 482. The 38 point decline is one of the largest recorded by a TIMSS participant.

A Progress Report on the A+ Countries

Cross-sectional data must be interpreted cautiously, and great care must be exercised when using them for predictive purposes. As Finland illustrates, a simple rule to remember is that sometimes things change.

Here is another example of that lesson, this time provided by a group of nations. The “A+ countries” are six nations that scored at the top of the 1995 TIMSS rankings in eighth-grade math. They are Belgium (Flemish community),⁷ Czech Republic, Hong Kong, Japan, Korea, and Singapore. Much hoopla was made about them when the 1995 TIMSS scores were released. In 2008, they were referenced as exemplars in the Final Report of the National Mathematics Advisory Panel. William H. Schmidt, Richard T. Houang, and colleagues have published a number of studies featuring a rubric based on the A+ countries’ math curriculums.⁸ The idea is that other countries should be more like the A+ countries. A 2012 study by Schmidt and Houang declared the Common Core mathematics standards comparable to the A+ countries’ curriculums in both focus and coherence. Moreover, they found that states with 2007 math standards similar to those of the A+ countries—again, using the same rubric from 1995—did very well on the 2007 NAEP. The findings were presented as implying that the Common Core will make the U.S. more like the A+ countries.⁹

Table 1-3 offers an update on the A+ countries. How are they doing? Let’s examine the table from the bottom-up. The Czech Republic left the TIMSS study after 2007, a year that saw its TIMSS score fall by 42 points from its performance twelve years earlier. Belgium (Flemish) has not participated in TIMSS since 2003. Its performance on TIMSS declined by 13 points before it left the study. The other four countries all took TIMSS in 2011. Hong Kong (+17) and Korea (+32) registered significant gains, Japan a significant decline (-11), and Singapore showed no significant change (+2). Of the six nations, then, two had statistically significant gains, three had statistically significant losses, and one scored about the same. The average score change for the six nations is -2.5 points, approximately equal to the average change for the 20 nations that participated in both 1995 and 2011. Put another way, the average A+ country made no more progress in math achievement than any other country in TIMSS.

Giving letter grades to entire nations may seem silly to many people but since the A+ designations have attained such widespread acceptance, readers are asked for their tolerance. It’s clear that A+ is no longer the appropriate grade for all of these countries.¹⁰ Korea and Hong Kong added to their outstanding 1995 scores and still deserve an A+. Singapore, too, although not making significant gains, surely preserves its A+ status by being one of only three nations with a 600+ scale score. Then things get dicey. Flemish Belgium was slipping when it left TIMSS in 2003. Its fourth graders did participate in 2011, however, and did well, scoring 549. That’s significantly higher than the U.S. at 541 and about the same as Florida at 545. But it represents no progress from the Belgian fourth graders’ previous TIMSS scores. Call Flemish Belgium a question mark—maybe an A- or B+, but definitely not an A+. We don’t know for sure without more recent eighth-grade data.

Japan’s score of 570 warrants an A, not an A+, and the downward trend is notable. Compare Korea with Japan. They both scored 581 in 1995. In 2011, Korea scored 43 points higher. The decline in the Czech Republic’s scores is the most dramatic, 42 points. The 2007 score of 504 is statistically indistinguishable from the international average of 500. Like Flemish Belgium, the Czech Republic fourth graders did participate in TIMSS 2011, scoring 511, a 30-point decline from 1995. The Czech Republic gets a C+ or B-.

Conclusion

What should we make of this? In 1995, six high achieving nations were described as “A+” to spur the U.S. towards greater math achievement. Their math curriculums were held up as ideals. And yet, since 1995, the U.S. gain of 17 points in eighth-grade mathematics is only exceeded by one A+ nation, Korea, and matched by another, Hong Kong. The other four A+ countries made less progress than the U.S. So in terms of gains, the U.S. should not look to the A+ countries for guidance. That said, five of the six A+ countries continue to lead the world in eighth-grade math achievement, and they continue to score significantly higher than the U.S.

The divergence of gain scores and status scores illustrates a problem that will be addressed in both remaining parts of this report. The tendency is for observers, when test scores are released, to zero in on the top performers, to ask what it is that the leading nations are doing, and then to urge the rest of the world to do those things. That response is understandable—but it is also potentially misleading. Causality is difficult to determine from cross-sectional data. Curriculum undoubtedly plays a role, but much more work needs to be done identifying potential curriculum effects in international data and testing well-formulated hypotheses with longitudinal models. Ideally, randomized trials would be conducted on the best curriculum programs, to tease out unobserved influences on learning. Those influences include a culture that places great value on academic success, parenting practices that promote achievement, and peers who award status based on working hard at school. They surely play a part in why some nations are “A+” while others only aspire to be.

« Introduction

Part II: The Resurgence of Ability and Persistence of Tracking »

Part I Notes:

1. In this section, the following rule was applied to ease the reading of the text. Subnational units, such as Hong Kong, may be referred to as “nations” or “countries.”

2. “2011 NAEP-TIMSS Linking Study,” National Center for Education Statistics, http://nces.ed.gov/timss/naeplink.asp.

3. Linking NAEP and international tests has been attempted before. See Gary W. Phillips, International Benchmarking: State Education Performance Standards (Washington, DC: American Institutes for Research, October 2010) and “Global Report Card,” Jay P. Greene and Josh B. McGee, http://globalreportcard.org/.

4. See “The Finnish Miracle,” Hank Pellissier, http://www.greatschools.org/students/2453-finland-education.gs.

5. Jenny Anderson, “From Finland, an Intriguing School Reform Model,” New York Times, December 12, 2011, http://www.nytimes.com/2011/12/13/education/from-finland-an-intriguing-school-reform-model.html?pagewanted=all&_r=0.

6. Pasi Sahlberg, Finnish Lessons: What Can the World Learn from Educational Change in Finland? (New York: Teachers College Press, 2011).

7. The Flemish, French, and German speaking communities operate separate school systems.

8. William H. Schmidt and Richard T. Houang, “Lack of Focus in the Mathematics Curriculum: Symptom or Cause?” in Lessons Learned: What International Assessments Tell Us about Math Achievement, ed. Tom Loveless (Washington: Brookings Institution Press, 2007).

9. William H. Schmidt and Richard T. Houang, “Curricular Coherence and the Common Core State Standards for Mathematics,” Educational Researcher 41, no. 8 (2012): 294-308.

10. Regression to the mean is possible, but the variance of the A+ countries’ gain scores suggests it’s unlikely.

Downloads

2013 Brown Center Report on American Education

Video

The 2013 Brown Center Report on American Education: How Well Are American Students Learning?

Authors

Tom Loveless

The Resurgence of Ability Grouping and Persistence of Tracking

Tom Loveless — Mon, 18 Mar 2013 00:00:00 -0400

This study examines the use of ability grouping and tracking in America’s schools. Recent NAEP data reveal a resurgence of ability grouping in fourth grade and the persistent popularity of tracking in eighth-grade mathematics. These trends are surprising considering the vehement opposition of powerful organizations to both practices. Although the current study will not delve into the debate—it is interested in what schools are doing, not why or whether they should do it—discussion is offered at the end of the article on implications of the findings for the controversy surrounding the topic.

Ability grouping and tracking are often confused. They both attempt to match students with curriculum based on students’ ability or prior performance, but the two practices differ in several respects. Tracking takes place between classes, ability grouping within classes. Tracking primarily occurs in high school and sometimes in middle school. In tracked academic subjects, students are assigned to different classrooms, receive instruction from different teachers, and study a different curriculum. The names of high school courses signal curricular differences. Advanced math students in tenth grade, for example, may take Algebra II while others take Geometry, Algebra I, or Pre-Algebra. Advanced tenth graders in English language arts (ELA) may attend a class called “Honors English” while other students attend “English 10” or “Reading 10.” Excellent science students may take “AP Chemistry” while others take a course simply called “Chemistry” or “General Science.” History may also be tracked, as when Advanced Placement courses are offered in U.S. or European history that not all students take. Some middle and high schools do not track at all, creating instead classes that are heterogeneous in ability. Students of all abilities study the same material.

What Tracking is Not

Perhaps the best way to clarify what tracking is, because of widespread misconceptions, is by describing what it is not. Tracking is decided subject by subject. Students are not assigned to college preparatory or vocational tracks that then dictate coursework all through high school; that practice died out in the U.S. in the late 1960s and early 1970s.¹¹,¹² European and Asian school systems still practice a form of this type of tracking (they call it “streaming”), typically in the final two or three years of secondary schooling.¹³ Students take placement exams and based on the scores are selected into separate schools with markedly different post-secondary destinations rather than attending different classes at the same school.¹⁴ Exam-based selection into high schools was common in the U.S. in the 19th century and the early part of the 20th century, but fell to the wayside. The comprehensive high school—with all students of a particular community attending the same school and then divided into distinct tracks within the school—came to be enshrined as the American model.

Ability Grouping

Ability grouping typically is an elementary school practice. Most elementary classes feature a single teacher with a classroom of students who are heterogeneous in ability. To create more homogeneity, teachers may divide students into small instructional groups reflecting different levels of ability, most often for reading in the primary grades (K–3) and perhaps for reading or math in later grades (4–6).¹⁵ While the teacher provides instruction to one group, the other students work independently—engaged in cooperative group activities or computer instruction or completing worksheets to reinforce skills. The teacher rotates among the groups so that each student receives a dose of teacher-led instruction in these small settings.

Researchers from Johns Hopkins conducted a comprehensive survey of ability grouping and tracking in 1986. The study analyzed national data augmented by an in depth survey of Pennsylvania schools. Several interesting patterns were uncovered that still hold true today. Disaggregating the data by grade level revealed that ability grouping is most prominent in first grade and then slowly recedes over subsequent grades. Ability grouping and tracking are inversely related; the school system’s strategies for creating groups that are as homogeneous as possible shift over the K-12 grade span. Tracking is rare in the elementary grades and, after increasing dramatically in middle school (in mathematics, in particular) peaks towards the end of high school. It is rare for students, once grouped between classes by tracking, to be grouped again within classes by ability grouping.¹⁶

Because the groupings are within-class (and often decided by a single teacher), ability grouping is more flexible than tracking. Groups may be reshuffled periodically to reflect changes in student performance. Ability groups might study from different levels of the same textbook series or use the same book and move at a different pace (with enrichment activities for the faster groups until the others catch up). Instead of the formality of transcript designations for high school courses, ability groups often take the names of animals—redbirds, bluebirds, sharks, dolphins, and the like—or the names of the books in the reading series that the students are using.

The most popular alternatives to ability-grouped instruction are whole class instruction, in which all students in the same classroom receive the same instruction, and the creation of small heterogeneous groups. Sometimes cooperative learning strategies are employed with heterogeneous groups, but cooperative learning can be used with any small group regardless of the criterion by which it is formed. Success for All, for example, is a popular program combining cooperative learning with small ability groups that are frequently reorganized to reflect student progress.¹⁷

Controversy

In the 1970s and 1980s, a barrage of studies criticized tracking and ability grouping. Race and class figured prominently in the debate. Grouping students by ability, no matter how it is done, will inevitably separate students by characteristics that are correlated statistically with measures of ability, including race, ethnicity, native language, and class. Critics argued that tracking and ability grouping do not separate students into socioeconomic status-related groups by accident. Ray C. Rist’s “Self-Fullfilling Prophecy in Ghetto Education” (1970) followed a group of kindergarten students through the first few years of school and noted how the composition of reading groups rarely changed, consistently reflecting students’ socioeconomic status (SES).¹⁸ The SES differences are hardened, Rist argued, as teachers develop different expectations for groups of low and high performing students, even if those groups are given innocuous sounding names to mask their status.¹⁹ James Rosenbaum’s Making Inequality (1976) described working class youth at a New England high school who were channeled into vocational and remedial tracks that were nothing more than boring, academic dead ends.²⁰

In 1985, Jeanie Oakes’ classic book, Keeping Track, was published. Oakes drew on data from several junior and senior high schools. Building on the social reproductionist theories of Samuel Bowles and Herbert Gintis’s Schooling in Capitalist America, Oakes argued that although tracking is typically justified by educators as a strategic response to student heterogeneity, the practice is undergirded by normative beliefs regarding race and class—and politically defended by white, middle-class parents to protect privilege. Black, Hispanic and poor children populate remedial classes; middle-class white children populate honors courses. Tracking and ability grouping are not mere bystanders to social injustice, Oakes and other critics charged. Such practices don’t just mirror the inequalities of the broader society. They reproduce and perpetuate inequality.²¹

This critique had a profound effect on policy and practice. In the 1990s, several prominent political organizations passed resolutions condemning tracking, including the National Governors Association, the American Civil Liberties Union, the Children’s Defense Fund, and the NAACP Legal Defense Fund. Some states urged schools to reduce tracking and ability grouping, most notably California and Massachusetts. A surprising implementation story ensued. Although the call to detrack was not accompanied by conventional incentives—the big budgets, regulatory regimes, and rewards and sanctions that draw the attention of policy analysts—detracking was, in a field famous for ignored or subverted policies, adopted by a large number of schools.²²

Surveys of Ability Grouping

How much did ability grouping decline? A 1961 national survey revealed that about 80% of elementary schools grouped students by ability for reading instruction.²³ A three-group format was the dominant approach, with students organized into high, middle, and low performing groups. Although subsequent national surveys of ability grouping are scarce until the John Hopkins study in the mid-1980s (mentioned above), carefully crafted studies of local practice reported similar frequencies. Eighty percent or more of elementary schools used within-class
ability groups.²⁴

Then things changed. A mid-1990’s survey of a random sample of pre-K through fifth grade teachers reported startlingly different results. When allowed multiple responses, only 27% of teachers reported using ability grouping for reading instruction. Another 56% of teachers indicated that they used flexible grouping. Some of the teachers with flexible grouping may have utilized ability as a criterion for grouping.²⁵ Whole class instruction was by far the most popular organizing strategy, with 68% of teachers reporting its use. Removing the overlapping responses makes it clear that ability grouping served a subordinate role as a method of organizing students. When teachers were held to one response and asked to identify their primary organizational approach, the order was: whole-class instruction (52%), flexible grouping (25%), and ability grouping (16%).

A more recent survey suggests ability grouping has regained favor among teachers. Barbara Fink Chorzempa and Steve Graham (2006) surveyed a national random sample of first through third grade teachers. Their questionnaire asked questions similar to the Baumann et al. survey of the 1990s, but also included questions about why teachers ability group. Three times as many teachers (63%) said they use ability grouping as the earlier survey. The authors explain that the discrepant findings may stem from the different grade levels of teachers in the two surveys. Pre-K and fourth- and fifth-grade teachers, who are included in the earlier
survey but not in the latter, may be less likely to employ ability grouping than first through third-grade teachers, the target population of the latter survey. Interestingly, the top reason teachers gave for using ability grouping was “that it helps them meet students’ needs;” however, respondents also expressed concern about the quality of instruction in low ability groups.²⁶ About 20% of teachers did not ability group at all because the practice was banned by district or school policy.

Is ability grouping in decline or on the rise again? How about tracking? Let’s turn to NAEP data to shed light on these questions.

NAEP Data on Ability Grouping

Table 2-1 displays NAEP data on ability grouping in fourth grade reading. Teachers were asked on what basis they create instructional groups (ability, interest, diversity, and other) with “not created” also an option. Bear in mind that asking fourth-grade teachers about ability grouping, as compared to sampling teachers of several elementary grades, has both an upside and a downside in elucidating trends. The upside is that grade level is held constant over several surveys. This is important because we know ability grouping varies by grade level. The downside is that fourth grade isn’t where the action is on ability grouping—that’s first grade, where unfortunately NAEP does not collect data. Fourth grade is well after ability grouping’s apogee and somewhere near the midpoint of its diminishing use by elementary teachers.

Table 2-1 is revealing. The percentage of students placed into ability groups for reading instruction skyrocketed from 1998 to 2009, from 28% to 71%. And the percentage of students whose teachers did not create ability groups fell from 39% in 1998 to 8% in 2009. In other words, the odds of a fourth grader being ability grouped in reading were less than 50-50 in 1998 but by 2009 had increased to about 9 to 1. The question was not asked prior to 1998.

Table 2-2 shows the frequency of ability grouping in fourth-grade mathematics. Teachers were asked if they create math groups based on ability. This question was asked twice before 1998 and in 2011, so it gives a deeper historical perspective than the question on reading. Math ability grouping dips from 1992 to 1996 (48% to 40%), stays about the same until 2003 (42%), and then accelerates from 2003 to 2011 (reaching 61% in 2011).

The NAEP data support the general finding of a drop in ability grouping in the 1990s and a resurgence in the 2000s. The rebound is more subdued in math than in reading. It is apparent by 2000 in reading (it may have begun even before then; the data start in 1998) but does not begin in math until after 2003. In the years for which data are available for both reading and math (2000, 2003, 2007, 2009), the two subjects have comparable frequencies in 2000 (39% in reading and 41% in math), but reading is more often grouped in subsequent years. In the last year with data on both subjects, 2009, 71% of fourth grade students were ability grouped for reading and 54% for math.

NAEP Data on Tracking

Table 2-3 displays NAEP data on tracking in 8th grade. Note that unlike ability grouping, which is a classroom level practice and consequently a topic for teacher surveys, tracking is a school level practice and a topic for surveys of school principals. Although the wording of the survey item varies slightly from year to year, NAEP asks principals whether students are assigned to classes based on ability so as to create some classes that are higher in average ability or achievement than others. The question is asked sporadically and about different subjects in different years.

Math has the most data, surveyed ten times from 1990–2011. Tracking in math shows a slight dip in the 1990s and an increase in the 2000s, but most of the fluctuations are too small to consider significant. The trend is essentially flat, with about three-fourths of students attending tracked math classes over the past two decades. Typically, this means schools offer an algebra class for some eighth graders and a pre-algebra class for those who are not yet ready for formal algebra (see table 3-2 for enrollment statistics). Sometimes a third class is offered, perhaps geometry for students who took algebra in seventh grade or a basic math class for students several years behind.

Data on the other subjects are spotty. They exhibit much less tracking than math and greater variation over time. In 1990, principals reported that 60% of students were in tracked ELA classes, a statistic that declined over the next several years, hitting a low of 32% in 1998. The 43% frequency of tracking reported in 2003 is an increase from 1998; however, because it was the last time the question was asked in that subject, it is impossible to tell whether an enduring rebound in ELA tracking had begun. Science and history have even less data, with both subjects registering their highest figures in 1990 and then indicating diminished tracking after that. Science seems to show a rebound from 1994–2000. For all four subjects, the least amount of tracking occurred between 1994 and 1998, when the detracking movement was in full bloom.

The national pattern is consistent with previous studies of California and Massachusetts. In those two states, detracking was most intense in the early to mid-1990s, but differences among the subjects emerged. Mathematics resisted detracking while heterogeneously grouped classes became the norm in ELA, science, and history. In a 2009 survey of Massachusetts schools with eighth grades, for example, in math only 15.6% of schools offered heterogeneously-grouped classes; 49.2% offered classes with two ability levels; and 35.2% offered three levels. In other subjects, tracking had almost disappeared—72.7% offered only heterogeneously-grouped classes in ELA, 89.8% in history, and 86.7% in science.²⁷

Discussion

This study has explored trends in the use of ability grouping and tracking by American schools. It used NAEP data to examine the frequency that fourth graders are assigned to groups and eighth graders assigned to classes based on ability or prior achievement. The investigation focused on what schools are doing, not on whether tracking or ability grouping is a good idea.

NAEP data from 1990 to 2011 were examined. Ability grouping in fourth grade decreased in the 1990s and then increased markedly in the 2000’s, with the rebound apparent in both reading and math. In reading, ability grouping has attained a popularity unseen since the 1980s, used with over 70% of students. As for tracking, it has remained commonplace in eighth-grade mathematics for the past two decades, with about three-quarters of students enrolled in distinct ability-level math classes. Tracking in ELA declined sharply from 1990 to 1998, and although there was a rebound in 2003, NAEP has not surveyed schools on tracking in ELA since then. And NAEP data are too sparse in other subjects to determine trends.

Do these trends matter? Why should anyone care about tracking and ability grouping? Although the debate today is more subdued than in the 1980s and 1990s, it does continue. A research review on the NEA website blasts both tracking and ability grouping as discriminatory.²⁸ Scholars continue to wrangle over the wisdom of both practices. Effectiveness and equity persist as the dominant themes of this literature. A 2010 meta-analysis of high quality studies calculated a positive effect size of 0.22, equal to about one-half year of learning, for within-class grouping in reading instruction.²⁹ A 2010 study of data from the Early Childhood Longitudinal Study (ECLS), on the other hand, found “students who are lower grouped for reading instruction learn substantially less, and higher-grouped students learn slightly more over the first few years of school, compared to students who are in classrooms that do not practice grouping.”³⁰ That finding is especially relevant to closing achievement gaps between students who may populate high and low groups.

The controversy offers a very important lesson about how education policy gets implemented in schools. Schools are not merely the last step of a vast organizational ladder, not simply the education system’s operational frontline, ready to put in place the policies that are passed down from above. Finley Peter Dunne famously observed that the U.S. Supreme Court “follows the election returns.” Court decisions not only reflect the U.S. Constitution but public opinion as well. Our schools are another institution with an ear to the ground. Educators are aware of public debates and are influenced when particular school practices become controversial.

Figure 2-1 shows the number of times the term “ability grouping” appeared in Education Week from 1983 to December 2012. Consider this a proxy for media visibility over the past thirty years. The 135 appearances over these three decades represent an average of 4.5 mentions per year. The peak coverage occurred in 1993, with 20 mentions. The years immediately preceding 1993 show a gradual build up in coverage, with 5 mentions in 1989, 13 in 1990, 11 in 1991, and 13 in 1992. The years immediately after 1993 show a gradual decline—8 appearances in 1994, 5 in 1995, 7 in 1996, 5 in 1997, and 7 in 1998. The ten years from 1989–1998 are the only years with more than 5 annual mentions. Tracking and ability grouping were in the spotlight.

The data on media visibility are inversely related to the data on use. At the beginning of the 1990s, tracking and ability grouping were conventional practices but then declined —albeit with some lag time—when they were subjected to the most public scrutiny. The mentions in Education Week peaked in 1993. The use of ability grouping and tracking reached all time lows soon after that event. As the controversy died down in the 2000s, schools returned to both practices.

What else may have promoted the resurgence in the 2000s? Accountability systems, bolstered by the accountability provisions of No Child Left Behind, focus educators’ attention on students below the threshold for “proficiency” on state tests. That provides a statutory justification for grouping students who are struggling. The increased use of computer instruction in elementary classrooms cannot help but make teachers more comfortable with students in the same classroom studying different materials and progressing at different rates through curriculum. The term “differential instruction,” while ambiguous in practice, might make grouping students by prior achievement or skill level an acceptable strategy for educators who recoil from the term “ability grouping.”

A substantial number of teachers believe that heterogeneous classes are difficult to teach. The 2008 MetLife Survey of the American Teacher asked teachers to react to the following statement: “My class/classes in my school have become so mixed in terms of students’ learning ability that I/teachers can’t teach them.” Responses were: 14% “agree strongly,” 29% “agree somewhat,” 28% “disagree somewhat,” and 27% “disagree strongly.”³¹ The percentages are surprising given the questionnaire’s blunt assertion that heterogeneous classes are impossible to teach. Moreover, the 43 percent of respondents that either agree strongly or somewhat agree with the prompt is up from 39 percent on the same survey item in 1988. Teachers’ beliefs about the impact of achievement heterogeneity on instruction undergird the use of ability grouping and tracking.

Let’s look ahead. Will the uptrend in ability grouping continue? Not necessarily. The current period may be the lull before the storm. Theoretically, at least, the Common Core establishes a curriculum that most, if not all, students will study. It is unclear how students who have already mastered the Common Core standards before beginning a particular school grade will have their needs met under the new regime. The same goes for students who lag many years behind. Tracking and ability grouping have been common approaches to addressing such challenges. These two organizational strategies affect millions of students daily. Both practices shape aspects of schooling that we know to be important—the curriculum students study, the textbooks they learn from, the teachers who teach them, the peers with whom they interact. Despite decades of vehement criticism and mountains of documents urging schools to abandon their use, tracking and ability grouping persist—and for the past decade or so, have thrived.

« Part I: The Latest TIMSS and PIRLS Scores

Part III: Advanced Math in Eighth Grade »

Part II Notes

11. Tom Loveless, The Tracking and Ability Grouping Debate (Washington, DC: Thomas B. Fordham Institute, July 1, 1998).

12. Samuel R. Lucas, Tracking Inequality: Stratification and Mobility in American High Schools (New York: Teachers College Press, 1999).

13. Even Finland and Sweden, famous for egalitarian reforms, divide students for the final two years of secondary school. Germany begins tracking at age 11.

14. Alan Smithers and Pamela Robinson, Choice and Selection in School Admissions: The Experience of Other Countries, accessed March 4, 2013, http://suttontrust.com/research/choice-and-selection-in-admissions/1smithers-final-report.pdf.

15. Robert Dreeben and Rebecca Barr, “The Formation and Instruction of Ability Groups,” American Journal of Education 97, no. 1 (1988): 34-64.

16. See p. 36, Figure 5: James M. McPartland, J. Robert Coldiron, and Jomills H. Braddock II, School Structures and Classroom Practices in Elementary, Middle, and Secondary Schools, Report No. 14 (Baltimore: The Johns Hopkins University, 1987).

17. “Success for All—Home”, Success for All Foundation, http://www.successforall.org/.

18. Ray C. Rist, “Student Social Class and Teacher Expectations: The Self-fulfilling Prophecy in Ghetto Education,” Harvard Educational Review 40, no. 3 (1970): 411-451.

19. Ability grouping is called “setting” in Great Britain. Recent reports have been sharply critical of the practice, see: “Setting Harms Education of Some Young Children, Report Warns,” The Independent, May 16, 2008, http://www.independent.co.uk/news/education/education-news/setting-harms-education-of-some-young-children-report-warns-829312.html.

20. James E. Rosenbaum, Making Inequality; the Hidden Curriculum of High School Tracking (New York: John Wiley & Sons, 1976).

21. See: Jeannie Oakes, Keeping Track: How Schools Structure Inequality (New Haven: Yale University Press, 1985). Also see: Jeannie Oakes, Amy Stuart Well, and Associates, Beyond the Technicalities of School Reform: Policy Lessons from Detracking School (Los Angeles: UCLA Graduate School of Education & Information Studies, 1996).

22. The politics and policies of tracking reform are investigated in: Tom Loveless, The Tracking Wars: State Reform Meets School Policy (Washington: Brookings Institution Press, 1999).

23. Mary C. Austin and Coleman Morrison. The Torch Lighters: Tomorrow’s Teachers of Reading (Cambridge: Harvard University Graduate School of Education, 1961).

24. Rebecca Barr and Robert Dreeben, How Schools Work (Chicago, University of Chicago Press, 1983).

25. ECLS asked kindergarten teachers in 1999 the frequency with which they used ability groups in reading. Five response categories, ranging from 0 (never) to 4 (daily). 30% reported never using ability grouping. The average for all teachers was 1.64, indicating about once a week (1 = less than once a week; 2 = once or twice weekly). When the ECLS sample was in 3rd grade, 2001–2002, 50% of teachers employed ability grouping in reading, consistent with the NAEP figure for 4th grade in 2003 (47%). See p. 301, note 6 in Christy Lleras, and Claudia Rangel, “Ability grouping practices in elementary school and African American/Hispanic achievement.” American Journal of Education 115, no. 2 (2009): 279–304.

26. Barbara Fink Chorzempa and Steve Graham, “Primary-Grade Teachers’ Use of Within-Class Ability Grouping in Reading,” Journal of Educational Psychology 98, no. 3 (2006): 529-541.

27. Tom Loveless, Tracking, Detracking: High Achievers in Massachusetts Middle School (Washington, DC: Thomas B. Fordham Institute, 2009).

28. “Research Spotlight on Academic Ability Grouping,” NEA, http://www.nea.org/tools/16899.htm.

29. Kelly Puzio and Glenn Colby, The Effects of Within Class Grouping on Reading Achievement: A Meta-Analytic Synthesis (Evanston: Society for Research on Educational Effectiveness, 2010).

30. Christy Lleras and Claudia Rangel, “Ability Grouping Practices in Elementary School and African American /Hispanic Achievement,” American Journal of Education 115, no. 2 (2009): 279.

31. Dana Markow and Michelle Cooper, The Metlife Survey of the American Teacher: Past, Present and Future (New York: Metlife, 2008).

Downloads

2013 Brown Center Report on American Education

Video

The Resurgence of Ability Grouping and Tracking: A Return to Controversial Practices?

Authors

Tom Loveless

When Does a Policy Start?

Tom Loveless — Wed, 13 Feb 2013 11:30:00 -0500

The question in the title should be easy to answer. Everyone knows that policies are debated and adopted by a legislative body (Congress, state legislature, or city council), signed by an executive leader (president, governor, or mayor), and then implemented by an administrative unit (federal, state, or city department). Once implemented, a policy starts.

The purpose of this Chalkboard post is to convince you that the easy answer can be misleading. A second purpose is to persuade you that the establishment of a starting point can affect the judgment of whether a policy has been good or bad--or has had no effect at all.

Let’s begin by considering a famous example, No Child Left Behind (NCLB). The basic timeline is this: NCLB was proposed by President Bush in January 2001, debated in Congress that year, passed in December 2001, and signed into law in January 2002. Most state accountability plans were not approved by the U.S. Department of Education until 2003.

Take a look at NAEP (National Assessment of Educational Progress) data for fourth grade reading. (Please play along by setting aside the daunting problems with drawing causal inferences from NAEP).

Year	Score
2011	221
2003	218
2002	219
2000	213
2000^	217
1992^	217

^{^ No accommodations permitted.}

Notice that scores jumped from 2000 to 2002, an increase of 6 points in only 2 years. Even larger gains were registered from 2000 to 2003 on NAEP’s 4^th grade math test (9 points) and similar gains on the 8^th grade math test (5 points).

Not surprisingly, critics and supporters of NCLB have wrestled over the 2000-2003 time interval like troops battling over a strategically important hilltop. Critics start the clock on NCLB in 2003, a year after the law was signed. With 2003 established as a post-intervention baseline, one can see that the gains immediately before NCLB were greater than the gains after adoption. Progress slowed after NCLB started.

Not so fast, say supporters of NCLB. It’s also clear from the NAEP scores that schools made little headway in the 1990s. How about using an earlier baseline—before 2003? Holding schools accountable for student performance, the key component of NCLB, started in the states before 2003. In 1999, Education Week’s annual publication, Quality Counts, was on the theme, “Rewarding Results, Punishing Failure.” It reported that 48 states were testing students, 36 issued annual report cards on schools, and 19 rated school performance and identified poorly performing schools. In 2000, both George W. Bush and Al Gore campaigned on promises of tough school accountability. Bush’s plan was to elevate the Texas school accountability program to all states. That became NCLB. People forget that Gore’s plan gave poorly performing schools only two years to improve; if they didn’t, they would be shut down, converted to charter schools, or reconstituted with a new principal and teaching staff.

Accountability was a hot topic in 2000 and figured prominently in state and local reforms. In 2001, as Congress debated NCLB, more states passed test-based accountability programs and by the time NCLB was signed into law in 2002, at least half of the states had some kind of test-based accountability in place. In the summer of 2002, states identified 8,652 schools in need of improvement under NCLB’s provisions. None of this was secret.

A standard rule of interrupted time series analysis (the formal name for before-and-after comparisons) is that the time of the policy intervention should be precise. To use the 2003 NAEP as a baseline for post-intervention assumes that schools were unaffected by the 1999-2002 activities described above when the NAEP was administered in early 2003. That’s unlikely.

Serious policy analysts are aware of the problem and in their evaluations of NCLB have addressed it in a variety of ways. Thomas Dee and Brian Jacob compare states that already had NCLB-style accountability systems in 2002 with states that were adopting accountability for the first time under NCLB. They define the treatment as the number of years states did not have accountability systems between 1992 and 2002, effectively putting each state on its own clock (see here for the study and also here for an alternative approach less favorable to NCLB). Dee and Jacob find a significantly positive effect for 4^th grade math (effect size of 0.23), a small positive effect for 8^th grade math, and no effect for either 4^th or 8^th grade reading. The math finding holds up using several different NCLB starting points.

Let’s not linger over statistical techniques or get side-tracked by NCLB. The point is that a counter-intuitive idea—that policy effects can occur before a policy is adopted—must be taken seriously.

Consider anticipatory effects, changes in behavior in anticipation of a policy. If you’ve been following the national news lately, two vivid examples should come to mind. Sales of firearms have surged in the wake of several horrifying mass shootings. Gun owners fear that stricter gun control laws are on the horizon so they are buying more firearms. In December, 2012, corporations paid out a whopping $30.9 billion in special dividends, shifting payouts into 2012 to avoid the scheduled expiration of the Bush tax cuts and the potential for sharply higher rates in 2013. Corporations would rather reward shareholders earlier than risk more highly taxed dividends later.

Note that these are observable changes in behavior before policies have been passed, let alone implemented. Indeed, the policies inducing these behaviors may never be passed. This makes careful estimates of policy effects difficult. If later this year Congress passes stricter gun control measures and gun sales drop, will it be because of the passage of the law? Or because gun buying had been shifted ahead in time to avoid new restrictions?

Federalism is another factor to take into account. In education, we have a multi-leveled system of federal, state, district, and school policies. When the federal government adopts an education policy you can almost be sure that it is already being done somewhere in some fashion. As noted above, the core of NCLB—accountability—was already in place in many states when NCLB was signed into law.

The impact of federalism is well known in other policy domains, and policy analysts in those fields incorporate state-level sources of variation into their work. For example, the National Minimum Drinking Age Act of 1984 prohibited the purchase or public possession of alcoholic beverages by anyone under the age of 21. Federal highway funds were used to persuade states to go along with the wishes of Congress on what historically had been strictly a state concern. Many states already had a 21-year age limit (affecting the pre-policy period). For those that allowed purchases at an earlier age, the change was more difficult, and they dragged their feet implementing the law (affecting the post-policy period).

A similar phenomenon occurred with 1974 federal legislation establishing a national speed limit of 55 mph. Some states already had a 55 mph limit--a few even had 50 mph—but most had to lower their speed limits to gain compliance. Western rural states reluctantly adopted the limit, and in an obvious ploy to subvert the law’s intent, also adopted $5 fines for speeding. When did the national speed limit law truly start? It depends on the state. We are more certain of when it ended—in 1995 when Congress repealed the law and gave authority over speed limits back to the states.

So now consider what all of this means for the Common Core. Currently, forty-five states and the District of Columbia have adopted the Common Core and school districts have begun implementing them. Kentucky recently gave a test which state officials claim is faithful to the Common Core standards. Textbook publishers declare that their latest series are Common Core compatible. Curriculum guides have been written and teachers all over the country are attending professional development sessions. This is all in the name of the Common Core.

It sure seems like the Common Core has started. But important pieces are missing. The federally funded state consortia who are writing Common Core assessments have not completed their task. Cut points for student proficiency haven’t been set. No tests have been administered (other than Kentucky’s). No consequences based on performance will be implemented until 2014-2015 at the earliest.

Sometime down the road, someone will declare the Common Core a success or failure, and that judgment will probably be based, as NCLB evaluations have been based, on a before and after analysis using national data. Some will say the Common Core started in 2010, when the states adopted the standards. Others will say 2011 or 2012, when state and local curriculum changes began. And still others will declare 2015 as the Common Core’s start date, after the first assessments have been conducted.

What is the best answer? Not only is the best answer unclear—despite the fact that we are nearer to the starting point today than we will be in another decade—but it’s also highly probable that future advocates, both for and against the Common Core, will exploit that lack of clarity to focus on the time period most pleasing to them. Analysts should be thinking about Common Core’s possible starting points right now.

Authors

Tom Loveless

Image Source: © Jim Young / Reuters

International Tests Are Not All the Same

Tom Loveless — Wed, 09 Jan 2013 11:25:00 -0500

Last month, the latest results were released from the 2011 Trends in International Mathematics and Science Study (TIMSS).

The scores dispelled the myth that all international tests are the same.

TIMSS and Program for International Student Assessment (PISA) are quite different. TIMSS is curriculum-based, reflecting the skills and knowledge taught in schools. PISA assesses whether students can apply what they’ve learned to solve “real world” problems. PISA tests an age-based sample (15 year olds). PIRLS and TIMSS are grade-based (4th and 8th graders). PISA is overseen by representatives from participating governments who meet under the auspices of the Organization for Economic Cooperation and Development (OECD). TIMSS and PIRLS are governed by a consortium of researchers and government representatives known as the International Association for the Evaluation of Educational Achievement (IEA).

It is true that scores from the two tests are highly correlated at the national level. But the two tests do not measure the same learning. Scores from NAEP’s math and reading tests are also highly correlated when aggregated to the state level. No one would argue that NAEP’s reading and math tests measure the same learning.

Discrepant results from PISA and TIMSS are revealing. When releasing the latest PISA results in 2010, Angel Gurría, Secretary-General of the OECD called New Zealand a “high flier.” New Zealand is definitely not a high flyer on the TIMSS math tests, scoring only 486 in 4th grade and 488 in 8th grade--significantly below the international means of 500.

The PISA math assessment is based on a philosophy known as Real Mathematics Education (RME), championed by the Freudenthal Institute in the Netherlands. Jan de Lange of the Freudenthal Institute chairs the PISA expert group in mathematics. RME’s constructivist, problem solving orientation is controversial among mathematicians. In the U.S. in the 1990s, a coalition of mathematicians, parents, and local educators opposed similar types of curricula in what became known as the “math wars.” In 2010, New Zealand implemented national standards that are compatible with RME and PISA—and less compatible with TIMSS.

The contrast is stark between constructivist countries and those favoring more traditional math curricula. On PISA, New Zealand scores within 27 points of Korea (519 vs. 546). On TIMSS, New Zealand and Korea are separated by a whopping 125 points (488 vs. 613), a difference of nearly one full standard deviation between the two tests! Chinese Taipei outscores Finland by only 2 points on PISA (543 vs. 541, the scores are statistically indistinguishable)—but by 95 points on TIMSS (609 vs. 514).

Finland is a great country and makes a wonderful travel destination. It also has fine schools. But its reputation in education is a bit overblown, based primarily on high PISA scores and an aggressive educational tourism industry. Like New Zealand, Finland also has a math curriculum compatible with PISA. In 2011, Finland participated in TIMSS for the first time since 1999. Finland’s 2011 math scores are statistically indistinguishable from the U.S. at both 4th and 8th grades. While U.S. scores have improved since 1999, Finland’s have declined (both changes are statistically significant). TIMSS is normally given to 8th graders, but to help monitor progress, Finland gave TIMSS to a random sample of 7th graders in both 1999 and 2011. Those scores fell from 520 to 482, a huge decline (see Exhibit 1.6 in the TIMSS International Report).

Here’s another way to compare Finland to the U.S. Nine of our states took part in the 2011 TIMSS 8th grade math assessment. Four states scored statistically significantly higher than Finland (MA, MN, NC, IN), three about the same (CO, CT, and FL), and two lower (CA and AL). Based on this distribution of scores, had Finland participated in the 2011 NAEP as a U.S. state, its 8th grade math score probably would have been in the middle of the pack.

The U.S. press has not told the entire story about Finland. Not everyone in Finland has applauded the emphasis on real world math. In 2005, a petition signed by more than 200 university mathematicians complained that, despite the country’s high PISA scores, students were increasingly showing up for college unprepared in mathematics. An analysis of items on a Finnish matriculation exam revealed a sharp fall off in computation skills, particularly with problems involving fractions and exponents. The concern in Finland about the math curriculum mirrors that expressed in the 1990s about math reform in the U.S.

Next December, when the latest PISA scores are released, New Zealand will undoubtedly be a high flyer again. So will Finland. Please remember that PISA is not a curriculum-based test, and whether we want schools to teach a mathematics curriculum compatible with PISA is a proposition that many people find objectionable.

Authors

Tom Loveless

Image Source: © Stephane Mahe / Reuters

The 2012 Brown Center Report on American Education

Tom Loveless — Thu, 16 Feb 2012 00:00:00 -0500

The 2012 Brown Center Report on American Education distills the results of studies to examine the state of education in the United States. In particular, the report focuses on education policy, student learning measures, trends on achievement test scores and education reform outcomes.

Highlights from three of the studies featured in the report are:

Predicting the Effect of the Common Core State Standards on Student Achievement: The Common Core will have little to no effect on student achievement. The quality or rigor of state standards has been unrelated to state NAEP scores, Loveless finds. Moreover, most of the variation in NAEP scores lies within states, not between them. Whatever impact standards alone can have on reducing within-state differences should have already been felt by the standards that all states have had since 2003.

Measuring Achievement Gaps on NAEP: The Main NAEP consistently reports larger SES achievement gaps than the Long Term Trend NAEP. The study examines gaps between students who qualify for free and reduced lunch and those who do not; black and white students; Hispanic and white students; and English language learners and students who are not English language learners.
Misinterpreting International Test Scores: Educators & policymakers often misinterpret International Test Scores in three ways: 1) Dubious Conclusions of Causality, 2) The Problem With Rankings, and 3) The A+ Country Fallacy. The errors are usually committed by advocates of a particular policy position who selectively use data to support an argument, argues Loveless.

Tom Loveless discusses the report in this video:

Download Media

The Brown Center Report on American Education

Downloads

Full Report

Video

The Brown Center Report on American Education

Authors

Tom Loveless

Image Source: Catherine Ledner

@ Brookings Podcast: Can an American Education Compete Globally?

Tom Loveless — Fri, 08 Apr 2011 14:41:00 -0400

Tom Loveless says U.S. students have never scored in the top ranks of educational achievement internationally, and the latest results, while improved, show U.S. scores in the middle range once again. He discusses data from the Programme for International Student Assessment (PISA) and the emphasis that American culture places on non-academic pursuits that consume so much time for American schoolchildren, including part-time work, socializing and sports.

previous play pause next

mute unmute

@ Brookings Podcast: Can an American Education Compete Globally? 06:28

Right-click (ctl+click for Mac) on 'Download' and select 'save link as..'

Get Code

Copy and paste the embed code above to your website or blog.

Video

Lackluster Scores for U.S. Students in International Tests

Audio

@ Brookings Podcast: Can an American Education Compete Globally?

Image Source: © Robert Galbraith / Reuters

The 2010 Brown Center Report on American Education

Tom Loveless — Mon, 07 Feb 2011 00:00:00 -0500

Introduction

This edition of the Brown Center Report marks the tenth issue of the series and the final issue of Volume II. The publication began in 2000 with Bill Clinton in the White House and the Bush-Gore presidential campaign building toward its dramatic conclusion. That first report was organized in a three-part structure that all subsequent Brown Center Reports followed. Part I presents the latest results from state, national, or international assessments and alerts readers to important trends in the data. Part II explores an education issue in depth, sometimes by investigating different sources of empirical evidence than previous research, sometimes by posing a conventional question in an unconventional way. Part III analyzes a current or impending question regarding education policy. In all three sections, the studies strive to ask clear questions, gather the best available evidence, and present findings in a nonpartisan, jargon-free manner.

Part I of this year’s Brown Center Report focuses on international assessments. The latest data from the Programme for International Student Assessment (PISA) were released in December 2010. The performance of the United States was mediocre, and although notching gains in all three subjects, the country scored near the international average in reading literacy and scientific literacy and below average in mathematical literacy. The term “literacy” is a signal that PISA covers different content than most achievement tests, and, indeed, assesses different skills than are emphasized in the school curriculum. As the 2006 PISA Framework states, the knowledge and skills tested on PISA “are defined not primarily in terms of a common denominator of national school curricula but in terms of what skills are deemed to be essential for future life.”

Two myths of international assessments are debunked—the first, that the United States once led the world on international tests of achievement. It never has. The second myth is that Finland leads the world in education, with China and India coming on fast. Finland has a superb school system, but, significantly, it scores at the very top only on PISA, not on other international assessments. Finland also has a national curriculum more in sync with a “literacy” thrust, making PISA a friendly judge in comparing Finnish students with students from other countries. And what about India and China? Neither country has ever participated in an international assessment. How they would fare is unknown.

Part II of the report looks at state test scores on the National Assessment of Educational Progress (NAEP) in light of the recent Race to the Top competition. The federal program encouraged states to apply for $4.35 billion in new money by promising to pursue a reform agenda backed by the Obama administration. Twelve states (for this discussion, the District of Columbia will be called a state) won the grants. But are the states that won the grants the same states that have accomplished the greatest gains in student learning? Not necessarily.

Who’s winning the real race to the top? Both short- and long-term gains on NAEP are calculated with statistical controls for changes in the demographic characteristics of each state’s students. Eight states—Florida, Maryland, Massachusetts, District of Columbia, Kentucky, New Jersey, Hawaii, and Pennsylvania—stand out for making superior gains. At the other end of the distribution, Iowa, Nebraska, West Virginia, and Michigan stand out for underperforming. Five of the eight impressive states won grants, but three did not. And a few states won grants even though they are faring poorly in the race to boost student achievement. Some of the reasons why a program called Race to the Top could distribute grant money in this manner are discussed.

Part III looks at NAEP. In June 2010, the Common Core State Standards Initiative released grade-by-grade standards for reading and mathematics. Two consortia were awarded $330 million to write tests aligned to the standards, and a total of 46 states have signed to at least one group. As the only assessment administered to representative samples of American students, NAEP has called itself “the Nation’s Report Card” for decades.

How well does NAEP match up with the Common Core? We examined 171 public release items from the eighth-grade NAEP math test and coded them based on the grade level the Common Core recommends that the content be taught. The items registered, on average, two to three years below the eighth-grade mathematics recommended by the Common Core. More than 90 percent of the items from the “number” strand (content area) cover material below the eighth grade. Almost 80 percent of the items assessing “algebra” are, in fact, addressing content in the curriculum that is taught before eighth grade. With Common Core assessments on tap to begin in the 2014–2015 school year, policymakers and analysts alike need to start thinking now about how NAEP and the Common Core assessments can be reconciled so as to inform, not to confuse, the public about student achievement.

An overarching theme of this year’s report is that events in the field of education are not always as they appear to be—and especially so with test scores. Whether commentators perpetrating myths of international testing, states winning races while evidencing only mediocre progress, or an eighthgrade test dominated by content below the eighth grade, the story is rarely as simple as it appears on first blush. This report tried to dig beneath the surface and uncover some of the complexities of these important issues.

Downloads

Download the Full Report

Authors

Tom Loveless

Image Source: R. Michael Stuckey

NAEP and the Common Core Standards

Tom Loveless — Tue, 11 Jan 2011 08:58:00 -0500

INTRODUCTION

The following is a special advanced release of a Brown Center report on NAEP and the Common Core standards. The full version of the 2010 Brown Center Report on American Education will be published at a later date.

Unlike most countries, the United States does not have national education standards, no single set of expectations for what all American teachers should teach and all American students should learn. It never has. A question that the rest of the world considers foundational to its national school systems—deciding the content of the curriculum—sits in the hands of local authorities. That is because the United States has 50 state school systems. Heterogeneity extends to the deepest levels of schooling. Even students transferring from one teacher to another within the same school may, as a consequence, learn a different curriculum than their former classmates.

So it was an historical event when the Common Core State Standards in mathematics and reading were released in June 2010. Launched by the National Governors Association and the Council of Chief State School Officers, the Common Core Standards project brought together experts in both reading and math to develop a set of standards that would be, in what became a mantra, both “higher and fewer in number” than existing state standards.[1] The standards are voluntary—states choose whether to participate—but for the first time most American students will study a uniform curriculum through at least the eighth grade. A draft of the experts’ work circulated for several months, and, based on input from other experts and the general public, the standards were finalized.[2] In September 2010, two consortia were awarded federal grants totaling $330 million to develop annual assessments aligned with the Common Core standards, and as of December 2010, 43 states and the District of Columbia have signed on to those efforts.[3] The tests are due to be given for the first time in the 2014–2015 school year.[4]

The nation currently monitors the math achievement of fourth, eighth, and twelfth graders on the National Assessment of Educational Progress (NAEP).[5] Since 1990, the main NAEP has assessed mathematics proficiency in five content strands—number properties and operations, algebra, geometry, measurement, and data analysis/statistics/probability.[6] How well does NAEP match up with the Common Core standards in mathematics?

We tackled this question by analyzing NAEP items from the eighth-grade assessment. NAEP items are periodically released to the public to give an idea of the content of the test. For the current study, we coded all public release items from the algebra and number strands[7] based on the grade at which the Common Core recommends teaching the mathematics assessed by the item. The 2009 NAEP Framework in Mathematics calls for number and algebra items to comprise half of the eighth-grade assessment.[8] A total of 171 items were available, 98 from the number strand and 73 from algebra.[9] We were unable to code four items (two from each strand) because they assess skills not found in the Common Core.

A precursor to this study can be found in the 2004 Brown Center Report.[10] In that study, we coded the grade level of public release items labeled as “problem solving,” one of NAEP’s process strands (different from the content strands). Only problems involving the application of arithmetic were analyzed. At what grade level are students taught the arithmetic required to answer NAEP problem-solving items? We discovered that the mean fourth-grade NAEP item registered at 3.2 and the mean eighth-grade item at 3.7, suggesting that the typical item could be answered using arithmetic taught by the end of third grade. Primarily, this finding stems from NAEP’s reliance on whole number arithmetic in word problems. We found that approximately 70 percent of the eighth-grade items focused on whole numbers. Problems with fractions, decimals, or percents—forms of rational numbers taught after third grade—are not common on NAEP.[11]

The 2004 study used the Singapore Math program as a rubric to code the grade level of items, assigning a value according to the grade and semester in which the arithmetic of the item was taught. By using the Common Core and evaluating the entire context of items, the current study’s rubric produces higher grade-level estimates for items. Problems involving only simple arithmetic are classified at a higher grade level if they are posed in the context of more sophisticated topics that are taught at a later grade (e.g., coordinate plane, equations with two variables). Selected NAEP items are shown [in the full report available for download on this page].

[1] “Common Core State Standards Development Work and Feedback Group Announced,” News Release (Washington: National Governors Association Center for Best Practices and the Council of Chief State School Officers, July 1, 2009).
[2] See the Common Core State Standards Initiative website on About the Standards, http://www.corestandards.org/about-the-standards .
[3] Catherine Gewertz, “Common-Standards Watch: South Dakota Makes 44,” Curriculum Matters, Education Week, November 29, 2010.
[4] “Beyond the Bubble Tests: The Next Generation of Assessments,” Prepared Remarks of U.S. Secretary of Education Arne Duncan to State Leaders at Achieve’s American Diploma Project (ADP) Leadership Team Meeting, Alexandria, VA, September 2, 2010.
[5] The long-term trend NAEP test assesses students at ages 9, 13, and 17.
[6] See the NCES website on the NAEP Mathematics Framework, http://nces.ed.gov/nationsreportcard/mathematics/whatmeasure.asp .
[7] The number strand refers to the number sense, properties, and operations strand for the 1990–2003 NAEP mathematics framework and the number properties and operations strand in the current mathematics framework.
[8] National Assessment Governing Board, U.S. Department of Education, Mathematics Frameworks for the 2009 National Assessment of Educational Progress (Washington: 2008).
[9] See the NAEP Questions Tool, http://nces.ed.gov/nationsreportcard/itmrlsx/landing.aspx .
[10] Tom Loveless, The 2004 Brown Center Report on American Education: How Well Are American Students Learning? (Washington: The Brookings Institution, 2004), pp. 5–17.
[11] Theresa Smith Neidorf and others, Comparing Mathematics Content in the National Assessment of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) 2003 Assessments (NCES 2006–029). (U.S. Department of Education, 2006).

Downloads

Download the Full Report

Authors

Tom Loveless

Charter Schools: A Report on Rethinking the Federal Role in Education

Thu, 16 Dec 2010 00:00:00 -0500

Executive Summary

Charter schools offer choice to parents who would otherwise be constrained to having their children attend a residentially assigned traditional public school. The number of charter schools has increased steadily in the last decade, reflecting their popularity with parents and the general public. They vary substantially in their missions, the students they serve, and their effectiveness. Research suggests that charter schools are particularly effective in raising the achievement of low-income and minority students in urban areas. Charter schools are underfunded in comparison to traditional public schools and have particular challenges in finding and paying for school facilities. Authorizers of charter schools decide whether charter schools can enter the market, expand, or close, and they provide ongoing performance oversight. The school districts with which charter schools compete for resources and students are the most frequent authorizers of charter schools. The authorizing function seems very important in determining the quality of charter schools, but very little is known about the relationship between variations in authorizing and school quality.

The federal government’s role in charter schools has expanded of late and is likely to be an important element in the reauthorization of the Elementary and Secondary Education Act. The current federal role is a haphazard collection of laws, rules, funding preferences, and rhetoric that lacks coherence at the policy or action level. In that context, the Brown Center on Education Policy at Brookings gathered a group of prominent policymakers, practitioners, and researchers to address what the federal government should do if its policy were to increase the number of effective charter schools in the nation.

The recommendations for federal action advanced by these experts include: a) collecting and using more and better data on the performance of charter schools for purposes of authorizing, research, and informed parental choice; b) requiring states to provide equitable funding for charter schools relative to traditional public schools—including support for facilities; c) supporting higher standards for authorizing; d) revising rules and definitions that unintentionally disadvantage charter schools; e) promoting the growth as well as quality control of virtual charter schools; and f) finally and most importantly articulating and following through on a coherent policy with respect to charter schools.

Introduction

Charter schools are public schools of choice (rather than residential assignment) that are operated autonomously, outside the direct control of local school districts. Since the first charter school was established in 1992, the charter movement has grown to include over 4,900 charter schools in 39 states educating 1.6 million children.[i] In some cities, the penetration of charter schools is pronounced. In the District of Columbia, for example, over a third of public school students are now in the charter sector.[ii] New Orleans has an even higher concentration, with more than sixty percent of students attending a charter school.[iii]

The growth has been insufficient to meet the demand for charter schools. Many charter schools are over-subscribed, and few charter schools close for lack of adequate school enrollment. When the general public is surveyed, twice as many respondents say they favor charters as say they are opposed.[iv]

Charters do not have a single pedagogical identity. The best known chains, such as the Seed School, Uncommon Schools, and Knowledge is Power Program (KIPP), create highly structured routines with uniforms, strict rules, and numerous drills. But charters take many other forms, including single sex schools, schools for the performing arts, schools for science and technology, bilingual schools, schools for the disabled, schools for drop-outs, and virtual schools where learning takes place online.

The type of student entering a charter school is different from the traditional public school student. Relative to statewide averages, charter schools tend to attract a disproportionate number of students eligible for free or reduced price lunch as well as minority students, especially African Americans. Initial test scores of students at charter schools are usually well below those of the average public school student in the state in which the charter school is located.[v]

The Effectiveness of Charter Schools

The variety of charter schools is consistent with the original mission to provide new options to families and to promote innovative ways to organize a school and deliver a curriculum. But that same variety makes it difficult to draw conclusions about the instructional effectiveness of charter schools as a sector. Research findings vary widely, depending on the schools studied and the research methodology employed.

Nearly all large-scale studies that have examined the effectiveness of charter schools across many states have relied on statistical controls to handle differences in student background between students attending charter schools vs. regular public schools. Several of these studies find that students attending charter schools do no better than students attending regular public schools.[vi]

Critics of these studies point out that no amount of statistical adjustment for observed differences, such as correcting for divergence in the proportion of minority or low-income students attending regular vs. charter schools or adjusting for students’ prior achievement scores, can handle unobserved differences between parents and children attending the two types of school. For example, parents who enroll their children in charter schools may have different expectations for their children’s academic success than other parents. Or the students themselves may be different, e.g., students who transfer out of a regular public school into a charter school may have had particular problems adjusting to school. These unobserved differences in students and families may affect academic outcomes independent of the type of school students are attending.

Most of the research community agrees that the preferred research strategy for estimating the impact of attending a charter school is a randomized control trial. In such trials, two groups of students are compared. Both groups contain students whose families attempted to enroll the child in a charter school where there were more students applying for a spot than there were seats. For those oversubscribed schools a lottery is used to determine who is offered admission. By chance alone some students win admission and others do not. A comparison of academic outcomes for students who won vs. those who lost the lottery keeps everything about the two groups of students the same, on average, except the offer of admission into a charter school. Thus any difference in the academic outcomes for the two groups of students can only be due to the one thing on which they differ systematically, gaining admission to a charter school.

There are presently five randomized trials that have addressed the performance of charter schools. Four found positive charter school impacts on student achievement[vii] whereas one found no overall effect.[viii] The four studies finding positive impacts each involved charter schools serving minority populations, three in large urban school districts (Chicago, New York City, and Boston, respectively) and one in a smaller, low income city north of Boston. The study that found no overall impact examined charters across multiple states and types of locale. Interestingly, the multi-state study that found no overall impact nevertheless identified subgroup effects, such that students from poor, minority, urban backgrounds did better in charter schools in contrast to students from middle-class, suburban backgrounds, who did worse. Thus all the randomized trials are consistent in pointing to the success of charter schools in large urban areas.

One limitation of these randomized trials is external validity (i.e., the ability to generalize the results to other settings). Because there are few non-urban, sufficiently oversubscribed charter schools, the randomized trials have taken place primarily in large, urban areas with a high percentage of minority students. The results of the randomized trials may not extend to the areas outside of the major urban areas and more research using other methods is needed on the effectiveness of charter schools for non-urban areas.

The results also do not necessarily generalize to students whose families have not tried to gain admission for their children into a charter school. To the extent that the success of some charter schools depends on motivated parents who buy into the school’s approach and make the extra effort that may be required to get their child to measure up to the school’s demands, such schools might not succeed with students whose parents do not have that motivation or are unwilling to make that commitment. To date, this possibility has not been addressed with credible research.

A final limitation of these studies is that they focus on student achievement on reading and mathematics as measured on standardized tests. It is possible to expand the range of student outcomes, including long-term outcomes. For example, a study of charter high schools in Chicago and Florida found positive effects on both high school completion and college attendance.[ix]

In summary, the overall body of research on the academic effectiveness of charter schools suggests considerable variability in impact. Thus knowing that a school is organized as a charter school does not, in and of itself, say much about whether the school is good, bad, or mediocre. Some charter schools are unambiguously providing a more effective education for students than is provided by regular public schools serving similar students. Other charter schools are no better than the public schools with which they compete, and some are worse.

When the focus is on academic achievement, the variability in the success of charter schools raises important issues for researchers, policymakers, practitioners, and parents. For researchers, the challenge is to identify the active ingredients that differentiate more successful charter schools from less successful ones, with the awareness that those ingredients may involve interactions between what is being offered by a school and the characteristics of the students and families being served. For practitioners, the issue is how to use research findings to increase the effectiveness of the best charters and to raise the level of performance of lagging charter and public schools. For parents, the principal issue is how to be an informed consumer when making a choice of schools. A school’s organizational status as a charter school or a regular public school conveys far less information than needed by a parent to make the wisest choice.

Charter School Finance

Charters generally operate on a tighter public budget for current expenditures than traditional district schools, receiving by one estimate only about 80 percent of the per pupil amount received by district schools,[x] and by another only about 60 percent.[xi] The difference in charter school per pupil spending also varies widely from state to state.

Charter schools are also disadvantaged relative to traditional district schools when it comes to facilities. The vast majority of district schools operate in buildings that are publicly owned and were purchased and amortized many years ago. This leaves the district responsible only for expenses for operation and maintenance. In contrast, charter schools typically require new investment in the construction or lease of facilities.

Some but not all states allow or require school districts to make unused public school facilities available to charter schools – in some cases for free but often for rent. Some but not all states provide support for low-cost loans or bonding authority for charter school facilities construction. A few states provide a per-pupil facilities payment to charter schools that is intended to equalize the advantage that district schools have through their legacy of existing school facilities. Some states provide no facilities assistance whatsoever to charter schools. In such cases, facilities must be built, purchased, or leased by charter school operators in current dollars, as well as operated and maintained.[xii]

Issues surrounding facilities are among the most vexing faced by charter school operators. When the additional costs of facilities for charter schools are added to lower levels of reimbursement per pupil compared to traditional schools, charter schools operate at a significant public funding disadvantage.

Authorizing Authority

The authority to establish and operate a charter school varies from locale to locale and state to state. The most recent survey identifies 819 charter school authorizing bodies nationwide.[xiii] School districts authorize more charter schools (55 percent) than any other type of authorizer. Some observers believe this role for school districts involves an inherent conflict of interest since charter schools compete for students and resources with the school district that must authorize them. Other authorizing bodies include state education agencies and independent chartering boards (splitting about 30 percent of charter schools). Higher education institutions, non-profit organizations, and mayors/municipalities are responsible for authorizing the remaining 15 percent of charter schools. In some states, each of these authorizing mechanisms is present, whereas in other states authority resides solely with one entity.

Authorizers have a number of roles, including handling applications for charter school expansion or startup, contracting with charter school operators, providing performance oversight, and making decisions on renewal or closure. Authorizers vary substantially in how they carry out these roles. For example, some authorizers provide direct assistance to schools in meeting performance goals whereas others provide only guidelines and warnings. Some authorizers engage in rigorous application and renewal processes whereas others do not. Some authorizers provide ongoing oversight and evaluation whereas others are engaged in evaluation only at the point of a charter school’s application for renewal.

Most practitioners and policymakers in the field believe that the nature, independence, and operational procedures of authorizing bodies are a significant factor in determining the quality of charter schools. Researchers are only beginning to investigate this relationship. A recent analysis of authorizer types in Minnesota found no statistically significant relationship between the type of authorizer and mean levels of student achievement, although achievement in charter schools authorized by school districts was about 0.15 standard deviations lower than achievement in charter schools authorized by the state.[xiv] This study used statistical controls for the differences in the types of students served by different authorizer types, so the same cautions in interpretation apply to it as we described for the multi-state observational studies of the effect of attending a charter school vs. a regular public school. Much more research addressing the casual impact of differences in the types and practices of charter school authorizers is needed.

The Federal Role

Prior to the American Recovery and Reinvestment Act (ARRA) of 2009, federal involvement in charter schools was minor, with approximately $200 million appropriated annually through the Elementary and Secondary Education Act to make charter school grants to state education agencies and charter management organizations. During the 2008-2009 school year, federal involvement was approximately $1.40 per student.[xv]Neither the amount of funding nor the conditions of competition afforded an opportunity at the federal level to have much impact on charter schools.

ARRA provided the Secretary of Education with a $650 million innovation fund from which awards were to be made to education entities that had made significant gains in closing achievement gaps. The purpose of the awards was to expand the work of the award winners and to identify and document best practices that could be shared and taken to scale based on demonstrated success. The KIPP Foundation, a large national charter operator, was one of the big winners under the innovation fund competition, receiving $50 million to scale-up its leadership training model.

ARRA also provided the Secretary with roughly $4 billion to carry out a competition among states (Race to the Top) to support reform and innovation. Under the rules established for the competition, states had to meet a number of requirements for their support of charter schools to have a chance of winning an award. These requirements include lifting caps on the number of charter schools; establishing authorizing practices that hold charter schools accountable for student achievement; ensuring equitable per student funding; and providing facilities assistance.

These actions by the U.S. Department of Education presage the stance towards charter schools that the administration is likely to take in the reauthorization of the Elementary and Secondary Education Act. We are clearly at the beginning of a new era in federal policy towards charter schools.

Recommendations for Federal Action
In order to inform future actions on charter schools by Congress and the administration, the Brown Center on Education Policy, operating with the advice of its Charter School Task Force, convened a day long advisory meeting of leading charter school researchers, practitioners, and policymakers to address the question of what the federal government should do or refrain from doing to support the growth of effective charter schools.

The purpose of the meeting was to develop and harvest a list of ideas and recommendations that might be useful for federal action. There was no effort or intent to develop consensus recommendations. Rather, each participant was asked to put forward one or more recommendations for comment and discussion by all the participants. The following is a categorized and annotated list of the recommendations that generated interest among the participants and appeared to be actionable at the federal level.

Data Collection and Use

One of the main themes of the advisory meeting was the challenge of obtaining and using charter school data to inform research, policy, practice, and parental choice. One of the forms of data that is difficult to obtain but would be particularly important for research, policy, and practice is lottery results at the level of individual students. To the extent that charters are oversubscribed and have to use lotteries to determine who is admitted, having those lottery results recorded in a state’s longitudinal education data system would allow many important questions to be addressed that are currently challenging. For example, with lottery results and records of student achievement in hand, charter school authorizers could avail themselves of a valid estimate of the impact of a charter school when carrying out their oversight and renewal obligations. Currently authorizers rarely have any data available on student achievement other than the average performance of students in a charter school on end-of-the-year state achievement tests. But as we described previously, charter schools differ among themselves and from traditional public schools in the population of students they serve. A charter school serving a suburban middle-class population very likely looks much better on end-of-the-year state assessments than a charter school serving a population of urban poor and minority families. But what appears to be better performance may reflect little more than the advantaged background of the students being served. The availability of lottery data would allow the effectiveness of each school to be evaluated relative to the other schools serving the same student population.

The availability and accessibility of lottery data in state longitudinal databases would also be a boon to research. Consider, for example, the questions about the effects of authorizers that we raised previously. Examining lottery-based student achievement outcomes by type of authorizer or type of authorizer practice could shed considerable light on which forms of authorizing have impacts on student achievement. Lottery data can also be useful for studying other differences in educational experience for the students selected for the charter schools compared to students not admitted.

Connected with the availability and accessibility of lottery data is the quality of the lottery itself. Allowing charter schools to design and carry out their own admission lotteries is a recipe for undermining random assignment, both through naïveté and self-interest. For example, researchers who have sought lottery data from charter schools have encountered schools that claimed they held a lottery but did not. Further, hidden within what seems to be a fair lottery can be a variety of special admissions decisions, for example the admission of children of staff, or children who for whatever reason were treated as exceptions by school officials.

Even in the case of fair lotteries, it may have been advantageous for a school to conduct a more sophisticated lottery than simply drawing names out of a hat. For example, a lottery might be stratified to assure geographic or demographic balance when the size of the overall pool of applicants and admissions slots is too small for the law of averages to create a high likelihood of such balance.

Flowing from these observations are the following recommendations for the federal government:

Fair and independent lotteries. Receipt of federal funds to support charter schools at the state and local level should be contingent upon charter schools being subject to lottery rules that require the design and implementation of lotteries by entities that are qualified to carry out the task, operate with clearly documented procedures, and are independent of the charter schools in which the lotteries are being conducted. One way to achieve this goal would be to require states seeking funding for charter schools to have established such rules as a precondition for application for funding.
Availability of lottery data. Student participation in lotteries for admissions to any public school and the results of such lotteries should be a required student data element in state or district longitudinal data systems supported with federal funds. Competition for future federal statewide longitudinal data system grants or use of Title I funding to support state administrative data systems could be contingent on this condition.
Use of lottery data for oversight. The use of lottery data by authorizers to carry out their oversight and renewal roles pertaining to the effectiveness of charter schools in raising student achievement should be encouraged. This goal could be achieved using the same contingent-funding mechanism described above, or could be pursued through guidelines and technical assistance in partnership with non-governmental organizations.
Use of lottery data for research. Since 2005, nearly every state in the nation and the District of Columbia has received a substantial federal grant through the Institute of Education Sciences within the U.S. Department of Education to develop a statewide longitudinal data system. A statutory requirement of these awards is that the resulting data systems be used to facilitate research to increase student achievement and close achievement gaps. Yet many states have made no provision for researcher access to their longitudinal data systems or have allowed access only to those with the persistence and skill to strike a deal with a responsible state official. One of the principal rationales for charter schools is for them to serve as engines of education innovation. It is difficult to identify and reap the rewards of innovation without a serious and sustained research presence. It is time for the federal government to insist that recipients of statewide longitudinal data system grants demonstrate that they have met their obligations to facilitate research.
Increasing data detail. All parties interested in identifying and scaling-up successful charter school practices would benefit from better information. Currently, most data elements that find their way into statewide longitudinal data systems are driven by federal reporting requirements under the Elementary and Secondary Education Act. Thus these data systems contain information on student test scores, student race, language, and disability status, student eligibility for free- and reduced-price lunch, and school and district identifiers. Important information is missing, including such things as curriculum in use, teacher characteristics, and as we have previously noted, lottery results. The federal government, working in collaboration with interested states and national charter school organizations, might generate a template for additional data elements that states or charter school authorizers could include in their routine data collections for their statewide longitudinal databases.

Information to Support Choice

Charter schools are by definition schools of choice. The promise of education choice includes improving quality and efficiency through competition among schools, enhancing opportunity for students of low-income families who may otherwise be trapped in ineffective schools, and spurring innovation. But the promise of choice in public education is constrained by the quality and timeliness of information on school performance that is available to parents.

Under current federal law, school districts are required to produce school report cards, but the information they include is incomplete and sometimes misleading. For example, the report cards include the percentage of students in a school who score proficient on state tests, which is strongly correlated with students’ family background, rather than student gains over the course of the year, which better reflect the performance of the school itself. Information about teacher turnover, parental satisfaction, and other important measures of school performance is not included.

In addition, school districts have demonstrated that they cannot be trusted to help parents choose schools based on school performance. As evidence, a federal study found that half of all districts required to offer school choice due to low performance did not notify parents of their right to choose a new school until after the school year had already started, and many used language that was too complicated for parents to understand. Choice cannot work if parents are blindfolded.[xvi]

The federal government has a role to play in providing parents with timely, transparent school data to support choice. This is particularly important for charter schools, which always require parental choice. Specific recommendations for federal action include:

Report measures of school popularity. Federal requirements for school report cards under the Elementary and Secondary Education Act should be revised to include information on popularity of schools as revealed through the number of applications for admission received by charter schools and other schools of choice.
Report additional school performance data. School districts should be encouraged to report more data on school performance to parents than required under law (or the law should be broadened), with the new data elements being those that are empirically linked to improved student outcomes or valued by parents. Such data might include percentage of inexperienced teachers; truancy rates; availability of extracurricular activities, enrichment programs, and programs for children with special needs; and success of students at the next level of education, such as college enrollment rates for high schools. Encouragement to collect additional data could come in the form of developing and disseminating model reporting templates, recognizing exemplary information systems to support parental choice of schools, and providing support for research and development on the design of school choice information systems.

Facilities

One barrier to charter expansion is the availability of physical space. As one advisory meeting participant highlighted, an enthusiastic educator eager to start a charter school may not have the funding or the expertise in construction to identify, rehabilitate, or build new facilities. Recommendations for federal involvement in facilities include:

Incentives for facilities access. Provide incentives to districts to allow charters to take advantage of surplus district facilities, for example by giving districts that do so priority preference points in federal discretionary grant competitions around school improvement and reform.
Federal loans. Provide federal loan guarantees for facilities or providing direct loans for facilities that take advantage of the low Treasury rates.
Single federal facilities program. Combine the existing federal facilities funding programs into a single coherent and efficient program.

Funding

Charter schools are often provided less funding per pupil for operating expenses than traditional public schools. Further, charter schools are frequently on different schedules for receiving funding compared to traditional public schools. For example, while the principal of a traditional public school typically knows well in advance of a school year what his or her budget will be, the leader of a charter school may not have a clarity on budget until after final enrollment figures are obtained, and may need to spend money on supplies, materials, and personnel before a state allocation of funds is in hand.

Current federal definitions of charter schools and educational programs also adversely affect charter school funding. For example, the longer school days adopted by many charter schools preclude these charters from qualifying for grants for after school programs under the 21^st Century Community Learning Centers program because the definition of “after school” excludes a regular school day that lasts until late afternoon.

Another financial hurdle preventing the growth of highly effective charter schools is the financing of charter school authorization. Authorization is important as the authorizers determine which schools can open and provide oversight and accountability to ensure that poor performing charter schools are closed. The authorization process is a complex and expensive process and routinely underfunded.

Recommendations for federal action to create equitable public funding for charter schools financial inequities include:

Equivalent per pupil expenditures. Make Title I funding contingent on per pupil expenditures that are equivalent across all schools that are eligible for Title I funding, including charter schools.
Equivalent distribution timetable. Require that Title I funds be available for use by charter schools on the same timetable and with the same predictability as they are available for use by regular district schools.

Authorizing

Charter school authorizers decide whether charter schools can enter the market, expand, or must close; they enter into contracts for charter school services; and they provide ongoing performance oversight of charter schools. How well they do their jobs would seem to be a very important determinant of the quality of charter schools. Yet, very little is known about what works in authorizing, and authorizers typically are underfunded with respect to their responsibilities. Although school districts are the most prevalent authorizers, there is an inherent conflict between actions that support the growth of the charter sector and those that support traditional public schools. Recommendations for federal action include:

Charter authorizer funding. Set aside a portion of funding for charter schools to allow a separate competition for awards to charter school authorizers who propose to develop and implement rigorous oversight processes, or require states to adequately fund the authorization process through new requirements in the reauthorization of the Elementary and Secondary Education Act.
Rigorous authorizing process. Make the receipt of federal funding to support charter schools contingent on the presence of authorizing processes that are rigorous and have safeguards to prevent the interests of charter schools from being subverted by the interests of traditional public schools.
Funding research. Provide funding for research on the design and consequences of authorizing practices.

Unintended Consequences of Federal Definitions

An issue that resonated with the experiences of participants in the advisory meeting is the unintended consequences of the definition of a charter school in federal regulations. To qualify as a charter school under federal rules, a school must admit students on the basis of a single lottery for all applicants if more students apply for admission than can be accommodated.

One unintended consequence of this rule is a significant disincentive for charter school operators to expand, for example, by taking over the operation of a low-performing regular high school. Such a high school would not qualify as a charter school if the charter school operator wanted to give preference in admission to students from its existing middle school. It would only qualify as a charter school if every student applying for admission had the same chance of admission through a lottery regardless of where those students had attended middle school. However, the motive for many charter operators to operate a high school would be to build on the scaffold for student success created in earlier grades through their charter elementary and middle school. Many would not wish to take on a high school population that had not benefitted from that earlier preparation. Thus if a charter operator takes the path it considers educationally best by giving admission preference to its own middle school students in a high school for which it assumes responsibility, it loses any opportunity for federal charter school funding for that high school.

Another unintended consequence of the federal rule requiring a single lottery for a charter school is that it precludes the use of stratified lotteries that could be designed to create schools that have student bodies with more geographic or demographic diversity than would result from a simple lottery.

The recommendations for federal action are:

Lotteries for vertically integrated campuses. Change the federal definition of a charter school to allow the use of lotteries for vertically integrated campuses in the same way they are presently allowed for single campuses – thus just as a second grader admitted to a charter school in first grade would not be subject to a lottery to continue in third grade in that school, an eighth grader at a charter operator’s middle school would not be subject to a lottery to advance to ninth grade at that charter operator’s high school.
Stratified lotteries. Consider changing requirements for lotteries to allow stratification on variables that promote wider and more equitable access to schools of choice or that assure greater demographic or geographical balance between lottery winners and lottery losers than the simple flip of a coin.

Virtual Charter Schools

There are presently over 200 virtual charter schools in operation in the U.S.[xvii] Virtual charter schools offer the promise of increasing the productivity of the education system as well as providing more equitable access to advanced and high quality coursework, but simply having coursework online guarantees neither lower costs nor higher quality.

Online education at the college level is proving itself competitive with the classroom experience. According to a survey of colleges and universities, more than a quarter of all students in post-secondary schools were taking at least one course online in the fall of 2008.[xviii] In k-12 education, virtual education is developing more slowly, but policy makers in nearly every state are intrigued by its potential. For one thing, the cost per student of virtual education is, in the long run, almost certainly less than that provided in brick-and-mortar classrooms. According to one survey of 20 such schools in 14 states, the average per pupil cost of online learning in 2008 was roughly half that of traditional public schools.[xix]

Little is known from rigorous research about the quality of virtual charter schools. Studies in Ohio and California comparing home-based virtual charter schools to traditional public schools have found significantly lower student achievement for the virtual school students.[xx] However, the researchers acknowledge that differences in the student population may account for the lower achievement (e.g., students facing significant academic problems in the traditional school leaving to try the virtual school). The remaining available research, which is also observational and does not support causal conclusions, has found similar student achievement outcomes for virtual charter school students and their brick-and-mortar counterparts.[xxi] Overall, quality is likely to vary substantially by course, provider, and instructor, just as it does in traditional settings.

The authorizing function is perhaps even more critical for virtual charter schools than for brick-and-mortar charter schools. Virtual charter schools have low financial barriers to entry because they do not require physical classrooms. This provides a desirable cost advantage but also permits fly-by-night operators to enter the market. Authorizers of virtual charter schools need to have rigorous approval and oversight processes in place to assure that new virtual charter entrants are of acceptable quality and that existing virtual charter operators produce learning outcomes that are on par with what similar students achieve in traditional settings.

The potential for particularly strong conflicts of interest exist when local school districts have authorizing authority over virtual charter schools because traditional public schools are most likely to be disrupted by the efficiencies and conveniences provided by virtual charter schools. These self-interests are likely to manifest themselves through the creation of unreasonable barriers to entry or expansion of virtual charter schools.

The recommendations for federal involvement with virtual charter schools include:

Funding for research and development. Provide competitive funding for studies that examine the condition and effectiveness of virtual education in k-12, and for the development or improvement of virtual courseware.
Shared quality standards. Provide incentives to states to work collaboratively to establish shared standards for virtual charter schools and to create funding policies that would allow students to enroll in recognized virtual charter schools that have out-of-state home offices.

Alignment of Resources with Policy

Although the current and previous administrations have supported charter schools and charter school growth, there are sometimes conflicts between broad policy and particular actions. For example, the recent Edujobs legislation, intended to prevent the layoff of teachers, did not extend funding to charter schools. The recommendation for federal action is:

Alignment. The administration should have a clear policy on charter schools, and examine each piece of law, regulation, and guidance that affects charter schools with the aim of aligning them with its policy. The articulation by the administration of its charter school policy and alignment efforts, including the identification of legislative roadblocks to alignment, could undergird the reauthorization of the Elementary and Secondary Education Act, not only with respect to charters but also with respect to broader issues of parental choice in public education.

[i] National Alliance for Public Charter Schools (2010). Public charter school dashboard. Retrieved from http://www.publiccharters.org/dashboard/home
[ii] Turque, B. (2010, October 6). Charter school enrollment up nearly 6 percent. D.C. Schools Insider, The Washington Post.
[iii] Laskow, S. (2010). Necessity is the mother of invention. Newsweek, August 26, 2010.
[iv] Howell, W.G., Peterson, P.E., & West, M.R. (2009). The persuadable public: The 2009 Education Next-PEPG survey asks if information changes minds. Education Next, 9 (4): 20–29.
[v] Zimmer, R., Gill, B., Booker, K., Lavertu, S., Sass, T.R., & Witte, J. (2009). Charter schools in eight states: Effects on achievement, attainment, integration, and competition. Santa Monica, CA: RAND Corportation.
[vi] Braun, H., Jenkins, F., & Grigg, W. (2006). A closer look at charter schools using hierarchical linear modeling (NCES 2006-460). U.S. Department of Education, National Center for Education Statistics, Institute of Education Sciences. Washington, DC: U.S. Government Printing Office. CREDO (2009). Multiple choice: Charter school performance in 16 states. Stanford, CA: Stanford University. Zimmer et al., supra, note 5.
[vii] Hoxby, C. & Rockoff, J. (2005). School reassignment and the structure of peer effects. NBER Working Paper. Hoxby, C.M., Murarka, S., & Kang, J. (2009). The New York City charter schools evaluation project. NBER Working Paper. Abdulkadiroglu, A. Angrist, J., Cohodes, S., Dynarski, S., Fullerton, J., Kane, T., & Pathak, P. (2009). Informing the debate: Comparing Boston’s charter, pilot and traditional schools. Boston, MA: The Boston Foundation. Angrist, J.D., Dynarski, S.M., Kane, T.J., Pathak, P.A., & Walters, C.R. (2010). Inputs and impacts in charter schools: KIPP Lynn. American Economic Review: Papers and Proceedings 100: 1-5.
[viii] Gleason, P., Clark, M., Clark Tuttle, M., & Dwoyer, E. (2010). The evaluation of charter school impacts. Washington, DC: Mathematica Policy Research.
[ix] Booker, K., Sass, T.R., Gill, B., & Zimmer, R. (2010). The unknown world of charter high schools. Education Next, 10.
[x] Thomas B. Fordham Institute (2005). Charter school funding: Inequity’s next frontier. Washington, DC: Thomas B. Fordham Institute.
[xi] Center for Education Reform (2008). Charter school funding: Follow the money. Washington, DC: Center for Education Reform.
[xii] ECS StateNote (2003). Charter school finance. Denver, CO: Education Commission of the States.
[xiii] National Association of Charter School Authorizers (2010). The state of charter school authorizing 2009. Chicago, IL: National Association of Charter School Authorizers.
[xiv] Witte, J., Carlson, D., & Lavery, L. (in press). Charter school authorizers and student achievement. Economics of Education Review.
[xv] Sable, J. & Plotts, C. (2010). Documentation to the NCES common core of data public elementary/secondary school university survey: School year 2008-09 (NCES 2010-350 rev). U.S. Department of Education. Washington, DC: National Center for Education Statistics.
[xvi] Greene, J., Loveless, T., MacLeod, W.B., Nechyba, T., Peterson, P., Rosenthal, M., & Whitehurst, G., (2010). Expanding choice in elementary and secondary education, Washington, DC: The Brookings Institution.
[xvii] National Alliance for Public Charter Schools (2010). Public charter school dashboard. Retrieved from http://www.publiccharters.org/dashboard/home
[xviii] Allen, I.E. & Seaman, J. (2010). Learning on demand. Online education in the United States, 2009. The Sloan Consortium.
[xix] U.S. Census Bureau (2008). Public schools spent $9,138 per student in 2006. U.S. Department of Commerce.
[xx] Zimmer et al., supra, note 5. Zimmer, R., Buddin, R., Chau, D., Daley, G.A., Gill, B., Guarino, C.M., Hamilton, L.S., Krop, C., McCaffrey, D.F., Sandler, M., & Brewer, D.J. (2003). Charter school operations and performance: Evidence from California. Santa Monica, CA: RAND Corporation.
[xxi] Cavanaugh, C. (2009). Effectiveness of cyber charter schools: A review of research on learnings. TechTrends, 53, 28-31. Zimmer et al., supra, note 5.

Downloads

Download the Full Paper

Authors

Michelle Croft
Susan Dynarski
Caroline Hoxby
Tom Loveless
Mark Schneider
Grover J. "Russ" Whitehurst
John Witte

Publication: The Brookings Brown Center Task Group on Charter Schools

Race to the Top Assessments: Common Core Standards and Their Impact on Student Testing

Thu, 28 Oct 2010 13:00:00 -0400

Event Information

October 28, 2010
1:00 PM - 3:00 PM EDT

Falk Auditorium
The Brookings Institution
1775 Massachusetts Ave., NW
Washington, DC

The Obama administration's Race to the Top Assessment Program funds two consortia of states to develop student tests based on a common core of academic standards. In September, the Department of Education awarded grants to two groups of states – the Partnership for Assessment of Readiness for College and Careers and the SMARTER Balanced Assessment Consortium. Grants of approximately $170 million and $160 million, respectively, were awarded to implement new methods for assessing achievement in mathematics and English language arts.

With the majority of states having signed on to this effort, what are the challenges facing policymakers as they try to translate common standards into assessments? How will testing methods grounded in common standards be properly implemented? What sort of accountability is required? And will these new assessment systems be an effective means of measuring student progress?

On October 28, a panel of education experts, moderated by Senior Fellow Tom Loveless, explored these questions in a discussion, focusing on implementation challenges and how common core standards, recently adopted by 38 states and the District of Columbia, will be incorporated into this latest generation of student testing. During the event, short papers on the topic were presented and discussed and are available to the public on this page.

After the program, panelists took audience questions.

Audio

Race to the Top Assessments: Common Core Standards and Their Impact on Student Testing

Transcript

Event Materials

Participants

Moderator

Panelists

Gregory Cizek

Professor of Educational Measurement and Evaluation
University of North Carolina

Gary Phillips

Vice President and Chief Scientist
American Institutes for Research

Andrew Porter

George and Diane Weiss Professor of Education
University of Pennsylvania

Roberto Rodríguez

Special Assistant to President Obama for Education
White House Domestic Policy Council

Kris Ellington

Deputy Commissioner, Florida Department of Education
PARCC

Joseph Willhoft

Assistant Superintendent, Washington State Department of Education
SMARTER

The 2009 Brown Center Report on American Education

Tom Loveless — Wed, 17 Mar 2010 00:00:00 -0400

This year’s Brown Center Report contains studies taking a long view. Part I examines national test data going back to 1971 from the National Assessment of Educational Progress (NAEP). The study in Part II compares the 1989 test scores of more than 1,000 schools to the same schools’ scores in 2009. Part III compares the test scores of conversion charter schools from 1986, when they operated as traditional public schools, to those from 2008, when they operated as charter schools. The studies tackle perennial questions that, as often happens in education, manifest themselves as controversial topics on the contemporary scene: how to interpret trends in test scores, the distribution of achievement, school turnarounds, and charter schools.

Part I rejects the conventional reaction to the 2009 NAEP scores. Scores in fourth-grade math were unchanged from 2007 to 2009. Eighth-grade scores were up a little. Press articles featured expressions of disappointment and concern, primarily from protagonists who used the flat scores to support policy arguments. Part I places the 2009 scores in the context of the 19-year history of the main NAEP, and after comparing the latest scores with results from other equally trustworthy tests of U.S. math achievement, concludes that the hand-wringing is unwarranted.

So when is a purported NAEP trend really a trend? Part I continues by examining achievement gaps, not between two racial, ethnic, or socioeconomic groups, but between the nation’s highest- and lowest-achieving students. It focuses on the distribution of academic achievement instead of the direction of average achievement. The study is a follow-up to a 2009 Fordham Institute paper documenting that the gap between high- and low-achieving students has been shrinking in recent years. The data in Part I show that the trend, which began sometime around 1998 or 1999, is historically unprecedented and extends across subjects (reading and math), grades (fourth and eighth), and tests (long-term trend and main NAEP). It is also more pronounced in public schools than in private schools. The two analyses in Part I highlight the contrast between a trend indicated by data collected from several independent sources over an extended period of time and speculative assertions arising from “instant analysis” of a single set of test scores.

Part II asks a simple question: do schools ever change? The sample consists of 1,156 schools in California that offered an eighth grade in 1989 and 2009. Test scores from 1989 are compared to scores from 2009. The scores are remarkably stable. Of schools in the bottom quartile in 1989—the state’s lowest performers—nearly two-thirds (63.4 percent) scored in the bottom quartile again in 2009. The odds of a bottom quartile school’s rising to the top quartile were about one in seventy (1.4 percent). The reverse was true as well, with similar percentages of top quartile schools staying among the top performers (63.0 percent) or falling to the bottom quartile (2.4 percent). Changes in a school’s socioeconomic status had only a marginal statistical relationship with test score changes.

The persistence of test scores has major implications for today’s push to turn around failing schools. It can be done, but the odds are daunting. California certainly cannot be accused of inactivity in education reform from 1989 to 2009. Few states tried as many diverse, ambitious reforms that targeted every aspect of the school system—finance, governance, curriculum, instruction, and assessment. Not only have these efforts failed to elevate California from its low national ranking on key performance measures, but they have also had little effect on the relative ranking of schools within the state.

The study suggests that people who say we know how to make failing schools into successful ones but merely lack the will to do so are selling snake oil. In fact, successful turnaround stories are marked by idiosyncratic circumstances. The science of turnarounds is weak and devoid of practical, effective strategies for educators to employ. Examples of largescale, system-wide turnarounds are nonexistent. A lot of work needs to be done before the odds of turning around failing schools begin to tip in a favorable direction.

Part III looks at charter schools. Conversion charters are favored by the Obama administration as a restructuring strategy. Most charter schools are start-ups, begun from scratch by their founders. Conversion charters are schools that are traditional public schools and convert to charter school status. They typically continue to rely on their home districts for several functions (e.g., maintenance of buildings, managing pension obligations, transportation services) but are freed from regulations pertaining to curriculum and instruction. The idea is that schools can be more productive if they are allowed to tailor core educational operations to the needs of their students.

California has the largest number of conversions, and the study was able to collect data on two cohorts: 49 schools from 2004 and 60 schools from 2008. For both cohorts, test score data were also available from 1986, allowing a comparison of scores before and after the schools converted. The analysis is exploratory and mainly descriptive. No causal conclusions can be derived from the data.

What do we know about conversions? Test scores look similar before and after conversion. The 2004 cohort evidences a 2 to 3 percentile point advantage as charters, but the 2008 cohort’s scores declined slightly, less than 2 points, from 1986 to 2008. On several key characteristics, conversions look more like traditional public schools than start-up charters. Compared with start-ups, conversions are more concentrated in urban areas, have larger student enrollments, and serve greater numbers of Hispanic and black students. Teachers at conversions are more experienced and more likely to hold teaching certificates, particularly in bilingual education. It is clear that future evaluations of charter schools must differentiate between start-ups and conversions because of the significant institutional differences between the two types of charters.

To sum up, the studies in this year’s Brown Center Report focus on long-term changes. Part I analyzes NAEP data. Parts II and III examine California test scores from the 1980s and compare them to scores from recent years. Because of its long history of testing, California is currently one of the few states able to provide assessment data for such long-term comparisons. That will change as other states continue to test students annually. Creating rich archives of student performance data bodes well for school reform. Improving schools requires patience and persistence, what education professors Richard Elmore and Milbrey McLaughlin1 call “steady work.” It also requires good information to verify whether reforms have paid off, or, like many efforts in education, produced hopeful signs that soon vanish. The future looks bright if analysts’ capacity to peer into the past continues to improve.

Downloads

Authors

Tom Loveless

Image Source: Daniel Hurst

Expanding Choice in Elementary and Secondary Education

Tue, 02 Feb 2010 13:07:00 -0500

Executive Summary

Education choice exercises a powerful pull on parents of school children: Twenty-four percent report that they moved to their current neighborhood so their children could attend their current school; 15 percent of public school students attend parent-selected rather than district-assigned schools; the charter school and homeschooling sectors have grown from nothing to 2.6 percent and 3 percent of total enrollment respectively; private schools capture 11 percent of enrollment; and virtual schooling is poised for explosive growth. Consistent with these behavioral manifestations of the desire of parents to choose their children’s schools, schools of choice consistently generate more positive evaluations from parents than assigned schools.

Arguments for school choice include improving school quality and efficiency through competition among schools for students; enhancing opportunity for students from disadvantaged families who may otherwise be trapped in ineffective schools; and spurring innovation through the greater administrative autonomy likely to exist in schools of choice. Opponents of choice theorize that it will stratify students by family background, result in niche schools that do not convey the nation’s common heritage, provide taxpayer support for religious instruction, and nullify the advantages of standardization in curriculum, teacher preparation, and management that accrue when schooling systems are designed to deliver a common educational experience across a universe of schools. Opponents of choice also argue that many traditional public schools perform superbly and that those that do not can be improved through better resource allocation and management.

Advocates and opponents of choice typically lock horns over idealized systems of schooling that do not presently exist in the U.S. Thus choice advocates frequently espouse voucher systems that would be similar to federal Pell grants at the postsecondary level. Parents would be able to choose any school they wished for their child, public or private, with government writing the check. In contrast, advocates for traditional schooling envision a system in which every school is good enough to ensure that families’ place of residence and income no longer correlate with the quality of the schools to which their children have access.

It is important to note that both the hopes of the advocates of idealized versions of choice and the fears of the detractors diverge from empirical reality. Charter schools and voucher programs are strongly favored by advocates of choice, but studies of the effects of charter schools on student achievement tend to show that on average charters nationally are performing in the same ballpark as traditional public schools, notwithstanding demonstrations that oversubscribed charter schools in Boston and New York City have generated above average academic gains. Studies of voucher programs, including those in Milwaukee, New York City, Dayton, and the District of Columbia, have found some positive effects, but the differences are not large or across the board. At the same time, concerns that voucher programs or charter schools would deplete the budgets of traditional schools, or result in skimming of the most qualified students, or destroy cultural cohesion or learning of common academic content have been unrealized.

The corresponding reality of public schooling is that the quality of schools is substantially correlated with geography and parental income and likely to remain so in the foreseeable future. While there have been improvements in performance in some large urban school districts and prospects for more, not even the strongest advocate of traditional public schools can maintain that we are close to a point at which a parent living in a low-income area can consign her child to the closest neighborhood school with confidence that the school will be as good, on average, as any other school within a reasonable geographic radius of her home, much less good enough to secure her child’s educational future.

We think the situation on the ground with respect to choice is so different from the idealizations that it warrants a new and different perspective on policy. Choice is most frequently realized within the public sector using the mechanisms of residence, magnet schools, and open enrollment systems, whereas the voucher-like systems applauded by choice advocates and feared by opponents are extremely rare. Further, the charter sector is neither large enough nor sufficiently prepared to go to scale to represent a threat to the traditional system of public schools.

Our policy recommendations are framed within the realities of large variation in the quality of public schools, widespread selection of schools by choice of place of residence, and choice being exercised predominantly within the public sector. These realities offer opportunities for common ground between advocates for choice and advocates for public schools. The goals these communities can share are providing more educational opportunity for children from disadvantaged backgrounds and reducing the number of low performing schools. The mechanisms they can share are: a) a system that affords parents as much choice as possible within the universe of taxpayer supported students and schools, b) portals by which parents can readily access rich information on the performance of schools that is framed to be useful in exercising choice, and c) a funding system that supports the growth of parentally preferred schools and school systems, including virtual education programs.

Specifically, to support the expansion of choice we recommend that:

choice be exercised through systems in which parents have more options than at present (with the expansion of virtual education programs being a promising means to that end);
admission into particular schools within choice systems be open;
selection into oversubscribed schools and programs be determined by lottery (which could be conducted using weights to enhance socioeconomic or geographic balance when that is a desired goal);
choice systems not include a default (all parents would have to choose);
all schools supported with public funds within choice systems be subject to the same standards and assessment regimen under which traditional public schools within a state are required to operate in order to provide transparency for choice;
the popularity of schools as revealed through parental preferences be reflected in funding formulas so that more popular schools garner additional resources to meet enrollment demand; and
substantially undersubscribed schools be restructured or closed. In order to ground the exercise of choice in valid and easily used information on the characteristics and performance of education programs, we further recommend that:
school systems be required to provide timely and relevant information to parents to support choice; • one or more choice navigation websites be developed with the support of federal funds that would be independent of education providers; and
school systems be incentivized to link these choice navigation websites to their parental choice systems.

The choice navigation sites would provide substantially more information on the performance of individual education programs than is presently available to parents (via expanded data collections and enhanced investment in an information infrastructure by the federal government); allow parents to create rankings of programs based on the parents’ own dimensions of preference; and give parents access to decision support tools that would aid in considering dimensions of the performance of schools and education programs that have been linked empirically to better student outcomes.

We recognize that meaningful choice and competition can be constrained even when nominal choice is available, for example because all the schools in a district are low performing, or because transportation to higher performing schools is unavailable, or because all schools are homogeneous. We also recognize that both nominal and meaningful choice are constrained in school districts with small populations, many of which are rural. We suggest means for enhancing meaningful choice, for example, by having multiple operators of schools within urban areas, expanding inter-district choice, subsidizing transportation costs when parents choose schools out of the neighborhood, stimulating the formation of quality charter schools, and fostering virtual education by a variety of operators, including nationally chartered providers.

To support the enhancement of meaningful school choice, we recommend:

the development of a metric of the extent of choice at the school district level that would be available to the public and policymakers; and that
school districts with both low levels of choice and low levels of performance be especially encouraged at the federal level to increase their levels of choice.

Our recommendations do not represent advocacy for any particular type of education institution or program. Rather, school choice should be a democratic process that benefits from the informed participation of parents. Our recommendations are suitable to a range of schooling designs, from a school district in which there are no choices other than district-run public schools, to a system of charter schools, to a division of courses between traditional and virtual schools, to a voucher-based open market in which all providers are on an equal footing, and to many variations in between.

A traditional school district could follow our recommendations by instituting an open enrollment plan at all of its schools, giving additional funding for expansion to oversubscribed schools, closing manifestly unpopular schools, providing transportation to students so that residence does not prevent the exercise of choice, making accredited virtual courses fully count towards graduation, and linking the choice system to a high-quality choice navigation website. Our recommendations are equally applicable to an open market in which public, private, charter, and virtual schools compete on an equal footing for students and the tax revenues that are attached to them.

Our position is that whatever the education delivery design the public has chosen to put in place in a particular school jurisdiction, parents should be afforded the maximum degree of choice, provided with valid information on the performance of the education programs that are available, and have their preferences for education programs reflected in the funding of those programs.

We believe the best evidence suggests that a) parents, including those with low levels of education, can make choices of schools for their children that are sensitive to school performance; b) students from low-income backgrounds benefit from their parents’ decision to send them to higher performing schools; c) the form in which information is presented to parents has important effects on their choice of schools; and d) parental choice can create a competitive market for better schools if the growth of preferred schools and the closure or restructuring of unpopular schools is provided for.

Evidence also suggests that there will be substantial variation in the impact of choice systems on parental behavior, student outcomes, and competition among schools depending on the design of the choice systems and the education options that are available. Poorly designed systems may create greater stratification of schools, reduce educational opportunity for disadvantaged students, and have no systemic competitive effects. Thus, the power of choice to increase educational achievement and opportunity is very much in the details of the design and implementation of choice systems. Because the knowledge base on which to construct school choice systems is far from mature, our final recommendation is that:

the federal role in advancing choice be carried out in a learning context — thoughtful variation in the design of choice systems should be encouraged, systematic data on effects should be collected, and redesign should follow naturally from what has been learned.

Downloads

Download Full Report »

Authors

Michelle Croft
Jay Greene
Tom Loveless
W. Bentley MacLeod
Thomas Nechyba
Paul Peterson
Meredith Rosenthal
Grover J. "Russ" Whitehurst