Brookings: Centers - Brown Center on Education Policy

Do Americans Know How Well Their State’s Schools Perform?

Martin R. West — Wed, 22 May 2013 11:00:00 -0400

Among the most common rationales offered for the Common Core State Standards project is to eliminate differences in the definition of student proficiency in core academic subjects across states. As is well known, the federal No Child Left Behind Act of 2002 (NCLB) required states to test students annually in grades 3-8 (and once in high school), to report the share of students in each school performing at a proficient level in math and reading, and to intervene in schools not on track to achieve universal student proficiency by 2014. Yet it permitted states to define proficiency as they saw fit, producing wide variation in the expectations for student performance from one state to the next. While a few states, including several that had set performance standards prior to NCLB’s enactment, have maintained relatively demanding definitions of proficiency, most have been more lenient.

The differences in expectations for students across states are striking. In 2011, for example, Alabama reported that 77 percent of its 8^th grade students were proficient in math, while the National Assessment of Educational Progress (NAEP) tests administered that same year indicated that just 20 percent of Alabama’s 8^th graders were proficient against NAEP standards. In Massachusetts, on the other hand, roughly the same share of 8^th graders achieved proficiency on the state test (52 percent) as did so on the NAEP (51 percent). In other words, Alabama deemed 25 percent more of its students proficient than did Massachusetts despite the fact that its students performed at markedly lower levels when evaluated against a common standard. U.S. Secretary of Education Arne Duncan has gone so far as to accuse states like Alabama of “lying to children and parents” by setting low expectations for student performance.

There’s no doubt that the definition of proficiency in many states provides a misleading view of the extent to which students are prepared for success in college or careers. Yet whether the way in which states define proficiency matters for student achievement is far from clear. As Tom Loveless demonstrated in the 2012 Brown Center Report on American Education, the rigor of state proficiency definitions is largely unrelated to the level of student achievement on the NAEP across states. Similarly, Russ Whitehurst and Michelle Croft have shown that the quality of state standards (as assessed by third party organizations) is unrelated to NAEP scores, a finding confirmed by the Harvard Kennedy School’s Josh Goodman in an analysis that examined the effects of changes in the quality of standards within states over time. The lack of a systematic relationship between either the rigor or the quality of state standards and student achievement casts doubt on claims that higher and better standards under the Common Core will, in and of themselves, spur higher student achievement.

Less attention has been paid to whether the rigor of state standards matters for public perceptions of the quality of the schools in their states and local communities. If using a more lenient definition of proficiency leads citizens to evaluate their schools more favorably, then the advent of common expectations under the Common Core could alter public perceptions quite dramatically – perhaps increasing pressure for reform in regions of the country in which state proficiency definitions have provided an inflated view of student accomplishment. Is such an outcome likely?

To shed light on this question, I use data from two surveys conducted in 2011 and 2012 under the auspices of Education Next and the Program on Education Policy and Governance at Harvard University. In each year, my colleagues and I asked a nationally representative sample of roughly 2,500 Americans to grade the public schools in their local community on a standard A-F scale. In the figures below, I examine whether the average grade the residents of each state assigned to their local schools is associated with the share of 2011 8^th graders deemed proficient by the state’s own test and by the NAEP. To the extent that differences in the definition of proficiency from one state to the next interfere with citizens’ ability to discern the performance of their local schools, we should see that the average grades citizens assign their schools hew more closely to proficiency rates as determined by state tests than by the NAEP.

The figures demonstrate the opposite. Figure 1a shows that average citizen ratings of local schools across states are only weakly correlated with 8^th grade proficiency rates on state tests. Although the relationship is statistically significant, it is quite small in size: a 10-percentage-point increase in the share of students deemed proficient is associated with an increase in citizen ratings of just 0.03 points on a GPA-style scale (i.e., A=4.0; F=0). Figure 1b, in contrast, reveals a markedly stronger relationship between citizen ratings and NAEP proficiency rates, with a 10-percentage-point increase in proficiency associated with an increase in citizen ratings of 0.17 grade points.

Figure 1a: Relationship between the Average Grades Assigned to Local Public Schools and Proficiency Rates on State Tests

Figure 1b: Relationship between the Average Grades Assigned to Local Public Schools and Proficiency Rates on the National Assessment of Educational Progress (NAEP)

Source: Author’s calculations based on data from the 2011 and 2012 EdNext-PEPG Surveys, state education agency websites, and the NAEP Data Explorer.

Notes: Average grades are reported on a standard GPA scale (i.e., A=4, F=0). State and NAEP proficiency rates are the average of 8^th grade proficiency rates in math and reading. The regression analyses used to generate fitted values are weighted by the inverse of each observation’s estimated variance to account for differences in the number of respondents from each state; unweighted regressions yield substantively similar results.

A simple regression of the average grades citizens assign to local schools in each state on NAEP and state proficiency rates simultaneously confirms that average grades (1) are strongly correlated with NAEP proficiency rates and (2) after controlling for NAEP proficiency rates, have no relationship whatsoever with proficiency rates on state tests. An increase in NAEP proficiency rates of 32 percentage points – the difference between Washington DC and Massachusetts – is associated with an increase in citizen ratings of more than a half of a letter grade. Holding NAEP scores constant, a difference in state test proficiency rates matters not at all.

In short, this evidence suggests that Americans have been wise enough to ignore the woefully misleading information about student proficiency rates generated by state testing systems when forming judgments about the quality of their state’s schools. This does not mean that they ignore state testing data altogether. Indeed, Matthew Chingos, Michael Henderson and I have shown that, within a given state, the grades citizens assign to specific elementary and middle schools are highly correlated with state proficiency rates in those schools. Nor does it necessarily imply that information from the NAEP has a causal effect on perceptions of school quality. The relationship between NAEP performance and the grades citizens assign their schools could easily be driven by other variables, such as the prosperity level of the state, that influence student achievement levels and could also influence school grades. Yet it does suggest that the implementation of the Common Core, by providing information about performance against a common standard, may have less of an impact on public perceptions of school quality than many have projected.

Authors

Martin R. West

Image Source: © Jim Young / Reuters

Does Expanding School Choice Increase Segregation?

Matthew M. Chingos — Wed, 15 May 2013 11:00:00 -0400

Advocates of expanding the educational options available to students from low-income families raise not only social justice arguments—pointing to the choices made by families that can afford to live close to a good public school or pay private-school tuition—but also the theory that competition induced by expanded school choice will be “the proverbial rising tide that lifts all boats.” Breaking the ironclad link between residence and school attended will, proponents argue, force schools to compete for students and resources in ways that increase the quality of education provided.

But critics of school choice policies argue that these reforms will lead to increased segregation by race and class as more motivated families move to better schools, leaving the most disadvantaged students behind in the worst public schools. Criticism has often focused on charter schools given the growth in the charter sector in recent years. Nationwide, charter enrollment grew from 1 to 3 percent of all students between 1999-2000 and 2009-10. Charters make up a much larger share of the market in several places, including 11 percent of Arizona students and 37 percent in the District of Columbia.

Charter critics point to reports showing differences in the demographic characteristics of charter school students and their counterparts in traditional public schools as evidence that choice leads to segregation. For example, a 2010 report by UCLA’s Civil Rights Project found that black charter school students were twice as likely to attend schools that enrolled fewer than 10 percent non-minority students as their counterparts in traditional public schools. This type of analysis says little about segregation because it compares charter schools to all schools nationwide, when charter schools tend to be located in areas with large concentrations of minority students. A reanalysis of the data used in the UCLA report found much smaller differences between charter and traditional public schools once more appropriate comparisons were made between the two groups of schools.

But any comparison of the demographics of students in charter and traditional public schools provides at best an incomplete picture of segregation because segregation resulting from school choice policies would occur primarily across schools, not within schools.^[1] The existence of charter schools could alter the composition of traditional public schools (by drawing students away from them), thereby compromising comparisons between the two sectors as a source of information about the effect of choice on segregation. However, a RAND study found that, in most states, students tend to transfer between traditional public and charter schools with similar racial compositions.

I provide new evidence on this question based on an analysis of nine years of data from the Common Core of Data, the federal government’s annual census of all public schools. For each of the more than 3,000 counties in the U.S., I calculate an “exposure index” that measures the share of non-minority students at the schools attended by the average under-represented minority student.^[2] The average minority student in the U.S. attends a school that is 33 percent non-minority. In other words, the typical minority student attends a majority-minority school. Likewise, the typical student eligible for free or reduced-price lunch (a proxy for economic disadvantage) attends a school where almost two-thirds of students are also eligible for a subsidized lunch.

A naïve examination of the relationship between this measure of (de)segregation and the percentage of students enrolled in charter schools appears to show that the critics are right: more choice is associated with minority students attending less diverse schools. For the 2010-11 school year, a 10-percentage-point increase in charter enrollment is associated with a decline of 16 percentage points in minority students’ exposure to non-minority students. A similar but weaker relationship exists along class lines (as measured by free lunch eligibility).

Of course, this relationship ignores the fact that charters tend to locate in areas that serve large shares of disadvantaged students and members of minority groups. As a result, this simple correlation tells us nothing about whether charters increase segregation or just tend to locate in areas where the schools are already segregated. This is the same methodological flaw that compromised the findings of the UCLA study.

A better approach to the question of whether choice increases segregation is to look at changes over time. Did areas that saw large increases in choice experience larger increases in segregation than areas that saw smaller increases in choice? This kind of analysis does not conclusively measure the causal effect of choice on segregation, but by examining the same locales over time it represents a clear improvement over the cruder approach of comparing different locales at the same point in time. For example, it takes into account any unmeasured factors, such as the degree of residential segregation, to the extent that those factors remain constant over time.

Figure 1 shows the relationship between the change in charter enrollment and the change in minority exposure to non-minority students between 2002-03 and 2010-11.^[3] The cloud of points suggests little relationship between these two factors, and a regression analysis confirms that this is the case.^[4] There is actually a slight positive (and statistically significant) relationship between choice and diversity, but it is very weak and is not also found in the free-lunch data.

Figure 1. Change in Minority Exposure to Non-Minority Students vs. Change in Charter Enrollment, U.S. Counties, 2002-03 to 2010-11

I also used an alternative measure of segregation called a “dissimilarity index” and obtained similar findings: no consistent relationship between changes in charter enrollment and changes in segregation. Finally, I conducted a more sophisticated panel data analysis that uses all nine years of data to estimate the relationship between charter enrollment and segregation using only the changes within counties over time^.^[5] Once again, using both the exposure and dissimilarity indices, the results consistently indicated no meaningful relationship between choice and segregation.

The lack of any consistent relationship between charter enrollment and segregation does not eliminate the possibility that such a relationship exists, but suggests that it is unlikely. For there to be a relationship, it would have to be the case that counties where charter enrollment increased experienced an increase in segregation as a result but then adopted policies (or experienced other changes) that counteracted the increase in segregation. In my view, that is not a very plausible explanation for these results.

There is no doubt that the high level of segregation in American society, including in our schools, is an important problem in its own right. The findings reported here indicate that it is unlikely that charter schools—a prominent effort to increase school choice, especially for students from disadvantaged backgrounds—are making the problem worse. But school choice policies come in a variety of flavors which may have different effects on the demographic makeup of schools. There may be examples of poorly designed choice programs that have increased segregation. For example, a choice system that is complicated and difficult to navigate may advantage affluent, educated parents at the expense of other parents.

Conversely, perhaps carefully designed choice policies can play a role in lessening the segregation of schools by race and class. For example, a simple, streamlined process that allows families to choose any school in a large urban district—and uses a fair method for allocating spaces at oversubscribed schools—could be a way to weaken the link between residential and school segregation that has plagued our school system since the end of legally mandated segregation more than 50 years ago.

[1] Of course students can also be segregated within schools, such as through the classrooms to which they are assigned or courses they decide to take, but that type of segregation is not usually the focus of critics of school choice policies.

[2] I define under-represented minority to include American Indian, black, and Hispanic students.

[3] The average county experienced an increase of charter enrollment of 1 percentage point, with a standard deviation of 4 percentage points. Weighted by student enrollment, the average increase is 2 percentage points with a standard deviation of 4 percentage points.

[4] The regression analysis and line in Figure 1 are both weighted by the number of minority students in each county (using the average of 2002-03 and 2010-11)

[5] This analysis pooled data from all years and included both year and county fixed effects.

Authors

Matthew M. Chingos

Policymakers Get Serious About Student Loan Interest Rates

Matthew M. Chingos and Beth Akers — Fri, 10 May 2013 11:25:00 -0400

At about this time last year, we saw President Obama and Republican challenger Mitt Romney engage in a pandering contest on student loan interest rates. Cheap political theater produced a shortsighted political solution—a one-year extension of the 3.4% interest rate on subsidized federal student loans.

That one-year “fix” is due to expire on July 1, setting up another round of debate about whether to extend the lower rate once again or come up with a permanent solution. Under current law, Congress sets the interest rates on loans (which are then fixed for the life of the loan). This leads to political fights over the interest rate on a regular basis, especially when market rates become out-of-sync with the rate set by Congress.

This time around, the Obama administration and several members of Congress have produced serious proposals, most of which propose allowing the interest rates on federal student loans to vary with market conditions rather than having a fixed rate that is set by Congress. An excellent summary of these proposals appears in today’s Inside Higher Ed. The key elements of each of the proposals (and current law) regarding the federal Stafford loan program are:

1) Obama administration proposal: interest rate varies with market rates (10-year Treasury rate plus 0.93% for subsidized loans and 2.93% for unsubsidized loans) but is fixed for the life of the loan. There is no cap on interest rates.

2) House Republican proposal: interest rate varies with market rates (10-year Treasury plus 2.5% for subsidized and unsubsidized loans) and varies over the life of the loan (as the Treasury rate increases or decreases). Interest rates are capped at 8.5%.

3) Sens. Reed and Durbin proposal: same as House Republican proposal, except market rate is defined as the 91-day Treasury rate plus a percentage determined by the Education Secretary to cover administrative costs, and the cap is 6.8%.

4) Sen. Warren proposal: one-year fix in which the rate on subsidized loans is set at the rate the Federal Reserve changes to banks (currently 0.75%).

Sen. Warren’s proposal should be quickly dismissed as a cheap political gimmick. It proposes only a one-year change to the rate on one kind of federal student loan, confuses market interest rates on long-term loans (such as the 10-year Treasury rate) with the Federal Reserve’s Discount Window (used to make short-term loans to banks), and does not reflect the administrative costs and default risk that increase the costs of the federal student loan program.

Setting aside this one embarrassingly bad proposal, the remaining proposals raise a set of questions that need to be answered in order to select the ideal policy:

First, should the interest rate on federal student loans be set by Congress or allowed to fluctuate with the market? Market rates reflect the cost of borrowing to the government. Consequently, rates below-market rates indicate a subsidy to students. In our view, subsidies of college-going should be administered through programs that bring about the greatest changes in enrollment behavior, such as grant programs, and not through subsidies to interest rates that are much less transparent. Indexing the interest rate to the market also has the advantage of lessening the role of politics in student loan programs.

Second, should the interest rate be fixed for the life of the loan or allowed to vary with the market? In the market for other kinds of loans, such as home mortgages, consumers can choose between fixed- and variable-rate loans. But many students are not sophisticated consumers of financial products. In our view, the federal program is best operated with a fixed-rate model because it shields the student from the risk that the rate will increase in the future (usually at the cost of a higher interest rate to make up for that risk). Although the actual risk associated with a variable rate loan may be small, fear of this uncertainty might discourage some students from taking the loans that they need to enroll in postsecondary education.

Third, should there be a cap on student loan interest rates? One of the criticisms of a move to market-based interest rates is that times of extraordinarily high market rates will make college inaccessible to many students (by making it prohibitively expensive to borrow). In our view, a cap on interest rates is a reasonable approach to ensure student access to college and to make a market-based system politically feasible.

Where does that leave us? It turns out the ideal policy is also a political compromise: it takes the market-based proposal of both President Obama and the House Republicans, the fixed-rate proposal of the President, and the interest rate cap of the House Republicans and Senate Democrats. Of course there are still details to be worked out, such as how much should be added to market interest rates to finance the administrative costs and default risk of the federal student loan program. But this is a rare example where proposals from our two political parties seem close enough that compromise on a good policy should be possible.

Authors

Image Source: © Jonathan Ernst / Reuters

Obama’s Early Education Proposals Leave Federal Efforts Fragmented and Incoherent

Chris Herbst — Wed, 08 May 2013 11:00:00 -0400

The release of President Obama’s budget reignited the debate over the potential benefits of public investment in early childhood education. The centerpiece of his proposal is a $75 billion federal-state partnership to provide all low- and moderate-income four-year-olds with high-quality, full-day pre-K.^[i] But equally important is what the President proposed—or, rather, didn’t propose—for the Child Care and Development Fund (CCDF), a vital child care subsidy program serving 1.7 million low-income children each month at a cost of $10 billion per annum.^[ii] By comparison, Head Start spends about $7 billion on 900,000 children each year.^[iii]

As I will explain below, the President’s budget is disappointing because it misses an opportunity to fix two structural flaws with the CCDF: its lack of integration with the larger early care and education system and its disproportionate emphasis on supporting parental employment.

When it was created in 1996, the CCDF was intended to help low-skilled mothers transition from welfare to work. In practice, this is accomplished in two ways. First, eligibility for child care assistance is conditioned on fulfilling a state-defined work requirement, which typically includes participation in paid employment, job training, or education. Second, the CCDF invokes the principle of “parental choice,” in which subsidized parents are allowed to purchase child care from most legally-operating providers, including those not subject to states’ child care regulations. Together, these design features underscore a longstanding tension between the dual goals of U.S. child care policy: to support parental employment and promote child development.

How has the CCDF performed in relation to these goals? There is little doubt that the child care subsidy system has been effective at increasing employment among disadvantaged mothers. Recent studies provide consistent evidence that mothers receiving subsidies are more likely to be employed, to be working without receiving welfare, and to be engaged in standard work (i.e., work performed between 8 a.m. and 6 p.m. Monday through Friday) than their unsubsidized counterparts.^[iv] Importantly, the CCDF has also allowed low-skilled mothers to invest in their own human capital. A recent study finds that subsidized mothers are more likely to enroll in college-level courses and participate in job training programs.^[v]

But on the second goal—enhancing child well-being—the evidence is less positive. Over the past few years, my colleague, Erdal Tekin, and I have studied the impact of CCDF-funded child care subsidies on preschool-aged children’s health and development. Our research examines over 10 dimensions of child well-being using several nationally representative datasets and a variety of methodological techniques. The results are strikingly consistent: receipt of CCDF child care subsidies is associated with worse health and developmental outcomes for low-income children. In particular, we find that children receiving subsidized care in the year before kindergarten score lower on tests of reading and math ability and display more behavior problems when they enter kindergarten than their unsubsidized counterparts.^[vi] Subsidized children are also more likely to be overweight and obese.^[vii] Equally troubling is that the negative effects do not stop with the child: subsidized mothers engage in lower-quality interactions with their children and are more likely to show symptoms consistent with clinical anxiety and depression.^[viii]

Our results beg the obvious—and important—question: why does the CCDF fail to promote child and family well-being? Admittedly, our research is not well-equipped to provide definitive answers, but there is scattered evidence from a variety of sources that may allow policymakers to pinpoint the culprits. The three most plausible explanations are:

1. The challenges of parental choice and low-quality child care. By maximizing flexibility in the selection of child care providers, the CCDF allows low-skilled parents to move quickly into paid work or education and job training programs. But there is a downside to parental choice: parents—regardless of education or income level—often do not have enough information to distinguish between low- and high-quality providers.^[ix] When parents cannot make informed decisions, child care providers have little incentive to make costly quality investments. This ultimately forces high-quality, high-cost providers out of the market, leaving only those willing to offer mediocre services. This is one explanation for the widespread quality problems plaguing the U.S. child care market,^[x] and it provides context for the growing number of studies finding that non-parental child care settings are detrimental to child development.^[xi] Parents receiving CCDF-funded subsidies are therefore victims of and unwitting accomplices to the information gap in the child care market: their choices are restricted primarily to low- to mediocre-quality providers, and the inability to make informed decisions only exacerbates the quality problem.

2. An overemphasis on parental employment. For all intents and purposes, the CCDF is a labor market policy. It was created in service of welfare reform legislation aimed at solving the “problem” of low employment rates among single mothers. The law’s solution was to make eligibility for cash and child care assistance conditional on fulfilling a work requirement. In my view, child care policy should not be used to fix a labor market problem. At best, the CCDF is a blunt instrument with which to boost mothers’ employment, especially in comparison to the alternatives available to policymakers (e.g., the Earned Income Tax Credit). At worst, the CCDF may have unintended effects on the child care market that are harmful to child well-being. For example, if parents lose eligibility for subsidies whenever they become separated from a job, such instability could undermine child development by severing productive child-teacher relationships and exposing children to comparatively low-quality care while the parent is looking for work. The work requirement is also problematic for child care providers: those relying heavily on subsidized children may experience revenue shortfalls when parents lose eligibility, thereby reducing the incentive to make costly quality improvements.^[xii]

3. Low reimbursement rates. The final explanation focuses on the subsidy reimbursement rate, or the maximum amount a state or local agency pays child care providers to serve subsidized children. The CCDF attempts to provide low-income families with “equal access” to high-quality care by recommending that reimbursement rates be set at the 75^th percentile of the local child care price distribution. As such, the CCDF gives states substantial flexibility to establish lower rates. In fact, only one state—New York—now sets its reimbursement rate at the 75^th percentile, and 18 states have not updated their reimbursement structure in at least five years.^[xiii] This abysmal record not only prevents families from purchasing high-quality child care; it also reduces the resources available to child care providers to make costly quality enhancements.

Enter the President’s budget. In my view, it fails to harmonize a highly fragmented early care and education system. In fact, the proposal may ultimately exacerbate fragmentation by creating what amounts to a two-tiered system for low-income children. On the one hand, there will be a set of heavily subsidized, high-quality programs for 3- and 4-year-olds (through Head Start and the new pre-K initiative). For children ages 0 to 3, on the other hand, we will continue to have a system of low-quality child care propped up in part by an underfunded CCDF.

One of the potential implications of this bifurcated system is the following: as 4-year-olds flock to pre-K, the CCDF will serve growing numbers of children ages 0 to 3. My research (with Erdal Tekin) suggests that the young children served by the subsidy program will already be behind developmentally as they move to pre-K. So rather than preparing children for kindergarten, the new pre-K program will expend valuable resources trying to undo the effects of the low-quality care to which subsidized children were exposed during the first three years of life.

What can policymakers do to improve the child care subsidy system? I will outline a few broad principles that should be incorporated into a redesign of the CCDF. In my view, the primary problem with child care in the U.S. is the low average quality available to parents. Thus, child care policy should shift away from its current focus on increasing parental employment to one that enhances child development. It can do so in the following ways:

Divorce the child care subsidy system from the welfare system. Employment-based child care subsidies represent a misguided approach to child care policy. By necessity, such a system places few restrictions on the quality of care that parents may purchase. And the mandated work requirement is clearly the wrong policy tool for solving the quality problem. The decoupling of child care and welfare policy will signal that the goal of the former is to neither promote nor inhibit parental employment. Doing so may also increase the odds of reform, as the subsidy program will no longer be seen as an appendage of an unpopular welfare system.
Provide parents with strong incentives to purchase high-quality care. One way to accomplish this is through a means-tested voucher—available to working and non-working parents—whose value is increasing in the quality of care purchased. This is already happening to some extent: 32 states have higher reimbursement rates for providers meeting higher-quality standards. But even the states with the highest tiered benefit levels often do not reach the 75^th percentile recommendation.^[xiv] Therefore, subsidy benefits need to be increased substantially. In addition, eligibility should be expanded to reach families up to at least 200 percent of the federal poverty line.
Inform parents about the potential benefits of high-quality care. Part of the quality problem originates with parents. Given that they are often unable to discern levels of child care quality or are unwilling to pay more high-quality services, states and the federal government should engage in an aggressive public information campaign to inform parents about the importance of early child care. The campaign should target families inside and outside the subsidy system so that the shift in demand is large enough to compel providers to invest sufficient resources into quality enhancement. Parents also need a better understanding of the accreditation system, and they should have easy access to the local child care resource and referral database.

Enacting these proposals will certainly increase the price tag of the President’s early education plan. But the U.S. has already tried to do child care policy on the cheap, and the results are not positive. It is time to get the CCDF on the right track by focusing on quality.

[i] The projected 10-year cost of the President’s Preschool for All proposal is $75 billion. It would begin in FY 2014 with a bit more than a $2 billion expenditure.

[ii] http://www.acf.hhs.gov/sites/default/files/occ/2009_final.pdf

http://www.acf.hhs.gov/sites/default/files/occ/final_overview_allyears11508_compliant.pdf

[iii] http://eclkc.ohs.acf.hhs.gov/hslc/mr/factsheets.

[iv] Herbst, C.M. (2010). The Labor Supply Effects of Child Care Costs and Wages in the Presence of Subsidies and the Earned Income Tax Credit. Review of Economics of the Household, 8(2), 199-230. Available here.

Herbst, C.M. (2008). Do Social Policy Reforms Have Different Impacts on Employment and Welfare Use As Economic Conditions Change? Journal of Policy Analysis and Management, 27(4), 867-894. Available here.

Tekin, E. (2007). Single Mothers Working at Night: Child Care Subsidies and Standard Employment with Implications for Welfare Reform. Economic Inquiry, 45(2), 233-250. Available here.

[v] Herbst, C.M. & Tekin, Erdal. (2011). Do Child Care Subsidies Influence Single Mothers’ Decision to Invest in Human Capital? Economics of Education Review, 30(5), 901-912. Available here.

[vi] Herbst, C.M. & Tekin, E. (2010). Child Care Subsidies and Child Development. (2010). Economics of Education Review, 29(4), 618-638. Available here.

Herbst, C.M. & Tekin, E. (2010). The Impact of Child Care Subsidies on Child Well-Being: Evidence from
Geographic Variation in the Distance to Social Service Agencies. National Bureau of Economic Research Working Paper No. 16250. Available here.

[vii] Herbst, C.M. & Tekin, E. (2011). Child Care Subsidies and Childhood Obesity. Review of Economics of the Household, 9(3), 349-378. Available here.

Herbst, C.M. & Tekin, E. (2012). The Geographic Accessibility of Child Care Subsidies and Evidence on the Impact of Subsidy Receipt on Childhood Obesity. Journal of Urban Economics, 71(1), 37-52. Available here.

[viii] Herbst, C.M. & Tekin, E. (2012). Child Care Subsidies, Maternal Well-Being, and Child-Parent Interactions: Evidence from Three Nationally Representative Datasets. National Bureau of Economic Research Working Paper No. 17774. Available here.

[ix] Mocan, N. (2007). Can Consumers Detect Lemons? An Empirical analysis of Information Asymmetry in the Market for Child Care. Journal of Population Economics, 20, 743-780. Available here.

[x] National Research Council and Institute of Medicine. (2000). From neurons to neighborhoods:

the science of early child development. Committee on Integrating the Science of Early Childhood Development. Jack P. Shonkoff and Deborah A. Phillips (Eds.). Washington, DC. National Academies Press.

National Institute of Child Health and Human Development (NICHD). (2000). Characteristics and quality of child

care for toddlers and preschoolers. Applied Developmental Science, 4, 116-141.

National Association of Child Care Resource and Referral Agencies. (2013). We Can Do Better: NACCRRA’s

Ranking of State Child Care Center Regulations and Oversight. Available here.

[xi] Bernal, R. & Keane, M. (2011). Child care choices and children’s cognitive achievement: The case of single mothers. Journal of Labor Economics, 29, 459-512. Available here.

Herbst, C.M. (2012). The Impact of Non-Parental Child Care on Child Development: Evidence from the Summer Participation “Dip.” Discussion Paper No. 7039. Bonn, Germany: Institute for the Study of Labor (IZA). Available here.

[xii] This article from The Washington Post describes the financial pressure child care providers face when they locate in low-income communities and serve subsidized children.

[xiii] Schulman, K. & Blank, H. (2012). Downward Slide: State Child Care Assistance Policies 2012. Washington, DC: National Women’s Law Center. Available here.

[xiv] Ibid.

Authors

Chris Herbst

Time-to-Graduation Too Often Overlooked

Beth Akers and Matthew M. Chingos — Wed, 01 May 2013 11:00:00 -0400

Providing information about college outcomes to students and their families as they shop for college is critical (as Brown Center scholars have argued here and here). The availability of information has the potential to aid individual students by helping them to select a college that will best serve them, and to improve the system by allowing market forces to keep prices in line with value. However, these benefits are only generated to the extent that consumers know how to use the information to make good decisions. Financial outcomes such as graduate earnings and employment rates are one important piece of information, but clearly not the only metrics that prospective students should consider.

Graduation rates are often cited as a key measure that students should consider when selecting a college. This is undoubtedly true, but too often this only refers to six-year graduation rates which are only a partial measure of completion patterns. Among the cohort of first-time, full-time students entering four-year degree programs in 2005, 39 percent graduated within four years. And about 60 percent eventually graduated (within six years),^[i] which means that about 35 percent of graduates take longer than necessary to complete their degrees.

Extended enrollment is very costly, perhaps more so than most students realize. Although some students take longer to graduate due to breaks in enrollment, others simply spend more time enrolled. With the average net price across all four-year, degree-granting institutions at $20,374 during the 2010-11 academic year, an additional year of enrollment would translate into an extra $234 per month in student loan payments if a student were to finance the cost.^[ii] Extending enrollment to six years would mean an additional $469 per month in loan payments compared to if the student had graduated on time.

In addition to the direct costs of extended enrollment, students forgo earnings while they remain in school. Although the actual opportunity costs vary greatly by individual, these costs will generally be even larger than the direct costs of attendance. These statistics highlight that the time it takes to complete a degree, just like earnings and employment outcomes, can have important financial consequences.

The benefits of finishing college on time are clear, but it is less obvious that time-to-degree is a dimension on which students and parents need to pay close attention when shopping for college. One might assume that general metrics of college quality are good proxies for all sorts of outcomes, including time-to-degree. For example, perhaps institutions with good six-year graduation rates also have good four-year graduation rates so it doesn’t really matter which one students use. But it turns out that this is not the case. The average time-to-degree varies widely, even within institutions that seem to be of similar quality based on other measures.

Figure 1 shows the relationship between four- and six-year graduation rates among all four-year, degree-granting institutions eligible for federal student aid. Since the number of students who graduate from an institution after six years is very small, we can think of six-year graduation rates as a decent proxy for the overall graduation rate (which is not observable due to natural data limitations). In this figure it is apparent that the rate of on-time graduation varies immensely even within the set of institutions with similar overall graduation rates. A student choosing among a group of institutions with similar six-year graduation rates ought to know that the chance of graduating on time is likely to vary significantly across those colleges.

Figure 1: Graduation and On-Time Graduation Rates

Source: IPEDS 2011 database.

Figure 2 shows the relationship between a second measure of institution quality, average SAT/ACT score, and the share of graduates who finish on time (the ratio of the four-year graduation rate to the six-year graduation rate). Again, it is apparent from this figure that among schools of similar quality, as measured by average SAT/ACT score, the share of graduates who finish on time varies widely. For a 100 point range of SAT score surrounding the mean (1040), the mean rate of on-time graduation was 65 percent. For this set of 485 schools, the percent of graduates who finish on time varies from 49 percent at the 25^th percentile to 79 percent at the 75^th percentile. This is a surprisingly large range considering the similar SAT/ACT scores of their students. It is not obvious from visual inspection that on-time graduation is positively related to average SAT/ACT score. In other words, students and parents cannot necessarily expect that schools reporting higher average SAT/ACT scores will also produce faster time-to-degree. However, there is less variation in the share of graduates who finish on time among schools that report average SAT/ACT scores near the upper end of the possible range.

Figure 2. Time-to-Degree and Institutional Quality

Source: IPEDS 2011 database.

Since general metrics of institution quality (like average SAT/ACT score and six-year graduation rate) are not a sufficient substitute for the rate of on-time graduation, colleges need to be evaluated on time-to-degree explicitly. Of course student characteristics play an important role in determining a college’s on-time graduation rate, but Institutions also have important effects. A recent working paper by Sarah Cohodes and Josh Goodman reports some of the first rigorous, quasi-experimental evidence that college quality has a substantial impact on the likelihood that a student will graduate, in either four or six years.

In sum, information matters. Specifically, information about a school’s track record for getting students through their programs on time matters. As students and parents shop for college, they should be aware of differences between schools on this dimension. Policy makers should also recognize these differences and prioritize making this measure of quality available to consumers. Publishing information about on-time graduation rates to the public through online tools like the College Scorecard is important. As consumers gain a better understanding of the metrics that are important to them, it will create incentives for schools to help students graduate on time, replacing the previous incentive to keep tuition-paying students as long as possible. Perhaps most importantly, higher rates of on-time graduation will reduce the financial burden of higher education on both individual students and the tax payers who support them through financial aid programs and direct subsidies to institutions.

[i] Administrative data from Ohio indicate that only a very small fraction of students graduate after six years (p. 33 of Bowen, Chingos and McPherson, Crossing the Finish Line: Completing College at America’s Public Universities, Princeton University Press, 2009).

[ii] This calculation is based on a loan with a 6.8% interest rate and a 10-year term.

Authors

Image Source: © Jessica Rinaldi / Reuters

Teacher Value-Added: Do We Want a Ten Percent Solution?

Grover J. "Russ" Whitehurst — Wed, 24 Apr 2013 11:00:00 -0400

A recent article in the New York Times describes how the statistical revolution that has swept professional baseball in the last decade has become so pervasive that long-time radio broadcasters are being replaced by announcers who can communicate to fans about advanced statistics. Most education reformers would select Waiting for Superman as their favorite education film. But for those whose passion is boosting teacher quality, the hands-down winner is the 2011 film about the use of statistics in baseball, Moneyball. Indeed, these reformers see a day in which district leaders who don’t understand and use advanced statistics to shape their teacher workforce will be as anachronistic as baseball broadcasters who can’t fluently discuss B.A.B.I.P. (batting average on balls in play).

We are clearly moving in that direction, at least in intent. Spurred on by the Obama administration’s $4.5 billion Race-to-the-Top state grant program and the subsequent No Child Left Behind (NCLB) state waivers, over two-thirds of the states in the nation have made commitments to the federal government to institute teacher evaluation systems that sort teachers into different levels of performance with associated consequences. For example teachers persistently in the top tier in terms of evaluation scores would be paid more and teachers persistently in the bottom tier would be replaced.

This sounds like a statistical revolution in that decisions are to be based on data rather than politics, intuition, union contracts, or the way it has always been done. And it certainly seems promising compared to existing practice in which teachers are seldom subject to an evaluation and all get passing scores, pay is determined by years on the job rather than performance, and almost no-one ever is dismissed for being a bad teacher. But the devil is in the details. A new generation of teacher evaluation systems won’t work without the right data being used smartly.

The Brown Center at Brookings is in the middle of a project in which we are examining the actual design and performance of new teacher evaluation systems in several large urban school districts scattered across the country. We’re asking whether there are significant differences in the design of these systems across districts, whether any such differences have meaningful consequences in terms of the ability to identify exceptional teachers, and whether there are practical ways that districts might improve the performance of their systems. We’ll have a lot more to say about this project later this year. Here I want to share some initial findings that are interesting to me and that may be of use to the many districts and states around the country that are just starting to create, design, and implement new teacher evaluation systems. This is work that our cooperating districts have been at for a couple of years. Lessons learned from them should help those just getting started.

Most of the action isn’t in value-added

You would think from the majority of the media coverage of teacher evaluation and the wrangling between teacher unions and policy officials that the new teacher evaluation systems being implemented around the country are principally about judging teachers based on their students’ scores on standardized tests. “Value-added” is the shorthand for this and has been a bone of contention in almost every effort to replace existing teacher evaluation systems, which declare everyone a winner, with new systems that are designed to sort teachers into categories of effectiveness.

For example, the nine day teachers strike in Chicago in the fall of 2012 was reported as having been driven by teachers’ objection to a proposed “evaluation system that judged them by the test scores of their students.”^[i] The Chicago Teachers Union felt they won a major concession in the final contract because the proportion of a teacher’s evaluation based on test scores was reduced to 30% from the 45% proposed by the City. Similarly, the union representing teachers in New York State stalled the state’s agreement under its NCLB waiver to institute a teacher evaluation system statewide. The main sticking point was whether 40% of a teacher’s evaluation would be based on test score gains of the teacher’s students, as proposed by the state. The final agreement reduced this number to 20%.

In the districts we’re working with less than 20% of teachers can be evaluated based on their students’ test scores. Why? Under NCLB, states have to administer annual tests in language arts and mathematics at the end of grades 3-8. These are the “tested grades and subjects.” Third graders haven’t been tested before the end of third grade. With only a score at the end of the year and no pretest their gains can’t be calculated. Gain scores can be computed for 4^th though 8^th graders by subtracting their score at the end of the previous grade from their score at the end of their present grade. But by 6^th grade students are in middle school, which means that they have different teachers for different subjects. Thus their gain scores on mathematics and reading can’t be allocated to a single teacher. Thus only 4^th and 5^th grade teachers in self-contained classrooms who remain the teacher of record for a whole year can be evaluated based on the test scores of the students in their classrooms. Every other teacher has to be evaluated some other way.

It gets worse if a district makes the reasonable decision to increase the reliability of its evaluation system by requiring at least two years of value-added data on a teacher as the minimum for making a high stakes decision such as denial of tenure. Because large numbers of teachers move between grades, schools, and in and out of the profession, particularly in big urban districts, the proportion of the teacher workforce that can be evaluated with two years of value-added data may fall to only about 10%.

Returning to Chicago and assuming that no more than 20% of Chicago teachers could be evaluated based on the test score gains of their students, the Chicago strike was about whether test scores would carry at most a weight of .09 (as originally proposed by the City) or .06 (as eventually agreed to by the City and the Union) in the overall evaluation system for all teachers.

If you like your coffee black, I can see you making a fuss if someone tries to add some milk. But having a bare-knuckle fight over whether it is going to be 6 or 9 drops of milk doesn’t make a lot of sense. Either the parties in these disagreements don’t understand the minor role that value-added can play in teacher evaluation systems given the small proportion of teachers on which it can be calculated, or the war is about something else with value-added simply being a convenient symbol.

The something else is likely meaningful evaluation at all. Student test score stats, flawed though they are, happen to provide the best predictions of future teacher performance and later student outcomes that are currently available. Even though student test score gains attributable to individual teachers can only be calculated for a small proportion of the teacher workforce, these stats are the anchor for the rest of the teacher evaluation system. For example, in the Gates Foundation’s Measures of Effective Teaching project, the validity of teacher evaluation scores based on classroom observations is assessed by their correlation with value-added scores from the same teachers. This is also how the Brown Center has previously proposed that the performance of all teacher evaluation systems be evaluated. Test scores are the one component of the evaluation system that has a known property from teacher to teacher, school to school, and district to district. Without it, at least for now, the meaning of any other component of the evaluation system is easily challenged. So it is that those who want to reform teacher evaluation want value-added and those who prefer the status quo don’t. It isn’t about how many drops of milk to add to the coffee, even though it seems to be – it’s about whether there will be any milk at all.

I want milk in the coffee – value-added adds value – but we need to pay more attention to the quality of the coffee itself. Those who advocate for meaningful teacher evaluation should be investing in and fighting for classroom observation systems and other sources of information on teacher performance, including student ratings of teachers, that are good enough to be used in high stakes decisions about teachers.

The districts we’re working with all use home-grown classroom observation systems that almost surely could be improved, and they’re using processes for collecting classroom observations that differ substantially across districts. For example, some districts have only building principals carrying out classroom observations, others have only master teachers doing this work, and others have a mix. Some conduct six classroom observations a year for each teacher, while others carry out only two. Do these different design decisions have consequences for the performance of the evaluation system? We need to know.

All of the classroom observation systems across the districts with which we are working are one-size fits all, which means that the high school algebra teacher is being evaluated on the same generic skill set as the kindergarten teacher. I’m sure that in addition to assessing generic teaching skills we need content-specific and grade-specific observation systems – does the math teacher know how to teach math and does the kindergarten teacher know how to create a classroom environment that is appropriate to 5-year-olds?

There is a lot of work to be done to provide school districts with the building blocks of evaluation systems that are good enough both to withstand the political and legal challenges they will face and to identify exceptional teachers reliably. This is an effort that must be carried out in the trenches. It lacks the glamour of the headline reform, which is replacing everybody-is-a-winner systems with systems that are predicated on there being meaningful differences among teachers in effectiveness. But if we don’t attend to building evaluation systems that work well for all teachers, not just those for whom value-added can be calculated, the headline reform is at risk of failing.

[i] http://www.nytimes.com/2012/09/15/education/optimism-for-a-deal-on-chicago-teachers-contract.html?pagewanted=all&_r=0

Authors

Grover J. "Russ" Whitehurst

Image Source: © Jim Young / Reuters

Do Math and Science Teachers Earn More Outside of Education?

Martin R. West — Wed, 17 Apr 2013 15:34:00 -0400

The urgency of improving American students’ skills in math and science is hardly in dispute. Performance in these subjects is increasingly critical to individual and national economic success, yet far too few of our students graduate from high school equipped for post-secondary work in technical fields. For example, the ACT reports that just 46 percent of high school graduates taking its college entrance exams in 2012 met college-readiness benchmarks in math; fewer than one in three did so in science. Among all 17-year-olds, the most recent data from the National Assessment of Educational Progress shows that as many as 36 percent lack even a basic level of math proficiency.

Improving the caliber of our math and science teachers is essential to changing this picture. A large body of evidence confirms that teacher effectiveness is a key determinant of students’ academic progress. Indeed, it is likely the case that, as President Obama has said, “The quality of math and science teachers is the most important single factor influencing whether students will succeed or fail in science, technology, engineering and math.”

Unfortunately, the same labor-market trends that have made math and science skills increasingly valuable to students may make it increasingly difficult to attract teachers with the talent and training necessary to address the challenge. Despite a recent wave of reform, the vast majority of school districts nationwide continue to pay teachers based on salary schedules that fail to differentiate among teachers based on their subject-area expertise. To the extent that teachers with technical skills have better earnings opportunities in other industries, this approach can be expected to produce fewer – perhaps even a shortage – of qualified candidates for math and science teaching jobs.

Data consistently show schools struggling to recruit and retain effective candidates for teaching positions in these subjects. In 2003-04, for example, 27 percent of schools with a math teaching vacancy reported that filling that vacancy was “very difficult” or ultimately unsuccessful, as compared with just four percent of schools with vacancies in elementary classrooms (see Figure 1). Even taking into account the fact that general elementary vacancies are more common, schools were more than four times as likely to have a difficult or unsuccessful search in math. The data on vacancies in the biological and physical sciences show much the same pattern.

Source: Author’s calculations based on U.S. Department of Education, National Center for Education Statistics, Schools and Staffing Survey, 2003–04, Public School, BIA School, and Private School Data Files.

Do these patterns reflect the nature of the job opportunities available to individuals with math and science skills in the broader economy? To shed new light on this question, my Brookings colleague Matthew Chingos and I used a unique administrative database to follow the careers of almost 32,000 high school teachers employed by Florida public schools between the 2001-02 and 2006-07 school years, roughly 3,500 of whom left teaching for a new job in the state during that time. Quarterly earnings data from the state’s unemployment insurance system enabled us to compare the earnings of teachers in different subject areas both while they were teaching and in their new careers.[1]

Figure 2 compares the earnings of math and science teachers to those of English teachers for the same group of teachers before and after they left the classroom.[2] Among those who left teaching for jobs other industries, math and science teachers earned 15 percent and 12 percent more, respectively, than did former English teachers after leaving. While they were teaching, these same math and science teachers earned no more than English teachers.[3]

Notes: * significant at p<0.01. N=3,456. The chart presents coefficient estimates from regressions of log annual earnings on district fixed effects (to capture variation in local labor market conditions) and the following subject area indicators: math, science, social studies, foreign languages, and multiple subjects, with teachers of English classes the omitted category. Stayers are those who remained as classroom teachers in Florida public schools from 2001–02 through 2006–07; leavers are those who left for non-teaching jobs within Florida. We exclude teachers who left classroom teaching for other positions in Florida public school districts, teachers who exited the Florida workforce altogether, and likely retirees (those 55 and older). We ignore earnings experienced during an individual’s first year outside of the classroom to allow for transitions between teaching and non-teaching jobs. All observations are weighted by the probability the individual was working full-time, as estimated based on the nationally representative 2004-05 Teacher Follow-up Survey.

This pattern strongly suggests that any efforts by Florida districts to provide better pay for teachers in the high-demand subjects of math and science are insufficient to offset the differences in outside earnings opportunities across subject areas. Although our evidence is limited to one state, data from other sources suggests that the same is true elsewhere. For example, the Institute for Education Sciences reports that more than one-quarter of math and science teachers who left the profession in 2004-05 said that the opportunity to earn better salary and benefits was a “very important” or “extremely important” consideration, as compared with just 13 percent of teachers in other subject areas.

There is a strong case, then, for modifying teacher compensation systems to offer larger salaries for math and science teachers as a means to improve teacher quality—and student achievement—in these subjects. This is not to say that compensation is the only factor influencing decisions to enter and remain in the profession. Research conducted by the University of Pennsylvania’s Richard Ingersoll, among others, shows that general working conditions, the degree to which teachers have classroom autonomy, and other non-monetary factors are at least as important a consideration as salaries in explaining teacher attrition. Yet ample evidence confirms that salary levels strongly influence on teachers’ career paths.

Unfortunately, while offering higher salaries to teachers in high-demand subject areas might initially appear to be less controversial than other forms of differentiated compensation such as merit pay, public opinion data suggests otherwise. In the 2007 Education Next-Program on Education Policy and Governance survey, my colleagues and I found that just 33 percent of Americans would prefer to offer a larger salary increase to teachers “in subject areas where there are shortages, such as math and science” rather than a smaller salary increase to all teachers. In contrast, a majority supported offering a larger salary increase to “teachers who work in challenging schools.” A recent survey of teachers in Washington state conducted by the University of Washington-Bothell economist Dan Goldhaber similarly found that 59 percent of teachers opposed differentiating teacher compensation by subject area.

My conversations with current and former educators suggest that their reluctance to embrace higher salaries for math and science teachers stems in large part from a concern that this approach would imply that other subject areas are somehow less important – and that the contributions made by teachers in those subjects therefore have less value. This concern is understandable. Yet the outside labor market is a reality that school districts cannot simply ignore. By not allowing teacher compensation to vary with outside earnings opportunities, we implicitly ask individuals with strong math and science skills to make a larger financial sacrifice to enter and remain in the profession. It is students who lose out when too few of them prove willing to do so.

President Obama’s recent 2014 budget proposal includes $80 Million in competitive grants to support programs that recruit and train talented math and science educators for high-need schools and $35 Million to pilot a STEM Master Teacher Corps through which effective teachers in technical fields would be rewarded for taking on new leadership roles. These proposals’ prospects in Congress of course remain uncertain. Even if they are enacted, however, they will face an uphill battle absent broader reforms to teacher compensation systems—reforms that may be encouraged from Washington but will ultimately require action by state legislatures and local school boards. Advocates in these venues seeking to make teacher compensation systems more rational—paying more to teachers with stronger outside earnings potential—will need to develop new strategies to overcome the opposition to the idea from both the public and the profession itself.

[1] Details on the construction of this dataset and our analytic sample are available in Chingos and West (2012), which shows that elementary and middle school teachers who are more effective in raising student achievement earn more in other occupations than do their peers.

[2] To assign teachers to subjects, we first computed the percentage of each teacher’s time spent on instruction in each subject in each year (as a percentage of their total time in academic courses that year) and averaged these percentages over all available years. Teachers spending at least 60 percent of their time in a given subject were assigned to that subject; those who did not were assigned to a “multiple subjects” category. Given the prevalence of “out-of-field” teaching, this method likely introduces error in our measurement of teachers’ true qualifications that would bias the analysis towards a finding of no differences in earnings across subject areas.

[3] We also find that science (but not math) teachers are heavily over-represented among teachers who left for a job elsewhere. Assuming that teachers with the best outside opportunities are more likely to leave for other industries, our results will understate the extent to which science teachers as a whole have better earnings opportunities than other teachers outside of teaching.

Authors

Martin R. West

Common Sense: Using Common Finals to Measure Postsecondary Student Learning

Matthew M. Chingos — Thu, 11 Apr 2013 10:00:00 -0400

College completion rates in the U.S. are stubbornly low despite the large and rising returns to a college degree. Efforts to increase student success in college have largely ignored a potentially key factor: the instruction that students receive in the sequence of courses that add up to a college education. Little evidence exists about how well students learn the material taught in these courses, largely because student performance is assessed using exams developed by instructors and thus cannot be compared to students at other institutions or even in other sections of the same course at the same college.

The lack of direct measures of student learning in higher education severely hampers efforts to measure the quality of instruction delivered in different classrooms. Improving the quality of instruction may represent a promising path to increasing the number of students who earn high-quality degrees by decreasing frustration and failure, and improving the skills of college graduates. But it is nearly impossible to improve instructional quality without being able to measure it.

This report describes a sophisticated set of common final exams implemented in two developmental algebra courses at Glendale Community College in California. These common finals enable instructors and administrators to compare student performance across different sections, and have earned broad faculty support by being implemented in a way that strikes a balance between standardization and the preservation of faculty autonomy.

I show how data from common finals can be used to measure how much students learn in sections of the same course taught by different instructors, and how instructor characteristics such as education and full-time status are related to student mastery of algebra. These results are limited in scope to the two courses at a single institution represented in my data, but serve as a “proof of concept” of the kind of analyses that are made possible by the adoption of common final exams.

I conclude with four policy recommendations aimed at moving forward efforts to assess and improve the quality of postsecondary instruction and ultimately increase the number of students who earn high-quality credentials:

First, more departments at more postsecondary institutions should adopt common final exams in their large, multi-section, introductory courses. The exams should be developed by faculty and reflect a consensus among professors about what students ought to be able to do after completing these introductory courses.
Second, campus administrators should encourage and provide support for these efforts, such as financial support to cover the modest costs of developing and implementing common finals as well as financial incentives to departments that undertake these efforts. Public university systems and higher education associations such as the American Council on Education and the Association of Public and Land-grant Universities could help coordinate efforts across member institutions.
Third, administrators should directly address concerns that common finals will be used to evaluate faculty. Some faculty may worry that test-score data will be used in ways that are unfair, and others may be resistant to any form of evaluation that represents a departure from business as usual. But some faculty may support learning-based measures as an alternative to sole reliance on student course evaluations.
Finally, higher education researchers and practitioners should work to continuously improve common finals. Pre-tests could be developed and administered at the beginning of the semester so that student learning is measured as growth over the course of the semester. Ways to assess student learning in courses other than large, multi-section courses also need to be developed for use in settings such as introductory lecture courses taught by a single instructor.

Editor's Note: The report contains an overview of this issue, and policy recommendations. For the more detailed analysis that led to these recommendations, see the technical paper at the link below.

Read the report » (PDF)
Read the technical paper » (PDF)

Downloads

Authors

Matthew M. Chingos

Ask the Students

Thomas J. Kane — Wed, 10 Apr 2013 11:00:00 -0400

Student surveys are ubiquitous in higher education as a means of evaluating teaching. (In fact, they are often the only source of feedback on classroom instruction for college professors.) But, until recently, they were quite rare in K-12 education. As state and district leaders redesign their teacher evaluation systems, they should consider adding student surveys to the set of measures included in teacher evaluation systems. As we learned in the Gates Foundation’s Measures of Effective Teaching project, student surveys have a number of advantages:

Relationship to student achievement gains: We tested the predictive power of student surveys by comparing a teacher’s score on the Tripod Survey (developed by Ron Ferguson at the Harvard Kennedy School of Government) to their effectiveness in raising test scores with a different group of students or in a different academic year. After adjusting for measurement error, the correlation was between 0.3 and 0.4 in mathematics and 0.1 and 0.3 in English Language Arts. In other words, the teachers who scored higher on the student surveys saw higher achievement gains.
Reliability: The student surveys were the most reliable of the measures we tested (that is, least volatile from year to year), especially in middle school. The reliability of student surveys derives from the power of averaging. Even if an adult is a more discerning evaluator of a teacher’s practice than the typical elementary or middle school student, classroom observations typically average over one or two observers. However, the typical elementary classroom has roughly 20 students and the typical middle school teacher works with 75 to 100 students, spread across multiple sections. In addition, rather than averaging over 2 or 3 lessons, students are present for 180 days.
Improving Practice: Although student achievement gains or “value-added” measures provide predictive power (that is, they help identify teachers likely to see similar student achievement gains with future students), they offer little diagnostic power for identifying specific aspects of a teacher’s practice which deserve attention. In contrast, student surveys, like formal classroom observations, offer the chance to identify areas where a teacher could improve. The power of student surveys and formal classroom observations to drive changes in practice could be enhanced by aligning the language of the surveys with the language of the teaching standards.
Cost and Coverage: Relative to the cost of observations by trained adults, or the cost of adding new assessments in untested grades and subjects, student surveys are a relatively low-cost way of providing additional sources of data for individual teachers. In the MET study, the youngest students we surveyed were in fourth grade and the oldest were in 10^th grade. In these grades, the student surveys could be used to provide additional coverage in subjects such as social science, science, history, art, etc. where student assessments are often available. Future work should investigate the predictive validity and reliability of student surveys in younger grades.
Emotional Salience: One of the potential strengths of student surveys is that they are measured in a currency that teachers inherently value— the perspective of students. A merit pay system attaches financial incentives to other measures—such as classroom observations or student achievement gain measures— to artificially attach value to those measures. However, to the extent that teachers inherently value what their students have to say, and care about whether their students rank them relative to their peers in responding to statements such as “We use time well and we don’t waste time” or “When I turn in homework, I get useful feedback which helps me improve,” then it may not be necessary to attach financial incentives to provoke the desired responses from teachers.

There are only a few places to look for independent sources of feedback on a teacher’s practice. Student achievement gains or “value-added” measures are valuable when they are available, but less than a quarter of teachers work in tested grades and subjects. Classroom observations by principals are another source, but it is costly to add observations by other observers from outside the school. Student surveys are a natural place to turn for an additional source of feedback for teachers. Outside the tested grades and subjects, student surveys may be the only source besides the teacher’s principal. As such, student surveys would be a valuable source for balancing or confirming those judgments.

Of course, we must be mindful that attaching high stakes for teachers to information from student surveys may introduce pressures to distort those measures. After all, some college professors have been known to chase higher student evaluation scores by being easy graders. One of the best ways to reduce this tendency is to use multiple sources of information, and not just one metric, for making important decisions about teachers. Meanwhile, through the MET project, we’ve learned what types of relationships to expect between student survey measures, student achievement gains and observations. States and districts should monitor the relationships among the various measures. If students or teachers begin abusing the student surveys (or another one of the measures), an early warning sign would be the breakdown of those relationships.

The following relevant reports can be found at www.metproject.org:

The Bill & Melinda Gates Foundation, Learning about Teaching: Research Report (Seattle, WA: The Bill & Melinda Gates Foundation, 2010)

Thomas J. Kane and Douglas O. Staiger, Gathering Feedback for Teaching: Research Paper (Seattle, WA: The Bill & Melinda Gates Foundation, 2012)

Thomas J. Kane, Daniel F. McCaffrey, Trey Miller, Douglas O. Staiger, Have We Identified Effective Teachers?: Validating Measures of Effective Teaching Using Random Assignment (Seattle, WA: The Bill & Melinda Gates Foundation, 2013)

Kata Mihaly, Daniel F. McCaffrey, Douglas O. Staiger and J.R. Lockwood, “A Composite Estimator of Effective Teaching” RAND Working Paper, January 8, 2013.

Authors

Thomas J. Kane

Image Source: © Jim Young / Reuters

Ability Grouping, Tracking, and How Schools Work

Tom Loveless — Wed, 03 Apr 2013 11:00:00 -0400

The 2013 Brown Center Report on American Education was released two weeks ago. One of the studies is on ability grouping. A key finding is that elementary teachers are using ability grouping again. Ability grouping is the practice of dividing classes into small instructional groups, especially for teaching reading. According to data collected by the National Assessment of Educational Progress (NAEP), the frequency of ability grouping’s use in fourth grade reading instruction rose about two and a half times, from 28 percent in 1998 to 71 percent in 2009.

This year marks the 30^th anniversary of the publication of How Schools Work by Rebecca Barr and Robert Dreeben, a book in which ability grouping plays an important role. I became aware of the book at the University of Chicago in 1988 as a Ph.D. student. Robert Dreeben was my program advisor and dissertation chair.

Ability grouping is one method by which educators differentiate instruction. The term “differentiation” refers to the many ways that schools try to tailor different learning experiences to children’s varying levels of performance. In the 1980s, I earned a masters degree in special education and taught both learning handicapped and gifted students. Differentiation was in my blood when I arrived at Chicago.

Differentiation was also under fire. Ability grouping and tracking were becoming taboo. The popular research at that time, which was predominantly qualitative and impressionistic, condemned tracking and ability grouping for harming black, Hispanic, and economically disadvantaged students. This literature often depicted teachers as stupid or evil: stupid by robotically following tradition and unwittingly imposing harmful practices on students; evil by harboring race- or class-based prejudices that manifested in low expectations for many kids.

That is what made How Schools Work so refreshing. The book honors teachers in a profound way, not in a “you are all saints and we love you” way, but in a manner much more meaningful—by studying teachers’ work. Barr and Dreeben followed a group of Chicago first grade teachers as they taught reading. A wealth of data was collected so that hypotheses could be tested empirically. In How Schools Work, readers discover that first grade reading groups operate within a grand organizational scheme: groups nested in classrooms, classrooms housed within schools, schools situated within a big urban district. Seemingly routine tasks of teaching are transformed into thoughtful, important activities. Teachers do not appear to be stupid or evil. They appear to be professionals engaged in purposeful activities.

In 1988, “The Formation and Instruction of Ability Groups,” was published in the American Journal of Education. Adam Gamoran, a Chicago graduate student at the time, worked on the project producing this paper. Dreeben and Barr describe as “technological” the ways in which teachers form groups and then instruct them; not technological in the sense of using computers or electronic media but in the sense of applying craft knowledge in the pursuit of an occupational end, in this case, the goal of organizing a classroom full of first graders so that they can be taught how to read.

The notion that teaching is primarily intuitive (“teachers are born not made”) was directly refuted. When they teach reading, teachers must juggle four inputs, each with its own constraints --student aptitude, the difficulty of reading materials, time devoted to instruction, and coverage of curriculum. The combination of these four inputs must be expertly managed to optimize learning. Sure, sometimes teachers have to fly by the seat of their pants while teaching, but for most of time, they employ craft knowledge to attain just the right mix. Kids do in fact learn how to read, and first grade, more than any other grade, is where that wonderful accomplishment can be observed while it happens.

Teachers aren’t perfect. They can make mistakes. They can form groups that are too large, too small, or too unwieldy in composition; move groups too fast or too slow; teach from a curriculum that is too demanding or too easy; or fail to provide enough time for instruction. They can also be unfair – even bigoted – but that’s not the norm.

It is heartening to note that as the use of ability grouping is increasing a new generation of researchers is bringing sophisticated statistical techniques (and open minds) to bear on questions involving both ability grouping and tracking. Tracking, the middle and high school practice of grouping students into separate classes as opposed to grouping students within a class, has always drawn the most scholarly attention. And the most opprobrium.

In a recent NBER working paper, Courtney A. Collins and Li Gan classify Dallas schools as sorted or non-sorted based on the heterogeneity of classes in math or reading achievement. The study also considers heterogeneity in the dispersion of students identified as gifted and talented, limited English speaking, or special education. Sorting is found to produce significantly positive effects in both reading and math -- and for both high and low achievers. The researchers conclude:

This study has valuable policy implications because unlike many school policy variables, the composition of classes can often be changed with little need for increased funds. A school with a fixed number of classrooms and teachers can increase efficiency by rearranging students in the most effective way possible. This study suggests that creating classes with lower levels of dispersion of score or ability level may improve the achievement outcomes for students across the score distribution (Collins and Gan, 2013, page 20).

The study joins a long line of research dating back to at least the 1920s. The overriding concerns have been to determine whether tracking and ability grouping are good or bad (whether they produce positive effects) and whether they are equitable (even if some students benefit, is it at the expense of others). The evidence on these questions is mixed. To adequately summarize the literature would require a series of posts, and I will return to this topic in the future. The main point I would like to make in concluding this post pertains to the renewed popularity of tracking and ability grouping, not to whether either practice is warranted by research.

In the late 1980s and into the 1990s, powerful groups condemned ability grouping and tracking, among them, the National Governors Association, the NAACP Legal Defense Fund, and the Children’s Defense Fund. The use of ability grouping dropped significantly in the 1990s. Tracking in middle schools declined in all subjects but math. According to the NAEP data reported in the Brown Center Report, ability grouping has made a strong comeback in the past decade.

The resurgence of ability grouping accentuates the need for new research questions. If educators are going to use ability grouping again, how should they employ this tool so as to maximize potential benefits and minimize potential harms? How large should groups be? How many groups should a teacher create, and how much time should be spent with each one? Do low achieving groups require more direct instruction than high achieving groups? How often should students be assessed and regrouped? Are different curricula more effective with different groups? Notice the thrust of these inquiries. Such questions are directed towards producing new knowledge on the craft of teaching and to guide teachers in improving their practice, not towards the policy question of whether to group or not to group.

A fine example of this kind of study is provided by Carol McDonald Connor and colleagues at Florida State University. The researchers conducted a randomized field trial of software that organizes first grade reading instruction. The algorithm employed by the software considers each child’s entering skill level and progress made during the school year to recommend several dimensions of instruction, including assignment to small, homogenous ability groups, the amount of time spent on code- versus meaning-focused literacy, and teacher/child versus child-managed delivery. The targets for these recommendations are dynamic; that is, they change in response to periodic assessment of children’s progress. Children in the experimental classrooms gained about two months in reading achievement over those in the control group.

I hope the new generation of researchers will take up more questions like those in the FSU study. The debate over tracking and ability grouping has gone on for nearly a century. Research has not answered the key questions in dispute, at least not to the protagonists’ satisfaction. It’s time for some different questions. How should researchers proceed? A good place to start is reading How Schools Work. It’s just as fresh and illuminating today as when it was published thirty years ago.

Authors

Tom Loveless

Image Source: © Jim Young / Reuters

Do School Districts Matter?

Grover J. "Russ" Whitehurst, Matthew M. Chingos and Michael R. Gallaher — Wed, 27 Mar 2013 00:00:00 -0400

EXECUTIVE SUMMARY

School districts occupy center stage in education reform in the U.S. They manage nearly all public funding and are frequently the locus of federal and state reform initiatives, e.g., instituting meaningful teacher evaluation systems. The most charismatic leaders over the last decade, people such as Michelle Rhee and Joel Klein, have received considerable national media attention. Financial compensation for district leaders is high, with many being paid more than the chief state school officers who oversee the entire systems in which they serve. Some private philanthropies pour money into initiatives to improve district performance. Others invest in ways that suggest that they too think districts are important but as impediments to rather than instruments of reform.

Despite the centrality of school districts in all the ways described, we know very little from existing research about how important they are to student achievement relative to other institutional components for delivering education services, including teachers and schools. Neither do we have information on the size of the differences in effectiveness among districts or whether there are districts that show exceptional patterns of performance across time, e.g., moving from low to high performing.

We begin to fill these information gaps in the present report by analyzing 10 years of data involving all public school students and school districts in Florida and North Carolina. We find that school districts account for only a small portion (1% to 2%) of the total variation in student achievement relative to the contribution of schools, teachers, demographic characteristics of students, and remaining individual differences among students. Within just the institutional components affecting student achievement, the effect of schools is about twice that of districts whereas the effect of teachers is about seven times larger than that of districts.

Even though district effects are only a small piece of the pie that represents all the influences on student achievement, there are still differences among the academic achievement of demographically similar students in higher and lower performing districts in North Carolina and Florida that are large enough to be of practical and policy significance. Combining the data from both states, 4^th and 5^th grade students in a district that is at the 70^th percentile in district effectiveness are more than 9 weeks ahead of similar students in a district at the 30^th percentile of effectiveness in their learning of reading and math. There are also districts that have displayed exceptional patterns of performance in terms of student achievement over the last decade, including districts that beat their demographic odds every year, districts that consistently underperformed, districts that had nose-dive declines, and districts that experienced transformative growth. These findings provide an empirical justification for efforts to improve student achievement through district-level reforms and should be a tantalizing fruit for those who want to better understand why some districts are better than others and translate that knowledge into action.

Download the report » (PDF)
Download the full technical paper » (PDF)

Downloads

Authors

Image Source: Lucy Nicholson / Reuters

How Important Are School Districts? Keynote Address by Michelle Rhee

Wed, 27 Mar 2013 10:30:00 -0400

Event Information

March 27, 2013
10:30 AM - 12:00 PM EDT

Falk Auditorium
Brookings Institution
1775 Massachusetts Avenue NW
Washington, DC 20036

Many popular education reforms focus on improving school districts whereas others, such as charter schools, are premised on school districts being the problem rather than the solution.

On March 27, Russ Whitehurst and Matthew Chingos from the Brown Center on Education Policy at Brookings presented findings from their new study examining the importance of school districts to student achievement. The study found that district effects on student achievement are smaller than the effects of schools and teachers but still large enough to be of practical and policy significance. For example, students in a district that is at the 70th percentile in district effectiveness will be more than 9 weeks ahead of similar students in a district at the 30th percentile of effectiveness on math and reading scores.

After Russ Whitehurst and Matthew Chingos presented their findings Michelle Rhee of Students First took to the podium to share anecdotes of her time as Chancellor of D.C.’s school district from 2007 to 2010.

Rhee and Whitehurst then went on to discuss her experiences as DCPS Chancellor, tackling questions about the role charter schools play in education reform, and what impacts poverty can have on student achievement. Rhee also reflected on how she would tackle her role as Chancellor differently if she were to do it again.

How important are school districts? … School districts occupy a really central place in the American educational system. They manage nearly all of the $500 billion a year of public funds…they are the recipients of federal funds, they are the recipients of state funds. If you follow the money, you’d certainly think that districts were terrifically important. - Grover "Russ" Whitehurst

Students in the best districts in North Carolina by the end of 4th and 5th grades have learned a whole year's worth more than students in the worst performing districts. - Matthew Chingos

The problem is not the kids. The problem is not the teachers. The problem is that the kids and teachers and principals are forced to operate in this incredibly antiquated bureaucracy that is driven by these rules that make absolutely no sense - Michelle Rhee

If you look at social mobility rates in this country, they are actually near the bottom internationally, which means that if you are a child born into poverty in this country, the likelihood that you will ever escape poverty is not good. That to me goes against every ideal we have as a country. That is so un-American.” - Michelle Rhee

Video

Audio

How Important Are School Districts?

Good News about Low-Income Students

Caroline M. Hoxby — Wed, 27 Mar 2013 11:00:00 -0400

In "The Missing One-Offs: The Hidden Supply of Low-Income, High-Achieving Students" (forthcoming in Brookings Papers on Economic Activity[1]), Christopher Avery and I study every student in the high school graduating class of 2008 who scored at the 90^th percentile or above on the SAT or ACT and whose high school GPA was A- or above. These high-achievers, who make up about 4 percent of American students, are well qualified for admission at very selective colleges.[2] We use individual data on millions of students and follow them from high school, through college applications, and onwards as they enroll in college, persist, and graduate.

The evidence gives us five pieces of good news to relate.

1. There are between 8 and 15 times as many high-achieving students from low-income families as college admissions staff at selective colleges thought there were. Admissions staff did not get the number wrong intentionally. They simply believed what they saw among their applicants: at least several high-achieving, high-income students for every high-achieving, low-income student. However, the ratio of high-achieving, high-income students to high-achieving, low-income students in the population is only about two-to-one!

This is good news because it strongly suggests that income is not destiny. Children from low-income families do make it into the ranks of the very high-achieving. Yes, it would be even better if the ratio were one-to-one, meaning that high-income and low-income students were equally likely to be high-achieving. However, the ratio is much better than anyone thought.

2. We show that very selective colleges offer high-achieving, low-income students such generous financial aid that they could attend these colleges and pay less than they currently pay to attend the non-selective colleges in which they often enroll. The very selective colleges offer them resources that are more generous by an order of magnitude and ask them to pay less. It is not every day that someone offers you ten times as much for a lower price.

This is good news and not just for the low-income students who currently enjoy generous financial aid. It means that very selective colleges are deeply committed to giving all academically qualified students an opportunity to get one of the most resource-intensive educations in the world.

3. We find that when high-achieving, low-income students apply to very selective colleges, they are admitted and enroll and graduate at the same rates as high-income students with the same level of achievement. Specifically, we find that if a low-income and high-income student with the same achievement apply to the same college, they thereafter have outcomes (matriculation, persistence, on-time graduation) that are so similar that they cannot be distinguished statistically.

This is good news because it suggests that low-income students take full advantage of very selective colleges' educational resources once they apply. This evidence should also put an end to the myth that low-income students usually fail to thrive at very selective colleges because they differ from the modal student. Believers in this myth routinely regale us with anecdotes about low-income students who struggle unduly at selective colleges. These anecdotes must be unrepresentative because, remember, we examine all high-achieving, low-income students in our study. Myth-believers will simply have to come around and accept the fact that low-income, high-achieving students succeed at top colleges to about the same extent as their higher-income counterparts. Again, income is not destiny.

4. Very selective colleges work hard to find and recruit low-income, high-achieving students using an array of methods: visiting high schools, inviting students to visit campuses (and providing financial aid to help them do so), coordinating with college mentoring programs, even creating their own college mentoring programs. We show that all this effort has borne fruit. Specifically, we demonstrate that colleges successfully recruit the high-achieving, low-income students of whom they are aware. Indeed, high schools that have a critical mass of low-income, high-achieving students are so often visited that they are "fully tapped." Similarly, high-achieving, low-income students who live near a very selective college tend already to enroll. The data also suggest that high school counselors who advise numerous low-income, high-achieving students are fairly expert advisors about college: their students exhibit college application behavior that is about as sophisticated as that of high-income, high-achieving students.

This is good news because it indicates no one is being lax or deliberately ignoring low-income, high-achieving students. Nevertheless, one might ask why, if everyone is trying so hard, are the vast majority of low-income, high-achieving students not applying to very selective colleges? We show that they are so dispersed across high schools and localities that there is no cost-effective way for colleges to find them using the traditional methods listed above. The students' dispersion is also the reason why their counselors likely do not develop expertise about very selective colleges. Suppose a counselor has 350 advisees (the normal number in the U.S.) and only encounters a strong candidate for very selective colleges once a year or every few years. It is likely that the counselor will develop skills to help her other 349 advisees--many of whom may be struggling to stay in school or attend any postsecondary institution--rather than skills that will help the rare "one-off."

5. Our study would not exist if it were impossible to identify low-income, high-achieving students. Put another way, it cannot both be true that we put together the data for our study and that it is impossible to use such data to develop novel tools to inform students about their college-going opportunities.

This is good news because it means that we must be on the verge of inventing tools that could make all American students understand their educational opportunities more clearly. In truth, we are not "on the verge": we have already made progress in inventing the tools. We wrote most of "The Missing Students" a few years ago and then kept the study under wraps until we had tested some tools. Stay tuned!

"The Missing Students" reveals something like buried treasure. The U.S. has a trove of low-income students who are extremely well prepared for college and who have been little-recognized up to now. We know that they could afford very selective colleges because they are already paying more to attend non-selective institutions. We see no evidence to suggest that they would fail if they were to attend very selective colleges: The low-income, high-achieving students who do attend persist and graduate at the same rates as their high-achieving counterparts. (The current study does not actually demonstrate that the missing students would do just as well as the students who have already "been found," but we test that hypothesis in forthcoming work.) We also see no evidence to suggest that colleges and counselors were lax. They used the tools at their disposal. Today, we can invent new tools, and we know that it would be worthwhile trying to do so.

I have been amazed that some commentators on "The Missing Students" have managed to spin the study so heavily that it comes off sounding like bad news. To me, that is like a person complaining about discovering buried treasure because it means that he will have to buy a shovel and do a little digging.

I am optimistic that we can learn how to inform students who are already prepared to rise above their economic circumstances. This is a manageable task yet one that could potentially improve economic mobility in the U.S. by a substantial amount. After all, it is not just the missing students themselves who might benefit from resource-rich college education. Their success might inspire other low-income students to achieve while in high school. They might serve as a conduit of information about colleges and high-skill careers for the schools and neighborhoods in which they grew up. It seems likely that the low-income students who succeed in becoming high achievers have insights into their schools and neighborhoods that would make them exceptionally good at improving the prospects of other low-income children.

[1] http://www.brookings.edu/research/interactives/2013/low-income-high-achieving-hoxby-avery

[2] Only 4 percent of students meet these criteria because not all students take a college assessment exam. Recent studies of universal SAT or ACT testing suggest that very few students who would meet these criteria do not, in fact, take one of the tests. The students who meet our criteria are well qualified for very selective institutions because they have test scores and grades that match those of students who enroll in colleges that Barron's classifies as "Most Competitive" or "Highly Competitive Plus" in its Profiles of American Colleges 2008.

Authors

Caroline M. Hoxby

Image Source: © Brian Snyder / Reuters

Is Tracking and Ability Grouping Making a Comeback?

Tom Loveless — Wed, 20 Mar 2013 15:00:00 -0400

On Monday, we released the annual 2013 Brown Center Report on American Education. The report contains three individual studies: one on the latest in international testing progress, one on tracking and ability grouping, and one on advanced math in eighth grade.

Below is a first look at a video we put together in which I discuss the details of Part II, on the resurgence of ability grouping and tracking.

Download Media

The Resurgence of Ability Grouping and Tracking: A Return to Controversial Practices?

Feel free to submit your thoughts and comments below.

Video

The Resurgence of Ability Grouping and Tracking: A Return to Controversial Practices?

Authors

Tom Loveless

2013 Brown Center Report on American Education: How Well Are American Students Learning?

Tom Loveless — Mon, 18 Mar 2013 00:00:00 -0400

Editors' Note: The introduction to the 2013 Brown Center Report on American Education appears below. Use the Table of Contents to navigate through the report online or download a PDF of the full report.

Table of Contents

PART I: The Latest TIMSS and PIRLS Scores
PART II: The Resurgence of Ability Grouping and Persistence of Tracking
PART III: Advanced Math in Eighth Grade

INTRODUCTION

This is the twelfth edition of the Brown Center Report. The structure of the report remains the same from year to year. Part I examines the latest data from state, national, or international assessments. This year the focus is on the latest results from the Progress in International Reading Literacy Study (PIRLS) and Trends in International Math and Science Study (TIMSS) released in December, 2012. The U.S. did relatively well, posting gains in reading, math, and science. Finland made headlines by registering declines from the last time it took the TIMSS math tests. At both fourth and eighth grades, the scores of Finland and the U.S. are now statistically indistinguishable in math. Part I also looks at the so-called “A+ countries,” named that because they were the top nations on the first TIMSS, given in 1995. Part I offers “A Progress Report on the A+ Countries,” and finds that, surprisingly, three of the six have registered statistically significant declines since 1995. Despite that, most of the A+ countries still score among the world’s leaders. The exception is the Czech Republic, which scored at approximately the international average the last time it took TIMSS in 2007.

Part II explores a perennial theme in education studies—the topics that never seem to go away in terms of research and debate. This year it’s on the controversial topics of tracking and ability grouping. An analysis of data from the National Assessment of Educational Progress (NAEP) documents a resurgence of ability grouping in fourth grade reading and mathematics. Tracking remains persistent in eighth-grade math, with about three-fourths of students in tracked classes. As readers are surely aware, both practices have been attacked for decades as inequitable, and many school analysts thought their use had diminished. Ability grouping was dominant for a long time in the elementary grades. Reading groups were the norm through most of the twentieth century and then declined dramatically in the 1990s. They are now coming back—and back strongly.

Part III is on a prominent policy or program. This year’s analysis is on the national push for eighth graders to take algebra and other high school math courses. Algebra is now the single most popular math course in eighth grade. The study in Part III uses state variation in enrollment rates to ask the question: what has happened to the NAEP scores of states that boosted their eighth-grade advanced-math enrollments? The study uncovers no relationship between change in state NAEP scores and change in enrollments. States boosting advanced math taking are no more likely to show NAEP gains than other states.

A second analysis uncovers some evidence consistent with the idea that advanced math courses are being “watered down,” that the mean achievement levels of advanced courses fall as enrollments go up. Again, change in NAEP score is the outcome of interest. The study shows that states that are more selective in math placements—not aggressively accelerating eighth graders into advanced courses—are more likely to show achievement gains in those courses.

There is one intriguing divergence from this finding: eighth-grade geometry classes. Geometry sits at the peak of the hierarchy of eighth-grade math courses, enrolling the nation’s best math students (about 5%). Presumably, these are students who took algebra in seventh grade. Increases in eighth-grade geometry enrollments evidence no association with changes in mean achievement for the course, not what one would expect if unprepared students were being accelerated into the course. This suggests that schools are implementing two different types of acceleration, one based on the age or grade of students, the other based on students’ preparation and readiness for advanced work. The analyses in the study are only correlational and cannot confirm or reject causality. Part III concludes with a discussion of hypotheses for future study to improve both strategies.

Download Media

Part I: The Latest TIMSS and PIRLS Scores »

Downloads

2013 Brown Center Report on American Education

Video

Authors

Tom Loveless

Advanced Math in Eighth Grade

Tom Loveless — Mon, 18 Mar 2013 00:00:00 -0400

As recently as 1990, taking algebra in eighth grade was unique. That has changed dramatically in recent years, and now more eighth graders take algebra than any other math class. Enrollment in eighth-grade algebra—and in other advanced math classes—varies by state. This section of the Brown Center Report exploits that variation to study the relationship of states’ enrollment in advanced math classes and scores on NAEP. The research question is whether a relationship exists between changes in advanced math enrollments and changes in 8th grade NAEP scores. Do states that boost advanced enrollments experience a concurrent increase in achievement? A second analysis uses the same technique to look at the potential that advanced courses are being “watered down.” Are rising enrollments associated with lower mean achievement in advanced classes?

Background

In 1982 Robert Moses was awarded a MacArthur Fellowship. He used the money to start The Algebra Project, a community-based effort to bring algebra to historically underserved middle school students—primarily, children from low income households and students of color. Moses called algebra “the new civil right,” an invocation of equity that cast course taking in a new light.³² The Clinton Administration tied the equity theme to international competitiveness and pushed for more students to take algebra before high school. “Around the world, middle students are learning algebra and geometry,” President Clinton observed. “Here at home just a quarter of all students take algebra before high school.”³³

Algebra soon came to be known as a “gatekeeper” course, a class standing like a sentry at the gateway to college. Take it and pass it and your odds of attending college were good. Take it and fail it and at least you had been exposed to challenging mathematics. Don’t take it at all and your chances of attending college were near zero. Algebra’s place in the typical high school math sequence enhanced its importance. Assume that college-going students should get some calculus under their belts in the senior year. In most high schools, a student who takes Algebra I in ninth grade has three remaining years to take Algebra II, Geometry, Pre-Calc/Trigonometry, and then Calculus. That’s four courses. Something has to give. Many schools change the order of the courses, and some mix in statistics with one of the year’s offerings, but the fact remains: if taking Calculus as a senior in high school is the goal, taking Algebra I in ninth grade means there are four courses to complete in three years. Taking algebra in eighth grade opens up an additional year for advanced math.

Equity, international competiveness, and practical concerns about course sequences converged in the mid 2000s to boost the campaign for eighth-grade algebra. An “algebra for all” movement emerged that pushed universal, mandatory eighth-grade algebra. Minnesota established a new high school graduation requirement that, beginning with the class of 2015, all students must complete an Algebra I credit by the end of eighth grade. California used its school accountability formula to promote eighth-grade algebra, offering a choice of two eighth-grade math assessments (algebra and general eighth-grade math) but then, in the formula for calculating Academic Performance Index (API), discounting the performance level of students taking the general math test (for example, downgrading to “basic” those students who took the test and scored “proficient”). That incentive motivated schools to dramatically increase eighth-grade algebra enrollments, and although the AYP rule was later tossed out by the courts, California ranks as the top state in the nation for eighth-grade algebra and advanced math enrollments.³⁴

NAEP Data on Advanced Math Enrollment

Table 3-1 illustrates the steady increase of U.S. eighth-grade enrollment in advanced mathematics courses. The data are taken from the NAEP eighth-grade math assessment. Students are asked: “what mathematics class are you taking this year?” The category “advanced mathematics” combines several responses, including Algebra I, courses that stretch Algebra I content over two years (whether it’s the first or second year of such a course), and courses that typically are more advanced than Algebra I, including Algebra II and Geometry. This amalgamated response is noisy and receives further discussion below.

In 1990, only 16% enrolled in an algebra course, compared to 20% in pre-algebra and 61% in 8th grade math. In this paper, the latter two courses are referred to as “basic.”
By 2011, nearly half (47%) of all eighth graders took algebra or a more advanced course. Only 48% were in a basic math course, down from 81% in 1990. The advanced math percentage may be understated in Table 3-1 for the years prior to 2000 as that was the first time geometry, advanced algebra, and algebra stretch classes were response categories in the NAEP questionnaire for eighth graders.³⁵ Moreover, some students—both then and now—may mistakenly believe they are in an algebra or geometry class when in fact they are not. Notwithstanding these data limitations, advanced math enrollments clearly rose substantially from 1990 to 2011.³⁶

More and more students are taking advanced math classes earlier and earlier. Is this a good idea?

Research on the Efficacy of Eighth-grade Algebra

The National Educational Longitudinal Study of 1988 (NELS) offers researchers a trove of information collected from a randomized sample of students. Several studies have used NELS data to investigate what happens when students take advanced math early in an academic career, whether eighth or ninth grade.³⁷ Researchers found gains for students taking algebra earlier rather than later, including—and this is important for the equity goal—low performing students. A recent meta-analysis of research on the topic (by Mary K. Stein and colleagues) reaffirmed that positive finding, with the caveat that “achievement gains occurred in settings where policies were accompanied by strong supports for struggling students, particularly more time for algebra instruction. We do not have strong evidence that universal algebra policies lead to achievement gains minus those strong supports.”³⁸

More recent evaluations of policies expanding algebra enrollment have raised cautionary flags. Chicago mandated that all ninth graders take what had been regarded as college preparatory classes, including algebra. Evaluators followed students for several years and concluded, “Although more students completed ninth grade with credits in algebra and English I, failure rates increased, grades slightly declined, test scores did not improve, and students were no more likely to enter college.”³⁹ Studies of California’s algebra policies found a trade-off: rising enrollments but also a rising number of failures. In North Carolina, researchers from Duke uncovered negative results after studying a Charlotte-Mecklenburg initiative to expand algebra in eighth grade: lower scores on the Algebra I test and then lower pass rates in Geometry and Algebra II in subsequent years.

Why have the more recent studies produced bleaker findings than suggested by the earlier work? The Duke researchers believe selection bias skewed the earlier findings. Stronger math students take algebra in eighth grade, and although they indeed may benefit academically from the course, that does not mean that weaker students will also benefit from taking algebra earlier. “Once this selection bias is eliminated, the remaining causal effect of accelerating the conventional first course of algebra into earlier grades, in the absence of other changes in the math curriculum, is for most students decidedly harmful.”⁴⁰

The Stein et al. meta-analysis and the Duke team’s policy recommendations, although different in emphasis, do share a small patch of common ground. Stein et al. say that without “strong supports” achievement gains cannot be expected. And the Duke researchers foresee harmful effects “in the absence of other changes in the math curriculum.” One is contingently positive, the other contingently negative. The common ground that they share is in forecasting the potential for a neutral effect.

Let’s return to NAEP and see what its data have to say about state efforts to encourage enrollment in advanced math courses in eighth grade.

Analytical Method

Are eighth-grade enrollments in advanced math related to states’ math scores on NAEP? To answer this question, an obvious first step is simply to examine the list of states, their NAEP scores, and the percentage of each state’s students taking algebra, geometry, and other advanced math courses in eighth grade. There is no clear relationship. In 2011, the correlation between states’ advanced math enrollments and NAEP achievement is 0.07, indistinguishable from 0.00. States with more eighth graders taking advanced math classes are no more likely to register a higher NAEP score in math than states with lower enrollments in those classes.

This kind of cross-sectional analysis is a reasonable place to start, but it’s limited to revealing correlations between variables at a single point in time. That can be misleading. A study in the 2007 Brown Center Report, for example, showed how the number of instructional minutes that nations devote to math instruction is unrelated, on a cross-sectional basis, to national math achievement. In 1995, the correlation was 0.05. In 2003, the correlation was -0.20. Neither statistic is significantly different from 0.00. But when nations are examined longitudinally, and data from the two cross-sections are modeled as change variables, the question under scrutiny is shifted to whether national changes in instructional minutes from 1995 to 2003 are related to changes in test scores over the same time period. The correlation for that relationship is 0.42, which is statistically significant. Countries that increased the amount of time devoted to math instruction tended to experience a rise in TIMSS math scores; those countries that decreased the time devoted to math instruction tended to see their scores fall.

Why is the analysis of change variables beneficial? Two reasons. The first is that the technique helps to control for bias introduced by omitted variables (including selection), a shortfall plaguing cross-sectional analyses of achievement. In the case of instructional minutes, for example, school systems might strategically decide to place low achieving students in longer classes to help them catch up. That would make it appear that more instruction is associated with lower achievement. Assuming that omitted variable bias is present at both the beginning and end points of the time interval under study—and the relationship to the dependent variable (the outcome of interest) remains consistent over the interval—such bias washes out in the calculation of change (see Gustaffson, 2007, for further explanation and applications to other educational questions).⁴¹

The second benefit of this approach is that it poses a question paramount to policy analysis. Considering whether to adopt policy X leads to the question: if we adopt policy X, what is the expected change in outcome Y? What will happen? The cross-sectional question is this: what is the relationship of policy X to outcome Y at one point in time? One often hears of cross-sectional analyses showing something along the lines of “a one-standard deviation change in X would result in the following change in Y,” but the prediction is only inferred, there being no observations of change (or data from different time periods) in the data set.

Analysis of Change Using NAEP Scores

The relationship between change in policy and change in outcome is the subject of the analysis below. The time period examined is 2005 to 2011. Be aware, notwithstanding the improvement over cross-sectional analysis, that the analysis is still only correlational and thus confined to generating plausible hypotheses for more rigorous research designs. No causality is asserted here.

Table 3-2 shows the tail end of the long term trend sketched in Table 3-1—enrollment gains in advanced math classes and declines in basic classes. The slow, steady national trend masks considerable variation among the states. In 2005-2011, the average state increase in advanced math enrollments (as a proportion of eighth graders) was 5.5%, with a standard deviation of 8.4%. The top four states that boosted advanced enrollments were Minnesota (35%), and Pennsylvania, Virginia, and Washington (all with 17%). In contrast, two states stand out for going against the national trend with shrinking advanced math enrollments: Nevada (-22%) and Georgia (-17%).

In terms of specific courses, forty-five states boosted enrollments in Algebra I, while only three states shrank enrollments and three stayed the same (in this discussion of NAEP scores, the District of Columbia is considered a state). Twenty-eight states decreased enrollments in general math, twenty increased, and three stayed the same. In general, course enrollments behave like a tube of toothpaste—squeeze on one end and the other end bulges. States with rising advanced math enrollments experienced shrinking enrollments in basic courses. And vice versa. The two states singled out for declining enrollments in advanced math courses illustrate the point. Their basic math enrollments rose. Nevada’s pre-algebra enrollments jumped 27%. Georgia’s percentage of students in general math rose 33%.

Is there a relationship between states’ change in course enrollments and change in NAEP scores? Did states experience gains on NAEP concurrent with increases in eighth graders taking advanced math? A series of correlation coefficients were computed to investigate these questions (see Table 3-3). The first model examines the relationship of advanced math enrollments and NAEP composite scores. The correlation coefficient (r = -0.01) is statistically indistinguishable from 0.00.

The NAEP composite score may assess mathematics too broadly to pick up the effects of emphasizing advanced math, which primarily involves boosting algebra. Fortunately, NAEP reports scores on specific content areas assessed within the test (called “strands”), including algebra and geometry. So the second model uses the NAEP subscore for the algebra strand as the achievement variable, which should be more sensitive to increased knowledge of algebra. Again, no significant relationship is found.

The third and fourth models use change in Algebra I enrollments as the course variable, instead of advanced math, in case aggregating several courses into the “advanced” category has muddied the waters. The change in composite NAEP score serves as the achievement variable in the third model and the change in the algebra strand score as the achievement variable in the fourth model. Neither correlation attains statistical significance.

Models five and six repeat the same treatment with geometry. Change in geometry course taking in eighth grade is used as the course variable—and the models calculate whether it is correlated with change in the NAEP composite in model five and change in geometry score in model six. Neither correlation is statistically significant.

In addition to the correlations reported here, multivariate regressions were run with three covariates controlled (also variables representing change)—change in state rates of child poverty, English language learners, and black and Hispanic students—demographic characteristics that are known correlates of state NAEP scores. The Great Recession unfolded during the time period under study, and some states, for example, witnessed growing rates of child poverty more than other states. If states experienced demographic changes, that could skew the results. It turned out not to be the case. None of the regression models were statistically significant.

In sum, no evidence was found in NAEP scores of a relationship between states raising enrollment in advanced math courses and raising achievement. States that increased the percentage of students taking algebra or geometry in eighth grade were no more likely to post NAEP gains than states with decreased enrollments in those two courses.

Do Rising Enrollments Water Down Advanced Math Courses?

Whether advanced math courses are watered down because of increasing enrollments is an important question. The notion is that filling advanced classes with academically weaker students than in the past could diminish the amount of learning that the courses are able to impart. That could help to explain the neutral correlations reported above. It could also help to explain the neutral—or even negative effects—revealed by recent evaluations of policies promoting universal algebra in eighth and ninth grades. NAEP data can only go so far in indicating whether watering down is taking place, but they do offer interesting insights into how course-shifting and achievement may be related.

Table 3-4 reports correlations between enrollment change and change in the mean achievement of students taking each course. Data from four courses are displayed. Again, the percentage of a state’s eighth graders taking each course serves as the enrollment variable. The courses are arranged hierarchically. Geometry is typically offered for the most advanced students and general math for the weakest ones. Three correlations are statistically significant.

Is there evidence of watering down? Yes, but not in all advanced courses. Let’s start with the results supporting the watering down hypothesis. Increases in Algebra I enrollments are negatively associated with achievement gains (r = -o.34, p < .05). Let’s be clear what that means. The average state registered a 5.6 NAEP scale score gain among its Algebra I students. The NAEP scores for students in Algebra I classes did not go up as much in states that raised enrollments in Algebra I (+5.2) as in states that either held enrollments constant or decreased them (+9.2). For Pre-Algebra, rising enrollments are also negatively associated with test scores (r = -0.34, p < .05). Both correlations are consistent with the watering down hypothesis if students who would otherwise be placed in lower courses are migrating upward to higher courses. We cannot tell whether that is happening using NAEP data. And, to issue an important warning once again, correlations do not prove causality.

The strongest correlation involves General Math (r = 0.47, p < .01). The positive association is also consistent with the watering down hypothesis. If the overall trend is to move students into upper-level courses—and schools are selective in the students they accelerate—General Math courses, as they shrink, should be increasingly dominated by the students who struggle the most at math. These courses presumably would have lost their best students. Falling enrollments would therefore be associated with falling scores. General Math classes that manage to keep the students who are being accelerated elsewhere would, comparatively, register higher scores.

Geometry complicates matters. Its correlation coefficient (0.27) is inconsistent with the watering down story. Geometry sits at the top of the course hierarchy. Any indiscriminate acceleration of students upward (an inextricable assumption of the watering down argument) should ultimately result in a negative association of enrollment gains and achievement scores in the course at the top. And yet, Geometry’s correlation coefficient has a positive sign and approaches statistical significance. Although statistically indistinguishable from 0.00 (p = .11), that could be due in part to the reduced number of states with data. Only thirty-six states have sufficient numbers of eighth-grade geometry students to produce a NAEP score.

Another possibility involves the noisy NAEP course variables. Perhaps more “real” geometry students are included in the NAEP course category for geometry in 2011 than in 2005—in other words, a larger proportion who are actually in a geometry class and not mistaken about their math course. As indicated in Table 3-2 above, only 5% of eighth graders were enrolled in Geometry in 2011, an increase from 4% in 2005. The mean NAEP score for geometry students was 290 in 2005 and 308 in 2011, a sharp increase of 18 points. The one-percentage-point gain in students seems to have packed a punch in terms of NAEP scores. The “real” geometry students probably took Algebra I in seventh grade. Much like algebra for eighth graders three or four decades ago, geometry is reserved for today’s very best math students.

Discussion

This study analyzed variation in state enrollment patterns to test whether rising enrollments in advanced eighth-grade math courses are correlated with achievement gains on NAEP. No evidence was found that they are. States with rising percentages of eighth graders taking Algebra I, Geometry, and other advanced math classes were no more likely to raise their NAEP scores from 2005-2011 than states with declining percentages of eighth graders in those courses.

A second analysis, again looking at changes in policy and test scores over time, investigated whether boosting the percentage of students in higher level courses is associated with decreases in the mean scores of those courses—suggesting a watering down effect. The evidence is consistent with watering down in all but one course. Negative correlations were found for Algebra I and Pre-Algebra. In those courses, mean achievement gains declined as enrollments increased. Achievement gains in general math courses were positively associated with enrollment changes. All three of these correlations are statistically significant and supportive of the watering down hypothesis.

Geometry diverges from the other courses. A positive association was found that, although statistically indistinguishable from 0.00, suggests at least a neutral relationship between rising enrollment and changes in NAEP scores. If schools were indiscriminately accelerating students into eighth-grade geometry, one would expect a negative correlation.

None of these findings can confirm or reject causality, but they are useful in generating hypotheses for future study. They also shed light on the findings from previous research. For example, a key finding from evaluations of California’s algebra policy is that universal algebra produces trade-offs. Many students benefit from the extra challenge. Rates of algebra enrollment for historically under-enrolled populations (in particular, low SES students) have increased. The raw number of students passing end of course exams has also increased. But the downside is that the number of students failing algebra goes up as well; and the failing students, too, are disproportionately low SES students.⁴² One study from California suggests that many of the failing students would have been better off spending an additional year preparing for algebra instead of taking it.⁴³ These kinds of trade-offs, when aggregated to the state level, could produce a neutral net effect.

The analysis of whether accelerating students into advanced classes is watering down achievement points to two different types of acceleration. One is selective and decided on an individual basis. Each student’s math skills are evaluated and a determination is made whether a more advanced math course is appropriate or not. That kind of acceleration appears to be occurring in eighth-grade geometry—and presumably in seventh-grade algebra. Students who would benefit from a more rigorous course are promoted. Mean test scores for eighth-grade geometry rise, or at least stay the same, despite rising enrollments.

The second type of acceleration is non-selective and group based. Students are advanced based on a characteristic independent of prior achievement or preparedness (e.g., grade level or age). Future research should compare these two types of acceleration and investigate who, when it comes to selective acceleration, should be accelerated and when. With age- or grade-based acceleration, a set of early indicators is needed (the universal algebra approach) that would identify students needing support and the type of support most beneficial for them. If the trade-offs of group acceleration are indeed real, then the policy goal should be to minimize negative effects and maximize benefits.

A final note on the Common Core. No one knows how gifted students’ needs will be met in the Common Core Era. Taking algebra in eighth grade is the new normal, and taking algebra in the seventh grade is rapidly becoming the new normal for gifted math students. In California, 8.1% of seventh graders (nearly 38,000 students) took the algebra end of course exam in 2012. If Common Core means the same curriculum for all, a time will surely come when exceptional math students need an uncommon curriculum that is appropriate for them.

« Part II: The Resurgence of Ability and Persistence of Tracking

Part III Notes

32. Background information on the Algebra Project is available at www.algebra.org.

33. Remarks by President Clinton, Education Roundtable, Springbrook High School, Silver Spring, MD, March 16, 1998. Available at http://www.gpo.gov/fdsys/pkg/WCPD-1998-03-23/pdf/WCPD-1998-03-23.pdf.

34. History of California’s algebra policy can be found in: Algebra Policy in California: Great Expectations and Serious Challenges (Oakland: EdSource, May 2009). Also see Tom Loveless, The Misplaced Math Student: Lost in Eighth-Grade Algebra (Washington, DC: The Brookings Institution, 2008).

35. The category “other” received about a 3% response rate before 2000 so the number of students taking more advanced classes was probably very small.

36. Jill Walston and Jill Carlivati McCarroll, Eighth-Grade Algebra: Findings from the Eighth-Grade Round of the Early Childhood Longitudinal Study, Kindergarten Class of 1998-1999 (ECLS-K) (Washington, DC: National Center for Education Statistics, October 2010).

37. See David Stevenson, Kathryn S. Schiller, and Barbara Schneider, “Sequences of Opportunities for Learning,” Sociology of Education 67, no. 3 (1994): 184-198; Adam Gamoran and Eileen C. Hannigan, “Algebra for Everyone? Benefits of College-Preparatory Mathematics for Students with Diverse Abilities in Early Secondary School,” Educational Evaluation and Policy Analysis 22, no. 3 (2000): 241-254; Julia Smith, “Does an Extra Year Make Any Difference? The Impact of Early Access to Algebra on Longterm Gains in Mathematics Achievement,” Educational Evaluation and Policy Analysis 18 (1996): 141-153.

38. See Mary Stein, Julia Kaufman, Milan Sherman, and Amy Hillen, “Algebra: A Challenge at the Crossroads of Policy and Practice,” Review of Educational Research 81, no. 4 (2011): 453-492.

39. Elaine Allensworth, Takako Nomi, Nicholas Montgomery, and Valerie E. Lee, “College Preparatory Curriculum for All: Academic Consequences of Requiring Algebra and English I for Ninth Graders in Chicago,” Educational Evaluation and Policy Analysis 31, no. 4 (2009): 367-391.

40. Charles T. Clotfelter, Helen F. Ladd, and Jacob L. Vigdor, The Aftermath of Accelerating Algebra Evidence from a District Policy Initiative (Washington, DC: National Center for Analysis of Longitudinal Data in Education Research, American Institutes for Research, 2012).

41. Jan-Eric Gustafsson, “Understanding Causal Influences on Educational Achievement through Analysis of Differences over Time within Countries,” in Lessons Learned: What International Assessments Tell Us about Math Achievement, ed. Tom Loveless (Washington: Brookings Institution Press, 2007).

42. Trish Williams, Edward Haertel, and Michael W. Kirst, Improving Middle Grades Math Performance: A Closer Look at District and School Policies and Practices, Course Placements, and Student Outcomes in California. Follow-Up Analysis (Mountain View: EdSource, 2011).

43. Jian-Hua Liang, Paul Heckman, and Jamal Abedi, “What Do the California Standards Test Results Reveal About the Movement Towards Eighth-Grade Algebra for All?” Educational Evaluation and Policy Analysis 34, no. 3 (2012): 328-343.

The Latest TIMSS and PIRLS Scores

Tom Loveless — Mon, 18 Mar 2013 00:00:00 -0400

In December 2012, the latest international test scores were released. The Trends in International Math and Science Study (TIMSS) is given every four years, and the Progress in International Reading Literacy Study (PIRLS) is given every five years. The latest results came from the 2011 administration of both tests, a unique event. Because of their asynchronous schedules, the two tests share the same year only once every twenty years. Forty-nine nations and nine benchmarking participants took part in PIRLS, which is given at fourth grade, and 63 nations and 14 benchmarking participants took part in TIMSS, which is given at both fourth and eighth grades.¹

U.S. National Achievement

The U.S. did reasonably well in all three subjects—reading, math, and science. In reading, the U.S. scored 556 on the international scale. All of the tests discussed in this section have a mean of 500 and a standard deviation of 100. Only four countries scored statistically significantly higher on the reading test. (In the discussion below, the term “significant” is used as shorthand for statistical significance at p<.05). Hong Kong led the world at 571, followed by the Russian Federation (568), Finland (568), and Singapore (567). The U.S. score for 2011 represented a 14-point gain since 2001 (significant).

In math, U.S. fourth graders scored 541, near the middle of second-tier countries on TIMSS. The top-tier countries were five Asian nations: Singapore (606), Korea (605), Hong Kong (602), Chinese Taipei (591), and Japan (585). The U.S. fourth-graders’ score represents a 23-point gain since 1995 (significant). Eighth graders in the U.S. scored 509, which is significantly higher than the 500 international average—but just barely. The 509 score is a 17-point improvement over the 1995 U.S. score (a significant gain).

In science, U.S. fourth graders scored 544, with six countries scoring at significantly higher levels. The fourth-grade gain of 2 points since 1995 is not statistically significant. Eighth graders scored 525, significantly above the international average and significantly below students from eight other nations. The 12-point gain since 1995 is statistically significant.

To sum up, the latest international scores are mostly positive for the U.S. American students scored above the international average on all five assessments of grade-subject pairings. For four out of the five tests, the gains since 1995 are statistically significant. Despite these encouraging results, there is much room for improvement. Over the past decade, countries joining TIMSS have been economically developing nations or, in the case of the Middle East, nations possessing abundant national wealth but lacking a tradition of public schooling. Such compositional changes can make international averages easier to surpass. Leading the world in reading, math, or science remains a challenge for the U.S.

State Achievement on TIMSS

Nine states took part in the TIMSS assessment (see Table 1-1). Let’s focus on eighth-grade mathematics as that is the only test on which all nine participated. As points of reference, be reminded that the international average for the test was 500, the U.S. national score was 509, and the top scoring nation was Korea at 613.

Massachusetts led the pack with a 561 score, followed by Minnesota (545) and North Carolina (537). Five of the states had taken TIMSS before, and three registered statistically significant gains from the first time they participated. As indicated in Table 1-2, the TIMSS scores map reasonably well onto NAEP scores. Because NAEP was also given in 2011, the National Center for Education Statistics was able to conduct a NAEP-TIMSS linking study.² Items from TIMSS and NAEP were embedded in the same booklets so that items from both tests were taken by the same student at the same time. Results of the study will be released later in 2013. The hope is that future analysts will be able to calculate, with reasonable precision, projected state TIMSS scores based on NAEP scores, allowing local leaders to place state performance in an international context.³

Finland

Finland generated headlines from TIMSS. The “Finnish Miracle” story was called into question. In recent years, the popular press has been filled with stories about Finland’s wonderful education system. Educational tourism took many observers to Finland to see schools firsthand. Tales abounded of no homework, no high stakes tests, no tardy bells, a short school day, and the national belief that requiring children to start school before age seven violates “children’s right to be children.”⁴ Visitors marveled at the relaxed, home-like atmosphere—fireplaces in lounges, kids going shoeless, teachers called by their first names.⁵ The current worldwide angst (especially evident in the U.S. and Great Britain) over achievement, productivity, and rising test scores pursued through reforms such as school choice and accountability furnishes such a stark contrast that it has even drawn a derogatory acronym—GERM—from a Finnish scholar. That stands for Global Educational Reform Movement.⁶

One problem. Finland’s reputation is based largely on its performance on PISA, a very different test from TIMSS. The gap between the U.S. and Finland on PISA is statistically significant in mathematics literacy. On the 2011 TIMSS, however, Finland and the U.S. had statistically indistinguishable scores in both fourth and eighth-grade mathematics.

Look again at Table 1-1. Finland’s score of 514 in eighth-grade mathematics places it near the middle of the list of states. The scores of Alabama and California are the only two states scoring statistically significantly below Finland; the scores for Colorado, Connecticut, and Florida are about the same as Finland; and four states—Massachusetts, Minnesota, North Carolina, and Indiana—scored significantly higher than Finland. If Finland had been a U.S. state in 2011, it probably would have scored in the middle of the pack on NAEP. More troubling for the Finns, their TIMSS scores have declined significantly. Finland’s seventh graders took the test in 1999, scoring 520, and again in 2011, scoring 482. The 38 point decline is one of the largest recorded by a TIMSS participant.

A Progress Report on the A+ Countries

Cross-sectional data must be interpreted cautiously, and great care must be exercised when using them for predictive purposes. As Finland illustrates, a simple rule to remember is that sometimes things change.

Here is another example of that lesson, this time provided by a group of nations. The “A+ countries” are six nations that scored at the top of the 1995 TIMSS rankings in eighth-grade math. They are Belgium (Flemish community),⁷ Czech Republic, Hong Kong, Japan, Korea, and Singapore. Much hoopla was made about them when the 1995 TIMSS scores were released. In 2008, they were referenced as exemplars in the Final Report of the National Mathematics Advisory Panel. William H. Schmidt, Richard T. Houang, and colleagues have published a number of studies featuring a rubric based on the A+ countries’ math curriculums.⁸ The idea is that other countries should be more like the A+ countries. A 2012 study by Schmidt and Houang declared the Common Core mathematics standards comparable to the A+ countries’ curriculums in both focus and coherence. Moreover, they found that states with 2007 math standards similar to those of the A+ countries—again, using the same rubric from 1995—did very well on the 2007 NAEP. The findings were presented as implying that the Common Core will make the U.S. more like the A+ countries.⁹

Table 1-3 offers an update on the A+ countries. How are they doing? Let’s examine the table from the bottom-up. The Czech Republic left the TIMSS study after 2007, a year that saw its TIMSS score fall by 42 points from its performance twelve years earlier. Belgium (Flemish) has not participated in TIMSS since 2003. Its performance on TIMSS declined by 13 points before it left the study. The other four countries all took TIMSS in 2011. Hong Kong (+17) and Korea (+32) registered significant gains, Japan a significant decline (-11), and Singapore showed no significant change (+2). Of the six nations, then, two had statistically significant gains, three had statistically significant losses, and one scored about the same. The average score change for the six nations is -2.5 points, approximately equal to the average change for the 20 nations that participated in both 1995 and 2011. Put another way, the average A+ country made no more progress in math achievement than any other country in TIMSS.

Giving letter grades to entire nations may seem silly to many people but since the A+ designations have attained such widespread acceptance, readers are asked for their tolerance. It’s clear that A+ is no longer the appropriate grade for all of these countries.¹⁰ Korea and Hong Kong added to their outstanding 1995 scores and still deserve an A+. Singapore, too, although not making significant gains, surely preserves its A+ status by being one of only three nations with a 600+ scale score. Then things get dicey. Flemish Belgium was slipping when it left TIMSS in 2003. Its fourth graders did participate in 2011, however, and did well, scoring 549. That’s significantly higher than the U.S. at 541 and about the same as Florida at 545. But it represents no progress from the Belgian fourth graders’ previous TIMSS scores. Call Flemish Belgium a question mark—maybe an A- or B+, but definitely not an A+. We don’t know for sure without more recent eighth-grade data.

Japan’s score of 570 warrants an A, not an A+, and the downward trend is notable. Compare Korea with Japan. They both scored 581 in 1995. In 2011, Korea scored 43 points higher. The decline in the Czech Republic’s scores is the most dramatic, 42 points. The 2007 score of 504 is statistically indistinguishable from the international average of 500. Like Flemish Belgium, the Czech Republic fourth graders did participate in TIMSS 2011, scoring 511, a 30-point decline from 1995. The Czech Republic gets a C+ or B-.

Conclusion

What should we make of this? In 1995, six high achieving nations were described as “A+” to spur the U.S. towards greater math achievement. Their math curriculums were held up as ideals. And yet, since 1995, the U.S. gain of 17 points in eighth-grade mathematics is only exceeded by one A+ nation, Korea, and matched by another, Hong Kong. The other four A+ countries made less progress than the U.S. So in terms of gains, the U.S. should not look to the A+ countries for guidance. That said, five of the six A+ countries continue to lead the world in eighth-grade math achievement, and they continue to score significantly higher than the U.S.

The divergence of gain scores and status scores illustrates a problem that will be addressed in both remaining parts of this report. The tendency is for observers, when test scores are released, to zero in on the top performers, to ask what it is that the leading nations are doing, and then to urge the rest of the world to do those things. That response is understandable—but it is also potentially misleading. Causality is difficult to determine from cross-sectional data. Curriculum undoubtedly plays a role, but much more work needs to be done identifying potential curriculum effects in international data and testing well-formulated hypotheses with longitudinal models. Ideally, randomized trials would be conducted on the best curriculum programs, to tease out unobserved influences on learning. Those influences include a culture that places great value on academic success, parenting practices that promote achievement, and peers who award status based on working hard at school. They surely play a part in why some nations are “A+” while others only aspire to be.

« Introduction

Part II: The Resurgence of Ability and Persistence of Tracking »

Part I Notes:

1. In this section, the following rule was applied to ease the reading of the text. Subnational units, such as Hong Kong, may be referred to as “nations” or “countries.”

2. “2011 NAEP-TIMSS Linking Study,” National Center for Education Statistics, http://nces.ed.gov/timss/naeplink.asp.

3. Linking NAEP and international tests has been attempted before. See Gary W. Phillips, International Benchmarking: State Education Performance Standards (Washington, DC: American Institutes for Research, October 2010) and “Global Report Card,” Jay P. Greene and Josh B. McGee, http://globalreportcard.org/.

4. See “The Finnish Miracle,” Hank Pellissier, http://www.greatschools.org/students/2453-finland-education.gs.

5. Jenny Anderson, “From Finland, an Intriguing School Reform Model,” New York Times, December 12, 2011, http://www.nytimes.com/2011/12/13/education/from-finland-an-intriguing-school-reform-model.html?pagewanted=all&_r=0.

6. Pasi Sahlberg, Finnish Lessons: What Can the World Learn from Educational Change in Finland? (New York: Teachers College Press, 2011).

7. The Flemish, French, and German speaking communities operate separate school systems.

8. William H. Schmidt and Richard T. Houang, “Lack of Focus in the Mathematics Curriculum: Symptom or Cause?” in Lessons Learned: What International Assessments Tell Us about Math Achievement, ed. Tom Loveless (Washington: Brookings Institution Press, 2007).

9. William H. Schmidt and Richard T. Houang, “Curricular Coherence and the Common Core State Standards for Mathematics,” Educational Researcher 41, no. 8 (2012): 294-308.

10. Regression to the mean is possible, but the variance of the A+ countries’ gain scores suggests it’s unlikely.

Downloads

2013 Brown Center Report on American Education

Video

The 2013 Brown Center Report on American Education: How Well Are American Students Learning?

Authors

Tom Loveless

The Resurgence of Ability Grouping and Persistence of Tracking

Tom Loveless — Mon, 18 Mar 2013 00:00:00 -0400

This study examines the use of ability grouping and tracking in America’s schools. Recent NAEP data reveal a resurgence of ability grouping in fourth grade and the persistent popularity of tracking in eighth-grade mathematics. These trends are surprising considering the vehement opposition of powerful organizations to both practices. Although the current study will not delve into the debate—it is interested in what schools are doing, not why or whether they should do it—discussion is offered at the end of the article on implications of the findings for the controversy surrounding the topic.

Ability grouping and tracking are often confused. They both attempt to match students with curriculum based on students’ ability or prior performance, but the two practices differ in several respects. Tracking takes place between classes, ability grouping within classes. Tracking primarily occurs in high school and sometimes in middle school. In tracked academic subjects, students are assigned to different classrooms, receive instruction from different teachers, and study a different curriculum. The names of high school courses signal curricular differences. Advanced math students in tenth grade, for example, may take Algebra II while others take Geometry, Algebra I, or Pre-Algebra. Advanced tenth graders in English language arts (ELA) may attend a class called “Honors English” while other students attend “English 10” or “Reading 10.” Excellent science students may take “AP Chemistry” while others take a course simply called “Chemistry” or “General Science.” History may also be tracked, as when Advanced Placement courses are offered in U.S. or European history that not all students take. Some middle and high schools do not track at all, creating instead classes that are heterogeneous in ability. Students of all abilities study the same material.

What Tracking is Not

Perhaps the best way to clarify what tracking is, because of widespread misconceptions, is by describing what it is not. Tracking is decided subject by subject. Students are not assigned to college preparatory or vocational tracks that then dictate coursework all through high school; that practice died out in the U.S. in the late 1960s and early 1970s.¹¹,¹² European and Asian school systems still practice a form of this type of tracking (they call it “streaming”), typically in the final two or three years of secondary schooling.¹³ Students take placement exams and based on the scores are selected into separate schools with markedly different post-secondary destinations rather than attending different classes at the same school.¹⁴ Exam-based selection into high schools was common in the U.S. in the 19th century and the early part of the 20th century, but fell to the wayside. The comprehensive high school—with all students of a particular community attending the same school and then divided into distinct tracks within the school—came to be enshrined as the American model.

Ability Grouping

Ability grouping typically is an elementary school practice. Most elementary classes feature a single teacher with a classroom of students who are heterogeneous in ability. To create more homogeneity, teachers may divide students into small instructional groups reflecting different levels of ability, most often for reading in the primary grades (K–3) and perhaps for reading or math in later grades (4–6).¹⁵ While the teacher provides instruction to one group, the other students work independently—engaged in cooperative group activities or computer instruction or completing worksheets to reinforce skills. The teacher rotates among the groups so that each student receives a dose of teacher-led instruction in these small settings.

Researchers from Johns Hopkins conducted a comprehensive survey of ability grouping and tracking in 1986. The study analyzed national data augmented by an in depth survey of Pennsylvania schools. Several interesting patterns were uncovered that still hold true today. Disaggregating the data by grade level revealed that ability grouping is most prominent in first grade and then slowly recedes over subsequent grades. Ability grouping and tracking are inversely related; the school system’s strategies for creating groups that are as homogeneous as possible shift over the K-12 grade span. Tracking is rare in the elementary grades and, after increasing dramatically in middle school (in mathematics, in particular) peaks towards the end of high school. It is rare for students, once grouped between classes by tracking, to be grouped again within classes by ability grouping.¹⁶

Because the groupings are within-class (and often decided by a single teacher), ability grouping is more flexible than tracking. Groups may be reshuffled periodically to reflect changes in student performance. Ability groups might study from different levels of the same textbook series or use the same book and move at a different pace (with enrichment activities for the faster groups until the others catch up). Instead of the formality of transcript designations for high school courses, ability groups often take the names of animals—redbirds, bluebirds, sharks, dolphins, and the like—or the names of the books in the reading series that the students are using.

The most popular alternatives to ability-grouped instruction are whole class instruction, in which all students in the same classroom receive the same instruction, and the creation of small heterogeneous groups. Sometimes cooperative learning strategies are employed with heterogeneous groups, but cooperative learning can be used with any small group regardless of the criterion by which it is formed. Success for All, for example, is a popular program combining cooperative learning with small ability groups that are frequently reorganized to reflect student progress.¹⁷

Controversy

In the 1970s and 1980s, a barrage of studies criticized tracking and ability grouping. Race and class figured prominently in the debate. Grouping students by ability, no matter how it is done, will inevitably separate students by characteristics that are correlated statistically with measures of ability, including race, ethnicity, native language, and class. Critics argued that tracking and ability grouping do not separate students into socioeconomic status-related groups by accident. Ray C. Rist’s “Self-Fullfilling Prophecy in Ghetto Education” (1970) followed a group of kindergarten students through the first few years of school and noted how the composition of reading groups rarely changed, consistently reflecting students’ socioeconomic status (SES).¹⁸ The SES differences are hardened, Rist argued, as teachers develop different expectations for groups of low and high performing students, even if those groups are given innocuous sounding names to mask their status.¹⁹ James Rosenbaum’s Making Inequality (1976) described working class youth at a New England high school who were channeled into vocational and remedial tracks that were nothing more than boring, academic dead ends.²⁰

In 1985, Jeanie Oakes’ classic book, Keeping Track, was published. Oakes drew on data from several junior and senior high schools. Building on the social reproductionist theories of Samuel Bowles and Herbert Gintis’s Schooling in Capitalist America, Oakes argued that although tracking is typically justified by educators as a strategic response to student heterogeneity, the practice is undergirded by normative beliefs regarding race and class—and politically defended by white, middle-class parents to protect privilege. Black, Hispanic and poor children populate remedial classes; middle-class white children populate honors courses. Tracking and ability grouping are not mere bystanders to social injustice, Oakes and other critics charged. Such practices don’t just mirror the inequalities of the broader society. They reproduce and perpetuate inequality.²¹

This critique had a profound effect on policy and practice. In the 1990s, several prominent political organizations passed resolutions condemning tracking, including the National Governors Association, the American Civil Liberties Union, the Children’s Defense Fund, and the NAACP Legal Defense Fund. Some states urged schools to reduce tracking and ability grouping, most notably California and Massachusetts. A surprising implementation story ensued. Although the call to detrack was not accompanied by conventional incentives—the big budgets, regulatory regimes, and rewards and sanctions that draw the attention of policy analysts—detracking was, in a field famous for ignored or subverted policies, adopted by a large number of schools.²²

Surveys of Ability Grouping

How much did ability grouping decline? A 1961 national survey revealed that about 80% of elementary schools grouped students by ability for reading instruction.²³ A three-group format was the dominant approach, with students organized into high, middle, and low performing groups. Although subsequent national surveys of ability grouping are scarce until the John Hopkins study in the mid-1980s (mentioned above), carefully crafted studies of local practice reported similar frequencies. Eighty percent or more of elementary schools used within-class
ability groups.²⁴

Then things changed. A mid-1990’s survey of a random sample of pre-K through fifth grade teachers reported startlingly different results. When allowed multiple responses, only 27% of teachers reported using ability grouping for reading instruction. Another 56% of teachers indicated that they used flexible grouping. Some of the teachers with flexible grouping may have utilized ability as a criterion for grouping.²⁵ Whole class instruction was by far the most popular organizing strategy, with 68% of teachers reporting its use. Removing the overlapping responses makes it clear that ability grouping served a subordinate role as a method of organizing students. When teachers were held to one response and asked to identify their primary organizational approach, the order was: whole-class instruction (52%), flexible grouping (25%), and ability grouping (16%).

A more recent survey suggests ability grouping has regained favor among teachers. Barbara Fink Chorzempa and Steve Graham (2006) surveyed a national random sample of first through third grade teachers. Their questionnaire asked questions similar to the Baumann et al. survey of the 1990s, but also included questions about why teachers ability group. Three times as many teachers (63%) said they use ability grouping as the earlier survey. The authors explain that the discrepant findings may stem from the different grade levels of teachers in the two surveys. Pre-K and fourth- and fifth-grade teachers, who are included in the earlier
survey but not in the latter, may be less likely to employ ability grouping than first through third-grade teachers, the target population of the latter survey. Interestingly, the top reason teachers gave for using ability grouping was “that it helps them meet students’ needs;” however, respondents also expressed concern about the quality of instruction in low ability groups.²⁶ About 20% of teachers did not ability group at all because the practice was banned by district or school policy.

Is ability grouping in decline or on the rise again? How about tracking? Let’s turn to NAEP data to shed light on these questions.

NAEP Data on Ability Grouping

Table 2-1 displays NAEP data on ability grouping in fourth grade reading. Teachers were asked on what basis they create instructional groups (ability, interest, diversity, and other) with “not created” also an option. Bear in mind that asking fourth-grade teachers about ability grouping, as compared to sampling teachers of several elementary grades, has both an upside and a downside in elucidating trends. The upside is that grade level is held constant over several surveys. This is important because we know ability grouping varies by grade level. The downside is that fourth grade isn’t where the action is on ability grouping—that’s first grade, where unfortunately NAEP does not collect data. Fourth grade is well after ability grouping’s apogee and somewhere near the midpoint of its diminishing use by elementary teachers.

Table 2-1 is revealing. The percentage of students placed into ability groups for reading instruction skyrocketed from 1998 to 2009, from 28% to 71%. And the percentage of students whose teachers did not create ability groups fell from 39% in 1998 to 8% in 2009. In other words, the odds of a fourth grader being ability grouped in reading were less than 50-50 in 1998 but by 2009 had increased to about 9 to 1. The question was not asked prior to 1998.

Table 2-2 shows the frequency of ability grouping in fourth-grade mathematics. Teachers were asked if they create math groups based on ability. This question was asked twice before 1998 and in 2011, so it gives a deeper historical perspective than the question on reading. Math ability grouping dips from 1992 to 1996 (48% to 40%), stays about the same until 2003 (42%), and then accelerates from 2003 to 2011 (reaching 61% in 2011).

The NAEP data support the general finding of a drop in ability grouping in the 1990s and a resurgence in the 2000s. The rebound is more subdued in math than in reading. It is apparent by 2000 in reading (it may have begun even before then; the data start in 1998) but does not begin in math until after 2003. In the years for which data are available for both reading and math (2000, 2003, 2007, 2009), the two subjects have comparable frequencies in 2000 (39% in reading and 41% in math), but reading is more often grouped in subsequent years. In the last year with data on both subjects, 2009, 71% of fourth grade students were ability grouped for reading and 54% for math.

NAEP Data on Tracking

Table 2-3 displays NAEP data on tracking in 8th grade. Note that unlike ability grouping, which is a classroom level practice and consequently a topic for teacher surveys, tracking is a school level practice and a topic for surveys of school principals. Although the wording of the survey item varies slightly from year to year, NAEP asks principals whether students are assigned to classes based on ability so as to create some classes that are higher in average ability or achievement than others. The question is asked sporadically and about different subjects in different years.

Math has the most data, surveyed ten times from 1990–2011. Tracking in math shows a slight dip in the 1990s and an increase in the 2000s, but most of the fluctuations are too small to consider significant. The trend is essentially flat, with about three-fourths of students attending tracked math classes over the past two decades. Typically, this means schools offer an algebra class for some eighth graders and a pre-algebra class for those who are not yet ready for formal algebra (see table 3-2 for enrollment statistics). Sometimes a third class is offered, perhaps geometry for students who took algebra in seventh grade or a basic math class for students several years behind.

Data on the other subjects are spotty. They exhibit much less tracking than math and greater variation over time. In 1990, principals reported that 60% of students were in tracked ELA classes, a statistic that declined over the next several years, hitting a low of 32% in 1998. The 43% frequency of tracking reported in 2003 is an increase from 1998; however, because it was the last time the question was asked in that subject, it is impossible to tell whether an enduring rebound in ELA tracking had begun. Science and history have even less data, with both subjects registering their highest figures in 1990 and then indicating diminished tracking after that. Science seems to show a rebound from 1994–2000. For all four subjects, the least amount of tracking occurred between 1994 and 1998, when the detracking movement was in full bloom.

The national pattern is consistent with previous studies of California and Massachusetts. In those two states, detracking was most intense in the early to mid-1990s, but differences among the subjects emerged. Mathematics resisted detracking while heterogeneously grouped classes became the norm in ELA, science, and history. In a 2009 survey of Massachusetts schools with eighth grades, for example, in math only 15.6% of schools offered heterogeneously-grouped classes; 49.2% offered classes with two ability levels; and 35.2% offered three levels. In other subjects, tracking had almost disappeared—72.7% offered only heterogeneously-grouped classes in ELA, 89.8% in history, and 86.7% in science.²⁷

Discussion

This study has explored trends in the use of ability grouping and tracking by American schools. It used NAEP data to examine the frequency that fourth graders are assigned to groups and eighth graders assigned to classes based on ability or prior achievement. The investigation focused on what schools are doing, not on whether tracking or ability grouping is a good idea.

NAEP data from 1990 to 2011 were examined. Ability grouping in fourth grade decreased in the 1990s and then increased markedly in the 2000’s, with the rebound apparent in both reading and math. In reading, ability grouping has attained a popularity unseen since the 1980s, used with over 70% of students. As for tracking, it has remained commonplace in eighth-grade mathematics for the past two decades, with about three-quarters of students enrolled in distinct ability-level math classes. Tracking in ELA declined sharply from 1990 to 1998, and although there was a rebound in 2003, NAEP has not surveyed schools on tracking in ELA since then. And NAEP data are too sparse in other subjects to determine trends.

Do these trends matter? Why should anyone care about tracking and ability grouping? Although the debate today is more subdued than in the 1980s and 1990s, it does continue. A research review on the NEA website blasts both tracking and ability grouping as discriminatory.²⁸ Scholars continue to wrangle over the wisdom of both practices. Effectiveness and equity persist as the dominant themes of this literature. A 2010 meta-analysis of high quality studies calculated a positive effect size of 0.22, equal to about one-half year of learning, for within-class grouping in reading instruction.²⁹ A 2010 study of data from the Early Childhood Longitudinal Study (ECLS), on the other hand, found “students who are lower grouped for reading instruction learn substantially less, and higher-grouped students learn slightly more over the first few years of school, compared to students who are in classrooms that do not practice grouping.”³⁰ That finding is especially relevant to closing achievement gaps between students who may populate high and low groups.

The controversy offers a very important lesson about how education policy gets implemented in schools. Schools are not merely the last step of a vast organizational ladder, not simply the education system’s operational frontline, ready to put in place the policies that are passed down from above. Finley Peter Dunne famously observed that the U.S. Supreme Court “follows the election returns.” Court decisions not only reflect the U.S. Constitution but public opinion as well. Our schools are another institution with an ear to the ground. Educators are aware of public debates and are influenced when particular school practices become controversial.

Figure 2-1 shows the number of times the term “ability grouping” appeared in Education Week from 1983 to December 2012. Consider this a proxy for media visibility over the past thirty years. The 135 appearances over these three decades represent an average of 4.5 mentions per year. The peak coverage occurred in 1993, with 20 mentions. The years immediately preceding 1993 show a gradual build up in coverage, with 5 mentions in 1989, 13 in 1990, 11 in 1991, and 13 in 1992. The years immediately after 1993 show a gradual decline—8 appearances in 1994, 5 in 1995, 7 in 1996, 5 in 1997, and 7 in 1998. The ten years from 1989–1998 are the only years with more than 5 annual mentions. Tracking and ability grouping were in the spotlight.

The data on media visibility are inversely related to the data on use. At the beginning of the 1990s, tracking and ability grouping were conventional practices but then declined —albeit with some lag time—when they were subjected to the most public scrutiny. The mentions in Education Week peaked in 1993. The use of ability grouping and tracking reached all time lows soon after that event. As the controversy died down in the 2000s, schools returned to both practices.

What else may have promoted the resurgence in the 2000s? Accountability systems, bolstered by the accountability provisions of No Child Left Behind, focus educators’ attention on students below the threshold for “proficiency” on state tests. That provides a statutory justification for grouping students who are struggling. The increased use of computer instruction in elementary classrooms cannot help but make teachers more comfortable with students in the same classroom studying different materials and progressing at different rates through curriculum. The term “differential instruction,” while ambiguous in practice, might make grouping students by prior achievement or skill level an acceptable strategy for educators who recoil from the term “ability grouping.”

A substantial number of teachers believe that heterogeneous classes are difficult to teach. The 2008 MetLife Survey of the American Teacher asked teachers to react to the following statement: “My class/classes in my school have become so mixed in terms of students’ learning ability that I/teachers can’t teach them.” Responses were: 14% “agree strongly,” 29% “agree somewhat,” 28% “disagree somewhat,” and 27% “disagree strongly.”³¹ The percentages are surprising given the questionnaire’s blunt assertion that heterogeneous classes are impossible to teach. Moreover, the 43 percent of respondents that either agree strongly or somewhat agree with the prompt is up from 39 percent on the same survey item in 1988. Teachers’ beliefs about the impact of achievement heterogeneity on instruction undergird the use of ability grouping and tracking.

Let’s look ahead. Will the uptrend in ability grouping continue? Not necessarily. The current period may be the lull before the storm. Theoretically, at least, the Common Core establishes a curriculum that most, if not all, students will study. It is unclear how students who have already mastered the Common Core standards before beginning a particular school grade will have their needs met under the new regime. The same goes for students who lag many years behind. Tracking and ability grouping have been common approaches to addressing such challenges. These two organizational strategies affect millions of students daily. Both practices shape aspects of schooling that we know to be important—the curriculum students study, the textbooks they learn from, the teachers who teach them, the peers with whom they interact. Despite decades of vehement criticism and mountains of documents urging schools to abandon their use, tracking and ability grouping persist—and for the past decade or so, have thrived.

« Part I: The Latest TIMSS and PIRLS Scores

Part III: Advanced Math in Eighth Grade »

Part II Notes

11. Tom Loveless, The Tracking and Ability Grouping Debate (Washington, DC: Thomas B. Fordham Institute, July 1, 1998).

12. Samuel R. Lucas, Tracking Inequality: Stratification and Mobility in American High Schools (New York: Teachers College Press, 1999).

13. Even Finland and Sweden, famous for egalitarian reforms, divide students for the final two years of secondary school. Germany begins tracking at age 11.

14. Alan Smithers and Pamela Robinson, Choice and Selection in School Admissions: The Experience of Other Countries, accessed March 4, 2013, http://suttontrust.com/research/choice-and-selection-in-admissions/1smithers-final-report.pdf.

15. Robert Dreeben and Rebecca Barr, “The Formation and Instruction of Ability Groups,” American Journal of Education 97, no. 1 (1988): 34-64.

16. See p. 36, Figure 5: James M. McPartland, J. Robert Coldiron, and Jomills H. Braddock II, School Structures and Classroom Practices in Elementary, Middle, and Secondary Schools, Report No. 14 (Baltimore: The Johns Hopkins University, 1987).

17. “Success for All—Home”, Success for All Foundation, http://www.successforall.org/.

18. Ray C. Rist, “Student Social Class and Teacher Expectations: The Self-fulfilling Prophecy in Ghetto Education,” Harvard Educational Review 40, no. 3 (1970): 411-451.

19. Ability grouping is called “setting” in Great Britain. Recent reports have been sharply critical of the practice, see: “Setting Harms Education of Some Young Children, Report Warns,” The Independent, May 16, 2008, http://www.independent.co.uk/news/education/education-news/setting-harms-education-of-some-young-children-report-warns-829312.html.

20. James E. Rosenbaum, Making Inequality; the Hidden Curriculum of High School Tracking (New York: John Wiley & Sons, 1976).

21. See: Jeannie Oakes, Keeping Track: How Schools Structure Inequality (New Haven: Yale University Press, 1985). Also see: Jeannie Oakes, Amy Stuart Well, and Associates, Beyond the Technicalities of School Reform: Policy Lessons from Detracking School (Los Angeles: UCLA Graduate School of Education & Information Studies, 1996).

22. The politics and policies of tracking reform are investigated in: Tom Loveless, The Tracking Wars: State Reform Meets School Policy (Washington: Brookings Institution Press, 1999).

23. Mary C. Austin and Coleman Morrison. The Torch Lighters: Tomorrow’s Teachers of Reading (Cambridge: Harvard University Graduate School of Education, 1961).

24. Rebecca Barr and Robert Dreeben, How Schools Work (Chicago, University of Chicago Press, 1983).

25. ECLS asked kindergarten teachers in 1999 the frequency with which they used ability groups in reading. Five response categories, ranging from 0 (never) to 4 (daily). 30% reported never using ability grouping. The average for all teachers was 1.64, indicating about once a week (1 = less than once a week; 2 = once or twice weekly). When the ECLS sample was in 3rd grade, 2001–2002, 50% of teachers employed ability grouping in reading, consistent with the NAEP figure for 4th grade in 2003 (47%). See p. 301, note 6 in Christy Lleras, and Claudia Rangel, “Ability grouping practices in elementary school and African American/Hispanic achievement.” American Journal of Education 115, no. 2 (2009): 279–304.

26. Barbara Fink Chorzempa and Steve Graham, “Primary-Grade Teachers’ Use of Within-Class Ability Grouping in Reading,” Journal of Educational Psychology 98, no. 3 (2006): 529-541.

27. Tom Loveless, Tracking, Detracking: High Achievers in Massachusetts Middle School (Washington, DC: Thomas B. Fordham Institute, 2009).

28. “Research Spotlight on Academic Ability Grouping,” NEA, http://www.nea.org/tools/16899.htm.

29. Kelly Puzio and Glenn Colby, The Effects of Within Class Grouping on Reading Achievement: A Meta-Analytic Synthesis (Evanston: Society for Research on Educational Effectiveness, 2010).

30. Christy Lleras and Claudia Rangel, “Ability Grouping Practices in Elementary School and African American /Hispanic Achievement,” American Journal of Education 115, no. 2 (2009): 279.

31. Dana Markow and Michelle Cooper, The Metlife Survey of the American Teacher: Past, Present and Future (New York: Metlife, 2008).

Downloads

2013 Brown Center Report on American Education

Video

The Resurgence of Ability Grouping and Tracking: A Return to Controversial Practices?

Authors

Tom Loveless

Redirecting Students to Less Demanding Colleges Not a Strategy for Success in the Sciences

Matthew M. Chingos — Wed, 13 Mar 2013 11:00:00 -0400

Last week, I wrote about the “mismatch” hypothesis, which posits that the beneficiaries of affirmative action in higher education will not be able to keep up in a demanding environment and would in fact be better off had they not been given a leg up in the admissions process. I showed that there is no credible evidence supporting the mismatch theory, and that the opposite finding prevails: students are most likely to graduate by attending the most selective institution that will admit them.

The focus of my analysis last week was a November 2012 NBER working paper by a team of economists from Duke University using data from the University of California Office of the President (UCOP). I conducted a reanalysis of the UCOP data and showed that more appropriate methods yield findings that are not consistent with the mismatch hypothesis.

Last month, three of the four authors of the earlier study released a new NBER working paper on the effects of mismatch on the chances that students, especially underrepresented minorities, will graduate with science degrees. In other words, instead of looking at overall graduation rates as they did in the earlier paper, they focused on earning a degree in the sciences. The new paper claims to find strong evidence that students with weaker academic preparation are much more likely to be successful at a lower-ranked campus: “Our estimates suggest that the vast majority of minority students who begin in the sciences at UC Berkeley would be more likely to graduate with a science degree had they enrolled in a less-selective campus.”

Do the findings of the new paper hold up to careful scrutiny any better than the findings of the previous paper? Only marginally. Figure 1 shows the graduation rates, both in the sciences and overall, for underrepresented minority students who intended to major in the sciences and had SAT scores that placed them in the bottom quarter of all UC applicants (1050 or less) in the period before California banned affirmative action. Students in this group that attended a more selective university were slightly more likely to graduate with a degree in any field (as would be expected based on last week’s analysis), but slightly less likely to graduate with a degree in the sciences.

Is this finding the result of affirmative action? The authors of the February 2013 NBER paper argue that it is: “in a period when racial preferences in admissions were strong, minority students were in general over-matched, resulting in low graduation rates in the sciences… In contrast, non-minority students are generally well-placed for graduating in the sciences.”

But a closer look at the UCOP data casts serious doubt on this conclusion. Table 1 shows the relationship between campus selectivity and graduation rates in the sciences for different groups of students. Specifically, I calculate the expected increase in the chance that a student will earn a degree in the sciences associated with a 100-point increase in the average SAT score of the campus, controlling for the student’s SAT score, high school GPA, parental education, and family income. I do not use the problematic Dale-Krueger method used by the NBER paper’s authors for the reasons I discussed last week. Following the authors’ practice of separating out students by whether they intended to major in the sciences cuts the data very thin and results in very imprecise (and non-robust) estimates, so I combine all students together but control for whether each student indicated an intention to major in the sciences.

The upper-left number in Table 1 indicates that, prior to the affirmative action ban, URM students with the lowest SAT scores were about one percentage point less likely to graduate with a degree in the sciences if they attend a more selective campus. This is similar to what we saw in Figure 1. The relationship is similar for URM students in the upper three SAT quartiles and for white students in all but the top quartile, although those relationships are estimated less precisely and consequently we cannot confidently reject the possibility that there is no relationship between science graduation rates and selectivity for those groups.

If affirmative action were the reason for the slight negative relationship between selectivity and science graduation rates for URM students, then we would expect the relationship to go away in the period following the passage of Prop 209, which banned affirmative action in California. But Table 1 shows exactly the opposite finding: the negative relationship was substantially larger in the three-year period following the ban, for both URM and white students. And the relationship was strongest for the most highly qualified URM students in the period following Prop 209.

A simple before-and-after comparison should not be taken as definitive, as other factors could have changed between the 1995-1997 and 1998-2000 periods. But the pattern of results casts doubt on the idea that affirmative action was the driver behind the 1995-1997 results given that even stronger negative relationships appear in the 1998-2000 data.

The fact still remains that some students are slightly more likely to graduate with a science degree if they attend a less selective university, although it is unclear to what extent this relationship is biased by unobserved student characteristics.[1] But in any case, it’s unclear what to make of this finding given that overall graduation rates go in the opposite direction. Is it better to have a higher chance of persisting in the sciences if it is accompanied by an increased risk of dropping out altogether? And is it better to have a science degree from a lower-ranked school rather than a non-science degree from an elite campus such as Berkeley or UCLA?

There is no doubt that graduation rates, both in the sciences and overall, are too low, especially for students from disadvantaged backgrounds. Anyone who thinks the U.S. needs to increase the number of students who earn science degrees should be troubled by the high percentages of students who start out interested in science but end up switching to another field or dropping out. Persisting in the sciences is especially challenging for students with weak academic preparation. Figure 1 shows success rates of around 20 percent for low-scoring URM students, a stubbornly low rate that doesn’t vary much by campus.

Policymakers and higher education practitioners clearly need to seek ways to increase the student success in the sciences. But redirecting students to less demanding institutions, either through changes to admissions policies or college counseling practices, is a dubious strategy that at best increases success rates in science by small margins at the cost of increased dropout rates overall.

[1] The negative relationship becomes stronger when control variables are added. If sorting on unobserved characteristics occurs in the same direction as the observed characteristics, then the true (causal) relationship will be larger in magnitude than the estimated relationship.

Authors

Matthew M. Chingos

Are Minority Students Harmed by Affirmative Action?

Matthew M. Chingos — Thu, 07 Mar 2013 11:00:00 -0500

Affirmative action is back in the news this year with a major Supreme Court case, Fisher v. Texas. The question before the Court is whether the Fourteenth Amendment’s Equal Protection Clause permits the University of Texas at Austin’s use of race in its undergraduate admissions process. The Court may declare the use of racial preferences in university admissions unconstitutional when it decides the case in the coming months, potentially overturning its decision in the landmark Grutter case decided a decade ago.

Accompanying the general subject of affirmative action in the spotlight is the “mismatch” hypothesis, which posits that minority students are harmed by the very policies designed to help them. Justice Clarence Thomas made this argument in his dissent in the Grutter case: “The Law School tantalizes unprepared students with the promise of a University of Michigan degree and all of the opportunities that it offers. These overmatched students take the bait, only to find that they cannot succeed in the cauldron of competition. And this mismatch crisis is not restricted to elite institutions.”

The mismatch idea is certainly plausible in theory. One would not expect a barely literate high-school dropout to be successful at a selective college; admitting that student to such an institution could cause them to end up deep in debt with no degree. But admissions officers at selective colleges obviously do not use affirmative action to admit just anyone, but rather candidates they think can succeed at their institution.

The mismatch hypothesis is thus an empirical question: have admissions offices systematically overstepped in their zeal to recruit a diverse student body? In other words, are they admitting students who would be better off if they had gone to college elsewhere, or not at all? There is very little high-quality evidence supporting the mismatch hypothesis, especially as it relates to undergraduate admissions—the subject of the current Supreme Court case.

In fact, most of the research on the mismatch question points in the opposite direction. In our 2009 book, William Bowen, Michael McPherson, and I found that students were most likely to graduate by attending the most selective institution that would admit them. This finding held regardless of student characteristics—better or worse prepared, black or white, rich or poor. Most troubling was the fact that many well-prepared students “undermatch” by going to a school that is not demanding enough, and are less likely to graduate as a result. Other prior research has found that disadvantaged students benefit more from attending a higher quality college than their more advantaged peers.

A November 2012 NBER working paper by a team of economists from Duke University comes to the opposite conclusion in finding that California’s Proposition 209, a voter-initiated ban on affirmative action passed in 1996, led to improved “fit” between minority students and colleges in the University of California system, which resulted in improved graduation rates. The authors report a 4.4-percentage-point increase in the graduation rates of minority students after Proposition 209, 20 percent of which they attribute to better matching.

At first glance, these results appear to contradict earlier work on the relationship between institutional selectivity and student outcomes. But the paper’s findings rest on a questionable set of assumptions, and a more straightforward reanalysis of the data used in the paper, which were provided to me by the University of California President’s Office (UCOP), yields findings that are not consistent with the mismatch hypothesis.

First, the NBER paper uses data on the change in outcomes between the three years prior to Prop 209’s passage (1995-1997) and the three years afterward (1998-2000) to estimate the effect of the affirmative action ban on student outcomes. Such an analysis is inappropriate because it cannot account for other changes occurring in California over this time period (other than simple adjustments for changes in student characteristics).

A key problem with the before-and-after method is that it does not take into account pre-existing trends in student outcomes. This is readily apparent in Figure 1, which shows that the graduation rates of underrepresented minority (URM) students increased by about four percentage points between 1992-1994 and 1995-1997, before the affirmative action ban. The change from 1995-1997 to 1998-2000 was smaller, at about three percentage points. The NBER paper interprets this latter change as the causal impact of Prop 209, but this analysis assumes that there would have been no change in the absence of Prop 209. If the prior trend had continued, then graduation rates would have increased another four points—in which case, the effect of Prop 209 was to decrease URM graduation rates by one percentage point.

Adjusting for student characteristics does not change this general pattern. The adjustment makes no difference in the pre-Prop 209 period, but explains about 36 percent of the increase in the immediate post-Prop 209 period (which is consistent with the NBER paper’s finding that changes in student characteristics explain 34-50 percent of the change). But if the 1992-1994 to 1995-1997 adjusted change was four points, and the 1995-1997 to 1998-2000 adjusted change was one point, then Prop 209 might be said to have a negative effect of three percentage points.

None of these alternative analyses of the effect of Prop 209 should be taken too seriously, because it is difficult to accurately estimate a pre-policy trend from only two data points. The bottom line is that there probably isn’t any way to persuasively estimate the effect of Prop 209 using these data. But this analysis shows how misleading it is in this case to only examine the 1995-1997 to 1998-2000 change, while ignoring the prior trend.

Second, the NBER paper finds that less-selective universities produce better outcomes among minority students with weaker academic credentials. This must be the case in order for “mismatch” to exist, but it runs counter to most prior research on the subject. The one exception is a 2002 study by Stacy Dale and Alan Krueger, which found no impact of college selectivity on earnings except among students from low-income families. However, the methodology of the Dale-Krueger study severely limits the relevance of its results for students and policymakers.

In order to control for unobserved student characteristics, Dale-Krueger control for information about the institutions to which students applied and were accepted. This takes into account potentially valuable information that is observable by admissions committees but not the researcher. But it is problematic because it produces results that are based on comparisons between students who attended more or less selective colleges despite being admitted to the same set of institutions. As Caroline Hoxby explains: “since at least 90 percent of students who [were admitted to a similar group of schools] choose the more selective college(s) within it, the strategy generates estimates that rely entirely on the small share of students who make what is a very odd choice.” In other words, the method ignores most of the variation in where students go to college, which results from decisions about where to apply.

The problem with the NBER paper is that it uses a variant of the Dale-Krueger method by controlling for which UC campuses students applied to and were admitted by. And the UCOP data are consistent with Hoxby’s argument: in 1995-1997, 69 percent of URM students attended the most selective UC campus to which they were admitted and 90 percent attended a campus with an average SAT score within 100 points of the most selective campus that admitted them (the corresponding figures for all UC students are 72 and 93 percent).

A more straightforward analysis is to compare the graduation rates of URM students with similar academic preparation and family backgrounds who attended different schools. The mismatch hypothesis predicts that URM students with weak qualifications will be more likely to graduate, on average, from a less selective school than a more selective one.

The data show the opposite of what mismatch theory predicts: URM students, including those with less-than-stellar academic credentials, are more likely to graduate from more selective institutions. I calculate graduation rates by individual campus that are adjusted to take into account SAT scores, high school GPA, parental education, and family income.[1] I restrict this analysis to URM students with SAT scores in the 900-990 and 1000-1090 range during the three years before Prop 209, which should be exactly the group and time period when mismatch is most likely to occur.

Figure 2 shows that for both of the low-scoring groups of URM students, graduation rates are higher at more selective institutions. Results for individual institutions vary somewhat, but the upward trend in Figure 2 is clear. I find a similar pattern of results in the period after Prop 209 was passed (not shown). The main limitation of this type of analysis is that it does not take into account unobserved factors such as student motivation that may be associated with admission decisions and student choice of institution. The Dale-Krueger method is meant to address this issue, but for the reasons explained above produces results that are not particularly informative.

A better solution is to find instances of students who attended institutions of differing selectivity for reasons unrelated to their likelihood of success. This is not possible with the UCOP data, but such quasi-experimental methods are used in two other studies that finds a positive relationship between selectivity and student outcomes. In a study published in 2009, Mark Hoekstra used a cutoff in the admissions process at a flagship state university to estimate the impact of attending that university on earnings. This strategy eliminates bias by comparing students who are very similar except that some were just above the cutoff for admission and others were just below. Hoekstra finds that attending the flagship increased earnings by 20 percent for white men.

In a more recent working paper, Sarah Cohodes and Joshua Goodman employed a cutoff-based approach to measure the effect of a Massachusetts scholarship that could only be used at in-state institutions. Students who won the scholarship were more likely to attend a lower quality college, which caused a 40 percent decrease in on-time graduation rates, as well as a decline in the chances of earning a degree at any point within six years.

These two studies do not directly address the mismatch question because they do not focus on the beneficiaries of affirmative action, but they show that taking into account students’ unobserved characteristics leaves intact the positive relationship between selectivity and student outcomes that has been consistently documented in the many prior studies that are less causally persuasive.

To truly put the mismatch theory to rest, rigorous quasi-experimental evidence that focuses on the beneficiaries of preferential admissions policies is needed. But the current weight of the evidence leans strongly against the mismatch hypothesis. Most importantly, not a single credible study has found evidence that students are harmed by attending a more selective college. There may well be reasons to abolish or reform affirmative action policies, but the possibility that they harm the intended beneficiaries should not be among them.

[1] Specifically, I estimate the coefficients on institutional dummy variables after including these control variables. For the controls I include dummy variables corresponding to the categories used in the UCOP data, as well as dummies identifying missing data on each variable so as not to lose any observations. The adjusted graduation rate for each institution is calculated as the difference in its coefficient estimate and Berkeley’s coefficient estimate plus Berkeley’s unadjusted graduation rate for the indicated group of students (i.e. Berkeley’s adjusted and unadjusted graduation rate are thus equal by construction).

Authors

Matthew M. Chingos