Editor's Note: The introduction to the 2015 Brown Center Report on American Education appears below. Use the Table of Contents to navigate through the report online, or download a PDF of the full report.

TABLE OF CONTENTS

Part I: Girls, Boys, and Reading

Part II: Measuring Effects of the Common Core

Part III: Student Engagement

INTRODUCTION

The 2015 Brown Center Report (BCR) represents the 14^th edition of the series since the first issue was published in 2000.  It includes three studies.  Like all previous BCRs, the studies explore independent topics but share two characteristics: they are empirical and based on the best evidence available.  The studies in this edition are on the gender gap in reading, the impact of the Common Core State Standards -- English Language Arts on reading achievement, and student engagement.

Part one examines the gender gap in reading.  Girls outscore boys on practically every reading test given to a large population.  And they have for a long time.  A 1942 Iowa study found girls performing better than boys on tests of reading comprehension, vocabulary, and basic language skills.  Girls have outscored boys on every reading test ever given by the National Assessment of Educational Progress (NAEP)—the first long term trend test was administered in 1971—at ages nine, 13, and 17.  The gap is not confined to the U.S.  Reading tests administered as part of the Progress in International Reading Literacy Study (PIRLS) and the Program for International Student Assessment (PISA) reveal that the gender gap is a worldwide phenomenon.  In more than sixty countries participating in the two assessments, girls are better readers than boys. 

Perhaps the most surprising finding is that Finland, celebrated for its extraordinary performance on PISA for over a decade, can take pride in its high standing on the PISA reading test solely because of the performance of that nation’s young women.  With its 62 point gap, Finland has the largest gender gap of any PISA participant, with girls scoring 556 and boys scoring 494 points (the OECD average is 496, with a standard deviation of 94).   If Finland were only a nation of young men, its PISA ranking would be mediocre.

Part two is about reading achievement, too. More specifically, it’s about reading and the English Language Arts standards of the Common Core (CCSS-ELA).  It’s also about an important decision that policy analysts must make when evaluating public policies—the determination of when a policy begins. How can CCSS be properly evaluated? 

Two different indexes of CCSS-ELA implementation are presented, one based on 2011 data and the other on data collected in 2013.  In both years, state education officials were surveyed about their Common Core implementation efforts.  Because forty-six states originally signed on to the CCSS-ELA—and with at least forty still on track for full implementation by 2016—little variability exists among the states in terms of standards policy.  Of course, the four states that never adopted CCSS-ELA can serve as a small control group.  But variation is also found in how the states are implementing CCSS.  Some states are pursuing an array of activities and aiming for full implementation earlier rather than later.  Others have a narrow, targeted implementation strategy and are proceeding more slowly. 

The analysis investigates whether CCSS-ELA implementation is related to 2009-2013 gains on the fourth grade NAEP reading test.  The analysis cannot verify causal relationships between the two variables, only correlations.  States that have aggressively implemented CCSS-ELA (referred to as “strong” implementers in the study) evidence a one to one and one-half point larger gain on the NAEP scale compared to non-adopters of the standards.  This association is similar in magnitude to an advantage found in a study of eighth grade math achievement in last year’s BCR.  Although positive, these effects are quite small.  When the 2015 NAEP results are released this winter, it will be important for the fate of the Common Core project to see if strong implementers of the CCSS-ELA can maintain their momentum.

Part three is on student engagement.  PISA tests fifteen-year-olds on three subjects—reading, math, and science—every three years.  It also collects a wealth of background information from students, including their attitudes toward school and learning.  When the 2012 PISA results were released, PISA analysts published an accompanying volume, Ready to Learn: Students’ Engagement, Drive, and Self-Beliefs, exploring topics related to student engagement.

Part three provides secondary analysis of several dimensions of engagement found in the PISA report.  Intrinsic motivation, the internal rewards that encourage students to learn, is an important component of student engagement.  National scores on PISA’s index of intrinsic motivation to learn mathematics are compared to national PISA math scores.  Surprisingly, the relationship is negative.  Countries with highly motivated kids tend to score lower on the math test; conversely, higher-scoring nations tend to have less-motivated kids. 

The same is true for responses to the statements, “I do mathematics because I enjoy it,” and “I look forward to my mathematics lessons.”  Countries with students who say that they enjoy math or look forward to their math lessons tend to score lower on the PISA math test compared to countries where students respond negatively to the statements.  These counterintuitive finding may be influenced by how terms such as “enjoy” and “looking forward” are interpreted in different cultures.  Within-country analyses address that problem.  The correlation coefficients for within-country, student-level associations of achievement and other components of engagement run in the anticipated direction—they are positive.  But they are also modest in size, with correlation coefficients of 0.20 or less. 

Policymakers are interested in questions requiring analysis of aggregated data—at the national level, that means between-country data.  When countries increase their students’ intrinsic motivation to learn math, is there a concomitant increase in PISA math scores?  Data from 2003 to 2012 are examined.  Seventeen countries managed to increase student motivation, but their PISA math scores fell an average of 3.7 scale score points.  Fourteen countries showed no change on the index of intrinsic motivation—and their PISA scores also evidenced little change.  Eight countries witnessed a decline in intrinsic motivation.  Inexplicably, their PISA math scores increased by an average of 10.3 scale score points.  Motivation down, achievement up.

Correlation is not causation.  Moreover, the absence of a positive correlation—or in this case, the presence of a negative correlation—is not refutation of a possible positive relationship.  The lesson here is not that policymakers should adopt the most effective way of stamping out student motivation.  The lesson is that the level of analysis matters when analyzing achievement data.  Policy reports must be read warily—especially those freely offering policy recommendations.  Beware of analyses that exclusively rely on within- or between-country test data without making any attempt to reconcile discrepancies at other levels of analysis.  Those analysts could be cherry-picking the data.  Also, consumers of education research should grant more credence to approaches modeling change over time (as in difference in difference models) than to cross-sectional analyses that only explore statistical relationships at a single point in time. 

Downloads

• Download the report

      

 

 

Editor's Note: The introduction to the 2014 Brown Center Report on American Education appears below. Use the Table of Contents to navigate through the report online, or download a PDF of the full report.

TABLE OF CONTENTS

Part I: Lessons from the PISA-Shanghai Controversy

Part II: Homework in America

Part III: A Progress Report on the Common Core 

INTRODUCTION

This year’s Brown Center Report on American Education represents the third installment of volume three and the 13^th issue overall since the publication began in 2000.  Three studies are presented.  All three revisit a topic that has been investigated in a previous Brown Center Report.  The topics warrant attention again because they are back in the public spotlight.

Downloads

• 2014 Brown Center Report on American Education

      

 

 

Editor's Note: The introduction to the 2014 Brown Center Report on American Education appears below. Use the Table of Contents to navigate through the report online, or download a PDF of the full report.

TABLE OF CONTENTS

Part I: Lessons from the PISA-Shanghai Controversy

Part II: Homework in America

Part III: A Progress Report on the Common Core 

INTRODUCTION

This year’s Brown Center Report on American Education represents the third installment of volume three and the 13^th issue overall since the publication began in 2000.  Three studies are presented.  All three revisit a topic that has been investigated in a previous Brown Center Report.  The topics warrant attention again because they are back in the public spotlight.

Downloads

• 2014 Brown Center Report on American Education

      

 

 

Editors' Note: The introduction to the 2013 Brown Center Report on American Education appears below. Use the Table of Contents to navigate through the report online or download a PDF of the full report.

Table of Contents

PART I: The Latest TIMSS and PIRLS Scores
PART II: The Resurgence of Ability Grouping and Persistence of Tracking 
PART III: Advanced Math in Eighth Grade

INTRODUCTION

This is the twelfth edition of the Brown Center Report. The structure of the report remains the same from year to year. Part I examines the latest data from state, national, or international assessments. This year the focus is on the latest results from the Progress in International Reading Literacy Study (PIRLS) and Trends in International Math and Science Study (TIMSS) released in December, 2012. The U.S. did relatively well, posting gains in reading, math, and science. Finland made headlines by registering declines from the last time it took the TIMSS math tests. At both fourth and eighth grades, the scores of Finland and the U.S. are now statistically indistinguishable in math. Part I also looks at the so-called “A+ countries,” named that because they were the top nations on the first TIMSS, given in 1995. Part I offers “A Progress Report on the A+ Countries,” and finds that, surprisingly, three of the six have registered statistically significant declines since 1995. Despite that, most of the A+ countries still score among the world’s leaders. The exception is the Czech Republic, which scored at approximately the international average the last time it took TIMSS in 2007.

Part II explores a perennial theme in education studies—the topics that never seem to go away in terms of research and debate. This year it’s on the controversial topics of tracking and ability grouping. An analysis of data from the National Assessment of Educational Progress (NAEP) documents a resurgence of ability grouping in fourth grade reading and mathematics. Tracking remains persistent in eighth-grade math, with about three-fourths of students in tracked classes. As readers are surely aware, both practices have been attacked for decades as inequitable, and many school analysts thought their use had diminished. Ability grouping was dominant for a long time in the elementary grades. Reading groups were the norm through most of the twentieth century and then declined dramatically in the 1990s. They are now coming back—and back strongly.

Part III is on a prominent policy or program. This year’s analysis is on the national push for eighth graders to take algebra and other high school math courses. Algebra is now the single most popular math course in eighth grade. The study in Part III uses state variation in enrollment rates to ask the question: what has happened to the NAEP scores of states that boosted their eighth-grade advanced-math enrollments? The study uncovers no relationship between change in state NAEP scores and change in enrollments. States boosting advanced math taking are no more likely to show NAEP gains than other states.

A second analysis uncovers some evidence consistent with the idea that advanced math courses are being “watered down,” that the mean achievement levels of advanced courses fall as enrollments go up. Again, change in NAEP score is the outcome of interest. The study shows that states that are more selective in math placements—not aggressively accelerating eighth graders into advanced courses—are more likely to show achievement gains in those courses.

There is one intriguing divergence from this finding: eighth-grade geometry classes. Geometry sits at the peak of the hierarchy of eighth-grade math courses, enrolling the nation’s best math students (about 5%). Presumably, these are students who took algebra in seventh grade. Increases in eighth-grade geometry enrollments evidence no association with changes in mean achievement for the course, not what one would expect if unprepared students were being accelerated into the course. This suggests that schools are implementing two different types of acceleration, one based on the age or grade of students, the other based on students’ preparation and readiness for advanced work. The analyses in the study are only correlational and cannot confirm or reject causality. Part III concludes with a discussion of hypotheses for future study to improve both strategies.

Part I: The Latest TIMSS and PIRLS Scores »

Downloads

• 2013 Brown Center Report on American Education

Video

• The 2013 Brown Center Report on American Education: How Well Are American Students Learning?
• The Resurgence of Ability Grouping and Tracking: A Return to Controversial Practices?
• Algebra and the Middle-schooler: the Impact of Advanced Math on Eighth Grade Students

      

 

 

The 2012 Brown Center Report on American Education distills the results of studies to examine the state of education in the United States. In particular, the report focuses on education policy, student learning measures, trends on achievement test scores and education reform outcomes. 

Highlights from three of the studies featured in the report are:

• Predicting the Effect of the Common Core State Standards on Student Achievement: The Common Core will have little to no effect on student achievement. The quality or rigor of state standards has been unrelated to state NAEP scores, Loveless finds. Moreover, most of the variation in NAEP scores lies within states, not between them.  Whatever impact standards alone can have on reducing within-state differences should have already been felt by the standards that all states have had since 2003.

• Measuring Achievement Gaps on NAEP: The Main NAEP consistently reports larger SES achievement gaps than the Long Term Trend NAEP. The study examines gaps between students who qualify for free and reduced lunch and those who do not; black and white students; Hispanic and white students; and English language learners and students who are not English language learners.

• Misinterpreting International Test Scores:  Educators & policymakers often misinterpret International Test Scores in three ways:  1) Dubious Conclusions of Causality, 2) The Problem With Rankings, and 3) The A+ Country Fallacy. The errors are usually committed by advocates of a particular policy position who selectively use data to support an argument, argues Loveless.

Tom Loveless discusses the report in this video:

Downloads

• Full Report

Video

• The Brown Center Report on American Education

      

 

 

Introduction

This edition of the Brown Center Report marks the tenth issue of the series and the final issue of Volume II. The publication began in 2000 with Bill Clinton in the White House and the Bush-Gore presidential campaign building toward its dramatic conclusion. That first report was organized in a three-part structure that all subsequent Brown Center Reports followed. Part I presents the latest results from state, national, or international assessments and alerts readers to important trends in the data. Part II explores an education issue in depth, sometimes by investigating different sources of empirical evidence than previous research, sometimes by posing a conventional question in an unconventional way. Part III analyzes a current or impending question regarding education policy. In all three sections, the studies strive to ask clear questions, gather the best available evidence, and present findings in a nonpartisan, jargon-free manner.

Part I of this year’s Brown Center Report focuses on international assessments. The latest data from the Programme for International Student Assessment (PISA) were released in December 2010. The performance of the United States was mediocre, and although notching gains in all three subjects, the country scored near the international average in reading literacy and scientific literacy and below average in mathematical literacy. The term “literacy” is a signal that PISA covers different content than most achievement tests, and, indeed, assesses different skills than are emphasized in the school curriculum. As the 2006 PISA Framework states, the knowledge and skills tested on PISA “are defined not primarily in terms of a common denominator of national school curricula but in terms of what skills are deemed to be essential for future life.”

Two myths of international assessments are debunked—the first, that the United States once led the world on international tests of achievement. It never has. The second myth is that Finland leads the world in education, with China and India coming on fast. Finland has a superb school system, but, significantly, it scores at the very top only on PISA, not on other international assessments. Finland also has a national curriculum more in sync with a “literacy” thrust, making PISA a friendly judge in comparing Finnish students with students from other countries. And what about India and China? Neither country has ever participated in an international assessment. How they would fare is unknown.

Part II of the report looks at state test scores on the National Assessment of Educational Progress (NAEP) in light of the recent Race to the Top competition. The federal program encouraged states to apply for $4.35 billion in new money by promising to pursue a reform agenda backed by the Obama administration. Twelve states (for this discussion, the District of Columbia will be called a state) won the grants. But are the states that won the grants the same states that have accomplished the greatest gains in student learning? Not necessarily.

Who’s winning the real race to the top? Both short- and long-term gains on NAEP are calculated with statistical controls for changes in the demographic characteristics of each state’s students. Eight states—Florida, Maryland, Massachusetts, District of Columbia, Kentucky, New Jersey, Hawaii, and Pennsylvania—stand out for making superior gains. At the other end of the distribution, Iowa, Nebraska, West Virginia, and Michigan stand out for underperforming. Five of the eight impressive states won grants, but three did not. And a few states won grants even though they are faring poorly in the race to boost student achievement. Some of the reasons why a program called Race to the Top could distribute grant money in this manner are discussed.

Part III looks at NAEP. In June 2010, the Common Core State Standards Initiative released grade-by-grade standards for reading and mathematics. Two consortia were awarded $330 million to write tests aligned to the standards, and a total of 46 states have signed to at least one group. As the only assessment administered to representative samples of American students, NAEP has called itself “the Nation’s Report Card” for decades.

How well does NAEP match up with the Common Core? We examined 171 public release items from the eighth-grade NAEP math test and coded them based on the grade level the Common Core recommends that the content be taught. The items registered, on average, two to three years below the eighth-grade mathematics recommended by the Common Core. More than 90 percent of the items from the “number” strand (content area) cover material below the eighth grade. Almost 80 percent of the items assessing “algebra” are, in fact, addressing content in the curriculum that is taught before eighth grade. With Common Core assessments on tap to begin in the 2014–2015 school year, policymakers and analysts alike need to start thinking now about how NAEP and the Common Core assessments can be reconciled so as to inform, not to confuse, the public about student achievement.

An overarching theme of this year’s report is that events in the field of education are not always as they appear to be—and especially so with test scores. Whether commentators perpetrating myths of international testing, states winning races while evidencing only mediocre progress, or an eighthgrade test dominated by content below the eighth grade, the story is rarely as simple as it appears on first blush. This report tried to dig beneath the surface and uncover some of the complexities of these important issues.

Downloads

• Download the Full Report

      

 

 

INTRODUCTION

The following is a special advanced release of a Brown Center report on NAEP and the Common Core standards. The full version of the 2010 Brown Center Report on American Education will be published at a later date.

Unlike most countries, the United States does not have national education standards, no single set of expectations for what all American teachers should teach and all American students should learn. It never has. A question that the rest of the world considers foundational to its national school systems—deciding the content of the curriculum—sits in the hands of local authorities. That is because the United States has 50 state school systems. Heterogeneity extends to the deepest levels of schooling. Even students transferring from one teacher to another within the same school may, as a consequence, learn a different curriculum than their former classmates.

So it was an historical event when the Common Core State Standards in mathematics and reading were released in June 2010. Launched by the National Governors Association and the Council of Chief State School Officers, the Common Core Standards project brought together experts in both reading and math to develop a set of standards that would be, in what became a mantra, both “higher and fewer in number” than existing state standards.[1] The standards are voluntary—states choose whether to participate—but for the first time most American students will study a uniform curriculum through at least the eighth grade. A draft of the experts’ work circulated for several months, and, based on input from other experts and the general public, the standards were finalized.[2] In September 2010, two consortia were awarded federal grants totaling $330 million to develop annual assessments aligned with the Common Core standards, and as of December 2010, 43 states and the District of Columbia have signed on to those efforts.[3] The tests are due to be given for the first time in the 2014–2015 school year.[4]

The nation currently monitors the math achievement of fourth, eighth, and twelfth graders on the National Assessment of Educational Progress (NAEP).[5] Since 1990, the main NAEP has assessed mathematics proficiency in five content strands—number properties and operations, algebra, geometry, measurement, and data analysis/statistics/probability.[6] How well does NAEP match up with the Common Core standards in mathematics?

We tackled this question by analyzing NAEP items from the eighth-grade assess­ment. NAEP items are periodically released to the public to give an idea of the content of the test. For the current study, we coded all public release items from the algebra and number strands[7] based on the grade at which the Common Core recommends teaching the mathematics assessed by the item. The 2009 NAEP Framework in Math­ematics calls for number and algebra items to comprise half of the eighth-grade assess­ment.[8] A total of 171 items were available, 98 from the number strand and 73 from algebra.[9] We were unable to code four items (two from each strand) because they assess skills not found in the Common Core.

A precursor to this study can be found in the 2004 Brown Center Report.[10] In that study, we coded the grade level of public release items labeled as “problem solving,” one of NAEP’s process strands (different from the content strands). Only problems involving the application of arithmetic were analyzed. At what grade level are students taught the arithmetic required to answer NAEP problem-solving items? We discov­ered that the mean fourth-grade NAEP item registered at 3.2 and the mean eighth-grade item at 3.7, suggesting that the typical item could be answered using arithmetic taught by the end of third grade. Primarily, this finding stems from NAEP’s reliance on whole number arithmetic in word problems. We found that approximately 70 percent of the eighth-grade items focused on whole numbers. Problems with fractions, decimals, or percents—forms of rational numbers taught after third grade—are not common on NAEP.[11]

The 2004 study used the Singapore Math program as a rubric to code the grade level of items, assigning a value according to the grade and semester in which the arithme­tic of the item was taught. By using the Com­mon Core and evaluating the entire context of items, the current study’s rubric produces higher grade-level estimates for items. Prob­lems involving only simple arithmetic are classified at a higher grade level if they are posed in the context of more sophisticated topics that are taught at a later grade (e.g., co­ordinate plane, equations with two variables). Selected NAEP items are shown [in the full report available for download on this page].

Downloads

• Download the Full Report

      

 

 

This year’s Brown Center Report contains studies taking a long view. Part I examines national test data going back to 1971 from the National Assessment of Educational Progress (NAEP). The study in Part II compares the 1989 test scores of more than 1,000 schools to the same schools’ scores in 2009. Part III compares the test scores of conversion charter schools from 1986, when they operated as traditional public schools, to those from 2008, when they operated as charter schools. The studies tackle perennial questions that, as often happens in education, manifest themselves as controversial topics on the contemporary scene: how to interpret trends in test scores, the distribution of achievement, school turnarounds, and charter schools.

Part I rejects the conventional reaction to the 2009 NAEP scores. Scores in fourth-grade math were unchanged from 2007 to 2009. Eighth-grade scores were up a little. Press articles featured expressions of disappointment and concern, primarily from protagonists who used the flat scores to support policy arguments. Part I places the 2009 scores in the context of the 19-year history of the main NAEP, and after comparing the latest scores with results from other equally trustworthy tests of U.S. math achievement, concludes that the hand-wringing is unwarranted.

So when is a purported NAEP trend really a trend? Part I continues by examining achievement gaps, not between two racial, ethnic, or socioeconomic groups, but between the nation’s highest- and lowest-achieving students. It focuses on the distribution of academic achievement instead of the direction of average achievement. The study is a follow-up to a 2009 Fordham Institute paper documenting that the gap between high- and low-achieving students has been shrinking in recent years. The data in Part I show that the trend, which began sometime around 1998 or 1999, is historically unprecedented and extends across subjects (reading and math), grades (fourth and eighth), and tests (long-term trend and main NAEP). It is also more pronounced in public schools than in private schools. The two analyses in Part I highlight the contrast between a trend indicated by data collected from several independent sources over an extended period of time and speculative assertions arising from “instant analysis” of a single set of test scores.

Part II asks a simple question: do schools ever change? The sample consists of 1,156 schools in California that offered an eighth grade in 1989 and 2009. Test scores from 1989 are compared to scores from 2009. The scores are remarkably stable. Of schools in the bottom quartile in 1989—the state’s lowest performers—nearly two-thirds (63.4 percent) scored in the bottom quartile again in 2009. The odds of a bottom quartile school’s rising to the top quartile were about one in seventy (1.4 percent). The reverse was true as well, with similar percentages of top quartile schools staying among the top performers (63.0 percent) or falling to the bottom quartile (2.4 percent). Changes in a school’s socioeconomic status had only a marginal statistical relationship with test score changes.

The persistence of test scores has major implications for today’s push to turn around failing schools. It can be done, but the odds are daunting. California certainly cannot be accused of inactivity in education reform from 1989 to 2009. Few states tried as many diverse, ambitious reforms that targeted every aspect of the school system—finance, governance, curriculum, instruction, and assessment. Not only have these efforts failed to elevate California from its low national ranking on key performance measures, but they have also had little effect on the relative ranking of schools within the state.

The study suggests that people who say we know how to make failing schools into successful ones but merely lack the will to do so are selling snake oil. In fact, successful turnaround stories are marked by idiosyncratic circumstances. The science of turnarounds is weak and devoid of practical, effective strategies for educators to employ. Examples of largescale, system-wide turnarounds are nonexistent. A lot of work needs to be done before the odds of turning around failing schools begin to tip in a favorable direction.

Part III looks at charter schools. Conversion charters are favored by the Obama administration as a restructuring strategy. Most charter schools are start-ups, begun from scratch by their founders. Conversion charters are schools that are traditional public schools and convert to charter school status. They typically continue to rely on their home districts for several functions (e.g., maintenance of buildings, managing pension obligations, transportation services) but are freed from regulations pertaining to curriculum and instruction. The idea is that schools can be more productive if they are allowed to tailor core educational operations to the needs of their students.

California has the largest number of conversions, and the study was able to collect data on two cohorts: 49 schools from 2004 and 60 schools from 2008. For both cohorts, test score data were also available from 1986, allowing a comparison of scores before and after the schools converted. The analysis is exploratory and mainly descriptive. No causal conclusions can be derived from the data.

What do we know about conversions? Test scores look similar before and after conversion. The 2004 cohort evidences a 2 to 3 percentile point advantage as charters, but the 2008 cohort’s scores declined slightly, less than 2 points, from 1986 to 2008. On several key characteristics, conversions look more like traditional public schools than start-up charters. Compared with start-ups, conversions are more concentrated in urban areas, have larger student enrollments, and serve greater numbers of Hispanic and black students. Teachers at conversions are more experienced and more likely to hold teaching certificates, particularly in bilingual education. It is clear that future evaluations of charter schools must differentiate between start-ups and conversions because of the significant institutional differences between the two types of charters.

To sum up, the studies in this year’s Brown Center Report focus on long-term changes. Part I analyzes NAEP data. Parts II and III examine California test scores from the 1980s and compare them to scores from recent years. Because of its long history of testing, California is currently one of the few states able to provide assessment data for such long-term comparisons. That will change as other states continue to test students annually. Creating rich archives of student performance data bodes well for school reform. Improving schools requires patience and persistence, what education professors Richard Elmore and Milbrey McLaughlin1 call “steady work.” It also requires good information to verify whether reforms have paid off, or, like many efforts in education, produced hopeful signs that soon vanish. The future looks bright if analysts’ capacity to peer into the past continues to improve.

Downloads

• Download Full Report
• Download News Release

      

 

 

Introduction

The watchword of this year’s Brown Center Report is caution—caution in linking state tests to international assessments—“benchmarking” is the term—caution in proceeding with a policy of “algebra for all eighth graders,” caution in gleaning policy lessons from the recent progress made by urban schools. State and local budget woes will restrain policymakers from adopting costly education reforms, but even so, the three studies contained herein are a reminder that restraint must be exercised in matters other than budgets in governing education well. All too often, policy decisions are based on wishful thinking rather than cautious analysis. As education evolves as a discipline, the careful analysis of high-quality data will provide the foundation for meaningful education reform.

The report consists of three sections, each discussing a separate study. The first section looks at international testing. Powerful groups, led by the National Governors Association, are urging the states to benchmark their state achievement tests to an international assessment, the Programme for International Student Assessment (PISA). After comparing PISA to the Trends in International Mathematics and Science Study (TIMSS), the other major international assessment in which the United States participates, the Brown Center analysis examines findings from a chapter of the 2006 PISA report that addresses student engagement. The chapter presents data on students’ attitudes, values, and beliefs toward science.

Benchmarking proponents argue that PISA offers policy guidance to American school officials by identifying the characteristics of successful school systems around the world. The Brown Center analysis calls that claim into question. The PISA report makes causal claims from cross-sectional data that cannot support such inferences. The chapter on student engagement presents inferences based on selective treatment of data, with policy recommendations going beyond the evidence adduced to support them.

Moreover, PISA poses questions that contain ideological bias. To define scientific literacy as encompassing beliefs as well as knowledge—a definition also embraced by skeptics of evolution—is a dubious position for any science assessment to take. PISA wants to assess whether students are capable of applying science to public policy. Fair enough. That capacity can be evaluated, however, without making a judgment about students’ political beliefs. PISA asks students whether they support several environmental policies and then creates an index of “responsibility for sustainable development” from the responses. Responses in favor of the policies are responsible; those opposed are not. That kind of questioning is inappropriate on a science assessment. Without serious reform, PISA is inappropriate for benchmarking.

The second section tackles another hot topic in policy circles—whether all eighth graders should take an algebra course. California recently adopted a universal eighth grade algebra policy that will be implemented in 2011, joining a Minnesota policy with the same objective and implementation date. Are all eighth graders prepared to take an algebra class? National data are examined from eighth grade math classes in 2005 to answer that question.

Low achievers in mathematics, those scoring in the bottom tenth of all students, function several years below grade level. A shocking percentage of these low achievers, 28.6 percent, were enrolled in advanced mathcourses—Algebra I, Algebra II, or Geometry—in 2005. A policy of algebra for all eighth graders will dramatically increase the proportion of these misplaced math students. Sample math items are presented to illustrate the large gaps in the misplaced students’ mathematical knowledge, in particular, their poor grasp of fractions, decimals, and percentages. The misplaced students are described in terms of demographic characteristics, the schools they attend, and the teachers who are instructing their math classes. The portrait is deeply troubling. The misplaced students are some of the nation’s most vulnerable youngsters. The analysis raises questions about the feasibility of an “algebra for all” policy until we know how to reduce the number of underprepared students and how to effectively teach algebra to students who struggle with basic arithmetic.

The final section of the report is a good news story. The 2001 Brown Center Report presented an analysis of academic achievement in big city school districts. That study compared test scores for school districts serving the top fifty cities in the 2000 U.S. Census to the average test score in the cities’ respective states. Not surprisingly, the big city districts lagged far behind. This year’s report replicates that study using the most recent achievement data. Big city schools have made significant gains. While all school districts have notched achievement gains, the big city districts made even larger gains than other districts. They are closing the gap with suburban and rural districts, slowly, to be sure, but they are clearly making progress.

The analysis does not hazard a theory as to why big city achievement is rising. One possible catalyst is mayoral control, a popular urban reform in recent years. The data neither support nor refute the effectiveness of mayoral control. Another possible influence is No Child Left Behind. The law targets low-performing students, and studies of test scores at both state and national levels have shown greater progress at the bottom of the achievement distribution than at the top. Having a disproportionate share of low achievers, big city schools benefit from that trend. As noted above, cross-sectional data are limited in what they can reveal about the causes of events, so whether NCLB has played a role in the progress of big city schools is merely speculative. In addition, not all big city districts have made gains.

A daunting obstacle to determining the drivers of academic trends is that there is no authoritative source that documents the policies that local districts have adopted, along with such details as when particular policies were started, when they were modified, what policies they replaced, and how they were implemented. The Brown Center Report ends with a call for a periodic national inventory of district policies across the country. We are getting much better at determining how well students are learning and tracking trends in test scores as they unfold over time. But policy analysis lags behind. Explaining why students are learning more or less—and really pinpointing the causes of trends in achievement—will take much more information about the policies and practices of our schools.


Download the Press Release »
View Tom Loveless's Presentation on the Use and Misuse of International Assessments »
View Tom Loveless's Presentation on Urban School Achievement »

Downloads

• Download

      

 

 

Introduction

The following is a special advanced release of a Brown Center report on eighth grade algebra levels.  The full report will be published in February 2009. 

Algebra in eighth grade was once reserved for the mathematically gifted student. In 1990, very few eighth graders, about one out of six, were enrolled in an algebra course. As the decade unfolded, leaders began urging schools to increase that number. President Clinton lamented, “Around the world, middle students are learning algebra and geometry. Here at home, just a quarter of all students take algebra before high school.”¹ The administration made enrolling all children in an algebra course by eighth grade a national goal. In a handbook offering advice to middle school students on how to plan for college, U.S. Secretary of Education Richard Riley urged, “Take algebra beginning in the eighth grade and build from there.”² Robert Moses ratcheted up the significance of the issue by labeling algebra “The New Civil Right,” thereby highlighting the social consequences of so many poor and minority students taking remedial and general math courses instead of algebra.³

The campaign was incredibly successful. Several urban school districts declared a goal of algebra for all eighth graders. In 1996, the District of Columbia led the nation with 53 percent of eighth graders enrolled in algebra. From 1990 to 2000, national enrollment in algebra courses soared from 16 percent to 24 percent of all eighth graders.

The surge continued into the next decade. Eighth-grade enrollment in algebra hit 31 percent nationally in 2007, a near doubling of the 1990 proportion. Today more U.S. eighth graders take algebra than any other math course.⁴ In July 2008, the State of California decided to adopt an algebra test as its eighth-grade assessment of student proficiency. The policy in effect mandates that all eighth graders will be enrolled in algebra by 2011.

At first glance, this appears to be good news. Transcript studies indicate that 83 percent of students who take geometry in ninth grade, most of whom completed algebra in eighth grade, complete calculus or another advanced math course during high school.⁵ Research also suggests that students who take algebra earlier rather than later subsequently have higher math skills.⁶ These findings, however, are clouded by selection effects—by the presence of unmeasured factors influencing who takes algebra early and who takes it late. Schools routinely assign incoming eighth graders to math courses based on how much math students already know. Moreover, it is no surprise that excellent math students want to take the most challenging math courses available to them and that low-achieving students avoid these courses as long as possible. Whether algebra for eighth graders is a good idea, especially for those who have not learned basic arithmetic, cannot be concluded from existing evidence. Studies that test for causality, such as experiments with random assignment of students to treatment and control groups, have not been conducted.

The push for universal eighth-grade algebra is based on an argument for equity, not on empirical evidence. General or remedial math courses tend to be curricular dead-ends, leading to more courses with the same title (for example, General Math 9, General Math 10) and no real progression in mathematical content. By completing algebra in eighth grade—and then completing a sequence of geometry as freshmen, advanced algebra as sophomores, and trigonometry, math analysis, or pre-calculus as juniors—students are able to take calculus in the senior year of high school. Waiting until ninth grade to take algebra makes taking calculus in high school more difficult. From this point of view, expanding eighth-grade algebra to include all students opens up opportunities for advancement to students who previously had not been afforded them, in particular, students of color and from poor families. Democratizing eighth-grade algebra promotes social justice.


Download the Full Report »
Download the Press Release »


_{1 Remarks by President Clinton, Education Roundtable, Springbrook High School, Silver Spring, Md., March 16, 1998. Available at http://www.ed.gov/inits/Math/timsroun. html.
2 Quoted in Matthew Bowers, “Virginia and the U.S. are Improving Slightly at Math, but We Lag Behind Our Economic Competitors in the Developed World,” The Virginian Pilot, March 28, 1997, p. B3.
3 Robert Moses, “Algebra, the New Civil Right,” in The Algebra Initiative Colloquium, Volume II, edited by Carol Lacampagne and others (U.S. Department of Education, 1995), pp. 53-67.
4 Data available on the main NAEP data explorer: http://nces.ed.gov/nationsreportcard/nde/. See also Jay Matthews, “Adding Eighth Graders to the Equation,” The Washington Post, March 12, 2007, p. B1.
5 Carolyn Shettle and others, America’s High School Graduates: Results from the 2005 NAEP High School Transcript Study (Department of Education, 2007), p. 11. Other than calculus, advanced math is defined as pre-calculus or AP statistics.
6 Julia B. Smith, “Does an Extra Year Make Any Difference? The Impact of Early Algebra on Long-term Gains in Mathematics Attainment,” Educational Evaluation and Policy Analysis, 18 no. 2 (1996): 141-153.}

Downloads

• Download

     

 

 

Introduction

This is the seventh edition of the Brown Center Report on American Education. As in the past, the report consists of three sections. The first section examines the latest test score data on math and reading achievement.This year the analysis focuses primarily on results of the 2007 National Assessment of Educational Progress (NAEP), including a discussion of NAEP achievement levels.

The second section investigates a general theme or trend in education. This year the second section scrutinizes enrollment patterns in private and public schools. Section three looks at an issue of policy relevance. International test data are examined to see whether a relationship exists between national math scores and the amount of time students spend learning mathematics in different countries.

In past Brown Center Reports, no effort was made to tie the three sections together. The studies stood on their own. This year, however, the studies do have something in common: they investigate phenomena in education that—at least at first blush—do not make sense. Data on the percentage of students performing at “proficient” on the NAEP performance levels, for example, are routinely cited as evidence that U.S. schools are underperforming. But as a 2007 study by Gary Phillips of the American Institutes for Research shows, every nation in the world—including high flyers such as Singapore and Japan—would have significant numbers of students falling below proficient if they were administered the NAEP test. How can that be?

The second section’s study of public and private school enrollment is precipitated by another oxymoron. Public opinion polls consistently show that the public considers private schools superior to public schools. Yet private school enrollment peaked around 1960 and has declined since then. People express a belief in public opinion surveys that they apparently contradict when selecting schools. What is going on?

The puzzle that is featured in the third section involves previous research on time and learning. When researchers have attempted to find a correlation between national test scores in mathematics and the amount of time different nations devote to teaching mathematics, no relationship has been found. Very odd. The correlation of homework and national test scores is stranger yet, with a negative relationship being the usual finding—the more homework given in math, the lower a nation’s test score. We analyze these relationships using a different approach and come up with different findings.

Increasingly, education’s most important questions are researched, debated, and decided with data. This is surely a healthy development as the field moves toward embracing the scientific methods that have benefited the intellectual disciplines that inform public policy. That said, with an election year now looming, it is important to be on the lookout for oversimplifications of either education’s most pressing problems or its most promising solutions. Simply marshalling good statistics is not enough. The educational enterprise is exceedingly complex, and many cross-currents exist in educational data—evidence supporting one hypothesis, when looked at from another angle, might be seen as supporting an alternative. Even with sound data, many mysteries remain in American education. This issue of the Brown Center Report explores three of them.

Download the Full Report
Download the Press Release
Download Presentation by Tom Loveless
Read More about the Launch Event and View Clips
Read More about the Brown Center Report on American Education (2000-2007)

Downloads

• Download
• Presentation
• Press Release

     

 

 

This report launches the second volume of the Brown Center Report on American Education. The five issues of volume one were published from 2000 to 2004. Volume one included regular reports on data from the National Assessment of Educational Progress (NAEP) and state assessments, analysis of student achievement in charter schools, a study of trends in homework, evaluations of the federal government's Blue Ribbon Schools Program, an investigation of the academic performance of high schools with powerhouse sports teams, analyses of student achievement in urban school districts and rural schools, a survey of exchange students from countries abroad to see what they think of American schools, and a survey of the mathematics preparation of middle school math teachers. Volume two will explore similar topics related to how well American students are learning.

As in volume one, the reports of volume two will be divided into three parts. Part one reviews the latest data on student learning in the U.S. In this issue, the most recent results from both NAEP tests, the long term trend and the main, receive attention for what they reveal about American students' progress in reading and mathematics. Part two looks at the "happiness factor" in education, analyzing international data to see whether students' self-confidence and enjoyment of math and the relevance of lessons that students experience in classrooms are correlated with higher math achievement. Do nations with happier students score higher on math tests than nations in which students are not quite as happy?

Part three looks at how states have responded to the No Child Left Behind Act. Several analysts have recently concluded that states are "racing to the bottom" by artificially inflating the number of students who demonstrate proficiency on state tests. It is indisputable that states report larger numbers of proficient students than the NAEP test. But the studies have overlooked some key questions. Is NAEP such a good test that it should be used as a benchmark for judging state assessments? Can NCLB be blamed for the discrepancies between reported levels of proficiency on NAEP and state tests? How large were the discrepancies before NCLB?

We hope readers will consider this issue of the Brown Center Report as interesting and provocative as previous ones. On a personal note, this also is the first issue that will be published without Paul DiPerna on hand for the release. After six and a half years in the Brown Center, Paul has moved on to bigger and better things at the Friedman Foundation in Indianapolis. We thank Paul for his tireless work on behalf of the Center and wish him a terrific career in Indianapolis.

PART I: The Nation's Achievement

PART II: The Happiness Factor in Student Learning

PART III: Are States Honestly Reporting Test Scores?

View the Powerpoint Presentation

Downloads

• Download
• Powerpoint Presentation

     

 

 

This is the fifth annual edition of the Brown Center Report on American Education. It analyzes the difficulty of items on the math portion of the National Assessment of Educational Progress (NAEP), examines the content training of middle school math teachers, and evaluates the Blue Ribbon Schools Program. The NAEP assesses student performance and periodically issues what is known as the Nation's Report Card. The No Child Left Behind Act requires that all teachers demonstrate mastery of the curriculum that they teach. The Blue Ribbon Schools Program singles out highly successful schools for national recognition. Measuring what students know, staffing classrooms with competent teachers, rewarding excellence—these are all activities of profound importance to the vitality of the nation's schools.

Most federal—including No Child Left Behind—honor federalism by leaving important powers in the hands of state and local officials. But final authority over NAEP and over Blue Ribbon Schools is held by officials in Washington, DC. They are truly federal government programs. And they are well-established. NAEP testing began in 1969. The Blue Ribbon Schools Program started in 1982. These programs have a track record.

The NAEP has publicly released more than 500 items from its mathematics tests. In the first section of this report, after reviewing test data released in 2004, we analyze a sample of NAEP items and discover that the mathematics required to solve many of the problems is extraordinarily easy. Most of the arithmetic one would need to know to solve the average item on the eighth grade NAEP is taught by the end of third grade. The second section of the report presents a survey of middle school math teachers, focusing on their educational background and professional development. A significant number of math teachers at this level lack formal undergraduate training in mathematics, and the professional development they are receiving appears to be inadequate to remedy the problem. The third section of the report examines the Blue Ribbon Schools Program. We replicate a study published in the first Brown Center Report, analyzing test data from schools that have won Blue Ribbons. In 2000, we found that at least one-fourth of the winning schools did not deserve to win the award, at least based on state test scores in reading and math. In this year's study, we find that the Blue Ribbon program today makes fewer errors. There is still room for improvement, but far fewer failing schools are receiving the awards.

A personal note. For eight years, Diane Ravitch has edited the Brookings Papers on Education Policy and organized the Brown Center's annual conference. Diane is now going to step down from those duties to spend more time on her own scholarship. She will maintain her non-resident senior fellow position at Brookings. The papers from the 2004 conference, featuring papers on eight great ideas in education, will be published early in 2005. Diane is our nation's leading education historian, and her penetrating analysis of contemporary educational problems is a national treasure. We at the Brown Center have been very fortunate to have worked with her—and look forward to working with her on future endeavors.

PART I
The Nation's Achievement/NAEP Math Items

PART II
The Content Training of Middle School Math Teachers

PART III
Blue Ribbon Schools Revisited

Downloads

• Download
• Powerpoint Presentation

     

 

 

This is the fourth edition of the Brown Center Report on American Education. The report premiered in the fall of 2000, as the presidential campaign between George W. Bush and Al Gore drew to an exciting finish. This year's report is published as Democratic candidates vie for their party's nomination for president. Education will figure prominently in the 2004 election, as it did in the last. President Bush will herald No Child Left Behind (NCLB) as the most important federal education legislation in a generation. Democrats will argue that without ample funding the law is an empty promise. They will point out that the Bush administration put the brakes on federal education spending just as NCLB was getting off the ground.

The partisan debate over how to improve American education has been following the same script for the past four years. Republicans stress holding schools accountable and make concessions on funding increases. They seek to neutralize the Democrats' traditional strength with voters on education issues. Democrats stress the need for providing more resources to schools and make concessions on accountability provisions. They seek to present education as a key plank in their party's domestic agenda for the country. This year's Brown Center Report examines several issues that are important to No Child Left Behind and ongoing efforts to improve American schools.

The first section of the report analyzes the latest data on student achievement and asks how the nation's students are doing in reading and mathematics. Achievement in rural schools receives a closer look.

The second section is a study of homework. Conventional wisdom is that higher academic standards, a bedrock of No Child Left Behind, have driven up the amount of students' homework. Stories of tired, over-worked kids abound. After examining several different sources of data on the topic, the study concludes that virtually no evidence exists that homework has increased in recent years, nor that the homework load has become—or ever was—overwhelming. The stories of children laboring under onerous amounts of homework appear to feature a small proportion of children who, though their predicaments are real, are not typical.

The third section of the report presents a follow-up of last year's study on charter schools. This year's study examines charters' test scores, with a special focus on achievement in conversion charters, schools that were previously regular public schools and converted to charter status, and charters managed by educational management organizations (EMOs), professional management firms. Both types of charter school can lay claim to a particular form of expertise. When a regular public school converts to a charter school, the most talented and experienced teachers and administrators usually stay on board. The very existence of educational management organizations is based on the premise that expert managers, who are usually not educators and come from the private sector, can employ their leadership skills to make schools more productive.

Readers should be informed that the author of the Brown Center Report, Tom Loveless, has had several professional affiliations with charter schools. These associations cannot change the data on which the following analysis is based, but they might have, in ways unknown, influenced the choice of questions in the research or the interpretation of the findings.

Downloads

• Download
• Powerpoint Presentation

     

 

 

As in the past, the report is divided into three independent sections. The first section reports on current trends in test scores in reading and mathematics. Arithmetic receives special consideration. A troubling body of evidence is presented that suggests students' computation skills have stagnated or even declined in recent years. Remarkably, the National Assessment of Educational Progress (NAEP), the nation's report card, does not report how well elementary grade students are performing in arithmetic.

The second section of the report revisits last year's study of high school culture. First, we replicate the 2001 survey of foreign exchange students with American students who have studied in high schools abroad, asking them also to compare U.S. high schools to high schools around the world. By compelling margins, American and foreign students agree that success at sports means much more to U.S. teenagers than to teens in other countries. Is this cause for concern? Does it interfere with the nation's efforts to raise academic achievement?

If holding athletic accomplishments in high esteem creates problems, one would expect them to surface in high schools with highly successful athletic teams. We present a study of high schools that are sports powerhouses, schools that in recent years have been the best in the nation in football, baseball, and basketball. It is clear that these schools are excellent at sports. What about academics? Is dominance in team sports attained at a cost to excellence in reading and mathematics? After analyzing test score data from dozens of states, Brown Center researchers are confident that the answer is no, excellence is not zero sum when it comes to sports and academics. There is no evidence that schools suffer academically when they excel in athletics. On state tests, the sports powerhouses score about as one would expect, no better or worse than non-powerhouse schools serving similar populations. And there is evidence, though only suggestive, that some schools are capable of making excellence at sports and excellence at academics mutually reinforcing.

The third section of the report looks at charter schools. Charters have proliferated across the country since the first few opened in Minnesota nearly a decade ago. There are now about 2,400 charters serving 250,000 students. Very little is known about academic achievement in charter schools, so we examined the test scores of charters from 1999 to 2001 in ten states. In a nutshell, charter schools performed about one-quarter standard deviation below comparable regular public schools on these three years of state tests. We do not know why charters performed at this level. They may have attracted students who were already low achieving, which explains why parents sought an alternative to the local public school. Thus, readers are cautioned that these test scores may or may not reflect the quality of education students have received and are receiving at charters. And we offer a few suggestions on how achievement in charters can be evaluated as fairly and accurately as possible in the future, especially with state accountability systems beginning to take hold.

View Powerpoint Presentation

Downloads

• Download
• 2002 Brown Center Report Presentation

     

 

 

Education commanded a prominent position in the 2000 campaign for president. Both candidates featured proposals for improving schools. Both talked about education on the stump, in campaign advertisements, and in televised debates. Within a few days of taking the oath of office, George W. Bush made education legislation the incoming administration's top priority. Secretary of Education designate Rod Paige stood at Bush's side as the president declared a vigorous new role for the federal government in school reform, an effort, Bush pledged, that would provide a better education for every American student and leave no child behind.

This pledge highlights a problem discussed in this report, the second edition of the Brown Center Report on American Education. The concern is with achievement gaps, the yawning disparities in achievement between the academically successful—be they students, schools, or nations—and the unsuccessful. Focusing on gaps demands that we think about achievement as not simply a phenomenon measured by test score averages. Statisticians would say that the distribution of achievement is now being taken into account. In the case of student achievement, for example, it means paying attention to the range of scores from top to bottom, the number of students performing at various levels of proficiency, and the learning differences that different scores represent. Focusing on gaps forces people to see education as a valued resource that is distributed differently to different students, inviting questions about the educational system's fundamental fairness and a sober consideration of what it will really take to ensure that students who struggle academically learn what they need to learn, of what—truly—is required so that no child is left behind.

Although varying in content from year to year, the Brown Center Report is organized by three consistent sections. The first section uses evidence released in the previous twelve months to evaluate student achievement in America's schools. This year, we investigate the enormous gap between the U.S. and other nations in mathematics achievement and analyze, in reading achievement, the gap between the nation's best and worst readers in fourth grade.

The second section explores a theme in depth. This year's theme is the culture of the American high school. A survey of foreign exchange students was conducted to get their impressions of American high schools. Responses indicate that the culture of high schools, and the low status of academic accomplishments within teen culture, may present formidable obstacles to the attainment of academic excellence.

The third section analyzes achievement in urban schools. Test score data from dozens of states were collected to determine how urban school districts are doing in comparison to rural and suburban districts in the same state. We present scores for 39 of the fifty most populous cities in the U.S. and analyze the performance of urban districts serving a substantial number of children in poverty. The data do not allow for any firm policy conclusions or recommendation, although the most popular urban school reforms of the 1990s are briefly discussed. This section's primary objective is to estimate the achievement gaps that urban schools must overcome to reach parity with their urban and suburban counterparts.

View the Powerpoint Presentation

Downloads

• Download
• 2001 Brown Center Report Presentation

     

 

 

There was a time when student test scores primarily concerned two groups: individual parents, as they received reports on their children's progress, and real estate agents, as they helped home buyers compare schools in different neighborhoods. Test scores were difficult to obtain and rarely discussed in public.

But now measures of student achievement are splashed across the front page of major newspapers, widely available on the Internet, and the subject of intense scrutiny and furious spin. Politicians closely watch test scores. From the race to the White House to the thousands of contests for local school boards, candidates stretch and bend scores to make them look as good or as bad as possible. Teachers unions and other organizations cite data to defend public schools and assure the public that all is well--or at least not as bad as everyone thinks. On the other side, critics of public schools publish voluminous studies documenting a steady decline in student performance.

What should the average citizen believe?
The purpose of this report is four-fold: to report on the direction of achievement in U.S public schools, that is, to determine whether it's going up, down, or sideways; to figure out whether any change that is detected is big, small, or insignificant; to dig under the numbers and uncover the policies and practices influencing the direction of student achievement; and, finally, to figure out whether the public is getting the full story on student learning. Americans spend $350 billion each year on elementary and secondary education. They deserve an accurate, non-partisan, no-holds-barred, data-driven account of what they're getting for their money.

The Brown Center Report will appear annually, this being the first edition. Although varying in content from year to year, the report will be presented in the same three sections. The first section will use the latest and best evidence available to evaluate student achievement in America?s schools. The second section will go into greater depth on a theme related to student learning. This year?s theme is mathematics achievement. The third section will evaluate the impact of policies and practices on student learning. This year?s topics are the use of calculators in math instruction and state and federal programs that single out exemplary schools for special recognition.

View the Powerpoint Presentation

Downloads

• Download
• The 2000 Brown Center Report Presentation