Brookings: Topics - Teachers

Education Technology: The Next Generation

Wed, 20 Mar 2013 14:00:00 -0400

Event Information

March 20, 2013
2:00 PM - 3:30 PM EDT

Falk Auditorium
Brookings Institution
1775 Massachusetts Avenue NW
Washington, DC 20036

Education technology is an accepted and integral component of reforming and improving the American educational system. The educational possibilities made possible by today's technology and mobile devices are expansive, with mobile phones apps, interactive games, distance learning programs, and environment software readily available to most students and teachers in the United States. Now that these tools are a common feature in the classroom, how can technology’s integration in education be expanded to best benefit students? How can educators incorporate the latest technologies to improve education and assess what proves effective? What future innovations can be expected in educational technology?

On March 20, Governance Studies at Brookings hosted a public forum to discuss the next generation of education technologies. A panel of experts discussed recent advances in educational technology and what new innovations are on the horizon. Participants can join the conversation on Twitter at hashtag #TechCTI.

Read the related paper: Education Technology Success Stories, by Darrell West and Joshua Bleiberg

Poor Students Can’t Afford Teacher Strike

Beth Akers and Matthew M. Chingos — Mon, 10 Sep 2012 11:51:00 -0400

Ninety-three years ago yesterday, the Boston police force went on strike, leaving the city unprotected while the state scrambled to find replacements. Governor Calvin Coolidge’s declaration of support for the city—he said that “There is no right to strike against the public safety, anywhere, anytime”—established his national reputation that ultimately led to the presidency.

Public outrage at labor actions that compromise public safety has historically been a bipartisan affair. Coolidge was a Republican but his actions earned the respect of Democratic President Woodrow Wilson, who hailed his re-election as Massachusetts governor as “a victory for law and order.” Nearly 20 years later, President Franklin Roosevelt shared his view that a strike by public employees of any sort is “unthinkable and intolerable.”

The impacts of the Chicago teacher strike that began today may not be as immediately obvious as the looting and vandalism that descended on Boston in 1919, but they are just as serious. Research from a large, urban school district found that teacher absenteeism has a negative impact on student learning in math.

But a strike doesn't leave students with substitute teachers—it leaves them without any school at all. Research on summer learning loss shows that being out of school has a disproportionate effect on low-income students. One recent study found that “while all students lose some ground in mathematics over the summer, low-income students lose more ground in reading, while their higher-income peers may even gain.” In other words, the consequence of being out of school is to increase the already unacceptably large achievement gap between low-income students and their affluent peers.

The American labor movement has made important contributions in areas ranging from workplace safety to child labor to employment discrimination. There are good reasons to believe that the public ought to accept higher coal prices resulting from a strike to protect the lives of miners. But the public should not tolerate damage to the education of disadvantaged students resulting from a strike over disagreements about teachers’ salaries, benefits, job security, and method of evaluation.

The Chicago Teachers Union’s differences with the city over how the public schools ought to be run may well be legitimate. But those battles should be fought in the court of public opinion and ultimately at the ballot box, not through strikes that come largely at the expense of poor children.

Authors

Image Source: © Stringer . / Reuters

Data Analytics and Web Dashboards in the Classroom

Tue, 04 Sep 2012 14:00:00 -0400

Event Information

September 4, 2012
2:00 PM - 3:30 PM EDT

Falk Auditorium
Brookings Institution
1775 Massachusetts Avenue, N.W.
Washington, DC 20036

Common school evaluation techniques suffer from several limitations: they provide little immediate feedback to students; require teachers to spend hours grading routine assignments; and fail to take advantage of innovative assessment techniques designed to improve the learning process. Digital technology, however, enables learning in real-time and provides immediate feedback to students from teachers. So-called “big data” allow educators to mine information for insights regarding student performance and learning approaches so they can analyze what students retained and which techniques are most effective for each student. By incorporating data analytics into the classroom, teachers can study learning in far more nuanced ways.

On September 4, the Center for Technology Innovation at Brookings hosted a forum to discuss how digital tools, such as online analytics and web dashboards, are transforming student evaluations and educational strategies. A panel of experts examined the potential for data mining, data analytics, and web dashboards to improve American education. After the program, speakers took audience questions.

This event was live Tweeted at hashtag #TechCTI.

Video

Full Event - Data Analytics and Web Dashboards in the Classroom

Audio

Data Analytics and Web Dashboards in the Classroom

Transcript

Transcript (.pdf)

Event Materials

Education Technology, Distance Learning & the Innovative American Classroom

Tue, 17 Jul 2012 10:15:00 -0400

Event Information

July 17, 2012
10:15 AM - 11:30 AM EDT

Falk Auditorium
Brookings Institution
1775 Massachusetts Avenue, N.W.
Washington, DC 20036

Education technology tools offer students improved access to resources, digital materials and a dynamic, personalized learning experience. Distance learning connects geographically-disadvantaged students with instructors, fellow classmates, course offerings, and educational experiences not accessible in their nearby brick-and-mortar schools.

On July 17, the Center for Technology Innovation at Brookings hosted a forum on the growing use and influence of distance learning in transforming American education. A panel of experts discussed the challenges and positive outcomes of integrating education technologies and distance learning techniques into academic instruction, as well as how these tools are transforming the definition of the American classroom.

Video

Full Event - Education Technology, Distance Learning & the Innovative American Classroom

Audio

Education Technology, Distance Learning & the Innovative American Classroom

Transcript

Transcript (.pdf)

Event Materials

20120717_distance_learning

Digital Schools : How Technology Can Transform Education

Darrell M. West — Thu, 31 May 2012 00:00:00 -0400

Brookings Institution Press 2012 160pp.

Nearly a century ago, famed educator John Dewey said that "if we teach today’s students as we taught yesterday’s, we rob them of tomorrow." That wisdom resonates more strongly than ever today, and that maxim underlies this insightful look at the present and future of education in the digital age.

As Darrell West makes clear in Digital Schools, today’s educational institutions must reinvent themselves to engage students successfully and provide them with the skills needed to compete in an increasingly global, technological, and online world. Otherwise the American education system will continue to fall woefully short in its mission to prepare the population to survive and thrive in a rapidly changing world.

West examines new models of education made possible by enhanced information technology, new approaches that will make public education in the post-industrial age more relevant, efficient, and ultimately more productive. Innovative pilot programs are popping up all over the nation, experimenting with different forms of organization and delivery systems.

Digital Schools surveys this promising new landscape, examining in particular personalized learning; realtime student assessment; ways to enhance teacher evaluation; the untapped potential of distance learning; and the ways in which technology can improve the effectiveness of special education and foreign language instruction. West illustrates the potential contributions of blogs, wikis, social media, and video games and augmented reality in K–12 and higher education.

Technology by itself will not remake education. But if today’s schools combine increased digitization with needed improvements in organization, operations, and culture, we can overcome current barriers, produce better results, and improve the manner in which schools function. And we can get back to teaching for tomorrow, rather than for yesterday.

Darrell M. West is the author of numerous books, including Brookings titles The Next Wave: Using Digital Technology to Further Social and Political Innovation (2011) and Brain Gain: Rethinking U.S. Immigration Policy (2010).

Praise for the book:

"As is expected from Brookings, Darrell West provides us with a clear, authoritative, non-dogmatic, up-to-date account of all the ways in which new technologies are altering the K-12 education landscape."—Paul E. Peterson, Director, Harvard University’s Program on Education Policy and Governance

"Darrell West’s book recognizes that if students don’t learn the way we teach, then we should teach the way they learn."—Larry Rosenstock, CEO and founding principal, High Tech High

"In this pithy volume, Darrell West offers wise words of both optimism and caution. He notes the promise of new technologies to improve schooling in the twenty-first century, but cautions that these advances will only deliver if accompanied by a tough-minded willingness to rethink the structure and culture of schools and school systems. Policymakers and educators alike would do well to heed the lessons West offers."—Frederick M. Hess, Director of Education Policy Studies, American Enterprise Institute

ABOUT THE AUTHOR

Darrell M. West

Downloads

Ordering Information:

{BE4CBFE9-92F9-41D9-BDC8-0C2CC479A3F7}, 978-0-8157-2244-1, $26.95 Order
{B98DCBB0-3580-4D55-ABD4-AB91E00585E6}, 978-0-8157-2245-8, $26.95 Order

Long-Stagnant Teacher Compensation Needs to be Upgraded

Michael Greenstone and Adam Looney — Mon, 14 Nov 2011 09:42:00 -0500

A good teacher can transform the lives of children. But most U.S. school districts have insufficient tools to encourage effective teaching.

The country is in the midst of a 30-year period of stagnation in student achievement. One of the things that could help turn things around is a new approach to how teachers are paid, one specifically aimed at attracting the best applicants and encouraging the most effective teachers to stay.

Great teachers not only transmit knowledge and passion, they also impart important attitudes about learning and teach valuable life skills that directly contribute to success later in life. For example, new evidence shows that having an above-average kindergarten teacher rather than a below-average one translates into a difference of more than $300,000 in lifetime earnings for a classroom of 20 students.

Unfortunately the current system doesn't make it easy to attract, recruit and retain effective teachers. In most states, public school teachers must have credentials that require a year or more of additional course work after college. Yet they step into jobs with relatively low salaries and considerable insecurity in these times of shrinking public budgets. That's why many reformers are calling for market-based approaches to fixing schools.

Let's start with compensation.

The Hamilton Project recently undertook an analysis of teacher salaries and found that they have been stagnant over the last three decades, while salaries for similarly skilled workers in other professions have increased. In the 1970s, teachers made about 7% less than non-teachers with similar education, experience and other characteristics, or about $3,800 per year in inflation-adjusted terms. Over the last decade, that pay gap has increased to about 19%. Today's teachers make about $11,000 less in annual earnings than other professionals with similar backgrounds.

This gap in pay is among the largest in the developed world, according to a recent report by the Organization for Economic Cooperation and Development comparing salaries of teachers with the average earnings of full-time workers with similar training and skills. Of the 27 countries studied, the United States ranked 22nd, demonstrating a more significant pay gap between teachers and non-teachers than in most other industrialized nations.

Of course, salaries are just one component of compensation, and public school teachers generally receive more favorable health and retirement benefits than do private-sector professionals. But whether these benefits have increased enough to offset the relative decline in wages is unclear from the available data.

More important, costly, backloaded benefits are not necessarily the best tools for attracting and retaining the most effective teachers. As one example, the pension plans available to most teachers today require that they stay in their school system for a long period of time to vest for retirement. Therefore, young teachers who enter the profession for a limited number of years or who want to change school districts will not reap the backloaded benefits in the current structure. This approach to compensation is out of step with the private market, which is competing for the same pool of talented individuals and can offer higher salaries and mobile retirement plans that allow for greater flexibility and near-term rewards.

Over the last three decades, educational attainment and achievement in the United States have stagnated and contributed to a decline in earnings for many Americans. Our current system is falling short, and it is increasingly clear that our approach to teacher compensation needs to be updated. There are several necessary components of such change, including the difficult task of identifying the teachers who spur higher student achievement. A key complement to such strategies is to reform teacher compensation policies to attract a broader pool of applicants to the profession and then pay the best teachers salaries that provide them with an incentive to remain in the profession.

Teachers play a crucial role in shaping our children and in this respect hold our nation's future in their hands. As we look to fill as many as 4 million teaching openings in the coming decade, we must reform our compensation systems to attract and retain the most effective teachers. If we fail to do so, we should not be surprised by the results.

Michael Greenstone is director of the Hamilton Project at the Brookings Institution and an economics professor at MIT. Adam Looney is policy director for the Hamilton Project and a senior fellow at the Brookings Institution. More of their research can be found at http://www.hamiltonproject.org.

Authors

Publication: Los Angeles Times

Education Technology: Revolutionizing Personalized Learning and Student Assessment

Thu, 06 Oct 2011 15:00:00 -0400

Event Information

October 6, 2011
3:00 PM - 4:30 PM EDT

Falk Auditorium
The Brookings Institution
1775 Massachusetts Ave., NW
Washington, DC

In a widely-quoted commentary, famed educator John Dewey predicted that “if we teach today’s students as we taught yesterday’s, we rob them of tomorrow.” This observation fits squarely with education policymakers’ call for re-engineering the modern classroom to harness the power of digital technologies. Education advocates envision schools where students master vital skills and critical thinking in a collaborative setting, teachers assess pupils in real time, and social media and digital libraries connect learners to a wide range of resources. What would digitized classrooms look like, and how could technology improve pupil engagement and mastery of concepts? How might educators scale up successful pilot projects?

On October 6, the Center for Technology Innovation at Brookings hosted a forum on education technology and its potential to transform the modern American classroom. Moderated by Darrell West, vice president and director of Governance Studies, a panel of experts analyzed how to best incorporate digital technologies into American classrooms and increase the use of adaptive learning and assessment. Discussants also examined how education technology can enhance student performance and engagement.

After the program, panelists took audience questions.

This event was followed on Twitter using #EdRev.

Video

Education Technology: Revolutionizing Personal Learning & Assessment

Audio

Education Technology: Revolutionizing Personalized Learning and Student Assessment

Transcript

Uncorrected Transcript (.pdf)

Event Materials

2011106_education_technology

Participants

Panelists

Chip Hughes

Executive Vice President of School Services
K12

Joanne Weiss

Chief of Staff to U.S. Secretary of Education, Arne Duncan
U.S. Department of Education

Nina Zolt

Co-Founder and Chief Learning Officer
ePals.com

Zoran Popović

Professor and Director, Center for Game Science, University of Washington

Online Teaching's Disconnect

John Villasenor — Wed, 28 Sep 2011 14:58:00 -0400

To the long list of threats to the quality of an American university education, we can now add another: the rush into online instruction. Universities across the country are under increasing pressure to offer more of their courses over the Internet. The University of California, where I teach, has started offering for-credit online undergraduate courses this year.

In theory, moving online is a win-win for all involved. Students receive instruction at the locations of their choosing, courses become more accessible to working students who can eliminate the overhead of commuting to class, cash-strapped universities broaden their reach and revenue base, and professors can earn extra compensation for putting their courses online.

But amid the enthusiasm for all that is gained, it is also important to look at what is lost when the classroom experience is piped through the Internet and delivered on a screen. The Internet is very efficient at conveying words and images from one place to another. But good university teaching is much more than that.

Teaching in the truest sense is what occurs when a committed instructor gets in a room with a group of equally committed students and engages them in an interactive, probing and challenging treatment of a subject. A good lecture or seminar has its foundation in words but gains its texture and flow from countless other subtle cues and interactions in the classroom. These include the body language of the students that an alert instructor will observe and use in modulating the pace and content of the discussion, the pauses and inflections in student questions that would escape capture by a microphone, and the dynamism that occurs because each student, sitting among different neighbors at a unique location in the room, experiences and engages with the class slightly differently.

A course is also made effective by the unscripted interactions that occur as students gather before and after the class, and by the simple fact that the physical act of getting to class requires at least some investment of time and energy. In short, attending a well-run class in person is immersive and engaging in a way that far exceeds anything that consumer technology can possibly hope to deliver now or in the foreseeable future.

I'll admit that there's a certain attraction to the idea of moving to Maui and teaching all my classes from the comfort of a video camera-equipped home office. In fact, on a small number of occasions over the years, I have lectured by live videoconference when an unavoidable business trip left me the choice between teaching by videoconference or not at all. Each time I do this I am struck by the near miracle of reaching across time zones and miles to see and hear my students in a sunlit classroom in California. I speak and write on the board; they take notes and ask questions. Business as usual.

But when the lecture ends, a button is pushed, and jarringly I am suddenly somewhere else — a campus in the evening on the East Coast, or a nearly empty building near midnight somewhere in Europe. And I always feel a pang of guilt because I know, and my students know, that a class taught by videoconference is a distant second choice to the here-and-now presence of a lecture, properly delivered, by a real person standing in front of them.

The national trend toward online university instruction has been bolstered by a Department of Education-funded report that analyzed nearly 100 studies and concluded that online instruction, in the words of the report's lead author, "actually tends to be better than conventional instruction."

Depending on how narrowly one defines "better," that may be true. Under certain conditions it undoubtedly is true — for example, for the working student who cannot travel to class and for whom online education opens a whole new world of previously inaccessible options. For these students, universities can and should work to create appropriate frameworks and programs to use online instruction to broaden their reach.

But policymakers, university teachers and administrators should acknowledge that scientific studies and budget pressures notwithstanding, something is lost when the classroom experience becomes virtual. As we strive to educate our university students in an increasingly competitive global economic climate, among the many costly and complex measures that are on the table for improving their educational experience, here's one that is refreshingly simple: Show up.

Instructors owe it to their students to be there in the classroom, and students owe it to themselves — and to the rest of us — to do their best to be there as well.

Authors

John Villasenor

Publication: The Los Angeles Times

Image Source: © Adam Hunger / Reuters

Are We Short-Changing our Future? The Economic Imperative of Attracting Great Teachers

Michael Greenstone and Adam Looney — Thu, 22 Sep 2011 12:13:00 -0400

Great teachers have the ability to transform and enrich the lives and living standards of Americans. According to recent research, a student’s kindergarten teacher has long-lasting influence on important lifetime outcomes, such as future earnings. These effects are so important that the difference between having an above-average kindergarten teacher and a below-average kindergarten teacher could translate into a difference of more than $300,000 in future earnings for a classroom of 20 students (Chetty et al. 2010). Therefore, continuing to attract and retain the most effective teachers is a necessary step in raising the achievement of American students. But attracting highly-effective teachers is an increasing challenge as today’s teachers are asked to do more than ever before and because the most salient form of teacher pay—salaries—has been in relative decline.

The State of U.S. Education

A strong educational system and the role of teachers have never been more important for America’s workforce. Less-skilled workers are disproportionately unemployed and have experienced declining wages, and the return to a college degree has never been higher.¹ Unfortunately, the United States is in a period of stagnating educational achievement. Despite a doubling of per-pupil expenditures and decades of education reform in the United States, student achievement has only inched forward. Test scores, as measured by the National Assessment of Educational Progress (NAEP), have barely budged in 35 years, and the United States ranks low relative to other developed countries on measures of achievement. For example, on the Program for International Student Assessment (PISA), which compares students across 34 countries, American 15 year-olds ranked a disappointing 25th in math, 17th in science, and 14th in reading.² These challenges contribute to low high school and college completion rates for American students. Since the 1970s, college completion rates for many groups have increased slowly or not at all and, if those with GED qualifications are excluded, the proportion of the population finishing high school has actually fallen. At the same time, the value of education in the labor market is at an all time high. Relative to someone with a high school diploma, for example, those with a college degree earn twice as much each year—equivalent to about $570,000 more over a worker’s lifetime.

The Importance of Effective Teaching

Almost everyone, from policymakers to parents to teachers, agrees that reforms in the K-12 educational system are necessary to developing a more educated workforce and a stronger economy. In the last 15 years, advances in data availability and computing power have unleashed an explosion of research that begins to identify what works and what doesn’t in education policy—all of which can help guide policymakers achieve better bang for the buck with education spending. For example, strong evidence supports the effectiveness of a subset of charter schools and expanding access to early childhood education. While there are certainly a variety of factors that affect student outcomes, including home environment and socioeconomic factors, this evidence suggests that there are many avenues to improving achievement through changes in the classroom.

In terms of delivering large potential gains, effective teaching is among the most important influences on student achievement. Research finds that teachers vary substantially in their effectiveness, even within the same school. These differences arise not just in the effect of teachers on test scores, but also in terms of their impact on the lives of children long after they leave the classroom—including impacts on their future employment prospects and earnings.

A primary challenge to improving teaching quality is continuing to recruit and retain the best and the brightest in our workforce to become teachers. While no one pursues a teaching career solely for the financial rewards, compensation does matter for attracting and keeping teachers with outside options and who must support their own families. Continuing to attract a broad pool of applicants has become harder over time in part because teaching salaries—the most visible form of compensation for new teachers—have declined relative to salaries in other professions.

The Relative Decline in Teacher Wages

The graph below shows that full-time teaching salaries have remained flat over time while the wages of comparable workers have increased substantially. After taking account of the experience, education, and other characteristics of workers, the pay gap between teachers and non-teachers has expanded. In the 1970s teachers made about 7 percent less than non-teachers, after controlling for education and other characteristics or about $3,800 per year. Over the last decade, that gap has increased so that teachers earn about 19 percent less than non-teachers—a difference of almost $11,000 a year.

Furthermore, compared to other countries, the relative salaries of American teacher appear low. In a recently released report, the OECD calculated the ratio of the average salaries of teachers with 15 years' experience to the average earnings of full-time workers with a college degree and found the United States ranked 22nd out of 27 countries.

Of course, in addition to salaries, public school teachers generally receive more favorable health and retirement benefits than many private-sector workers, though detailed data are hard to come by. Even though non-wage benefits have historically been more generous for teachers, it is not clear that these benefits have increased by enough to offset the relative decline in wages demonstrated above.

But the relevant question is whether the combination of relatively low salaries and relatively high deferred benefits is the right formula to attract and retain talented young people with many alternative career opportunities other than teaching. Given the evidence that great teachers have profound effects on the life-time outcomes of their students, it is important that compensation policies are designed to attract high-quality applicants and retain many of the outstanding teachers currently in the profession.

Conclusion

Over the last 30 years, educational attainment and achievement have stagnated and, in the face of an increasingly competitive global economy, have contributed to a decline in the earnings of many Americans. The question is not whether additional reforms to our educational system are needed, but which reforms provide the most cost-effective ways of improving student achievement.

In a forthcoming Hamilton Project strategy paper (for release on September 27th), we provide a dual-track approach to improving student performance:

1.The first approach examines opportunities for structural changes to America’s educational system—a new way of doing business. These include generalizing the best practices of top performing charter schools and changing the current systems for identifying, hiring, and retaining highly effective teachers.

2.The second approach focuses on smaller, cost-effective reforms that could be implemented without dramatically re-thinking how schools operate, such as student incentives, early childhood education, and managerial and organizational changes at the school and district levels.

The Great Recession has highlighted the importance of a strong education and yet many of our children are not graduating with the skills necessary to successfully compete in today’s global marketplace. To help address these issues, The Hamilton Project will release three new policy proposals and host a forum, “Promoting K-12 Education to Advance Student Achievement,” on September 27th in Washington, DC. In addition to broader discussions on the educational system, the event will highlight new proposals on the use of incentives in education; a new approach to accountability for teachers and students; and opportunities for organizational changes to improve student performance.

Education has traditionally been a key component of the American Dream—that each generation can do better than the last. A hallmark of that success has been the ability of any American family to give their child a high-quality education in public schools. In order to maintain this foundation, new policies and new ways of thinking will be necessary to ensure that America’s educational system is prepared for the demands, now and in the future, of an increasingly competitive world economy.

[1] Michael Greenstone and Adam Looney, “Trends: Reduced Earnings for Men in America”
[2] Howard L. Fleischman; Paul J. Hopstock; Marisa P. Pelczar; and Brooke E. Shelley, “Highlights From PISA 2009: Performance of U.S. 15-Year-Old Students in Reading, Mathematics, and Science Literacy in an International Context”

Authors

Publication: The Hamilton Project

Improving STEM Education in the United States

Darrell M. West — Mon, 12 Sep 2011 14:24:00 -0400

The need for better science, technology, engineering, and math (STEM) teacher training and investment was emphasized today at a Brookings Institution forum on the topic. Dr. Rebecca Blank, the Acting Secretary of Commerce, presented several Commerce reports showing the importance of STEM education for job creation and economic development, and significant underrepresentation in the field for women, African-Americans, and Hispanics. Its report on “Women in STEM: A Gender Gap to Innovation” found that STEM workers were 76 percent male and only 24 percent female. A new report released today on “Education Supports Racial and Ethnic Equality in STEM” noted that 74 percent of STEM workers are male, compared to 6 percent who are Hispanic, 6 percent African-American, and 14 percent Asian-American. She noted the importance of the United States doing a better job attracting students into STEM fields and the need to reach out to under-represented communities. Since STEM workers earn a premium of 25 percent over other workers and have only a 5.5 percent unemployment rate, there are strong economic incentives to get more people into STEM fields.

Jim Simons, the founder of Math for America and board chairman of Renaissance Technologies, discussed his non-profit’s interest in improving teacher training in high school STEM courses. He said we need “knowledgeable and inspiring teachers” and that today we have a “shortage of such teachers”. The way to make STEM teaching more attractive so instructors do a better job introducing students to science and math is “higher pay and better working conditions”. Math for America proposes bonuses and stipends for high school STEM teachers and has provided funding for this across the country. The organization helps 350 math teachers in New York City and hopes to raise that figure to between 700 and 800 in the near future.

Charles Vest is president of the National Academy of Engineering and MIT president emeritus. He pointed out that South Korea graduates more engineers than the United States and the China graduates 10 times as many as America. In many Asian countries, 21 percent of college graduates are engineers, compared to 12 percent in Europe and 4.5 percent in the United States.

Charles Giancarlo is managing director and head of value creation for Silver Lake Partners. He noted that Cisco (where he used to serve as executive vice president) employs 24,000 engineers and Silver Lake Partner’s companies employ 87,000. Yet the United States graduates only 86,000 engineers, indicating a mismatch between supply and demand. He also explained that 35 percent of graduates are foreign born, yet we only provide 85,000 H-1b visas for scientists and engineers so many foreign students who would like to stay in the United States are forced to return to their home country. This robs the United States of valuable talent and sources of future innovation and job creation.

Authors

Darrell M. West

Image Source: © Adam Hunger / Reuters

Passing Muster: Evaluating Teacher Evaluation Systems

Tue, 26 Apr 2011 10:51:00 -0400

Passing Muster Calculator
The calculator allows users to determine the percentage of the total teacher workforce who can be identified as exceptional based on the characteristics of the teacher evaluation system.

Download the calculator »
(.xls)

Executive Summary

U.S. public schools are in the early stages of a revolution in how they go about evaluating teachers. In years past there was little more than intuition and anecdote to support the view that teachers vary in their quality. The little data that was available came from ratings of teachers carried out by school principals, a process that typically resulted in nearly all teachers receiving uniformly high ratings. It is nearly impossible to discover and act on performance differences among teachers when documented records show them all to be the same.

A new generation of teacher evaluation systems seeks to make performance measurement and feedback more rigorous and useful. These systems incorporate multiple sources of information, including such metrics as systematic classroom observations, student and parent surveys, measures of professionalism and commitment to the school community, more differentiated principal ratings, and test score gains for students in each teacher’s classrooms. The latter indicator, test score gains, typically incorporates a variety of statistical controls for differences among teachers in the circumstances in which they teach. Such a measure is called teacher value-added because it estimates the value that individual teachers add to the academic growth of their students.

Value-added has a prominent role in new evaluation systems for several reasons, including a burgeoning research literature that demonstrates that value-added measures predict future teacher ability to raise student test scores better than principal ratings and teacher attributes such as years of experience or advanced coursework. Further, federal law and policy in the George W. Bush and the Obama administrations has incentivized states to develop the assessment systems and databases that allow value-added to be calculated, and to incorporate value-added information as a significant factor in evaluating teacher performance. For example, a commitment to developing new teacher evaluation systems incorporating value-added information was required of states competing for the billions of dollars of Race to the Top funds that were available under the American Recovery and Reinvestment Act during the first year of the Obama administration.

Although much of the impetus for new approaches to teacher evaluation comes from policymakers at the state and national levels, the design of any particular teacher evaluation system falls to the roughly 16,000 school districts and 5,000 independent public charter schools in the U.S. that have the responsibility for developing human resource policies and procedures for their instructional staff. Because of the immaturity of the knowledge base on the design of teacher evaluation systems and the local politics of school management, we are likely to see considerable variability among school districts in how they go about evaluating teachers—even as most move to new systems that are intended to be more informative than those used in the past.

If an individual state or the federal government wishes to require or incentivize local education agencies to evaluate teachers more rigorously and meaningfully, how can they do so while honoring each district’s authority to do it its own way? And how can individual school districts benchmark the performance of their teacher evaluation system against the performance of evaluation systems in other districts or against the previous version of their own evaluation system? In other words, how can teacher evaluation systems be compared, one to another?

This report addresses the comparison of teacher evaluation systems in the context of a particular administrative and legislative challenge: How a state or the federal government could achieve a uniform standard for dispensing funds to school districts for the recognition of exceptional teachers without imposing a uniform evaluation system on those districts. We address and provide practical procedures for determining the reliability of local teacher evaluation systems. We then demonstrate that the reliability of the evaluation system determines the proportion of teachers that a system can identify as exceptional. Thus a school district wanting to accurately recognize the top quartile of teachers as highly effective would only be able to identify some portion of the top 25 percent with confidence given the lack of perfect reliability in the measures of teacher effectiveness that are deployed by the district. Further the portion of the top quartile that could be identified would be greater in an identically sized district that has more reliable measures.

Our approach to answering the question of how the federal government or a state could dispense funds to local districts to reward exceptional teachers is that the amount of funding should be scaled to the reliability of each district’s evaluation system. Thus school districts with more reliable systems would be able to accurately identify a greater proportion of their teachers as exceptional and would receive funding in line with those numbers. The procedures we propose by which the reliability of a district-level teacher evaluation system could be reported to and evaluated by state or federal officials are straightforward and simple. They do not necessitate an intrusive federal or state role.

Although we provide a worked solution to a specific administrative challenge, i.e., state or federal funding for district-level recognition programs for exceptionally effective teachers, the underlying approach we offer has more general uses in a variety of circumstances in which decisions have to be made about teachers based on imperfect data. For example our approach is easily adapted to the district-level task of identifying low-performing teachers for intensive professional development or to the state- or federal-level task of setting minimal standards for the reliability of teacher evaluation systems. Further, by demonstrating how the reliability of non value-added measures of teacher performance such as classroom observations is an important component of the overall reliability of a teacher evaluation system, our approach provides a spur to the development of multi-faceted methods for evaluating how well teachers are doing their jobs.

Background

The vast majority of school districts in the U.S. presently use teacher evaluation systems that result in nearly all teachers receiving uniformly high ratings. For instance, a recent study by The New Teacher Project of twelve districts in four states revealed that more than 99 percent of teachers in districts using binary ratings were rated satisfactory whereas 94 percent received one of the top two ratings in districts using a broader range of ratings.[i] As Secretary of Education Arne Duncan put it, “Today in our country, 99 percent of our teachers are above average.”[ii]

The reality is far different from what these evaluations would suggest. We know from a large body of empirical research that teachers differ dramatically from one another in effectiveness. The failure of today’s evaluation systems to recognize these differences means that human resource decisions are not as productive or fair as they could be if they incorporated data that meaningfully differentiated among teachers. To put it plainly, it is nearly impossible to act on differences between teachers when documented records show them all to be the same.

A new generation of teacher evaluation systems seeks to make performance measurement and feedback more rigorous and useful. As such, the measures should demonstrate meaningful variation that reflects the full range of teacher performance in the classroom. New evaluation systems typically incorporate several sources of information on teacher performance. For example, the Hillsborough County Public School District in Florida utilizes classroom observations of teacher performance, student ratings of their teachers, direct assessments of teacher knowledge, and student test score gains in each teacher’s classrooms as components of their teacher evaluation system.[iii] The District of Columbia Public Schools evaluates teachers based on student test score growth in each teacher’s classroom, a classroom observation measure, a rating of commitment to the school community, student test score gains for the whole school, and a measure of professionalism that takes into account factors such as unexcused teacher absences and late arrivals.[iv]

Many of these new systems incorporate student test score growth in ways that aim to capture the contribution teachers make toward student achievement. This contribution is often referred to as teacher value-added. There are various methods for estimating teacher value-added, but all typically entail some variant of subtracting the achievement test scores of a teacher’s students at the beginning of the year from their scores at the end of the year, and making statistical adjustments to account for differences in student learning that might result from student background or school-wide factors outside the teacher’s control. These adjusted gains in student achievement, also known as value-added, are compared across teachers.

The prominence of value-added in new evaluation systems is a result of several influences. Among them is the commonsensical view that because the principal role of teachers is to enhance student learning, a central measure of their job performance should be how much their students learn. Another influence is a burgeoning research literature on teacher classroom effectiveness that has focused on value-added measures and demonstrated that those measures predict future teacher performance, as measured by value-added in subsequent years, better than teacher attributes such as years of experience or advanced coursework.

The broader reform community in education has taken up the cause of meaningful teacher evaluation grounded in value-added measures of effectiveness. Incentives for school districts to evaluate teachers based on value-added are central to the Teacher Incentive Fund that was authorized and funded during the George W. Bush administration and also to the Obama administration’s proposed replacement, the Teacher and Leader Innovation Fund. Further, the Obama administration made a state’s commitment to measuring teacher performance using value-added a requirement for success in the competition for $4.3 billion in the Race to the Top fund. The appeal of teacher value-added measures is further strengthened by their wide availability as a result of the No Child Left Behind Act’s requirement for testing of all students in reading and mathematics in grades 3-8 coupled with federal funding to states to develop longitudinal data systems to serve as central state repositories of the resulting student assessment data.

The availability of student test scores allows, within each state, a common and face-valid yardstick for measuring teacher effectiveness across all schools in the state—an attribute that is not present for the other presently available sources of information on teacher effectiveness that individual school districts might employ, e.g., supervisor ratings or classroom observations. Thus for a variety of reasons value-added measures of teachers’ contribution to student growth have come to be central to popular and powerfully driven efforts to improve U.S. public schools.

Researchers have pointed out that value-added estimates for individual teachers fluctuate from year to year and can be influenced by factors over which the teacher has no control. We have previously issued a report that describes some of the imperfections in value-added measures while documenting that: a) they provide one of the best presently available signals of the future ability of teachers to raise student test scores; b) the technical issues surrounding the use of value-added measures arise in one form or another with respect to any evaluation of complex human behavior; and c) value-added measures are on par with performance measures in other fields in terms of their predictive validity.[v] Our report recommended the use of value-added measures as a part of teacher evaluation but in the context of continuous improvement of those measures and awareness of their imperfections and limitations.

The present report offers advice on how to determine the degree to which an evaluation system is successful in the face of those imperfections and limitations. We address the connection between the reliability of an evaluation system and its ability to accurately identify exceptional teachers for special action, e.g., for a salary bonus if they are exceptionally good or for remedial action or dismissal if they are exceptionally bad. Reliability is not the only issue arising from the use of value-added measures. In particular, designers of evaluation systems and policymakers have to address biases that are introduced by differences in the contexts in which different teachers work. However, in this report, we focus on the issue of reliability.

We build our presentation around a proposal we put forth in a previous report, America’s Teacher Corps, calling for the creation of national recognition for teachers deemed effective based on approved state and local evaluation systems.[vi] The three design features of that proposal were:

Promoting the use of teacher evaluation systems to identify and reward excellence

Whereas most of the focus of teacher evaluation systems using value-added has been on the identification and removal of ineffective teachers, we believe that such systems can also have a major impact by identifying and promoting excellence through recognition of exceptionally strong classroom performance.

Flexibility on the components that would need to be a part of a teacher evaluation system and how those components would be weighted

There is no consensus on the degree to which teacher performance should be judged based on student gains on standardized achievement tests. Supporters of test-based measures would seek to expand standardized testing to virtually all grades and subjects and weight the results heavily in personnel decisions about teachers. Opponents question the validity of state assessments as measures of student learning and the accuracy and reliability value-added indicators at the classroom level. They typically prefer observational measures, e.g., ratings of teachers’ classroom performance by master teachers. Our proposal for a system to identify highly-effective teachers is agnostic about the relative weight of test-based measures vs. other components in a teacher evaluation system. It requires only that the system include a spread of verifiable and comparable teacher evaluations, be sufficiently reliable and valid to identify persistently superior teachers, and incorporate student achievement on standardized assessments as at least some portion of the evaluation system for teachers in those grades and subjects in which all students are tested.

Involving a light hand from levels of government above the school district

A central premise of our previous report is that buy-in from teachers and utilization of their expertise are most likely if the design of an evaluation system occurs at a level at which they feel they have real influence. In most cases this will be the local school district where they work. We expect wariness from teachers, even with respect to a system intended only to identify and reward excellence, if the design of that system is subject to considerable control from Washington or the state level. Further, we doubt that there is much of an appetite within Congress for the creation of a federal bureaucracy devoted to the fine-grained oversight of state and local teacher evaluation systems. And we doubt there is sufficient capacity within state-level education bureaucracies to carry out such oversight even if there is a political will to do so.

Suppose a state or the federal government wanted to fund a program whereby individual school districts could provide a bonus or other rewards to their exceptionally effective teachers. This requires a system of evaluation that meaningfully differentiates among teachers based on their performance. Similarly, suppose that a state wanted to encourage districts to differentiate the teaching profession so that new teachers started with one set of responsibilities but could be promoted into more complex and challenging roles as they demonstrated capability in the job. This reform, again, requires evaluations to determine different levels of teaching performance. Given the great variation in design and quality of district evaluation systems and the practical and political constraints on states or the federal government producing uniformity in those systems, how could state or federal funds for such recognition programs be fairly distributed?

In this report we address the question of how a state or the federal government could achieve a sufficiently uniform standard for dispensing funds for the recognition of exceptional teachers without imposing a uniform evaluation system on participating school districts. In particular, we address the role of the state or federal government in assessing the reliability of local evaluation systems. We demonstrate that the quality of the measures and the quantity of data affect reliability and determine the number of teachers a system can identify as exceptional. Instead of a school district wanting to recognize the top quartile of teachers being able to identify 25 of every 100 teachers as being in the top 25 percent, we show that when imperfections in the measurement system are taken into account, only some portion of the true top 25 percent can be identified with confidence. Further, that portion would be greater in an identical sized district that has better measures and more data.

Although we provide a solution to what may seem to be a narrowly-focused administrative challenge, i.e., funding a teacher recognition program from the state or federal level, the underlying approach we offer has more general uses to which we will turn in the final section of this report.

How state or federal teacher recognition programs can accommodate district evaluation systems of differing quality

A major source of debate about the methods of estimating teacher performance is the statistical reliability of such measures and whether they are sufficiently precise to support attaching consequences to them such as pay for performance and tenure decisions.[vii]

Our concern in this report is with the reliability of the evaluation system as a whole. We focus on the information that is necessary to determine the extent to which teacher evaluation systems are likely to result in classification errors (e.g., classifying teachers as highly effective when they are not or failing to classify them as highly effective if they are). For this discussion we will not address the potential problem of systematic bias in which the evaluations for some teachers are systematically too high or too low in comparison to the teachers’ true effectiveness. Clearly, a desirable evaluation system will adjust for differences in the classrooms and schools in which teachers work to reduce or eliminate such biases. We focus here primarily on the reliability (or precision) of the estimates.

No evaluation approach will be exact for all teachers and thus designers and those using the evaluations should consider the implications of the imprecision for the decisions they make. If designers were to dismiss any evaluation systems that had error in identification, they would have to dismiss all possible systems and end up with no evaluation at all. Given that error is a fact in evaluation, understanding the implications of this error and how error varies across different approaches to evaluation can be helpful in choosing an effective approach.

In what follows, we describe how policymakers can determine the number of teachers that would accurately and inaccurately qualify to be singled out for special treatment given the power of the system to predict future teacher performance and the level of teacher exceptionality that is the criterion for special treatment. We will describe how to estimate the extent of misclassification, as well as the average difference in later effectiveness between groups identified by the evaluation. We also address how the tolerance that policymakers are willing to permit for misclassification plays a role in the number of teachers that can be accurately identified as exceptional. These subjects go to the heart of the issue of the performance of district-level teacher evaluation systems relative to each other, and provide the basis for a solution to the challenge of building a fair system for distributing state or federal funds to support district-level programs for teacher recognition.

The factors that influence the accuracy of teacher identification systems

Using teacher performance measures to identify teachers for special treatment is, fundamentally, an exercise in prediction. For example, the use of measures of past performance of novice teachers to decide who will be tenured assumes that the better-performing novice teachers will be better teachers after receiving tenure than would the lower-performing teachers had they been given tenure. Likewise, the common district-level practice of selecting a small number of teachers as “master teachers” to serve as role models and supports for beginning or struggling teachers involves the implicit assumption that those teachers are persistently high performers who will continue to be stars in the classroom. Thus the use of teacher evaluation measures to identify different levels of teacher performance in one period as a basis for personnel action nearly always assumes that identification in one period signifies something about how teachers will perform in the future.

Our approach to judging the relative performance of teacher evaluation systems rests on determining their ability to predict future performance. We propose to judge teacher performance measures based on the degree to which they accurately estimate future teacher performance from past years of teaching, i.e., how reliable they are as a predictor of future effectiveness.

A more reliable measure is one that will yield similar answers when it is used to take more than one reading of the same phenomenon. We use the correlation of value-added measures from one period to the next as one component of a gauge of reliability, but the degree to which a performance measure in one period predicts performance in the future will depend on both the degree to which the measure is related to true performance and the extent to which true performance is stable from one period to the next. Differences between the measurement of performance and true performance are considered measurement error. If there is a significant amount of measurement error such that the performance measure is only loosely related to true performance, we would refer to it as “noisy.” If on the other hand true performance changes from one period to the next, even a perfect measure of performance in one period will not accurately predict performance in the next period. There is no reason to think that true teacher performance is completely stable from one period to the next, e.g., teachers who are quite effective one year may encounter problems at home or changed work conditions that lead them to be less effective in a subsequent year, and teachers may become more effective over time as a result of experience (learning on the job) and professional development.

For an evaluation system to be useful, the true performance of teachers must be sufficiently stable over a period of a few years for predictions of future performance from past performance to be worthwhile. This assumption is buttressed, in the case of value-added measures, by the fact that value-added measures from one period predict student achievement in future periods.[viii] It is also buttressed by anecdotal evidence that some teachers are simply more effective than other teachers and, as a result, parents work to get their children into these teachers’ classrooms. For this discussion, we will assume that true performance, while variable from year to year, is stable enough for there to be meaningful differences in the average effectiveness of teachers over time.

In addition to picking up variation in true performance from year to year, any measure of performance will have error, i.e., will be an imperfect reflection of true teacher effectiveness. However, while all measures have error, some measures are likely to capture enough of the true differences in teacher effectiveness to be useful. Indeed, the same studies that permit an inference that there is stability in the true performance of teachers also demonstrate that the measures used in those studies are sufficiently reliable to capture at least some of those true performance differences.

Because we can neither know the precise degree of error in a given measure of performance nor the actual stability of true teacher ability from one period to the next, the correlation of a school district’s measures of teacher performance from one period to the next cannot be judged against an absolute standard. Thus, our approach to judging the quality of evaluation systems must be relative: if we use common yardsticks we can demonstrate that some evaluations systems are more reliable than others and by what degree.

Value-added as the common yardstick

If we are to judge the quality of teacher evaluation systems relative to each other, there must be some common measure across those systems that is sensitive to true differences in teacher performance. Without such a common measure, the quality of teacher evaluations systems cannot be meaningfully compared across districts.

The focus of this paper is on using such a common measure to assess the reliability of evaluation systems. However, it is important to keep in mind that reliability is not useful unless a measure also has validity. To produce valid scores a measure must pick up differences in teacher performance that are important to student learning. Thus, while a teacher's height is strongly correlated from one year to the next, can be measured precisely, and is available for every teacher in the country, it would not be a good common measure for our purpose because it does not capture teacher performance. Similarly, suppose district A’s evaluation system produced scores for individual teachers based on a weighted average of years of teaching experience, route into teaching, certification status, receipt of advanced degrees, and principal ratings of performance on a pass-fail system. The correlation of these scores for the district’s teachers from one year to the next would be high, i.e., they would be very reliable. Suppose that district B deployed a very different evaluation system based on classroom observations, value-added test scores, and student surveys. The year-to-year correlation of evaluation scores would likely be much lower for district B than for district A. However, that would not mean that district B had the weaker evaluation system. In fact, the measures used by district A have been shown empirically to be only weakly related to classroom performance whereas those used by district B have a stronger evidence base. A system for determining whether a district’s evaluation system passes muster in terms of recognizing exceptional teacher performance should not be designed to favor year-to-year reliability disconnected from what is being measured.

If we are to judge the quality of teacher evaluation systems relative to each other, we have to have a common measure or a set of common measures across those systems that are sensitive to true differences in teacher performance. Without such common measures, it is difficult to meaningfully compare the quality of teacher evaluations systems. A number of different measures of teacher effectiveness have at least basic face validity for measuring teacher effectiveness, including: direct measures of teaching such as teachers' scores on observational protocols of teaching quality; measures of student learning while in a teacher's classroom such as value-added measures; principals' ratings; and survey-based assessments of teachers by students and parents.

Currently value-added measures are, in most states, the only one of these measures that is available across districts and standardized. As discussed above, value-added scores based on state administered end-of-year or end-of-course assessments are not perfect measures of teaching effectiveness, but they do have some face validity and are widely available. In our analysis below we use value-added as the metric for comparing the quality of evaluation systems; however, we are limited in our goal of comparing systems by the limitations of these measures. As other measures become widely available and as the tests on which the value-added measures are based become better aligned with societal goals, our ability to judge and compare systems of evaluation will improve

Note that we do not recommend that states or the federal government be prescriptive about the components that districts should include in their teacher evaluation systems or how they should be weighted or how the information from those systems should be used for high-stakes personnel decisions. We use growth in student achievement scores on standardized assessments as an outcome measure to judge the relative quality of teacher evaluation systems but should other standardized measures of teacher performance that have clear face validity come to be used in public schools statewide in the future, they could augment or replace the use of standardized achievement test gains without changing the conceptual and computational model we put forward. For example, one could imagine measures of student engagement or assessments of so-called soft skills finding their way into statewide assessments of student outcomes.

An approach based on relative strength of prediction

The goal of this report is to outline an approach for states to use in judging district-developed teacher evaluation systems. Better systems of evaluation will have a number of features—five that we outline here. First, they will differentiate teachers. If there is little differentiation between teachers’ evaluations (e.g., if 95 percent of teachers receive the same rating) then the system will not be useful. Second, this differentiation should not be driven by observable characteristics that are unlikely to be strong predictors of effectiveness. For example, if teachers were ranked based on their years of experience, there would be differentiation but that differentiation would not have face-validity as a measure of effectiveness, even though we might imagine reasons (e.g., ease) that could lead to school leaders differentiating teachers based on experience. Third, the evaluations should be predictive of future evaluations of effectiveness. That is, the system should be able to identify teachers who not only performed well in the current year but who continue to perform well in subsequent years. Fourth, the system should employ multiple measures that include not only value-added measures on state-level standardized tests but formal or informal observation measures and/or other measures of student progress. This need for multiple measures arises, at least in part, from the measurement error in value-added measures and the low proportion of teachers for which value-added measures are available. It also arises from the importance of goals for students that are not captured by available tests. Finally, the system should be applicable to all teachers. It is unlikely that all measures used in the evaluation system will apply to all teachers. For example, teachers serving some groups of special education students may not have appropriate value-added from student achievement measures because the tests do not sufficiently capture their contribution. However, even though these teachers do not have all the metrics, a good system will provide enough alternative metrics to reliably assess these teachers. In what follows, we concentrate on the fourth and fifth features of a strong evaluation system.

Value-added measures have the benefit of being based on student outcomes. If we believe that a teacher’s effectiveness is, by definition, the contribution that he or she makes to students, then a student performance measure is ideal. However, there are clear drawbacks with current value-added measures. In particular, they are usually based on student outcomes over a very narrow set of domains; they have substantial measurement error; and they are usually only available for a small subset of teachers. As a result, a system based only on value-added measures would not be helpful for identifying the most effective teachers in a district.

In order for districts to develop useful systems of evaluation they will need meaningfully to supplement the state test score data. One approach to this supplementation would be to develop other measures of student performance that more fully capture the range of outcomes that the district cares about and provides information on the learning in classrooms of a greater share of the district’s teachers. A second approach to this supplementation is to collect data other than student performance data that gives insights into teaching effectiveness. This data could include formal or informal assessments from school leaders or parents or students as examples. In a system that collects these three types of information, some teachers are likely to have data on all three types of measures—state value-added, additional value-added, and non-student-based measures—while other teachers may only have data on a single type of measure.

The measures based on information aside from the state tests have advantages. For example, non test-based measures are likely easier to collect for a wider range of teachers. In addition, they may provide direct information for teachers on how to improve. On the other hand, these measures are not directly linked to student benefits, which is the ultimate goal of teaching. Similarly, student performance-based measures that use information other than performance on the state tests allow a fuller coverage of content areas than do state test-based measures but may be based on invalid or unreliable instruments.

We see a two-pronged approach for how states judge the non state-test measures utilized by a district’s evaluation system. In the first prong, states may approve a measure or a type of measure for use as a core part of the evaluation system. For example, the state may approve any value-added measures based on student test performance, even those not using the required state exams. Alternatively, a state may choose a common observational protocol collected in a standard way to be used as a core part of any evaluation system. Yet, even with state allowance of measures other than value-added on the state tests, each district’s evaluation system will likely utilize a range of other measures such as principal evaluation. Whether a district’s evaluation system passes muster for the purpose of identifying effective teachers will depend in many cases on districts demonstrating the validity of these non state-test measures. We propose an approach that uses measures of teacher effectiveness for teachers that have both core (e.g., value-added on state tests) and non-core (e.g., principal evaluations) elements to validate the full range of measures in a teacher evaluation system for a full range of teachers.

For the rest of this report we will treat value-added measures as core measures for the purpose of assessing the reliability of the district system. However, policymakers may choose other approaches that either do not allow any non state-test measures of value-added or that allow some non state-test value-added measures to serve as the benchmark for assessing the reliability of the other measures used in the evaluation system.

Our model addresses the use of non value-added measures by examining how they perform in the grades and subjects in which both value-added and non value-added evaluation data are collected. We then extrapolate the findings from non value-added measures in the grades in which they are combined with value-added measures to their use in the grades and subjects in which value-added data cannot be calculated. Because the larger proportion of most districts’ teachers teach in untested grades and subjects, the performance of the non value-added components of a district’s teacher evaluation system has a strong influence on the number of highly-effective teachers who can be accurately identified as exceptional in our model. The importance of measuring teacher effectiveness in non-tested grades and subjects gives school districts a strong incentive to develop reliable and valid teacher evaluation measures in addition to value-added. These measures could be based on student performance on other tests, on other student work, and/or on the assessments of school leaders, peers, parents, or students.

The factors that influence the extent to which performance information about teachers in one period predicts their effectiveness in the future will vary from teacher to teacher and locality to locality. We propose judging the sufficiency of district-developed teacher evaluation systems by identifying just seven parameters, permitting the computation of the proportion of a district’s teacher workforce that could be identified accurately for special treatment using the same rules for all districts. This proportion then becomes a metric for assessing the effectiveness of the evaluation system. Specifically, the proportion of highly-effective teachers that will qualify for special treatment will depend on a) two critical values adopted by policymakers (what we call exceptionality and tolerance), b) three correlations calculated using teacher-level data for teachers within each district, and c) a count of the teachers in each district who are subject to the full evaluation system including value-added measures and a count of the number who are subject only to the non value-added components (e.g., because they teach in untested grades and subjects).

The first parameter that we need to identify, the exceptionality parameter, identifies the point in a distribution of teacher evaluation scores that singles out individual teachers as being exceptional and thus subject to some type of special treatment. The more stringent the definition of exceptionality, the fewer teachers who will be identified for special treatment.

Exceptionality is the cutoff in a teacher rank distribution that is used for decision-making. For example, to identify the “top 20 percent” or “bottom 5 percent” the exceptionality parameter would be 80 percent and 5 percent respectively.

As we have indicated previously, errors of measurement are endemic in nearly all assessments of complex human performance. Measures of teacher performance will not be nearly as reliable as measures of teacher height, for example. Because the degree of noise in measures of teacher performance can be quantified, policymakers are put in a position of needing to decide how tolerant they wish to be of two offsetting types of errors—over-inclusion and under-inclusion.

Some policymakers may wish to be very certain that the teachers identified as exceptional are truly exceptional. These policymakers would have low tolerance for classifying teachers as exceptional whose true performance level falls below the cutoff for exceptionality in the recognition program. These policymakers are intolerant of over-inclusion. Other policymakers may want to be very certain that errors of measurement do not result in excluding teachers from the exceptional category who truly belong there. They are tolerant of over-inclusion because they want to minimize errors of under-inclusion. Since errors of under-inclusion and over-inclusion are countervailing, policymakers can’t have their cake and eat it too. They have to decide whether to equate the probability of the two types of errors or to favor reduction in one type of error at the cost of increasing the other.

Statistically, the value of the tolerance parameter that produces the lowest number of classification errors creates an equal probability of errors of over- and under-inclusion. That value is .5, and we have set that as the default in our subsequent spreadsheet calculator. There might be occasions when policymakers would want to select a different value, perhaps to align the tolerance parameter for assessing the evaluation system itself with the one used to make decisions about individual teachers within a district system. That said, we expect there will be little reason to alter the default value at the state or federal level for the purposes of determining the degree to which districts pass muster. We introduce the tolerance parameter here primarily to expose the mechanics and conceptual model underlying our calculator and to drive home the point that errors of measurement are inescapable in systems that attempt to evaluate complex human performance and require decisions on how they are to be handled.

Tolerance represents how willing policymakers are to risk an error of over-inclusion. It is the probability that the average teacher in the group defined by the exceptionality parameter does not actually belong in the exceptional category based on his or her true performance.

The Correlation parameters are the simple correlations between a) the full evaluation scores for the teachers for whom value-added can be calculated in one year or more years (baseline) and their value-added test scores in the subsequent year (outcome); b) the evaluation scores of these same teachers calculated without the value-added component for the same baseline years(s) and their value-added scores in the outcome year; and c) the value-added scores of these same teachers in the outcome year and the year just preceding.

The first of these correlations, which we will refer to as the full evaluation correlation, indicates the strength of the observed relationship between teachers’ scores on a district’s full teacher evaluation system and subsequent student achievement gains (value-added). Higher values are more desirable. The components in the complete evaluation system would include value-added as well as non value-added components such as classroom observations and evaluations by administrators. The components would be weighted as they are in the evaluation system that is actually deployed or will be deployed by the district. The number of years of data used as a baseline for evaluating teachers would also be the one deployed by the district. It could be a minimum of one year. The full evaluation correlation indicates the degree to which this total, composite evaluation score from one or more previous years predicts value-added for the same teachers the next year.

Imagine an Excel spreadsheet: Each row represents a teacher in a district for whom full evaluation scores are available for the most recent year for which state-mandated student assessment scores are available and for as many preceding years as the district wishes to employ in generating a baseline evaluation score. Each teacher’s aggregate evaluation score for the baseline year(s) is represented in one column and that teacher’s aggregate value-added score in the subsequent and most recent year is represented in the second column. The full evaluation correlation is the simple Pearson’s r between those two columns of data.

The second correlation, which we will refer to as the non value-added correlation, indicates the strength of the observed relationship between teachers’ scores on the non value-added components of the district’s full teacher evaluation system in the baseline year(s) and value-added in those teachers’ classrooms in the outcome year.

Value-added cannot be calculated for teachers in untested grades and subjects. Such teachers represent the majority of the workforce in most districts. How can the evaluation system that is applied to these teachers be evaluated fairly across districts? The non value-added correlation addresses this question by assuming that the relationship that exists between value-added and an alternative teacher performance instrument, for example a classroom observation, for teachers for whom both measures are collected also applies in the case of teachers in subjects where value-added measures are not available. For instance, we would assume that the correlation between observationally-based ratings of teachers and value-added in math would be the same in history, where value-added measures are not available.^{^[*]}

Operationally, the non value-added correlation is collected for the same teachers in a district for whom the full evaluation correlation is calculated. These are the teachers who are subject to the complete evaluation system for whom value-added can be calculated in the outcome year. The correlation coefficient would be calculated between the composite of the non value-added components of the evaluation score for these teachers in the baseline year(s) and the value-added component of those teachers’ evaluation score for the next year, i.e., the outcome year. In order to increase the predictive power of the non value-added components, a district might choose to use more years of data on those components than it does for the value-added component. That is acceptable in our model. The only caveat is that the conditions the district sets on the data it uses to generate the correlation for the non value-added components have to be the conditions the district uses for its actual decision making system. The non value-added correlation coefficient estimates the degree to which the non value-added components of the teacher evaluation system, e.g., classroom observations and administrator ratings, predict future value-added.

The third correlation, which we will call the value-added correlation, estimates the reliability of the value-added measure itself. The first two correlations express the observed predictive relationship between evaluation scores and value-added. We are expressing these relationships with simple correlations that can range between r = .00 (no association whatsoever) and r = 1.00 (teachers’ value-added scores are perfectly predicted by their evaluation scores). But even if the true underlying relationship between evaluation scores in a baseline period and value-added the next were a perfect r = 1.00, we would not see anything close to that coefficient in the observed correlation unless the value-added measure itself were perfectly reliable. However, we know and have described previously many sources of noise in measures of value-added, e.g., student gain scores on standardized assessments capture imperfectly what students have learned from their teacher; teachers may improve from one year to the next. Thus the maximum values of the first two correlations (the full evaluation correlation and the non value-added correlation) are constrained by the reliability of the value-added measure itself.

Imagine early astronomers trying to fix the coordinates of dim stars at particular times of year using a crude telescope. Before deciding how much stars actually change their positions as observed from earth, it would be important to know the reliability of the telescope itself. If objects appear to move in the telescope based on errors of refraction in the glass itself, it would prudent to adjust for those errors before drawing conclusions about true celestial movements.

We use the correlation of value-added between the outcome year and the preceding year to estimate the reliability of the value-added measure. Just as we would want to assess the astronomer’s model of annual changes in the coordinates of stars by adjusting for what the telescope can reliably detect, so too do we want to assess a district’s evaluation system by asking how much it can account for of what it can detect as the persistent year to year contribution of teachers to student achievement.

We do this by adjusting upward the full evaluation correlation and the non value-added correlation to take into account the reliability in the value-added measure. For example, suppose the full evaluation correlation accounted for 20 percent of variability in the outcome year’s value-added scores, whereas value-added scores from the preceding year accounted for only 40 percent of the variability in the next year’s value-added scores. In this case, the full evaluation correlation would have accounted for 50 percent (20% / 40%) of the persistent relationship between value-added from one year to the next. The full evaluation correlation would be adjusted upward to reflect this. The technical details of this adjustment are described in the appendix.

Operationally, the value-added correlation is calculated on a district-by-district basis for the same teachers from whom the first two correlations are derived. We use a district-level adjustment rather than a standard adjustment for all districts because districts differ in the reliability of the value-added component of their evaluation systems for a number of reasons, including the statistical model that is used to adjust for student and school background variables, the quality of the state assessments, and the heterogeneity of the teachers for whom value-added has been calculated.

The Number parameters are a) the number of teachers in a district’s present workforce that is subject to the same full evaluation system that was used to calculate the correlation parameters, and b) the number of teachers in a district’s present workforce that is subject only to the non value-added components of the district’s evaluation system (the same non value-added components that were used to generate the correlation parameters).

Because the reliability of the teacher evaluation system differs for the teachers who are subject to the full system vs. only parts of the system, we need to know the number in each category to calculate the overall identification rate for exceptional teachers. Districts will typically have teachers in their workforce who do not fall into either the full evaluation or the non value-added evaluation category. For example, many novice teachers will not have had enough time on the job to be subject to the evaluation system, and part-time teachers or teachers serving primarily in administrative roles may not be subject to the system. Such teachers should not be included in calculating the number parameters, which is to say that our model for identifying exceptional teachers applies only to teachers who are subject to the evaluation system for which reliability can be calculated in our model. This means that if there are two identical districts except that one manages to include more teachers in its evaluation system than the other, then the more inclusive district will be able to identify more teachers as exceptional.

A worked example: Application to America’s Teacher Corps

We have previously described an earlier report from this task group recommending a program called America’s Teacher Corps (ATC). We will use some of the details in that proposal to work through an example of how the model we have proposed could be applied.

The ATC proposal recommended that the federal government provide funds to school districts to augment the salary of teachers who are in the top quartile of the performance distribution and serve in high poverty schools. Thus the exceptionality parameter in our model is set at the 75^th percentile for the ATC.

The ATC proposal did not touch on the issue of errors of over- and under-inclusion. For this example we will use the 50 percent tolerance value. As we have indicated previously, this particular tolerance value equalizes the number of over- and under-inclusion errors and minimizes the total number of errors. The 50 percent value also maximizes the average difference in teachers’ contribution to student achievement between teachers in each identified group, meaning that the value-added of ATC identified teachers is as large as possible relative to non-ATC teachers.

Having established the exceptionality and tolerance parameters, we can calculate how many teachers would be eligible for the ATC given the three correlation coefficients we have previously described (full evaluation, non value-added, and value-added) and the number of teachers in a district subject to the full evaluation system vs. only the non value-added components of the evaluation system. Districts will typically rely on a suite of performance measures, such as principal or peer observations and value-added. But regardless of the system they use to identify teachers, the proportion of teachers that will be ATC-eligible rises with the correlation between the performance metric in the baseline year(s) and teacher value-added in the next.

We illustrate what this might look like for actual school systems by drawing values for the required correlations from prior research. Studies of the stability of correlations of teacher evaluation measures across years find different values in different states. For example, Goldhaber and Hansen[ix] find much higher correlations in North Carolina for elementary school math and reading value-added estimates than are reported by Harris and Sass[x] using Florida data. To be conservative we will use lower range estimates in our example.

The Harris and Sass report found a correlation of 0.40 between evaluation scores from two baseline years and value-added in the subsequent year for Florida elementary school teachers, where the evaluation scores combined value-added and principal evaluations. We will use that value for our full evaluation correlation. The same study found a correlation of 0.18 between principal ratings of teachers and value-added scores in the subsequent year. We will use that as our non value-added parameter. The multi-district Measures of Effective Teaching project being carried out by the Bill and Melinda Gates Foundation found that teacher value-added in mathematics correlated 0.40 from one year to the next whereas the corresponding correlation in English language arts was 0.20.[xi] Later we describe why two years of baseline data are preferable to one, including that the correlations tend to be higher. But for our worked example, we will take the average of these two year-to-year correlations, 0.30, as the value for our value-added correlation.

We have developed a spreadsheet based on our model, which can be downloaded here.[+] Users enter values such as those chosen as illustrative above. An example of the spreadsheet using the values described above is presented in the next two figures. The first figure presents the portion of the spreadsheet in which values are entered by the user. The number of teachers subject to the total evaluation system vs. only the non valued-added portions of the district’s teacher pool is set at 400 and 600 respectively for this example.

Figure 1

The next figure, Figure 2, presents the results that are calculated based on the values entered and displayed in the Figure 1.

Figure 2

Under this scenario, this school district is capable of identifying 8.3 percent of its total workforce, i.e., 83 out of the 1000 teachers subject to the evaluation system, as sufficiently exceptional to be eligible for ATC status. This percentage is far less than the 25 percent that is set as the exceptionality parameter by policymakers. The difference is due to the unreliability of the district’s evaluation system as a measure of the persistent value-added performance of its teachers. The more reliable the evaluation system (i.e., the stronger the correlation of evaluation scores in one year with the persistent teacher value-added effect), the greater the proportion of teachers who can be identified, up to the theoretical limit of 25 percent that is set by the exceptionality parameter.

This relationship of number of teachers who can be identified and the reliability of the measures is highlighted in Figure 2 by the difference in the identification rate for teachers who are subject to the full evaluation system vs. teachers who are subject only to the non value-added components (in this case just principal ratings). Recall that the full evaluation correlation is 0.40 whereas the non value-added correlation is only 0.18. These differences lead to 17.7 percent of the teachers subject to the full evaluation system being identifiable as exceptional whereas only 2.1 percent of those subject to only the non value-added evaluation are identifiable.

The relationship between measure reliability and the number of teachers who can be identified as exceptional is illustrated for the full range of values of reliability in Figure 3. Here the values on the vertical axis represent the proportion of teachers who can be identified up to the exceptionality parameter used in our ATC example, 25 percent. The horizontal axis represents the first number column in the spreadsheet in Figure 2 and represents the amount of variance in the persistent teacher value-added estimate explained by the evaluation score: the lower the number, the lower the explanatory power—the higher the number, the better the prediction.

Figure 3

Although there is considerable measurement error in the evaluation system we are illustrating, the decisions it supports are robust under our model because the number of teachers who can be identified is directly determined by the reliability of the measures. For instance, in Figure 3 one can see that district A’s evaluation system which explains 50 percent of the teacher-level variance would be able to identify approximately 160 of the district’s 1000 teachers as being the top 25 percent; whereas, district B’s evaluation system only captures approximately 25 percent of the teacher-level variance and would only allow approximately 90 of the 1000 teachers to be identified.

What does this mean in terms of the actual performance of those who fall into the identified or not identified categories? Note from the spreadsheet in Figure 2 that for our illustrative case the average identified teacher subject to the full evaluation system is 1.3 standard deviations more effective in raising student achievement than the average non-identified teacher (the difference between the last two columns). A standard deviation of difference among teachers in their effectiveness corresponds to about a month of student learning during one year of elementary school.[xii] Thus in the scenario we have illustrated, which uses plausible values drawn from the empirical literature, the average progress for children in the classrooms of teachers in the year after their identification as exceptional teachers via the full evaluation system is equivalent to about 5 weeks more schooling than for students in the classrooms of the other teachers in the district.

This relationship is illustrated in Figure 4, which depicts the link between the degree to which the evaluation score predicts subsequent value-added and the difference in student gain produced by identified and non-identified teachers for the full range of possible values for the ATC case. Note that an evaluation system that approached a perfect correlation with the persistent component of teacher value-added would generate about a .25 standard deviation difference in student learning between identified and non-identified teachers in the year after identification. This is a difference of about one-quarter of a year of schooling.[±] This illustrates, again, the importance of developing sophisticated teacher performance evaluation systems, be they for the purpose of implementing the ATC concept or for the myriad high-stakes purposes promised, for instance, in states’ Race to the Top applications.

Figure 4

A technical presentation of the rationale and calculations for the material in the figures and the associated spreadsheet calculator is found in the appendix of this document.

Questions and Answers

Q1. Should districts identify exceptional teachers in different proportions in the tested vs. untested grades and subjects based on the different identification rates from your model?

A1. Doing so increases the accuracy of prediction but may be undesirable for other reasons.

Districts may well not wish to identify different proportions of teachers from the tested grades and subjects vs. the non-tested grades and subjects as exceptional simply because the prediction correlations are different for these two categories. There is no conceptual reason to believe that greater proportions of elementary school teachers are exceptional than high school teachers, for example. That more elementary school teachers can be identified than high school teachers within the same margin of error is simply an artifact of value-added being available for some elementary school grades but for few high school grades and subjects.

In many districts only about 20 percent of the teacher workforce is deployed in tested grades and subjects. Providing disproportionate access to recognition and reward for teachers who happen to be subject to a district’s full evaluation system could have unintended negative consequences such as incentivizing teachers to teach in the tested vs. non-tested grades and subjects or in undermining the sense of fairness in the evaluation process as a whole.

We expect that many districts would want to provide equitable access to incentive and recognition programs across all categories of teaching while recognizing that a higher rate of misclassification will occur for categories of teachers for whom the full suite of evaluation data are not available. The implicit tradeoff between equity of access and accuracy of prediction can be reduced by including a greater proportion of the teacher workforce in the full evaluation system and by including more years of baseline data for predictions for the non value-added components of the evaluation system.

The one requirement of our model is that districts use their evaluation scores within each teacher category to identify exceptional teachers. Thus a district may decide to provide a recognition program for junior teachers that recognizes a higher proportion of junior teachers than does a recognition program for more senior teachers. The reason for doing so might be to increase the retention rate of high performing junior teachers. This could be a worthwhile objective notwithstanding the fact that the full evaluation correlation is greater for senior teachers. Or a district might decide to provide equal probability of recognition to teachers in untested and tested grades and subjects despite the different misclassification rates that result. That is acceptable as we frame policy and administrative uses of our model, as long as teachers are selected in rank order within their category based on their evaluation score.

Q2. Would your model apply to selecting low performing teachers for special treatment?

A2. In general, yes, but some specifics would change.

Our model and the associated calculator can be used to estimate the number of teachers who could be identified and their contribution to student value-added for either tail of the distribution and for any value of exceptionality. For example, if a district wanted to identify the lowest 5 percent of its teachers in terms of their true contribution to value-added for intensive remediation, they could use our calculator, enter 95 percent as the exceptionality parameter, and simply change the numerical signs and descriptors for the output in the last two columns, e.g., “SDs above average” to “SDs below average.” Using all the other values entered into our previous worked example, but changing exceptionality from .25 to .95, generates the following output:

The necessary changes to the headings and signs to make the output relevant for low performers follow:

This output makes clear the difficulty of reliably identifying appreciable numbers of teachers in the extreme tails of the distribution for special treatment based on their evaluation score assuming correlations similar to those we have employed as examples. In this case whereas the target is 5 percent, or 50 teachers in a district of 1000, only 5 can be reliably identified. Here is a case in which a change in the tolerance factor might be required. For example, by increasing the tolerance value from 50 percent to 75 percent, the number of teachers who could be identified rises from 5 to 22, or to roughly half of the theoretical target of 50 teachers. Increasing the tolerance for over-identification while reducing the tolerance for under-identification may be a reasonable policy decision with respect to interventions for very low-performing teachers.

Q3. What is the optimal number of years of baseline evaluation data for making selection decisions and for passing muster?

A3. There is no optimal number of years of baseline data. The advantage of additional years of data or additional data in any form in improving prediction has to be weighed against the costs of collecting such data and the practical needs of decision making.

Our worked example is based on two or three years of data: one or two baseline years of evaluation data used to predict a subsequent year’s value-added. Some districts have more years of evaluation data on some teachers; some less. Inclusion of more data points for individual teachers will increase the precision of prediction. Further, using at least two years of data for the baseline period substantially reduces the biases that can occur in the assignment of students to teachers in a single year, e.g., an unusually disruptive student who depresses a teacher’s ability to create good conditions for learning for all the students in the classroom. However, the practical realities of personnel decisions may require time frames that are shorter than those that are optimal for the most accurate decisions, e.g., ten years of data would allow much more accurate predictions than two years of data but districts should not wait 10 years to evaluate their teachers. We leave these design details to districts with the understanding that the more data they collect the better their predictions are likely to be, and the better their predictions the greater the proportion of evaluated teachers that can be identified as exceptional under our model.

Q4. Which model for calculating value-added do you recommend?

A4. For the purposes of this report, we do not prefer any particular value-added model.

All teacher value-added models start with the gain scores of individual students on academic assessments that are associated with the time period when those students are the responsibility of a particular teacher. A large number of statistical adjustments are then possible to try to remove influences on student achievement that are not the effect of the teacher. For example, classrooms with more economically disadvantaged students may, on average, show higher or lower rates of growth than classrooms of more advantaged students, so value-added models often adjust for that variable and other student background characteristics that reflect the mix of students in a particular teacher’s classrooms. Adjustments can also be made for school characteristics, for district characteristics, for attendance, and other factors. There are typically several ways of calculating the effects of variables regardless of which variables are included in the value-added model. Further, models that are formally very similar can produce different results depending on the number of years of prior data on student achievement that are used as an adjustment for student background. There is as yet no clear consensus among the econometricians and statisticians who construct these models as to which is best. Further, models that may be superior technically can fail as management tools or face political or public challenges because they are difficult for anyone except specialists to understand.[≠]

Because of the unsettled nature of the engineering of value-added models and the tension between technical strength and public transparency we do not recommend a particular value-added model. However, we believe there is now a sufficient body of research to recommend that the most important adjustment in a value-added model is the inclusion of student achievement from prior years, e.g., adjusting for how much a 6^th grade teacher’s students learned in 5^th and 4^th grade.[xiii] Although the design of value-added models clearly matters, we believe that for the broad purposes of passing muster, which is intended to require only a light hand from the federal or state level, it is better to leave those design decisions to responsible education authorities.

Q5. Is the passing muster model useful for anything in addition to the task of identifying exceptional teachers?

A5. Yes. There are a number of possible uses for both policy and management decisions.

We have built our narrative around federal and state programs that provide incentives to local education agencies to recognize exceptional teachers. However, the model we have put forward could be used to create conditions for funding broader education programs that are predicated on school districts having meaningful teacher and principal evaluation systems. For example, the Obama administration has identified improved teacher and principal effectiveness as a centerpiece of its plans for reauthorization of the Elementary and Secondary Education Act (ESEA).[xiv] We can imagine Congress building a requirement into Title I or Title II of ESEA that districts report on the reliability of their evaluation system using our model as a foundation. With those data available for every school district in the nation, we could imagine states taking the next step and determining threshold values of evaluation reliability that districts would be expected to meet. We could also see the federal government incentivizing states or districts to reach threshold values or to demonstrate improvement in the reliability of their evaluation systems.

Technical Appendix: Standard-Setting for Value-added Based Teacher Identification Systems

This appendix provides the technical details underlying our approach to evaluating the reliability of a teacher evaluation system, and the related question of how state or federal teacher recognition programs can accommodate district evaluation systems of differing quality. We approach this question as a Bayesian decision problem: Based on an imprecise teacher evaluation, we must (1) form posterior beliefs about each teacher’s effectiveness and then (2) base decisions about recognition on these posterior beliefs in a way that minimizes mistakes. The key insight is that decisions should be made based on posterior beliefs, and these beliefs are more precise when the teacher evaluation system is more reliable allowing us to recognize more teachers while making fewer mistakes.

The appendix is organized into four sections. The first section formalizes our approach using a simple model. The second section shows how posterior beliefs can be formed about teacher effectiveness, and how these relate to the reliability of the evaluation. The third section develops a simple method that can be used to estimate the reliability of the teacher evaluations from district data. The final section shows how this information can be used to select teachers for recognition in a way that minimizes mistakes.

1. A Simple Model for the Evaluation of Teacher Evaluation Systems

Our goal is to identify teachers whose effectiveness, defined as the effect they have on student test scores, is above some threshold. For this example, we will consider a threshold at the 75th percentile of teacher effectiveness, so that only the top 25 percent of teachers would truly exceed this threshold if we had a perfectly reliable evaluation. We assume that the impact of teacher (j) on student test scores is the teacher effect (μj) in the following value-added model:

Equation 1:

Where Xijc represents variables that control for student (and potentially peer) factors such as prior test scores, μj is the teacher effect, θjc is an idiosyncratic effect on student scores in a given classroom (c) of students, and εijc is a student-level idiosyncratic residual. We will assume that the teacher effect is normally distributed with mean 0 and variance .

Districts cannot observe the teacher effect (μj) directly, but instead must evaluate teachers based on a variety of imperfect measures such as value-added estimates, classroom observations, etc. Suppose that a district uses whatever information they have available to create an overall evaluation score (Sj) for each teacher. This score could be formed in any way, such as based on informal reviews by principals, or formal composite measures based on multiple measures of teacher performance. We will assume that the evaluation score is continuous and normally distributed, but if this were not the case the ideas here would generalize (although some of the specifics would differ). For simplicity, we will also assume that the evaluation of all teaches is based on the same information—e.g., all teachers are evaluated based on classroom observations and value-added estimates.

2. Posterior Beliefs and the Reliability of Teacher Evaluation

We can summarize our posterior beliefs about μj after observing Sj as:

Equation 2:

Where the posterior mean is , and the posterior variance is . Thus, α represents the increase in average teacher value-added associated with a one point increase in the overall evaluation score, while represents the amount of remaining variation in the teacher effect among teachers with a given score. With an assumption that teacher effects are normally distributed, the mean and variance summarize the posterior distribution (again, the ideas here would generalize without normality). Note that we are assuming the same posterior variance for everyone, i.e., that the evaluation score (Sj) provides the same amount of information for all teachers.

A simple statistic summarizes the quality of Sj in terms of how much it improves our ability to predict the teacher effect:

Equation 3:

This statistic is analogous to the teacher-level R-squared statistic, i.e., the percentage of the variance in the teacher effect on student test scores that was explained by the evaluation score (Sj). The square root of this statistic is the correlation between the evaluation score (Sj) and the teacher effect on test scores (μj):

Equation 4:

The more strongly correlated the score is with the teacher effect, the more reliable is the evaluation. We discuss how to estimate this statistic from a district’s data below.

The R-squared statistic in Equation 3 is directly related to the posterior uncertainty about a teacher’s effectiveness. One can rearrange Equation 3 to write the posterior standard deviation as . Thus, the closer the R-squared is to one, the more precise our posterior beliefs are relative to the total variation across teachers.

3. Estimating the Reliability of Teacher Evaluations from District Data

There are a number of ways to use district data to estimate the correlation between the teacher evaluation score and teacher impacts on test scores. Plugging Equation 2 into Equation 1 yields:

Equation 5:

Note that this is just a value-added model that includes Sj as an additional regressor. In principal, one could estimate the posterior mean () and variance ( ) by estimating Equation 5 using hierarchical linear models (HLM), and then use these to calculate the correlation in Equation 4 (noting that ). This HLM approach could be difficult to implement in practice, since it requires student-level data, multiple classrooms per teacher (to separately identify the classroom-level effect from the teacher effect), and a teacher evaluation score that was constructed in a different year (so that it is not correlated with the errors in equation 5).

A simpler method is the following. Suppose that a district can provide 3 pieces of information for each teacher:

1. The evaluation score (S_j,t) from a given year (t).
2. A value-added estimate from the same year (VA_j,t). The value-added estimate for each teacher is the average residual over all of their students, using the residual from a value-added regression as specified in Equation 1:

So that the value-added estimate is equal to the teachers true impact on test scores (μ_j) plus some estimation error. 3. A similar value-added estimate from a different year (VA_j,s), where this could be the year after.

The simple correlation between the evaluation score and the teacher’s value-added in the same year will be biased because both the evaluation score and teacher’s value-added will be influenced by idiosyncratic factors that occurred during that year (i.e., because the evaluation score depends on this year’s performance, it will be correlated with the idiosyncratic student and classroom errors in Equation 1. To avoid this bias, we use the correlation between the evaluation this year and value-added in a different year: . However, this correlation is the correlation between the evaluation score and the noisy estimate of the teacher effect on test scores, not the correlation with the true teacher effect on test scores as defined in Equation 4. Because of the noise in the value-added estimate, this correlation will understate the true correlation.

It is straightforward to show that to correct for this bias, we must adjust the correlation for the proportion of the variation in the value-added measure that is due to the true teacher effect, as opposed to estimation error (the reliability of the value-added measure). One simple estimate of the reliability of the value-added measure is the correlation in the value-added measure from one year to the next (). Using this estimate for the reliability of the value-added measure, we can construct an adjusted estimate of the correlation between the evaluation score and the true teacher effect on test scores using:

Equation 6:

Thus, we can estimate the correlation between a district’s evaluation measure and teacher effects on test scores based simply on information about the correlation of the evaluation measure with value-added in a different year, and the correlation of value-added from one year to the next.

4. Selecting Teachers to Minimize Mistakes

For any given evaluation system, we can calculate the posterior distribution for each teacher based on the model just described and methods that will be described below. Given the posterior distribution, how should we decide whether any teachers (and if so, how many) should be eligible for top-25 percent recognition?

The general answer to this question from a technical perspective is that eligibility should just depend on the likelihood that a teacher exceeds a specified threshold based on each teacher’s posterior distribution. As discussed above, the posterior distribution for each teacher’s effect (μj) is a normal distribution with mean and variance . Therefore, we can use the posterior distribution to calculate the probability that each teacher is below the threshold for recognition. All that needs to be determined is the threshold above which teachers are recognized for exceptional performance (the exceptionality parameter) and the minimum probability of being below the threshold that one is willing to tolerate when identifying any teacher as exceptional (the tolerance parameter), i.e., the willingness to classify a teacher as exceptional who does not actually belong in the exceptional category based on his or her true performance.

The exceptionality parameter is a policy choice. For ATC, we have chosen to focus on recognizing teachers who are in the top quartile (top 25 percent) of all teachers in terms of raising student test scores. Therefore, we set the threshold at the 75th percentile of the distribution of teacher effects, which is equal to (based on the normal distribution).

The tolerance parameter is more of a technical choice, driven by the relative costs of making errors of omission (not rewarding a teacher who is truly in the top 25 percent) versus errors of commission (rewarding a teacher who is not actually in the top 25 percent). We chose a tolerance of 50 percent, meaning that to be selected for ATC the teacher must have at least a 50 percent chance of being in the top quartile (or, equivalently, every ATC teacher has no more than a 50 percent chance of lying below the 75th percentile of teacher effectiveness). A tolerance of 50 percent minimizes the total number of errors—the total number of teachers who are misclassified due to errors of omission or commission. If the cost of incorrectly rewarding a non-exceptional teacher were higher than the cost of incorrectly overlooking an exceptional teacher, one would want to set a lower tolerance. For example, a tolerance of 10 percent would ensure that any teacher selected for ATC had at least a 90 percent chance of having true performance above the 75th percentile. While this standard would reduce the chance of rewarding a teacher who is not actually in the top quartile, it would reward many fewer teachers and thereby increase errors of omission—omitting many more teachers whose performance was actually in the top quartile.

A tolerance of 50 percent results in a very simple selection rule: select any teacher whose posterior median (which is the posterior mean,, for a normal distribution) exceeds the threshold for recognition. The posterior mean will not exceed the threshold for many teachers when the evaluation () is an unreliable predictor of teacher performance. In the extreme, if the evaluation score was uncorrelated with teacher impacts on test scores (), then no teacher would qualify. At the other extreme, when the evaluation score is perfectly correlated with teacher performance (the R-squared in Equation 3 is equal to one), the posterior mean will be a teacher’s true impact on test scores and we will correctly identify all 25 percent of teachers in the top quartile for ATC.

More generally, for any given level of tolerance and exceptionality, there is a direct relationship between the reliability of the evaluation score, as summarized by the R-squared in Equation 3 (or, equivalently, the correlation in Equation 4), and the proportion of teachers who will be eligible for ATC. As the correlation between the evaluation scores and the teacher effects on test scores increases from zero to one, the proportion of teachers eligible for ATC increases from 0 to 25 percent. Figure 3 in the main body of the paper plots this relationship, and the lookup tables provide the calculation of what percent of teachers should be eligible for any level of tolerance, exceptionality, and reliability of the teacher evaluation.

One nice feature of this approach is that it is not an “all or nothing” approach that requires every district’s evaluation system to exceed a certain level of reliability. Instead, all districts can participate in ATC. Those with better evaluation systems will have more teachers eligible for ATC than those with worse evaluation systems.

While all of the discussion to this point has assumed that value-added estimates are available for all teachers, the system can be accommodated to include teachers evaluated by other methods (e.g., classroom observations, student evaluations) in untested grades and subjects. For these teachers, we would suggest using results from tested grades and subjects as a guide. In particular, for teachers in tested grades and subjects, the district could construct an evaluation score using the more limited information used in untested grades and subjects (e.g., only using classroom observations & student evaluations, but excluding value-added). Then, the correlation between this limited evaluation score and teacher impacts on test scores could be estimated as we have discussed, and this correlation would determine the proportion of teachers eligible for non-tested grades and subjects. Because evaluations would most likely be less reliable without value-added information, fewer teachers would be eligible in non-tested grades and subjects if decisions were based entirely on the reliability of prediction. However, we describe in the narrative of our report why districts might find it desirable to create equal opportunities for recognition among different categories of teachers even though this would create differences in the rate of classification errors in the different categories.

[*] This assumption could be evaluated by occasional testing of students in subjects that are not typically tested year after year.
[+] http://www.brookings.edu/reports/2011/0426_evaluating_teachers.aspx
[±] It is important to note that many reports of the strength of teacher effects based on value-added are based on estimates from the same year(s) of data, whereas our model and examples are based on predictions to the next year from data collected in the previous year. Correlations between value-added estimates in one year and performance in subsequent years decay over time, but are still statistically significant 9 years later. The correlations are much higher and the decay in magnitude stabilizes after 5 years if the initial estimates are based on two years rather than one year of data. Goldhaber, D. & Hansen, M. (2010). Assessing the potential of using value-added estimates of teacher job performance for making tenure decisions. (Washington, DC: CALDER, The Urban Institute).
[≠] For an example of the clash between ease of understanding and technical sophistication see: Michael Winerip, “Evaluating New York teachers, perhaps the numbers do lie,” The New York Times, March 7, 2011, p. A15.

[i] Weisberg, W., Sexton, S., Mulhern, J., & Keeling, D. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. New York: The New Teacher Project.
[ii] Gabriel, T. (2010, September 2). “A celebratory road trip by education secretary,” New York Times, p. A24.
[iii] Hillsborough County Public Schools, New evaluation form for teachers, (June 8, 2010), available at http://communication.sdhc.k12.fl.us/empoweringteachers/?p=568.
[iv] District of Columbia Public Schools (2010). IMPACT: The District of Columbia Public Schools effectiveness assessment system for school-based personnel 2010-2011. Group 1 general education teachers with individual value-added student achievement data. Washington, DC: DCPS.
[v] Glazerman, S., Loeb, S., Goldhaber, D., Staiger, D., Raudenbush, S., & Whitehurst, G.J. (2010). Evaluating teachers: The important role of value-added. Washington, DC: The Brookings Institution.
[vi] Glazerman, S., Goldhaber, D., Loeb, S., Staiger, D., & Whitehurst, G.J. (2010). America’s Teacher Corps. Washington, DC: The Brookings Institution.
[vii] Clotfelter, C., Glennie, E., Ladd, H., & Vigdor, J. (2006). Would higher salaries keep teachers in high-poverty schools? Evidence from a policy intervention in North Carolina. NBER Working Paper 12285. Cambridge, MA: National Bureau of Economic Research.
Harris, D.N. (2009). “Teacher value-added: Don’t end the search before it starts,” Journal of Policy Analysis and Management, 28(4), pp. 693-699. Hill, H.C. (2009). “Evaluating value-added models: A validity argument approach,” Journal of Policy Analysis and Management, 28(4), pp. 700-709.
Baker, E.L., Barton, P.E., Darling-Hammond, L., Haertel, E., Ladd, H.F., Linn, R.L., Ravitch, D., Rothstein, R., Shavelson, R.J., & Shepard, L.A. (2010). Problems with the use of student test scores to evaluate teachers. Briefing Paper 278. Washington, DC: Economic Policy Institute.
Glazerman, S., Loeb, S., Goldhaber, D., Staiger, D., Raudenbush, S., & Whitehurst, G.J. (2010). Evaluating teachers: The important role of value-added. Washington, DC: The Brookings Institution.
MET Project (2010). Learning about teaching: Initial findings from the measures of effective teaching project. Seattle, WA: Bill & Melinda Gates Foundation.
R othstein, J. (2011). Review of “Learning about teaching: Initial findings from the measures of effective teaching project.” Boulder, CO: National Education Policy Center.
Goldhaber, D. & Hansen, M. (2010a). “Is it just a bad class? Assessing the stability of measured teacher performance.” CEDR Working Paper #2010-3. Seattle, WA: University of Washington.
Kane, T.J. & Staiger, D.O. (2002). “The promise and pitfalls of using imprecise school accountability measures,” The Journal of Economic Perspectives, 16, 91-114.
Schochet, P.Z., & Chiang, H.S. (2010). Error rates in measuring teacher and school performance based on student test score gains (NCEE 2010-4004). Washington, DC: National Center for Evaluation and Regional Assistance, Institute of Educational Sciences, U.S. Department of Education.
[viii] Kane, T.J. & Staiger, D.O. (2008). Estimating teacher impacts on student achievement: An experimental evaluation. NBER Working Paper W14607. Cambridge, MA: National Bureau of Economic Research.
[ix] Goldhaber, D. & Hansen, M. (2010). Assessing the potential of using value-added estimates of teacher job performance for making tenure decisions. Washington, DC: CALDER, The Urban Institute.
[x] Harris, D.N. & Sass, T.R. (2009). What makes for a good teacher and who can tell? Washington, DC: CALDER, The Urban Institute.
[xi] MET Project (2010). Learning about teaching: Initial findings from the measures of effective teaching project. Seattle, WA: Bill & Melinda Gates Foundation.
[xii] Schochet, P.Z. & Chiang, H.S. (2010). Error rates in measuring teacher and school performance based on student test score gains (NCEE 2010-4004). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
[xiii] Kane, T.J. 2008. Estimating teacher impacts on student achievement: An experimental evaluation. Working Paper 14607. Cambridge, MA: National Bureau of Economic Research.
[xiv] U.S. Department of Education (2010, March 13). “A blueprint for reform,” available at http://www2.ed.gov/policy/elsec/leg/blueprint/publicationtoc.html

Downloads

Video

How to Standardize Teacher Evaluation Systems

Authors

Michelle Croft
Steven Glazerman
Dan Goldhaber
Susanna Loeb
Stephen Raudenbush
Douglas Staiger
Grover J. "Russ" Whitehurst

Publication: The Brookings Brown Center Task Group on Teacher Quality

Image Source: Yellow Dog Productions

Comparing Teacher Evaluation Systems

Tue, 26 Apr 2011 13:30:00 -0400

Event Information

April 26, 2011
1:30 PM - 3:00 PM EDT

Falk Auditorium
The Brookings Institution
1775 Massachusetts Ave., NW
Washington, DC

Although much of the impetus for new approaches to teacher evaluation comes from state and national policymakers, the design of teacher assessment systems falls to the roughly 16,000 school districts and 5,000 independent public charter schools in the U.S. The considerable variability among school districts in how they evaluate teachers’ performance raises questions: If an individual state or the federal government wishes to require or incentivize local education agencies to evaluate teachers more rigorously, how can they do so while honoring each district’s individual authority? And how can each school district benchmark the performance of its teacher evaluation system against the performance of evaluation systems in other districts, or against the previous version of its own evaluation system? How can teacher evaluation systems be compared, one to another?

On April 26, the Brown Center Task Force on Teacher Quality hosted an event on the release of a new report on teacher evaluations, the third in a series of proposals by the Brown Center on Education Policy at Brookings on improving teacher quality. The report addresses how a state or the federal government could achieve a uniform standard for dispensing funds to school districts for the recognition of exceptional teachers without imposing a uniform evaluation system on those districts. A panel of the report's authors presented the findings in the context of the controversies surrounding teacher evaluation and the promise that more meaningful teacher evaluation offers for enhancing student achievement.

After the program, the panelists took audience questions.

Audio

Comparing Teacher Evaluation Systems

Transcript

Download the Full Transcript (.pdf)

Event Materials

20110426_teacher_evaluations_transcript

Participants

Panelists

Steven Glazerman

Senior Fellow
Mathematica Policy Research

Dan Goldhaber

Director, The Center for Education Data and Research
University of Washington

Susanna Loeb

Professor of Education and Director, Institute for Research on Education Policy and Practice
Stanford University

Stephen W. Raudenbush

Lewis-Sebring Distinguished Service Professor and Chair, Committee on Education
University of Chicago

Douglas O. Staiger

John French Professor of Economics
Dartmouth College

We Need to Standardize Teacher Evaluation Systems

Grover J. "Russ" Whitehurst — Mon, 25 Apr 2011 16:28:00 -0400

The difference between getting a good teacher and a lousy teacher can be 10 percentile points in gain over one year, says Russ Whitehurst, director of the Brown Center on Education Policy at Brookings. Of all the things that are under the control of policymakers and schools, teacher quality is at the top of the list in terms of impact on student achievement, and so there is a great interest in evaluating teacher performance. On April 26, the Brown Center Task Force on Teacher quality hosts an event to release a new report on teacher evaluations accompanied by a calculator that will assist in making comparisons between and within districts. Whitehurst says we need better ways to place teacher evaluation systems on the same scale so state and federal governments can incentivize fairly and open the door to policies that depend on having good evaluation systems in place.

Video

How to Standardize Teacher Evaluation Systems

The False Promise of Class-Size Reduction

Matthew M. Chingos — Thu, 14 Apr 2011 00:00:00 -0400

Class-size reduction, or CSR, is enormously popular with parents, teachers, and the public in general. The latest poll results indicate that 77 percent of Americans think that additional educational dollars should be spent on smaller classes rather than higher teacher salaries. Many parents believe that their children will benefit from more individualized attention in a smaller class and many teachers find smaller classes easier to manage. The pupil-teacher ratio is an easy statistic for the public to monitor as a measure of educational quality, especially before test-score data became widely available in the last decade.

Policymakers across the nation, including those in at least 24 states, have taken these ideas to heart and enacted CSR initiatives at costs upward of billions of dollars. California allocated $1.5 billion per year in the late 1990s to reduce class size in the early grades. Florida has spent about $20 billion since 2002 reducing class size in every grade from kindergarten through high school. The federal government also has its own program, which provided $1.2 billion to $1.6 billion per year from 1999 to 2001 for CSR in grades K–3. This program was absorbed into Title II of the No Child Left Behind Act in 2001.

These policies, coupled with trends in local school districts, have produced a widespread reduction in the number of students per teacher over the past four decades. Figure 1 shows that the pupil-teacher ratio in public schools has fallen by about 30 percent since 1970. This trend partly reflects an increase in educational services to students with disabilities, as required by federal law beginning in 1975. But falling pupil-teacher ratios affected all students, as evidenced by the even steeper drop at private schools (which serve fewer disabled students). The trend at private schools also likely reflects the strong preference of parents for small classes and the greater incentive for private schools to respond to those preferences.

Parents, teachers, and policymakers have all embraced CSR as a strategy to improve the quality of public education. There is surprisingly little high-quality research, however, on the effects of class size on student achievement in the United States. The credible evidence that does exist is not consistent, and there are many low-quality studies with results all over the map. The most encouraging results for CSR come from a single experiment conducted in the 1980s, which found that a large reduction in class size in the early grades increased test scores, particularly among low-income and African-American students. But evaluations of large-scale CSR policies in California and Florida have yielded much less positive results, perhaps because of the need to hire so many (inexperienced and potentially less effective) new teachers.

The evidence on class size indicates that smaller classes can, in some circumstances, improve student achievement if implemented in a focused way. But CSR policies generally take exactly the opposite approach by pursuing across-the-board reductions in class size at the state or federal level. These large-scale, untargeted policies are also extremely expensive and represent wasted opportunities to make smarter educational investments.

Large-scale CSR policies clearly fail any cost-benefit test because they entail steep costs and produce benefits that are modest at best. But what about reductions in class size at the district or school level? When school finances are limited (as they always are), the cost-benefit test any educational policy must pass is not “Does this policy have any positive effect?” but rather “Is this policy the most productive use of these educational dollars?” Assuming even the largest class-size effects in the research literature, such as the STAR results that indicate that a 32 percent reduction in class size increased achievement by about 15 percent of a year of learning after one year, CSR will still fail this test because it is so expensive. Reducing class size by one-third, from 24 to 16 students, requires hiring 50 percent more teachers. Depending on how much extra space schools have, new facilities may need to be built to accommodate the additional classes.

There are certainly many policies that might be proposed as cost-effective alternatives to CSR, but one set of policies that stand out are those aimed at improving teacher quality. Researchers agree that teacher quality is the single most important in-school determinant of how much students learn. Stanford economist Eric Hanushek has estimated that replacing the worst 5 percent to 8 percent of teachers with average teachers would dramatically boost achievement in the United States. Investing less in CSR would free up resources that could be used to recruit and retain highly effective teachers. For example, schools might “treat different teachers differently,” or pay teachers differently based on their effectiveness in the classroom or the subject area they teach, as Robin Chait and Raegen Miller have suggested.

The fact that across-the-board CSR policies at the state or district level are not cost-effective does not mean that smaller classes should never be used, but rather that they should be reserved for use in special cases by individual schools. A principal may decide, for example, that a smaller class makes sense for an inexperienced teacher who needs support in developing skills to provide accommodations for students with disabilities. At the same time, the principal may want to assign a larger class to a highly effective veteran teacher, perhaps with some extra compensation for the additional work required. School districts should encourage this kind of creative management and enable it by collecting and providing to principals detailed data on their teachers and the classes they teach.

Download the full paper » (www.americanprogress.org)

This material [article] was published by the Center for American Progress http://www.americanprogress.org/ (online)

Authors

Matthew M. Chingos

Publication: Center for American Progress

The Year of Education Reform, with Gov. Chris Christie

Thu, 07 Apr 2011 11:00:00 -0400

Event Information

April 7, 2011
11:00 AM - 12:00 PM EDT

Jumeirah Essex House

New York, NY

School districts across the nation are grappling with the question of how to improve student performance in a time of fiscal austerity. Some reformers are challenging the idea of automatic tenure, arguing that teachers should be paid based on performance rather than seniority. Moreover, recent legislative battles involving teacher compensation in Wisconsin and Ohio have put the issue squarely in the public spotlight.

On April 7, New Jersey Gov. Chris Christie (R-NJ) addressed these questions and spoke of the need to reform the U.S. education system so that teachers are held accountable for student progress. He described the steps that New Jersey has taken to meet his goals, and outlined a proposal for public school districts to include peer evaluations in their annual assessments of teacher performance. Brookings Managing Director William Antholis welcomed the governor, while James D. Robinson III, General Partner and Co-Founder of RRE Ventures, provided introductory remarks. Grover J. "Russ" Whitehurst, director of Brookings's Brown Center on Education Policy, moderated the discussion that followed Gov. Christie's remarks.

Video

Audio

Governor Chris Christie

Transcript

Uncorrected Transcript (.pdf)

Event Materials

20110407_christie_education

Participants

Panelists

James D. Robinson III

General Partner and Co-Founder, RRE Ventures

The Hon. Chris Christie

Governor, New Jersey

Americans Want More Coverage of Teacher Performance and Student Achievement

E.J. Dionne, Jr., Darrell M. West and Grover J. "Russ" Whitehurst — Tue, 29 Mar 2011 13:59:00 -0400

EXECUTIVE SUMMARY

Americans want more media coverage of their local schools. In particular, they want more information than they now receive about teacher performance, student academic achievement, crime, and violence in their schools – and more as well about curricula, finances and reform efforts. While there is a great interest in receiving this information through new technological sources more so than ever before (http://pewresearch.org/pubs/1924/state-of-the-news-media-2011), Americans however, continue to rely on traditional media, particularly newspapers, for information on their schools. There is an imperative to improve both education journalism and the ways in which schools communicate directly to parents, students, and citizens.

In an earlier report (West, Whitehurst, and Dionne, 2009), we noted several problems with the way the media system reports on education, the most basic being that there little national coverage of education. During 2009, only 1.4 percent of national news coverage from television, newspapers, news Web sites, and radio dealt with education. In our content analysis, we found this paucity of coverage was not unique to 2009. In 2008, only 0.7 percent of national news coverage involved education, while 1.0 percent did so in 2007.

Of the education news that is reported, little relates to school policies and ways to improve the curriculum or learning processes. There was hardly any coverage of school reform, teacher quality, or other matters thought to be crucial for educational attainment. Instead, most 2009 stories focused on budget problems, school crime, and the H1N1 flu outbreak. The lack of news coverage of the actual work of schools represents a significant problem for the education area.

In a follow-up paper, we reported on interviews with a number of educational and media leaders, outlined new trends in education coverage, and described how major news organizations are re-imagining their futures (West, Whitehurst, and Dionne, 2010). We examined the development of niche publications, news aggregators, social media, and new content providers, and the emergence of alternative business models for funding news organizations. We also argued that education journalism is transforming itself into a new digital form that looks and behaves differently than traditional models. Digital technologies have altered the manner in which the traditional education news industry produces and disseminates information (Melton, 2009). Through the emergence of the Internet, social media, blogs, electronic news readers, smart phones, and cell phone texting, the cost of information transmission has dropped and the vehicles for communication have expanded dramatically. New content providers have found novel ways of delivering information to students, parents, and the general public.

The new ecosystem has clear strengths, including immediacy, interactivity, and diversity. But these virtues must be linked more effectively to the delivery of in-depth and substantive reporting. It is important to build on the strengths of new media platforms, but we also must find ways to develop high-quality coverage because it is crucial for democratic governance.

With the evolving mix of old and new media, how do Americans get information about elementary and secondary schools? How do they rate the performance of various content providers? What topics deserve additional attention? How would viewers and readers like to get information about schools in the future? And how do consumer differences in age, gender, race, income, region, and parental status affect their views about these issues?

In this report, we present the results of a national public opinion survey on education news. From December 6 to 19, 2010, we conducted telephone interviews with 1,211 adults aged 18 years or older (including an over-sample of parents) in the continental United States. We asked a series of questions about their education news consumption focusing on where respondents got their information, how they assessed media sources, and how they envisioned the future of education reporting and school communication. (See Appendix for description of survey methodology and questions.)

Respondents were clear on their priorities. Asked about the areas in which they wanted more coverage of their local schools, 73 percent wanted more information on teacher performance; 71 percent wanted more on student academic performance; 69 percent wanted more reporting on school crime or violence; 68 percent wanted more on school curricula; and 66 percent more both on school finances and school reform (66 percent). The matters about which respondents were least likely to seek more information were school athletic team performance (42 percent), and scandals or undesirable activities at their local schools (50 percent). The relatively smaller number of respondents seeking more coverage for school sports teams may not be surprising, since in many communities, athletics － particularly at the high school level － already receive substantial coverage.

The survey also asked respondents where they received education information now and how they would like to receive it the future. The most common sources of current education news were family and friends (75 percent), followed by daily newspapers (60 percent), school publications (56 percent), local television (54 percent), community groups (42 percent), national television (38 percent), Internet sites (37 percent), radio (33 percent), school specialty publications (28 percent), school Facebook or MySpace sites (14 percent), electronic newsreaders (11 percent), cellphone texts (nine percent), and blogs (nine percent). However, there were interesting age differences with young people being more likely than older people to rely on a range of Internet and electronic sources such as blogs, social media, electronic news readers, and cellphone texts.

The most highly regarded current news-providers were family and friends, with 62 percent rating that source positively. This was followed by school publications (45 percent), daily newspapers (44 percent), local television (38 percent), community groups (32 percent), school specialty publications (28 percent), Internet sites (25 percent), national television (24 percent), radio (24 percent), school Facebook sites (12 percent), electronic news readers (nine percent), blogs (nine percent), and cellphone texts (seven percent).

Young people were more likely than older respondents to have positive assessments of electronic outlets: 35 percent of those aged 18 to 29 gave excellent or good ratings to Internet news sites, compared to 9 percent of senior citizens. There were similar magnitudes of difference when it came to school Facebook sites (21 percent of the young but only four percent of older respondents have them positive ratings), blogs (11 percent positive from young people and five percent among those 65 years or older), electronic news readers (13 percent among the young, four percent among older respondents), and cell phone texts (10 percent versus two percent). Whites (46 percent) were more likely than non-whites (38 percent) to feel that newspaper coverage was excellent or good.

There were variations in respondents’ assessments of the overall quantity of the education news they received. Thirty-nine percent said they received too little information about elementary and secondary schools, 56 percent got “the right amount of information,” and only two percent said there was too much information. When asked about changes over time, 22 percent felt the quantity of news had increased, 60 percent felt it had stayed about the same, 10 percent believed that it had decreased. Thirty percent considered themselves well-informed about elementary and secondary schools in the community, 47 percent moderately informed, and 22 percent said they were poorly-informed.

When given a list of suggestions for improving the information they received on education information, 82 percent said their schools should communicate more though printed newsletters; 74 percent wanted to receive more school information through the Internet; and 71 percent wanted more from email communications. A noticeably large proportion, 67 percent, said they wanted to get more information through newspaper blogs and forums. The least popular areas for an expanded flow of education information were schools communication through Facebook pages (38 percent), through cell phone texts (32 percent) and cellphone texts from newspapers (23 percent).

Younger respondents were more interested in the newer technological approaches. Among younger respondents, 76 percent were interested in seeing schools communicate through the Internet, compared with 52 percent of older respondents. There were similar findings for newspaper blogs and forums (63 percent for the young, 42 percent for older respondents; school emails (80 versus 49 percent); cellphone texts (44 versus 22 percent); school Facebook pages (51 versus 30 percent); newspaper text alerts (70 versus 42 percent); newspaper cellphone texts (37 versus 18 percent), and newspaper blogs or forums (74 versus 56 percent). Non-whites were more interested than whites in expanded school communication through cellphone texts (42 for non-whites compared with 30 percent for whites), through Facebook (46 versus 35 percent), email alerts (65 versus 52 percent) and cellphone messages (40 versus 18 percent).

Downloads

Download the Full Report

Authors

Image Source: Sam Bloomberg-Rissman

First Step to Literacy: Getting Books in the Hands of Children

Anda Adams and Jacques van der Gaag — Fri, 28 Jan 2011 14:07:00 -0500

Being able to read and write is the most basic foundation of knowledge accumulation and further skill development. Without literacy, there can be no quality education. Presently, 1 in 5 adults is illiterate, two-thirds of whom are women. At the current pace, over 700 million adults worldwide will still not be able to read in 2015. [1] In global education discussions, literacy rates are most often reported for adolescents and adults, an ex post facto measure of the failure of primary school systems to impart basic skills in the most formative schooling years. It is clear that much needs to be done to provide these adolescents and adults with access to successful literacy programs. But we must also ensure that children with access to schooling are not growing up to be illiterate.

Children enrolled and regularly attending school for the first three grades should be able to read basic text. Evidence shows that acquiring this ability to read sets students up for further learning, enabling them to read and comprehend progressively more advanced materials and acquire additional knowledge.

As explained in our earlier policy brief, data from numerous countries show that children in school are failing to acquire the most basic of skills, measured as the ability to read words of connected text. We called for a global paradigm shift that places learning at the center of the global education discourse. This shift requires the major bilateral and multilateral actors to refocus their own efforts on supporting learning in the classroom and measuring progress by increased learning outcomes. There has been some progress here, such as USAID’s goal to improve reading skills for primary school children in its new education strategy and the World Bank’s Education Strategy 2020, Learning for All: investing in people’s knowledge and skills to promote development.

This shift of focus also requires substantial changes on the ground, including encouraging and supporting a culture of literacy and learning at the community level. For example, Gove and Cvelich highlight some main factors contributing to low reading levels, including a lack of support for teachers, limited instructional time, poorly resourced schools, the absence of books in the home and policies regarding the language of instruction. [2] In Mali, a recent survey found that three-quarters of grade 2 students did not have a textbook and no student had supplementary reading books at school. [3] In The Gambia, the vast majority of students who demonstrated a level of reading fluency said that they had books at home. Globally, in both developed and developing economies, a relatively consistent proxy for “parental commitment to education” is the number of books in the home. A 20-year study of 27 countries found that children growing up in homes with many books get three years more schooling than their peers who come from homes without books. [4] There is no one-size-fits-all solution to improving the quality of education in developing countries. However, there is plenty of room for innovation to address some of the biggest barriers to improving reading levels, including availability of appropriate reading materials at school and at home. In disadvantaged communities, where there are relatively few books and even fewer books in local languages and that deal with culturally-relevant topics, innovation is needed to help develop a robust culture of literacy.

One such innovation is Worldreader.org’s iRead pilot in Ghana, which has put hundreds of e-readers into children’s hands. A lot has been written on similar classroom technology in developing countries, which cite examples of supplying hardware to schools without plans for its educational use, promoting technology from a single company, insufficient planning for sustainability, and inadequate investment in time to train teachers and administrators who will be the purveyors of the technology initiatives in the classrooms. [5]

However, the important difference between this e-reader program and similar projects focused on putting computers in classrooms is that e-readers usually operate on the mobile phone system, which has exploded in developing regions over the last few years. In Kenya, more than 80 percent of the population has mobile phone network coverage and more than half of the population has purchased a mobile phone subscription. The GSM compatibility of e-readers allows for downloading of new reading materials wherever there is mobile phone coverage and sufficient funds available to purchase new texts. E-readers also have relatively low levels of energy consumption (a one-hour charge can last more than a week). In addition to gaining the support of community leaders and teachers from the beginning, the pilot began with intense in-service training for teachers in how to use e-readers to complement their existing curricula. While Worldreader.org has not solved all of the challenges posed by technology initiatives in education, it has taken some important steps toward addressing the barriers to project success. [6]

The organization has also tackled specific challenges that are impeding reading success in the early primary grades:

Additional support for emergent readers. E-readers provide additional support to teachers in teaching children how to read, an important supplement in primary school classrooms in low-income countries where there may be 40 or 50 students per teacher. In such cases, students are required to work independently or in small groups while the teacher is working with other students. The text-to-speech feature on e-readers can read books aloud to the student, exposing her to the written text as she hears it read aloud. Students can also use the downloaded dictionary while reading to look up unfamiliar words and continue to read without adult assistance.
Students and teachers get to choose. While paper books donated by schools, libraries, and individuals from around the world have helped to get written materials into low-resource schools in developing countries, e-books allow students and teachers in developing countries to choose which books they teach and read. Although choices now are restricted by the dominance of English in the e-book market, the potential for the expansion of the digital market represents a step toward greater agency for teachers and students.
Working with local publishers to increase access to books for emergent readers. Children learning to read need access to the types of books that engage their imagination and spark their interest. For children learning to read, this means stories with simple sentences in their local language. Yet, traditionally children’s books are not a good economic bet for publishers, particularly in developing countries. The high cost of printing the books are not recouped since so many families cannot purchase copies for their own household use. However, distributing books in e-reader format will actually allow publishers to reach more customers at a lower cost. To bring more books to the developing world through e-readers and e-books, Worldreader.org seeks to support a self-sustaining reading and publishing culture by working with local publishers to digitize books and materials to support local language curricula.
Portability can increase reading opportunities. Anecdotal reports from classroom teachers in the Ghanaian pilot frequently reference how students would not stop reading, pulling out their e-readers in between lessons, during recess and lunch, and after school with friends, parents and siblings. An International Association for the Evaluation of Educational Achievement study on reading literacy in 32 countries found that the amount of voluntary book reading that students did during out-of-school time was strongly positively related to students’ achievement levels. [7]

While the pilot is still in the early stages, the founders of the project are focused on the essential outcomes. Their USAID-funded impact study seeks to find out whether children are reading more than they were before the program and whether children read better than they were before the program. Measuring program success by understanding the impact on learning outcomes is a critical step for shifting the global education paradigm to one focused on learning.

[1] UNESCO. (2010). EFA Global Monitoring Report 2010: Reaching the Marginalized. Paris: UNESCO.
[2] Gove, A., and P. Cvelich, (2010). Early Reading: Igniting Education for All. A report by the Early Grades Learning Community of Practice. Research Triangle Park, NC: Research Triangle Institute.
[3] Evans, 2010
[4] M.D.R. Evans, Jonathan Kelley, Joanna Sikora, Donald J. Treiman. “Family scholarly culture and educational success: Books and schooling in 27 nations.” Research in Social Stratification and Mobility, 2010; DOI: 10.1016/j.rssm.2010.01.002
The study controls for education levels, occupations, and socio-economic status of the parents.
[5] For example, Trucano, M. “Worst practice in ICT use in education,” 2010, accessed at http://blogs.worldbank.org/edutech/worst-practice
[6] Some of the core challenges identified by Worldreader.org and others include the upfront costs of e-readers, need for on-going training and support to teachers, students, and communities, buy-in of school systems and local governments to deploy technology and content, insufficient relevant materials in e-book format, and consistent access to electricity and mobile networks.
[7] Elley, W.B. (Ed.). (1994). The IEA Study of Reading Literacy: Achievement and Instruction in Thirty-two School Systems. Oxford: Pergamon Press.

Downloads

Download Paper

Authors

Anda Adams
Jacques van der Gaag

Equitable Learning: Our Common Objective

Wed, 08 Dec 2010 13:00:00 -0500

Event Information

December 8, 2010
1:00 PM - 2:30 PM EST

Johnson Room
The Brookings Institution
1775 Massachusetts Ave., NW
Washington, DC

Improving the quality of education in developing countries is a complex challenge that requires input and cooperation by a multitude of stakeholders. As the frontline facilitators of education, teachers play a unique and important role in education; yet their voices and perspectives have not been well represented in the global dialogue to ensure equitable learning for all.

On December 8, the Center for Universal Education at Brookings hosted a private luncheon discussion on the role that the teaching profession plays in ensuring equitable learning for all. Susan Hopgood, president of Education International, and Angelo Gavrielatos, federal president of the Australian Education Union, provided opening remarks. Senior Fellow Rebecca Winthrop moderated the discussion with representatives of teachers’ unions, multilateral organizations, professional development organizations and alliances, advocacy groups and academics.

Learn more about the event »

Click on the thumbnail below to view a full-size image

CUE Director Rebecca Winthrop (center) with NEA President Dennis van Roekel (left), EI President Susan Hopgood, and AEU President Angelo Gavrielatos.

Participants

Moderator

Panelists

Susan Hopgood

President, Education International

Angelo Gavrielatos

Federal President, Australian Education Union

Evaluating Teachers: The Important Role of Value-Added

Wed, 17 Nov 2010 00:00:00 -0500

Executive Summary

The evaluation of teachers based on the contribution they make to the learning of their students, value-added, is an increasingly popular but controversial education reform policy. We highlight and try to clarify four areas of confusion about value-added. The first is between value-added information and the uses to which it can be put. One can, for example, be in favor of an evaluation system that includes value-added information without endorsing the release to the public of value-added data on individual teachers. The second is between the consequences for teachers vs. those for students of classifying and misclassifying teachers as effective or ineffective — the interests of students are not always perfectly congruent with those of teachers. The third is between the reliability of value-added measures of teacher performance and the standards for evaluations in other fields — value-added scores for individual teachers turn out to be about as reliable as performance assessments used elsewhere for high stakes decisions. The fourth is between the reliability of teacher evaluation systems that include value-added vs. those that do not — ignoring value-added typically lowers the reliability of personnel decisions about teachers. We conclude that value-added data has an important role to play in teacher evaluation systems, but that there is much to be learned about how best to use value-added information in human resource decisions.

Teacher evaluation at a crossroads

The vast majority of school districts presently employ teacher evaluation systems that result in all teachers receiving the same (top) rating. This is perhaps best exemplified by a recent report by the New Teacher Project focusing on thousands of teachers and administrators spanning twelve districts in four states.[1] The report revealed that even though all the districts employed some formal evaluation process for teachers, all failed to differentiate meaningfully among levels of teaching effectiveness. In districts that used binary ratings more than 99 percent of teachers were rated satisfactory. In districts using a broader range of ratings, 94 percent received one of the top two ratings and less than 1 percent received an unsatisfactory rating. As Secretary of Education Arne Duncan put it, “Today in our country, 99 percent of our teachers are above average.”[2]

There is an obvious need for teacher evaluation systems that include a spread of verifiable and comparable teacher evaluations that distinguish teacher effectiveness. We know from a large body of empirical research that teachers differ dramatically from one another in effectiveness. Evaluation systems could recognize these differences but they generally don’t. As a consequence, the many low stakes and high stakes decisions that are made in the teacher labor market occur without the benefit of formalized recognition of how effective (or ineffective) teachers are in the classroom. Is there any doubt that teacher policy decisions would be better informed by teacher evaluation systems that meaningfully differentiate among teachers?

There is tremendous support at both the federal and state levels for the development and use of teacher evaluation systems that are more discerning.[3] And the two national teachers unions, the AFT and the NEA, support teacher evaluation systems that recognize and reward excellence and improve professional development. This is consistent with their long-term support of the National Board for Professional Teaching Standards, which is designed to identify excellent teachers and provide them a salary bonus.

The latest generation of teacher evaluation systems seeks to incorporate information on the value-added by individual teachers to the achievement of their students. The teacher’s contribution can be estimated in a variety of ways, but typically entails some variant of subtracting the achievement test score of a teacher’s students at the beginning of the year from their score at the end of the year, and making statistical adjustments to account for differences in student learning that might result from student background or school-wide factors outside the teacher’s control. These adjusted gains in student achievement are compared across teachers. Value-added scores can be expressed in a number of ways. One that is easy to grasp is a percentile score that indicates where a given teacher stands relative to other teachers. Thus a teacher who scored at the 75^th percentile on value-added for mathematics achievement would have produced greater gains for her students than the gains produced by 75 percent of the other teachers being evaluated.

Critics of value-added methods have raised concerns about the statistical validity, reliability, and corruptibility of value-added measures. We believe the correct response to these concerns is to improve value-added measures continually and to use them wisely, not to discard or ignore the data. With that goal in mind, we address four sources of concern about value-added evaluation of teachers

Value-added information vs. what you do with it

There is considerable debate about how teacher evaluations should be used to improve schools, and uncertainty about how to implement proposed reforms. For example, even those who favor linking pay to performance face numerous design decisions with uncertain consequences. How a pay for performance system is designed—salary incentives based on team performance vs. individual performance, having incentives managed from the state or district level vs. the building level, or having incentives structured as more rapid advancement through a system of ranks vs. annual bonuses—can result in very good or very ineffective policy.[4]

Similar uncertainty surrounds other possible uses of value-added information. For example, tying tenure to value-added evaluation scores will have immediate effects on school performance that have been well modeled, but these models cannot predict indirect effects such as those that might result from changes in the profiles of people interested in entering the teaching profession. Such effects on the general equilibrium of the teacher labor market are largely the subject of hypothesis and speculation. Research on these and related topics is burgeoning,[5] but right now much more is unknown than known.

However, uncertainties surrounding how best to design human resource policies that take advantage of meaningful teacher evaluation do not bear directly on the question of whether value-added information should be included as a component of teacher evaluation. There is considerable confusion between issues surrounding the inclusion of value-added scores in teacher evaluation systems and questions about how such information is used for human resource decisions. This is probably because the uses of teacher evaluation that have gained the most public attention or notoriety have been based exclusively on value-added. For example, in August 2010, the Los Angeles Times used several years of math and English test data to identify publicly the best and the worst third- to fifth-grade teachers in the Los Angeles Unified School District. The ensuing controversy focused as much on value-added evaluation as the newspaper’s actions. But the question of whether these kinds of statistics should be published is separable from the question of whether such data should have a role in personnel decisions. It is routine for working professionals to receive consequential evaluations of their job performance, but that information is not typically broadcast to the public.

A place for value-added

Much of the controversy surrounding teacher performance measures that incorporate value-added information is based on fears about how the measures will be used. After all, once administrators have ready access to a quantitative performance measure, they can use it for sensitive human resources decisions including teacher pay, promotion, or layoffs. They may or may not do this wisely or well, and it is reasonable for those who will be affected to express concerns.

We believe that whenever human resource actions are based on evaluations of teachers they will benefit from incorporating all the best available information, which includes value-added measures. Not only do teachers typically receive scant feedback on their past performance in raising test scores, the information they usually receive on the average test scores or proficiency of their students can be misleading or demoralizing. High test scores or a high proficiency rate may be more informative of who their students are than how they were taught. Low test scores might mask the incredible progress the teachers made. Teachers and their mentors and principals stand to gain vast new insight if they could see the teachers’ performance placed in context of other teachers with students just like their own, drawn from a much larger population than a single school. This is the promise of value-added analysis. It is not a perfect system of measurement, but it can complement observational measures, parent feedback, and personal reflections on teaching far better than any available alternative. It can be used to help guide resources to where they are needed most, to identify teachers’ strengths and weaknesses, and to put a spotlight on the critical role of teachers in learning.

Full-throated debate about policies such as merit pay and “last in-first out” should continue, but we should not let controversy over the uses of teacher evaluation information stand in the way of developing and improving measures of teacher performance.

Some classification errors are worse than others

Recent reports by nationally visible education researchers and thinkers have urged restraint in the use of teacher evaluations based on student test scores for high stakes decisions. The common thread in these reports is the concern that value-added scores reported at the level of individual teachers frequently misclassify teachers in ways that are unfair to teachers, e.g., identifying a teacher as ineffective who is in fact average.[6]

There are three problems with these reports. First, they often set up an impossible test that is not the objective of any specific teacher evaluation system, such as using a single year of test score growth to produce a rank ordered list of teachers for a high stakes decision such as tenure. Any practical application of value-added measures should make use of confidence intervals in order to avoid false precision, and should include multiple years of value-added data in combination with other sources of information to increase reliability and validity. Second, they often ignore the fact that all decision-making systems have classification error. The goal is to minimize the most costly classification mistakes, not eliminate all of them. Third, they focus too much on one type of classification error, the type that negatively affects the interests of individual teachers.

Imagine the simplest classification system that could be fit on a continuous distribution of teachers’ value-added scores: A point on the distribution is selected as a cut point. Any teacher receiving a value-added score at or above that cut point is categorized as effective whereas any teacher with a score below that point is categorized as ineffective. Imagine further that value-added is measured with error, i.e., a teacher’s score does not capture perfectly the teacher’s true contribution to student learning. This error in measurement means that depending on where the cut point is placed, some truly effective teachers will be rated ineffective (they are false negatives) and some ineffective teachers will be rated effective (they are false positives). The other two classification outcomes are truly effective teachers so categorized (true positives), and truly ineffective teachers so categorized (true negatives).

To illustrate, the figure above represents the obtained evaluation scores of two categories of teachers: those who are truly effective (colored grey) and those who are truly ineffective (colored blue). The scores of the two groups of teachers are distributed normally around the mean for their group, with the spread of scores representing both true differences in teacher effectiveness and error in the measure used for evaluation. The cut point in the figure represents the point on the scale of teacher evaluation scores at which a manager chooses to treat the teachers differently in terms of a personnel action. Using tenure as an example, everyone who received an evaluation score at or above the cut point would receive tenure, whereas everyone scoring below the cut point would be dismissed or continue in a probationary status. In this instance, the majority of truly effective teachers received scores at or above the cut point – they are true positives – and a majority of truly ineffective teachers received scores below the cut point – they are true negatives. But there are also classification errors, i.e., truly effective teachers categorized as ineffective (false negatives) and truly ineffective teachers classified as effective (false positives).

The false positive rate and the false negative rate are inversely related and determined by where the cut point is placed on the distribution of scores. Thus, if the manager moved the cut point for granting tenure to the right in this figure, the false positive rate would go down whereas the false negative rate would go up. Likewise the true positive rate would go up and the true negative rate would go down.

Much of the concern and cautions about the use of value-added have focused on the frequency of occurrence of false negatives, i.e., effective teachers who are identified as ineffective. But framing the problem in terms of false negatives places the focus almost entirely on the interests of the individual who is being evaluated rather than the students who are being served. It is easy to identify with the good teacher who wants to avoid dismissal for being incorrectly labeled a bad teacher. From that individual’s perspective, no rate of misclassification is acceptable. However, an evaluation system that results in tenure and advancement for almost every teacher and thus has a very low rate of false negatives generates a high rate of false positives, i.e., teachers identified as effective who are not. These teachers drag down the performance of schools and do not serve students as well as more effective teachers.

In the simplest of scenarios involving tenure of novice teachers, it is in the best interest of students to raise the cut point thereby increasing the proportion of truly effective teachers staffing classrooms whereas it is in the best interest of novice teachers to lower the cut point thereby making it more likely that they will be granted tenure. Our message is that the interests of students and the interests of teachers in classification errors are not always congruent, and that a system that generates a fairly high rate of false negatives could still produce better outcomes for students by raising the overall quality of the teacher workforce.[7] A focus on the effects on teachers of misclassification should be balanced by a concern with the effects on students.

A performance measure needs to be good, not perfect

Discussions of teacher evaluation at the policy and technical levels often proceed in isolation from experience and evidence from other related fields. But we know a lot about performance evaluation in other labor markets, knowledge that should inform debates about value-added and teacher evaluation in general.

The correlation in test-based measures of teaching effectiveness between one school year and the next lies between .20 and .60 across multiple studies, with most estimates lying between .30 and .40.[8] A measure that has a correlation of .35 from one year to the next produces seemingly troubling statistics in line with our conceptual discussion of classification errors. For instance, only about a third of teachers ranked in the top quartile of value-added based on one academic year’s performance would appear in the top quartile again the next year. And ten percent of bottom quartile teachers one year would appear in the top quartile the next. Some of this instability is due to variation in teachers’ true performance from year to year and some of it is simply due to error in the measure.

It is instructive to look at other sectors of the economy as a gauge for judging the stability of value-added measures. The use of imprecise measures to make high stakes decisions that place societal or institutional interests above those of individuals is wide spread and accepted in fields outside of teaching.

The correlation of the college admission test scores of college applicants with measures of college success is modest (r = .35 for SAT combined verbal + math and freshman GPA[9]). Nevertheless nearly all selective colleges use SAT or ACT scores as a heavily weighted component of their admission decisions even though that produces substantial false negative rates (students who could have succeeded but are denied entry). Why would colleges use such a flawed selection instrument? Because even though the prediction of success from SAT/ACT scores is modest it is among the strongest available predictors. An entering class formed in part by the decision to admit those with higher SAT/ACT scores in preference to those with lower scores will perform better than a class formed without the use of that information.

In health care, patient volume and patient mortality rates for surgeons and hospitals are publicly reported on an annual basis by private organizations and federal agencies and have been formally approved as quality measures by national organizations.[10] Yet patient volume is only modestly correlated with patient outcomes, and the year-to-year correlations in patient mortality rates are well below 0.5 for most medical and surgical conditions. Nevertheless, these measures are used by patients and health care purchasers to select providers because they are able to predict larger differences across medical providers in patient outcomes than other available measures.[11]

In a similar vein, the volume of home sales for realtors; returns on investment funds; productivity of field-service personnel for utility companies; output of sewing machine operators; and baseball batting averages predict future performance only modestly. A meta-analysis[12] of 22 studies of objective performance measures found that the year-to-year correlations in high complexity jobs ranged from 0.33 to 0.40, consistent with value-added correlations for teachers.

Despite these modest predictive relationships, real estate firms rationally try to recruit last year’s volume leader from a competing firm; investors understandably prefer investment firms with above average returns in a previous year; and baseball batting averages in a given year have large effects on player contracts. The between-season correlation in batting averages for professional baseball players is .36.[13] Ask any manager of a baseball team whether he considers a player’s batting average from the previous year in decisions about the present year.

We should not set unrealistic expectations for the reliability or stability of value-added. Value-added evaluations are as reliable as those used for high stakes decisions in many other fields.

Ignoring value-added data doesn’t help

We know a good deal about how other means of classification of teachers perform versus value-added. Rather than asking value-added to measure up to an arbitrary standard of perfection, it would be productive to ask how it performs compared to classification based on other forms of available information of teachers.

The “compared to what” question has been addressed by a good deal of research on the other teacher credentials and characteristics that are presently used to determine employment eligibility and compensation. Here the research is quite clear: if student test achievement is the outcome,[14] value-added is superior to other existing methods of classifying teachers. Classification that relies on other measurable characteristics of teachers (e.g., scores on licensing tests, routes into teaching, nature of certification, National Board certification, teaching experience, quality of undergraduate institution, relevance of undergraduate coursework, extent and nature of professional development), considered singly or in aggregate, is not in the same league in terms of predicting future performance as evaluation based on value-added.

Consider a particular example that has arisen as a consequence of the deep recession: the need of districts to lay off teachers as a result of budget shortfalls. Managers in most industries would attempt to target layoffs so as to cause as little damage as possible to productivity — less productive workers would be dismissed or furloughed before more productive workers.

Suppose school district leaders were similarly motivated and had flexibility in deciding how to proceed. Imagine three possible approaches for deciding who should be dismissed. The first approach would employ the existing teacher evaluation system based on principal ratings, which identifies a few teachers as unsatisfactory but categorized the vast majority of teachers as satisfactory. The second approach would employ teacher experience, which has been found in a number of studies to have a statistically significant positive association with student achievement. The third approach would use teacher value-added scores to identify the lowest performing teachers.

Researchers have compared these three approaches using data from fourth and fifth grade public school teachers in New York City and simulating the elimination of enough teachers to reduce the budget by 5 percent.[15] A graph from that study, reproduced below, illustrates the results for student achievement if the positions of teachers with the lowest value-added scores were eliminated vs. the positions of teachers with the least experience. The horizontal axis is teacher effectiveness as indexed by student gains whereas the vertical axis is the number of teachers. Teacher effectiveness scores are those regularly calculated by the NYC public schools and could encompass teacher performance going back as far as four years.

Note that if teachers were laid off based on seniority they would be distributed across the full range of performance in terms of effectiveness in raising student test scores whereas teachers laid off based on low value-added scores would be at the bottom of the distribution. In other words, many more effective teachers would be retained were layoffs based on value-added than were they based on seniority. Principal ratings, not shown in the graph, perform better than teacher seniority in identifying teachers with low effectiveness in raising student achievement, but not nearly as well as value-added scores.

The question, then, is not whether evaluations of teacher effectiveness based on value-added are perfect or close to it: they are not. The question, instead, is whether and how the information from value-added compares with other sources of information available to schools when difficult and important personnel decisions must be made. For example, keeping ineffective teachers on the job while dismissing far better teachers is something most school leaders, parents, and the general public would want to avoid. Value-added is a better tool for that purpose than other measures such as teacher experience, certification status, seniority, and principal ratings, even though it is imperfect.[16]

Conclusion: Value-added has an important role to play

We have a lot to learn about how to improve the reliability of value-added and other sources of information on teacher effectiveness, as well as how to build useful personnel policies around such information. However, too much of the debate about value-added assessment of teacher effectiveness has proceeded without consideration of the alternatives and by conflating objectionable personnel policies with value-added information itself. When teacher evaluation that incorporates value-added data is compared against an abstract ideal, it can easily be found wanting in that it provides only a fuzzy signal. But when it is compared to performance assessment in other fields or to evaluations of teachers based on other sources of information, it looks respectable and appears to provide the best signal we’ve got.

Teachers differ dramatically in their performance, with large consequences for students. Staffing policies that ignore this lose one of the strongest levers for lifting the performance of schools and students. That is why there is great interest in establishing teacher evaluation systems that meaningfully differentiate performance.

Teaching is a complex task and value-added captures only a portion of the impact of differences in teacher effectiveness. Thus high stakes decisions based on value-added measures of teacher performance will be imperfect. We do not advocate using value-added measures alone when making decisions about hiring, firing, tenure, compensation, placement, or developing teachers, but surely value-added information ought to be in the mix given the empirical evidence that it predicts more about what students will learn from the teachers to which they are assigned than any other source of information.

[1] Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. New York, NY: The New Teacher Project.
[2] Gabriel, T. (2010, September 2). A celebratory road trip by education secretary, New York Times, p. A24.
[3] For instance, the Obama administration made state support of rigorous teacher evaluation systems a pre-condition for competition in Race to the Top, and has laid out a blueprint for the reauthorization of the Elementary and Secondary Education Act in which teacher effectiveness defined by evaluation of on-the-job performance is an important facet.
[4] Springer, M.G., Ballou, D., Hamilton, L., Le, V., Lockwood, J.R., McCaffrey, D., Pepper, M., & Stecher, B. (2010). Teacher pay for performance: Experimental evidence from the project on incentives in teaching. Nashville, TN: National Center on Performance Incentives at Vanderbilt University.
[5] Goldhaber, D. & Hannaway, J. (Eds.) (2009). Creating a new teaching profession. Washington, DC: The Urban Institute.
[6] For example, a policy brief from the Education Policy Institute on the problems with the use of student test scores to evaluate teachers, reports that value-added estimates “have proven to be unstable across statistical models, years, and classes that teachers teach.” The authors, buttress their recommendations not to use such scores with descriptions of research showing that “among teachers who were ranked in the top 20 percent of effectiveness in the first year, fewer than a third were in that top group the next year,” and that “effectiveness ratings in one year could only predict from 4 percent to 16 percent of the variation in such ratings in the following year.” And, a report from the National Academies of Science presents a range of views on the use of value-added but nevertheless concludes that “persistent concerns about precision and bias militate against employing value-added indicators as the principal basis for high-stakes decisions.” Likewise, reports from Rand , the Educational Testing Service , and IES remind us to be cautious about the degree of precision in estimates of teacher effectiveness derived from value-added measures.
[7] Of course, there are many tradeoffs that belie the simple calculus in our example. For instance, if an appreciable share of junior teachers were removed from the workforce in a particular district the pool of applicants might be too small to replace the dismissed teachers. From a district or student’s perspective it would be better to have lower quality teachers in the classroom than no teachers at all. Likewise, the calculus is not straightforward from a teacher’s perspective. For example an evaluation system that identifies nearly everyone as a winner and thereby avoids false negatives may lessen the opportunities for advancement of stronger teachers and reduce the public’s support for the teaching profession.
[8] Goldhaber, D. & Hansen, M. (2010). Is it just a bad class? Assessing the stability of measured teacher performance. CEDR Working Paper 2010-3. Seattle, WA: University of Washington.
[9] Camera, W.J. & Echternacht, G. (July 2000). The SAT I and high school grades: Utility in predicting success in college. New York, NY: The College Board.
[10] See http://www.leapfroggroup.org/ , http://www.hospitalcompare.hhs.gov/ , and http://www.qualityforum.org/Measures_List.aspx .
[11] For example, Dimick, J.B., Staiger, D.O., Basur, O., & Birkmeyer, J.D. (2009). Composite measures for predicting surgical mortality in the hospital. Health Affairs, 28(4), 1189-1198.
[12] Sturman, M.C., Cheramie, R.A., & and Cashen, L.H. (2005). The impact of job complexity and performance measurement on the temporal consistency, stability, and test-retest reliability of employee job performance ratings. Journal of Applied Psychology, 90, 269-283.
[13] Schall, T. & Smith, G. (2000). Do baseball players regress to the mean? The American Statistician, 54, 231-235.
[14] Although student scores on standardized achievement tests are obviously proxies for rather than the actual student outcomes that education is supposed to generate, it is important to remember that they are strong predictors of long term outcomes. For example, a large scale national study by the ACT found that eighth-grade achievement test scores were the best predictor of students’ level of college and career readiness at high school graduation —even more so than students’ family background, high school coursework, or high school grade point average.
[15] Boyd, D.J., Lankford, H., Loeb, S., & Wyckoff, J.H. (July, 2010). Teacher layoffs: An empirical illustration of seniority vs. measures of effectiveness. Brief 12. National Center for Evaluation of Longitudinal Data in Education Research. Washington, DC: The Urban Institute.
[16] Research related to this conclusion includes:
Goldhaber, D. D. & Hansen, M. (2009). Assessing the potential of using value-added estimates of teacher job performance for making tenure decisions. Working Paper 2009-2. Seattle, WA: Center on Reinventing Public Education.
Jacob, B. & Lefgren, L. (2008). Can principals identify effective teachers? Evidence on subjective performance evaluation in education. Journal of Labor Economics. 26(1), 101-36.
Kane, T. J., Rockoff, J.E., & Staiger, D.O. (2008). What does certification tell us about teacher effectiveness? Evidence from New York City. Economics of Education Review, 27(6), 615-31.

Downloads

Download the Full Paper

Authors

Steven Glazerman
Dan Goldhaber
Susanna Loeb
Stephen Raudenbush
Douglas Staiger
Grover J. "Russ" Whitehurst

Publication: The Brookings Brown Center Task Group on Teacher Quality

Image Source: Yellowdog Productions / Jane Sobel Klonsky