Evaluating Teaching: The Promise and the Pitfalls

The past few weeks have brought several more reminders of why it’s critical to develop fair and defensible ways to evaluate teachers—and the programs that prepare them. The first was a ruling by a California superior court judge in the Vergara case that the state’s teacher tenure system discriminates against minority and low-income students. (I spoke about the case on the Diane Rehm show, and may return to the topic in a future blog post.)  The second reminder came in the form of the latest release of so-called data on the quality of teacher preparation programs from the National Council on Teacher Quality—a project in which GW and many of the nation’s leading teacher preparation programs chose not to participate. (The flaws of the NCTQ data and rating methods are described in my blog post of June 18, 2013.) The third reminder concerns the Department of Education’s plan to press ahead with a system to rank teacher education programs in institutions of higher education, which has already engendered considerable debate.

Without question, teachers and teacher preparation programs must be held accountable for performance, just as we do for physicians, engineers, and other professions with life-changing impacts. Our society, which so deeply believes in the promise of education, deserves measurable evidence of how teachers and teacher educators are doing. Both the Vergara ruling and the NCTQ ratings highlight why it’s so important to find the right measures—to get the right data and get the data right.

This point was emphasized in Evaluating Teacher Preparation Programs, a 2013 report of the National Academy of Education for which I had the honor of being the lead author. I referenced this report in my thank-you letter to Secretary Duncan, who had so generously spent an afternoon with us on campus in March to encourage students to go into teaching. As I said in my note to the Secretary, the NAEd report argues for a balanced approach to evaluating teacher education programs that includes measures of inputs (such as faculty qualifications, quality of courses and student teaching, selectivity in admissions) and outcomes (such as job placement, retention rates, and teaching performance of program graduates as measured by the learning in their classrooms). The report suggests that designers of accountability systems for teacher preparation programs should consider the strengths and limitations of various measures, their intended and unintended incentives, and the anticipated benefits and potential risks to the teaching profession and to the education community generally.

One measure of performance has received considerable attention of late—the impact of individual teachers on the achievement of their students, often gauged through so-called “value-added” models, or VAMs. The research community has weighed in on the plusses and minuses of VAMs (here, here, and here). And this spring, Secretary Duncan announced plans to move ahead with new rules that would require states to develop rating systems for teacher preparation programs that emphasize outcome measures, including VAM measures of graduates’ impacts on student achievement. These ratings, along with other data, would be used to determine which programs are eligible for federal TEACH grants.

In preparing the NAEd teacher preparation report, our committee reviewed evidence on the effectiveness of VAMs in estimating teachers’ impact on student achievement and found it to be mixed. Questions remain about the validity of VAMs as indicators of program quality, and we therefore recommended against placing too much weight on them when making critical decisions about program accountability.

Recent research raises new questions about the validity of VAMs for decisions about teacher performance or the quality of the programs that prepared them. A study by Polikoff and Porter finds weak to nonexistent relationships between state-administered VAM measures and the content or quality of teachers’ instruction, leading the researchers to question the usefulness of VAM data in evaluating teacher performance. Condie, Lefgren, and Sims find that a large number of teachers are misranked by the typical VAM; they estimate that using VAMs in teacher retention policies will improve student outcomes, but not by as much as a policy of promoting teacher specialization across subjects. Bitler and colleagues used a VAM model to assess the effects of teachers on something they can’t change—students’ height. Though this may seem flippant, preliminary results presented at the AERA conference indicate that the magnitude of the teachers’ “effects” on height is nearly as large as their effect on math and reading achievement—which led the authors to question “the extent to which VAMs cleanly distinguish between effective and ineffective teachers.” 

So while the Administration is right to focus on strengthening the capacity of teacher preparation programs to supply high-quality teachers, the evidence continues to suggest caution about placing too much stock in VAMs within systems for evaluating these programs. But a missing piece in many of the critiques of VAMs is the “counterfactual”: how good or bad are other measures of teaching quality?  For example, conventional wisdom and the increasingly strident rhetoric of some educators who are wary of standardized testing tend to favor classroom observations. In fact, though, observational methods have their own potential pitfalls. As shown in research on workplace discrimination generally, extraneous factors and unconscious biases can undermine the reliability and validity of judgments about performance. In the specific case of classroom assessments of teaching, Pianta and colleagues caution that “to draw any conclusions from observational data, the instruments we are using must be subjected to extensive testing and evaluation” and note that “the research community is just beginning to subject classroom assessment tools to that type of useful scrutiny.”

A Brookings study released in May shows that compromises to the validity and reliability of classroom observations can stem from the distribution of students of varying academic proficiency. The evidence suggests that teachers in classrooms with relatively higher-performing students tend to fare better in observational assessments than teachers working with lower-performing students, independent of the effects the teachers actually have on the rate or magnitude of gains in their students’ learning. A legitimate question that arises from this research is whether an over-reliance on classroom observations might have the unintended effect of perpetuating disparities in the allocation of teaching resources and put unfair negative pressure on teachers who are working in the toughest environments. As Whitehurst et al. note, “We should not tolerate a system that makes it hard for a teacher who doesn’t have top students to get a top rating.” 

The bottom line, then, is that any measure of teaching quality is prone to be imperfect, and no single measure should ever be the sole basis for important decisions such as hiring, retention, promotion, or firing of teachers. The NAEd report reiterates that important principle—echoing decades of research and advocacy by the professional measurement and assessment community.

Accountability is a perfectly American concept—and what better time to remember that than on the eve of Independence Day!  But finding approaches to what might be called sensible accountability (as wisely suggested in the recent work of Greg Duncan and Richard Murnane) is one of our greatest contemporary challenges.

–MJF

July 3, 2014

Leave a comment

Filed under Uncategorized

Some Thoughts After Commencement

 

Friends and family have asked me how our commencement went.  My short answer (OK, maybe not so short…) is that it was a glorious day on the National Mall—sunny, temperate, filled with moments reaffirming the cultural, linguistic, racial, and religious mosaic of our education system generally and GW specifically.  Asked for details, I offer these vignettes:

  • In his opening prayer, Irfaan Nooruddin of the Islamic Ministry Services speaks of the promise of scholarly instruction in “realizing our potential for individual and communal transformation.”
  • Gabriel Felder, winner of this year’s student speaker competition and soon-to-be urban teacher with Teach for America, reminds his classmates to be grateful to “every mentor who pushed you to lead and not follow” and ends with a joyful “l’chaim and mazel tov!”
  • Maudine Cooper, an honorary doctorate recipient who graduated from Howard University when job opportunities for African Americans were scarce, describes how an early desire to help people in a small way grew into a career devoted to helping people in a big way.  “Don’t be afraid to do the right thing when it’s necessary—and sometimes when it’s not!” she advises the graduates.
  • Russ Ramsey, the first in his family to attend college, a class of ’81 GW graduate, and GW baseball hall-of-famer, says that three principles—“passion, purpose, and possibilities”—have guided him as he built a successful career in investment banking and gave back to his alma mater with fifteen years of service as a trustee and chairman of the board.
  • Commencement speaker José Andrés charms the crowd with reflections on his journey from Spanish immigrant in New York with a love of cooking and $50 in his pocket— just enough to get him half way to his destination before the cab driver dropped him off— to becoming a world-renowned chef.  The man responsible for introducing tapas to America and on a crusade to promote healthy eating advises the graduates that “If things don’t go as expected, make the unexpected work in your favor.  Change the name of the dish,” he tells his awe-struck audience.

Our GSEHD ceremonies, too, reaffirmed, through the words of our speakers and the diversity of our audience, the power of education to open a channel to the future, turn youthful enthusiasms into productive careers, and bring some unity to our increasingly fragmented society.  Carrie Morgridge, a remarkable innovator and philanthropist who brought us STEMosphere in March and returned to keynote our doctoral celebration, urged us to become “transformers—not reformers.”  Cora Marrett, deputy director of the National Science Foundation and our main speaker on Saturday morning, reminded us of the power of education in an era of increased inequality and encouraged us to become agents of change.  David Seuratt, our student speaker, inspired his peers with reflections on his own journey and, quoting Robert F. Kennedy, reminded us that “Each time a man stands up for an ideal, or acts to improve the lot of others, or strikes out against injustice, he sends forth a tiny ripple of hope, and those ripples build a current which can sweep down the mightiest walls of oppression and resistance.”  Sage advice from accomplished and dedicated GSEHD friends!

What next?  Teachers, school leaders, and other education professionals, including our graduates, shoulder much of the responsibility for turning the words of our speakers into a reality of educational progress.  Based on my interactions with our new GSEHD degree recipients, I’m confident that they’re up to the challenges, and optimistic about the future they will enable.

But a dean’s pride in his graduates isn’t enough.  As I said during our ceremonies, education is the most important investment a society makes.  And with so much at stake, it’s essential that we invest wisely, that we deploy our best scientific and cultural resources to the challenges we face.  It is simply not enough to want things to improve—we need to apply the best available evidence in the pursuit of innovations that are most likely to succeed, and we must be willing and able to monitor performance as we go along. 

This is what GSEHD is all about:  connecting rigorous research to the improvement of policy and practice.  I believe the students who received their diplomas last weekend are ready.  They represent the very best hopes for the continued application of sound research to the analysis and resolution of our biggest educational challenges. 

Congratulations to our new alumni and happy summer to all!

 

–MJF

May 27, 2014

1 Comment

Filed under Uncategorized

PISA: The Morning After

Yesterday was PISA Day, an opportunity for concerned educators and citizens to think about the latest round of results from this important international comparative assessment

Not surprisingly, at least to those of us who follow the rhetoric and reality of comparative data, U.S. performance was basically unchanged from three years ago; some other countries (e.g., Poland, Germany) improved; some of the traditional “stars” (e.g., Finland) experienced a decline; and policy makers and commentators were quick to pronounce on the meaning of the results.

PISA is a remarkable program, in terms of the breadth of its coverage (65 education systems, including selected states in the US and Shanghai as separate from China) and the care taken to provide reliable estimates of the math, reading, and scientific literacy of samples of 15-year olds.  We have come a long way since the early days of international comparative assessment, in terms of sampling methods, psychometric quality, and reporting of results. 

Interpretation, though, remains a challenge.  For descriptive purposes, PISA provides a trove of interesting information, which, along with TIMSS, NAEP, and PIAAC (another OECD project), should be studied by anyone who cares about the ongoing pursuit of improved educational opportunity locally, nationally, and globally.  The more complicated task, though, is deriving sound policy inferences from these descriptive data.  There is no clear enough pattern of relationships to infer anything definitive about the relative success of various reforms in the U.S. and elsewhere; about the relationship of test performance to national economic outcomes; or about what exactly we should do next as we struggle to expand access and high quality educational opportunities for our students. 

For example, it seems that Massachusetts again did better than the overall U.S. average and on par with some of the biggest international “winners.”  Florida fared more poorly.  So people who like what Massachusetts has been doing must be pleased, and would be inclined therefore to like what PISA measures.  People who like Florida’s hard-charging accountability reforms are surely disappointed, and some of them must now be skeptical about whether PISA is the right tool to gauge the effects of reform.  It is no small irony that some of the harshest critics of PISA (and testing generally) are willing to use the latest results to vindicate their claims about the success or failure of various reform initiatives. 

Similarly, there is the implicit (and in some cases explicit) attempt to tie PISA scores to our current or future economic stature.  Here, too, the hazards of inferring cause and making predictions from purely correlational and descriptive data are profound.  As I’ve written elsewhere, the U.S. performed near or at the bottom on the First International Mathematics assessment in 1964, and indeed our economic productivity growth declined in the subsequent decade (when many of those high school kids who had taken the test were in the labor market).  But countries that significantly outperformed us on the math test, e.g., Britain and Japan, experienced an even more dramatic productivity growth slowdown, suggesting that those test results could not, alone, explain much about the economy or predict much about the future.  Average annual labor productivity increased by about 4.2% in the U.S., between 1979 and 2011, compared to 2.5% in Germany, which outperforms us on PISA.  Again, it’s not easy to infer simple relations from these data.  Does academic achievement matter to economic outcomes?  Yes, without a doubt.  But so much else matters, and probably more, that to draw quick inferences – and again sound loud alarm bells about our impending economic doom and, worse yet, blame everything on schools and teachers – from results on one assessment of one age group is a recipe for the ultimate erosion of respect for what is otherwise a useful tool for comparative study.  I’ve tried to make this argument elsewhere.

Finally, there is the question about what to do next.  On this, I was inspired by the wise comment of John Jackson, of the Schott Foundation for Public Education, whom I met at a dinner with a number of education policy makers and researchers.  John asked whether it was possible that we in the U.S. had essentially “maxed out” on the impact of reform in terms of its effects on PISA scores, and if so what should guide us as we continue to work on educational improvement.  That’s the right question, and though I believe there is useful information in PISA I also believe that answering it will require considerably more nuance in our understanding of the results.  (To get a sense of the complexity of these issues, see the recent work of Martin Carnoy and Richard Rothstein). 

For me, the most important issue to focus on is not where we stand on average, but rather how to cope with the ravaging effects of growing economic inequality on educational opportunity and the life chances of our youth.  In other words, we need to work on the variance more than the mean, to acknowledge the effects of poverty, and to concentrate on policies and programs that can restore opportunity (my colleagues Richard Murnane and Greg Duncan are co-authors of a book with that title, due out early next year).  A good place to start would be with a sustained program of research and policy that builds on the foundations of work such as Whither Opportunity?.  If we care about the American dream – and really want to reaffirm the nation’s commitment to upward mobility and improved quality of life for all our people – we should not distract ourselves with foolish attempts to use a single assessment as the guide to policy and practice.  PISA poses important questions; the answers aren’t as obvious. 


MJF
December 4, 2013

Leave a comment

Filed under Uncategorized

Remembering JFK

On the bulletin board above my desk at home I have a faded photograph taken in 1960.  It’s me on my dad’s shoulders, welcoming John F. Kennedy to a campaign stop in Queens, holding a sign that says “Kennedy is the remedy.”  My dad’s lovely wit, an abridged version of the rush of emotion so many people were feeling about the young Kennedy.  Sure, he made mistakes, and in retrospect did things that today would likely have gotten him in big trouble.  But there’s no doubt he was an amazing, maybe unique, figure:  an optimist’s optimist, a patriot’s patriot, an American hero, the stuff of myth and dreams and desires.  That he listened to “Camelot” in the Oval Office says a lot about what was going on in this town in those early days of a decade whose massive social changes are now remembered simply as “the 60’s.”   What changes?  Oh, little things like the Civil Rights Act, the Voting Rights Act, the Elementary And Secondary Education Act, the beginning of desegregation (in the north as well as the south), not to mention the Beatles and Joan Baez and Simon and Garfunkel and…I could go on.

Our colleague who directs the GW Graduate School of Political Management, Mark Kennedy (no relation as far as I know), an accomplished politician in his own right and a distinguished teacher and statesman, wrote an extremely moving reminiscence of JFK and the assassination. It brought back all kinds of memories (I was in 6th grade). 

And I couldn’t agree more with Mark’s assessment:  if I may paraphrase, we need to hit the pause button in these frenzied times and figure out how to restore some respect for government and some civility of discourse even (especially) among people who disagree about the details but who share a basic love of what our country was and can still be.  Maybe that’s the best way to remember JFK.


MJF
November 21, 2013

1 Comment

Filed under Uncategorized

Congratulations to NCEHD Member Dr. Ericka Miller on her nomination to the White House Administration

President Obama has nominated our colleague and friend, Ericka Miller, to be the next Assistant Secretary for Postsecondary Education.  Ericka has been a member of our National Council of Education and Human Development (NCEHD), and has given us invaluable advice as we have worked on our vision and strategy for the future.  I applaud the President for this superb choice.  Ericka has an abiding commitment to the improvement of education and educational opportunity, and if confirmed will bring to this post a remarkable blend of content knowledge and experiential wisdom.  We are proud of Ericka and very happy for the world of education!


MJF
November 8, 2013

Leave a comment

Filed under Uncategorized

Reflecting on a New Challenge

I was honored by my colleagues to be inducted as the President of the National Academy of Education at a ceremony last Friday.  My wonderful family was at my side as I accepted this honor – and though I don’t get to tell them often enough, they deserve great thanks for their abiding support.

It will be a great pleasure to work with my GW colleagues, NAEd members and leaders, and other friends in the broad education research and policy community as we continue to advance the cause of improved education and increased opportunity. A challenge, no doubt, but one that I feel privileged to take on.

President Knapp (and Diane Knapp), as well as Provost Lerman (and Lori Lerman), were kind enough to attend the ceremony, as were a number of our senior leadership faculty and staff.  I’m grateful for President Knapp’s most generous introduction and welcome.  GW Trustee Titulola Williams-Davies and former trustee Bob Perry also joined us, and it was again a joy to see their enthusiasm for what’s happening in GSEHD.   And it was a special treat to have Undersecretary of Education Martha Kanter in our company.  Let me add my very special thanks to members of the GSEHD National Council who were present:  Laura Taddeucci Downs, Ericka Miller, Elizabeth Lodal, Kathy Manatt, Dorothy Moore, and Ed Vest continue to grace us with their wisdom and support.  My heartfelt thanks to all.

Following a memorable and moving introduction by my dear friend and mentor, Carl Kaestle (whom many of you will remember of course from his masterful “master class” earlier this year), I offered some reflections about the Academy, the world of education research and policy, and my hopes for the future.  As I contemplate this new role, I am glad to share a slightly edited version of my remarks from Friday’s induction ceremony.


MJF
October 29, 2013

[Abridged remarks from NAEd ceremony on October 25, 2013]
I have been thinking about the name of our organization.  Although I was an English major I am not really a big fan of deconstruction; but it is interesting to think about the words in our name, especially in the current policy environment.

The word “national” evokes, for some people, and maybe for more people today than in previous times, an anxiety about the excesses of federal involvement in those sacred rights of states and localities and individuals to keep doing their own thing.  It’s gotten to the point where one wonders, listening to the shrill rhetoric, if anyone was in class on the day when the principles of individual choice and social well-being were taught.  What can we do to remind ourselves that education is essentially a “public good?”

Continue reading

Leave a comment

Filed under Uncategorized

Bringing Teacher Rights Into Question

With increasing evidence about the importance of teacher quality as a correlate of student learning, it is not surprising that there is concern about the ways teachers are evaluated. Some researchers and policy analysts argue that teacher-evaluation systems across the country are inconsistent and often poorly conceived and implemented. Perhaps most significantly, there is seemingly little awareness surrounding the consequences of an evaluation process that holds the potential to unfairly dismiss a qualified educator. What rights — if any — do teachers have to stand up to unsound teacher evaluation systems?

In collaboration with Dr. W. James Popham (professor emeritus at the University of Calfornia, Los Angeles) GSEHD’s Dr. Marguerita K. Desander offers an enlightening examination of this issue in the Education Week commentary “Unfairly Fired Teachers Deserve Court Protection.” I encourage you to read the full commentary here.

For related reading, a recent series of reports by GSEHD’s Center on Education Policy offers detailed information regarding states’ efforts to implement new Common Core standards, as well as states’ efforts to prepare teachers and principals for Common Core standards.

Let me also note that on October 25 the National Academy of Education will release a report on the current and future of teacher preparation program evaluation. Watch this space for more details about a public seminar, which we will host here at GW.

Welcome to academic year 2013-14!

MJF
September 24, 2013

Leave a comment

Filed under Uncategorized