Friday, July 30, 2010

Study: Error rates high when student test scores used to evaluate teachers

Study: Error rates high when student test scores used to evaluate teachers

Valeria Strauss - Washington Post

I don’t actually understand all of a new statistical study about error rates when “value-added” student test scores are used to evaluate teachers, but I do get this: The rates high enough to give even supporters of such measures some pause about using them for high-stakes decisions.

There’s a more in-depth analysis of the report, which was undertaken for the Education Department’s Institute of Education Sciences, on Bruce D. Baker’s School Finance 101 blog.

But here’s my takeaway from the report, entitled "Error Rates in Measuring Teacher and School Performance Based on Student Test Score Gains:"

Value-added measures have become all the rage in evaluating teachers. What does that mean? As explained in a guest blog this year by by FairTest's Lisa Guisbond, these measures use student standardized test scores to track the growth of individual students as they progress through the grades and see how much “value” a teacher has added.

An emerging body of research has found that these value-added estimates based on a few years of data can be imprecise. How imprecise?

According to thenew report by Mathematica Policy Research:

If three years of data is used there is about a 25 percent change that a teacher who is “average” would be identified as significantly worse than average, and, under new evaluation systems, perhaps fired.

*If one year of data is used, there is a 35 percent chance of the same misidentification.

Considering that teachers are now being fired based partly on test scores -- D.C. Schools Chancellor Michelle Rhee just let go dozens of teachers based on such evaluations -- this error rate matters in a big way.

The report, written by Mathematica’s Peter Z. Schochet and Hanley S. Chiang, goes on to say that value-added estimates “in a given year are still fairly strong predictors of subsequent-year academic outcomes in the teachers’ classes.”

I wonder if the authors would like their evaluations to be based on such “fairly strong” criteria.

By the way, Baker’s analysis of the report mentions other major issues that he says undermine "the usefulness of value-added assessment for teacher evaluation and dismissal (on the assumption that majority weight is placed on value-added assessment)." According to Baker, they include:

*That students are not randomly assigned across teachers and that this non-random assignment may severely bias estimates of teacher quality.

*That only a fraction of teachers can even be evaluated this way, generally less than 20%.

Wednesday, July 28, 2010

Who's a Good Teacher? Walt Gardner

Who's a Good Teacher?
By Walt Gardner on July 28, 2010 8:10 AM

When Chancellor Michelle Rhee fired 241 teachers in Washington D.C. on July 23, the news was heralded as evidence that true accountability was finally a reality because the evaluation system used is considered one of the most rigorous in the nation. But like most controversial issues in education, there's more to the story than initially meets the eye.

The firings included 165 teachers for poor performance and the rest for lack of proper teaching credentials. These constituted 6 percent of the district's 4,300 teachers. Rhee put an additional 737 on notice that if they don't improve next year, they too could lose their jobs. George Parker, the head of the Washington Teacher's Union, said he will contest the firings because the evaluation system used was flawed.

To try to determine who's right, it's necessary to take a closer look at the actual way D.C. teachers are evaluated. They are observed five times a year by administrators and master teachers on the basis of coherent lesson plans and student engagement. After the first observation, teachers receive a detailed plan spelling out their weaknesses and are offered coaching to help them improve. Fifty percent of the evaluation of math and reading teachers in 4th through 8th grades is based on their students' growth on standardized achievement tests. (High school teachers will be included in the future.) Teachers are then placed into four categories.

Rhee acknowledged that she didn't know how many teachers were fired for low student achievement on standardized tests, and how many were dismissed for poor classroom performance. This is a crucial distinction. Despite what is widely believed, these are not necessarily interchangeable criteria. It is altogether possible for teachers to violate every principle of effective instruction while being observed and yet still post satisfactory - if not remarkable - results on the same tests. There is something about their personality and style that is likely responsible. The reverse is true as well. In Testing! Testing! (Allyn and Bacon, 2000). W. James Popham explains why it's risky to draw conclusions about effectiveness merely by observing teachers in action. He calls the research underlying the principles of effective instruction "tendency research" because teachers who follow them tend to be effective. But the results are far from conclusive.

Then there is the question of using growth - rather than proficiency - on standardized tests as a major determinant. In theory, it makes eminent sense. If teachers happen to inherit a class of students with low skills and are able to show progress in their learning, these teachers deserve credit, even though their students may still not have reached proficiency. By the same token, if teachers happen to inherit a class of Talmudic scholars, they don't deserve credit for how their students subsequently perform on the same tests unless the students have shown improvement. In other words, these teachers can't coast.

In practice, however, the growth model, which is similar to the value-added model, does not adequately control for factors outside the classroom. For example, teachers are not responsible for the influences, either positive or negative, that moving to a new neighborhood exert. Nor are teachers responsible for changes within the family in the form of divorce, unemployment, and death of a parent. These are not theoretical factors. They affect student learning in the classroom in ways that non-educators do not understand.

Finally, there is the element of non-cognitive outcomes that are not being evaluated. If teachers know that half their evaluation is based on standardized test scores, they will feel pressure to turn their classrooms into test preparation factories. This strategy will no doubt boost test scores, but it is not likely to develop a love of the subject. That's because it's possible to teach a subject well (high test scores), but teach students to hate the subject in the process (low affective scores). When that happens, are teachers to be lauded? If so, why? Isn't one of the goals of education to make students lifelong learners?

None of the above is meant to suggest that the teachers who were fired deserve to remain in the classroom. Instead, it is intended to emphasize that evaluating teachers is more complex than is appreciated. That's why I've long believed hindsight is the fairest way of evaluating teachers. So often, the influence of teachers doesn't show up until years after students graduate. With the passage of time and the insights of maturity, students are in a far better position to evaluate their teachers. For obvious reasons, the accountability movement doesn't allow this luxury. Nevertheless, it's something to think about.

Tuesday, July 27, 2010

The Audacity of Arne Duncan

By Jim Horn

Posted: 2010-07-25

On 7/24/10, this was posted on EPATA, a discussion list of people who care about education.

Arne Duncan stood before the NAACP convention last week to repeat his claim that "education is the civil rights issue of our generation." He also declared "the only way to equality in society is to achieve equality in the classroom." Since Mr. Duncan did not spell out what he meant by equality or civil rights, let's see if we can extrapolate his meaning from the policies he is pushing hard to be adopted across America, even if his heavy-handed forcing means ignoring the lawful Congressional role in making federal education policy.

Apparently, Mr. Duncan does not believe that the equality shortage in classrooms that we have known about ever since poor children started going to school can be helped by fair housing policies, better transportation policies, improved health policies, or new jobs policies, any of which we know could affect the poverty levels of urban and rural America, where rates are now the highest, after taxes, of any industrialized nation. According to Mr. Duncan, it would seem that policy shifts or new efforts in these areas are unimportant, for it is "only" in the classroom that we may hope to achieve equality.

Well, what kind of equality in the classroom would that be? Apparently, it is first and foremost a segregated kind of equality, a segregation that is aided by the spread of charter schools, which remains a top priority of the Administration. Two studies last year, in fact, showed incontrovertible evidence for the segregative effects of charter schools, whether run by non-profit or for-profit corporations.

So by ignoring segregation within charters, we must assume that the kind of equality that Mr. Duncan is talking about does not depend upon the sharing of social and cultural capital that occurs when socioeconomic classes are educated together, and it is the kind of equality that apparently pays no attention to the facility and funding advantages accrued when middle class parents lend their voices to decisions within the schooling community.

Secondly, it has become a harsh, punishing kind of equality centered on remediation, ever since the "let a thousand flowers bloom approach to charters" has been replaced by an urgency to ramp up and bring to scale the "no excuses" KIPP schools and the KIPP behave-alikes. In these "no excuses" schools, equality demands total compliance by children who go to school nine or more hours a day and then have 2 to 3 hours of homework each night. Plus Saturdays and part of the summer. In order to be equal in these school and, thus, make up for the poverty that puts these students behind, they must be willing to give up their childhoods, family, and friends for a chance at passing the necessary tests that may or may not prepare them for college some day. For even though the "no excuses" chain gangs remain the dominant model for corporate education reform, we know very little about how these children will fare in independent learning environments after years of total compliance and behavioral/psychological modification.

Thirdly, it is the kind of equality that denies the importance of the other massive inequalities within the communities where these poor children live. It is the kind of equality that does nothing to aid the child who must dodge bullets on the way home from a 9-hour school day, or who must return home to find nothing to eat. It is the kind of equality that refuses to enroll a child whose parents are not willing to sacrifice their child to a schooling regimen that parents of the leafy suburbs would consider abusive to children if it were their own being subjected to it.

Fourthly, it is the kind of equality that depends upon assessments that put poor children at a great disadvantage all along the line, for there is no standardized test used in schools today, whether in third or thirteenth grade, that does not demonstrate, on average, a direct correlation between family income and testing outcomes.

In short, it is the kind of equality that depends upon a race that has many starting lines but only one finish line, a race wherein the hordes of losers claim their place among those who deserve to be the "unequal," children who will be dropped out, pushed out, and eventually forgotten behind the walls of the workhouses and correctional facilities that mark the destination in the school to prison pipeline.

If Arne Duncan's views on equality are evidenced in his actions, it leaves us with a troubling realization. For to understand that for Mr. Duncan to be right in saying that "education is the civil rights issue of this generation," we must stand shamefaced in admitting that civil rights now demands from equality what we previously could expect only from oppression.

Monday, July 26, 2010

Killing Kindergarten

"It’s critical that children arrive in kindergarten with the cognitive, emotional, and social skills." Elanna S. Yalow

Please Ms. Yalow, tell me more. I thought kindergarten was supposed to help children develop cognitive, emotional, and social skills. When did we shift the responsibilities? It's this shift that is responsible for the over institutionalizing of children. Kindergarten is not supposed to be a rigorous academic experience. Go back to playing kitchen.

Friday, July 23, 2010

Hard Data Won't Change Educational Beliefs by Walt Gardner

The debate over how to improve educational quality for all students in this country is predicated on the assumption that empiricism rather than ideology will eventually prevail. But a recent op-ed in the Boston Globe by Joe Keohane calls that belief into question ("How facts backfire," July 11). "Facts don't necessarily have the power to change our mind," he wrote. "In fact, quite the opposite." Keohane goes on to cite a series of studies in 2005 and 2006 by researchers at the University of Michigan showing that facts can actually make misinformation stronger.

The reasons for this counterintuitive finding range from simple defensiveness to avoidance of cognitive dissonance. But whatever the cause, they have direct relevance to efforts now underway to turn around failing schools. The best example is the campaign being waged by the 10-year-old Bill & Melinda Gates Foundation. The latest issue of Bloomberg Businessweek has a cover story detailing the thinking behind the foundation's distribution of hundreds of millions of dollars toward this goal since its inception and for its plans to spend $3 billion more in the next five to seven years on educational reform ("Bill Gates' School Crusade," July 15).

What emerges from the reportage is that Bill Gates does not like to be confused by evidence. In 2000, for example, he doled out hundreds of millions to make high schools smaller in the stubborn belief that student body size is crucial to student achievement. Gates subsequently discovered to his chagrin that students at small high schools, for example, were more likely to graduate than their peers at big high schools, but they did no better on standardized tests. Never one to let the facts get in the way of his personal convictions, he is now betting that teacher quality is the solution. The foundation is investing $290 million over the next seven years in the Tampa, Memphis and Pittsburgh school districts in the mistaken belief that measuring student gains on standardized tests is the key to educational quality.

It apparently makes no difference to Gates that this strategy is not nearly as straightforward as he thinks. Cheating by educators and narrowing of the curriculum, for example, have already been well documented in connection with high-stakes tests. Nevertheless, Gates wants to replace underperforming teachers (based on progress on student test scores) with effective teachers (based on the same metric). He is convinced that teachers ranked in the top 25 percent for four consecutive years will be enough to close the black-white achievement gap. He offers no credible evidence to support his assertion, but that doesn't undermine his influence.

That's because big money has a way of making itself heard over hard data. In today's recession, school districts are so desperate to avoid layoffs and make other cuts to their programs that when the Gates Foundation or other financial powerhouses come calling it's hard to resist doing their bidding. Adding to their enormous clout are their friends in the Obama administration. Together, they are a formidable team.

Whether they will ever be open to other views about turning around failing schools is doubtful. Ideology is notoriously resistant to alteration even in the face of overwhelming evidence. This observation is true whether it applies to the richest man in America, the man in the White House or the man in the street.

Thursday, July 22, 2010

New "Common State Standards"

From Comments section of

Frazer112358 wrote:
Standards, standards, standards. Hogwash. We are focusing on the WRONG thing. Build rigorous standards and students will come. What deception is being foisted on the educational world!! The research is clear: a competent teacher makes the difference. Building a new baseball park is not going to make your players better. High quality coaching in a decrepit ballpark will. Let's continue hiding behind the smoke and mirrors and pretend that NOW that we have new "Standards" everything will be as it should be in la-la land. I'm disgusted with the continued focus on content standards. I do think we need them, and they are a step in the direction we want to take but they are not a Magic Bullet that will kill the ills that plague us.

Wednesday, July 21, 2010

The Tantalizing Vagueness of Teaching by Susan Ohanian

Posted: 2010-07-11

This essay, which speaks to today's issues of teacher excellence, was published in the July/August issue of Learning Magazine in 1986. So oldtimers say, "Yes we've been through this before," using the "been there" putdown as an excuse for their current silence. Read on: you'll see I complained about them then, just as I complain about them now.

These silent ones--who have nothing to lose but their consultant money--refuse to acknowledge that teacher professionalism is under greater peril than ever before.

I condemn their silence.

I was once enrolled in a whoop-de-doop NYU program funded by the National Endowment for the Humanities, wherein participants were tested for a tolerance for ambiguity. My score went off the chart, which means that although I can't remember how to multiply fractions, I'm very likely to be able to "out wait" the most recalcitrant of students. For thirty years I've wondered why this "skill" never comes up in discussions of teacher excellence.

NOTE: The one thing that's way out of whack since this article was written is the pay of sports figures. The minimum wage for a major league baseball player, for example, is $400,000. With one hundred million in the news for a basketball celebrity, there seems to be no maximum.

from Learning Magazine

In 1916, Robert Frost wrote his friend and fellow poet Louis Untermeyer that a poem "begins as a lump in the throat." Frost also noted that a poem is "at its best when it is a tantalizing vagueness." I feel the same way about teaching.

That's why I become uncomfortable and even irate when committee-persons insist that a good teacher's performance can be charted and graphed, and then rewarded accordingly.

If someone appears in my doorway and says, "I'm going to examine your anticipatory sets," I wonder if I have the right to make one phone call first. Teachers have been polite too long to managerial types, fellow with bulging briefcases of checklists with terms like praise as positive reinforcer, demonstration of mastery, time on task, and other slimy slugs of that ilk.

We are teachers. Teachers. Teachers. We traffic in words and ideas and feelings and hopes, not in goods or systems, not even in five-tiered career ladders. I hear the blizzard of words spewed forth about teacher competency and I want to ask, "Why me?"

Do we see such high-level scrutiny of other professions--of doctors, for example? After all, their collegial cover-ups are notorious--and life-threatening. Yet I can't pick up my morning newspaper and find out what my surgeon's operating mortality rate is. Nor do I know what percentage of his diagnoses are correct. Would the AMA sponsor the idea of box scores for doctors' performances--or merit pay?

But if teachers can prove they are competent, say our education managers, society will pay them accordingly. Hah. American values simply aren't skewed that way. Sports figures might easily get $500,000--or $5,000,000--to hit around a little white ball. But teachers can't command much--mainly because we don't have any special skills that are observable. What we do looks fairly easy; most people feel they could do the teaching part, though they acknowledge that putting up with the kids all day might be a bit difficult. Our real skills, of course, are secret. Nobody ever knows when we hit a home run or a grand slam, and that's why we can never be paid what we're worth.

It doesn't take the perception of a parsnip to realize that teachers didn't enter the calling to get rich. Yet these education managers persist in ignoring--and even eliminating--the very qualities that did lead us into the fold. They talk a lot about skills, for example, but don't mention a sense of humor or a tolerance for ambiguity or an enjoyment of children.

Behaviorists have long insisted that they can deliver the carefully delineated subskills of learning. Now they are marketing a similar package for teaching. Some administrators label this move to standardize teaching as a clarion call for excellence. A lot of us veteran teachers see it as an ultimately catastrophic worship of systems at the expense of people.

I am particularly bothered by the growing popularity of teacher evaluation forms that are supposed to be objective, systematic, impersonal. Such forms tell no more about essential teacher qualities than do a box of jujubes. The time is past due for teachers to stand up and say, "No! Your checklists and timetables tun counter to what goes on in a vital, stimulating, nurturing classroom." The time is past due for the professors of education to step down from their ivory towers and to become involved in what's happening in the schools. If they form one more committee or draft one more recommendation, let it be on the importance of human relations in the classroom. Let them form a task force to protect teachers. . . and children.

The education managers who hand out competency tests and who write up official classroom observations make a critical mistake. They insist that prospective teachers should prove what they know. But we veteran teachers realize that the hard part of being a teacher has nothing to do with facts. Yes, teachers need to know where the apostrophes should land, but more important, they need to be nurturing human beings. They must be optimistic and enthusiastic about the possibilities of the children in their care. They must be flexible and able to bounce back after sixty-three defeats--ready and even eager to try again.

I'm not much interested in seeing how a teacher carefully structures her lesson so that the kids stick to the objectives and the bell always rings in the right place--just after she makes her summary and gives the prelude for what will come tomorrow. I want to find out if that teacher is tough and loving and clever and flexible. I want to be sure she's more nurturing than a halibut.... What does she do when a kid vomits (all over those neat lesson plans)? Or an indignant parent rushes in denouncing the homework? Or the worst troublemaker breaks his arm and needs special help? Or the movie projector bulb burns out, and the replacements have to come from Taiwan? Or somebody spots a cockroach under her desk?

A teacher's talents for dealing with crises aren't easily revealed on an evaluation report or rewarded on a salary schedule. And neither are those special moments that a teacher savors. So don't yield to the number crunchers--even when they dangle a golden carrot in front of you. Remember that the most wonderful joys of teaching happen in the blink of an eye and are often unplanned and unexpected. You can miss their importance and lose their sustenance if your eyes are glassily fixed on the objective you promised your principal you'd deliver that day. When you maintain a sharp eye and the ability to jump off the assigned task, the rewards are many--when a child discovers a well-turned phrase; or a mother phones and says, "Our whole family enjoyed the homework. Please send more"; or the shiest child in the room announces she wants to be the narrator in the class play; or the class bully smiles quietly over a poem. Our joy is in the daily practice of our craft, not in the year-end test scores or the paycheck. When outside experts ignore this, then we must stop and remind ourselves. We must talk, not of time on task but of the tantalizing vagueness and the lumps in the throat, the poetry and true purpose of our calling.

Friday, July 16, 2010

Anthony Cody on the Enemy

Certainly teachers can be part of the solution, but not so long as we, and the organizations that represent us, have been defined as the problem. It has taken me a long time to get to the point where I would write something as strong as this, because I have been trying very hard to connect and dialogue with this administration. But as I try to explain, we cannot connect when we have been defined as the source of the problem.

I do not think shameful mismanagement of urban districts is an adequate explanation for the systematic inequities we see in student outcomes. Issues like working conditions and the ability to attract and retain teachers in high needs schools are directly related to poverty and funding inequities. So we are back to this as a critical issue that must be faced.

When the "big bold idea" of the administration is to take funds that have historically been provided to correct inequity, and turn them into competitive grants, where we will have winners and losers, something is very wrong.

You may see this as polarizing, but when you are in a fight, it becomes necessary to frame the issues or they are framed for you. Teachers have been framed by NCLB since 2001, when GW Bush indicted us for the crime of the achievement gap. Duncan and Obama are continuing the prosecution, as we see with the continued demand to close schools and fire teachers.

Teachers can be part of the solution, but I refuse to continue to allow others to define the problems for us.
by: AnthonyCody | July 16, 2010 12:07 AM

Thursday, July 1, 2010

Listen to Me.

Tuesdays at 11:00 am you can hear me try to converse with people that want to do away with public education.

Too Much Money

Teachers make too much money. True or False?

False. The average salary of a starting teacher in this area is around $31,000.00 a year. The average babysitter in this area makes five dollars an hour per child. If teachers made babysitter wages they would make $135, 000.00 a year. Do the math. The average classroom has 25 students. The school day is about six hours long. The school year is 180 days a year. At baby sitter wages a teacher would make $30.00 a day per child. Multiply that by 25 students and a teacher making babysitter wages would make $750.00 a day. Multiply that by 180 days and a teacher making babysitter wages would make $135, 000.00. Keep this in mind the next time you feel compelled to complain about teacher salaries. $31,000.00 a year is a bargain compared to paying a babysitter.

Merit for Teachers

Merit pay for teachers based on student achievement is a ridiculous idea unless teachers can be guaranteed that all students are heterogeneously grouped and randomly assigned to classrooms. If this doesn’t happen then teachers and schools that are already meeting achievement goals based on an a geographic economic distribution model (property taxes) will receive all the “merit” pay. The schools that have the most to lose will eventually lose more and continue to be seen as failing. Moral question. Why do we continue to punish teachers and students that operate at the fringes of society? Giving more to those that already have more is morally wrong—merit pay is morally wrong.