Tag Archives: Value-Added Modeling

“All Politics is Local,” The Saga of the Annual Professional Performance Review (APPR), Assessing Teacher Performance and the Underside of Law-Making in Albany

Mike Schmoker, the author of Focus, wrote a prescient article in Education Week, “Why I’m Against Innovation in Education” at the same time that Bill Gates and Mark Zuckerberg announced yet another richly funded innovation,  “The State of the Art Ideas for Schools,” Schmoker writes,

I’m against innovation in education—as currently conceived and conducted. I’m not against small-scale educational experimentation, where new methods are tested, refined, and proved before they are widely implemented. But I’m against our inordinate obsession with what’s new at the expense of what works—with exceedingly superior (if much older) evidence-based practices

 “Inordinate obsession” is the appropriate term; education policy has been driven by billionaires, economists, statisticians and psychometricians, experts on one field setting policy in another field. A prime example is the work of Raj Chetty, who uses “big data,” statistical tools to analyze huge datasets. Chetty and others, in “The Long-Term Impacts of Teachers: Teacher Value-Added and Student Outcomes in Adulthood” concludes that high Value Added teachers have a substantial positive impact on students into adulthood; however, warns that VA should not be used for teacher evaluation

… our study shows that great teachers create great value and that test score impacts are helpful in identifying such teachers. However, more work is needed to determine the best way to use VA for policy. For example, using VA in teacher evaluations could induce counterproductive responses that make VA a poorer measure of teacher quality, such as teaching to the test or cheating. There will be much to learn about these issues from school districts that start using VA to evaluate teachers.

 In spite of Chetty’s caveat states across the nation hopped on the “assess teachers through student test results” bandwagon. The NYS Board of Regents held a summit, experts from across the nation, expressing opinions on the use of student data as assessment tools. The experts warned that the use of Value-Added was ill-advised; teachers teach different kids each year, as well as different grades and different subjects, the errors of measurement, plus/minus ten, twenty, thirty percent makes the data useless. Another vampire idea, a refuted idea that refuses to die; popping up again and again.

The New York State Race to the Top application, in exchange for $700 million, included a multiple measures plan, teachers rated by a combination of supervisory observations and student test scores. Without going too far into the weeds, the current system, called a matrix, combines supervisory observations and student learning objectives (SLO) also referred to as measurements of student learning (MOSL).

Three years ago the governor agreed to a four year moratorium on the use of student test scores and this year the commissioner was in the early phases of constructing an alternative plan. The commissioner has used a consultative process, task forces or work groups, the names are interchangeable, to propose changes in state policies.

Apparently the state teacher union (NYSUT) had been working with the Assembly leadership to craft a plan, a bill was introduced the day before the state teacher union convention and passed, with only one negative vote a few days later. The commissioner was clearly stunned, and not happy. While changing the law is the responsibility of the legislature, the commissioner is the leader, the CEO of the state education establishment amd would expect to be part of the bill drafting process.

A summary of the changes and the opinions of the stakeholders, read here .

The bill passed in the Assembly was introduced as the Senate, a “same as” bill, and, sponsored by the Senate education chair; however, not so fast. Chalkbeat, the online education website muses over the future of the bill.

Senate Majority leader John Flanagan has a dilemma, the bill is popular with parents as well as teachers, the Long Island Opt Out Facebook page proudly boasts 25,000 members, a number of them in Republican senatorial districts. Flanagan needs cover for his Republican colleagues, and his presser speculates over whether the bill will increase the number of tests, clearly an appeal to Opt Out parents.

Since it was first introduced, the State Education Department, the New York State School Boards Association, and the New York Council of School Superintendents have raised concerns that the legislation as written could inadvertently open the door to even more testing than we have now.  Nobody – not students, not parents, not teachers, nor myself or my legislative colleagues – wants that outcome.  With this in mind, we are performing an extensive review of this legislation to determine the best path forward. 

The School Boards Association also questions the bill, a bill that requires negotiations with the collective bargaining agents

We are concerned that if enacted, proposed APPR legislation that has passed the Assembly would result in additional student testing. 

Unless the state wants to forfeit federal ESSA funds, it still must administer grades 3-8 ELA and math state assessments. Under the proposed APPR legislation, students could have to take both the state tests as well as alternative assessments that would be used for teacher and principal evaluation purposes.

In addition, we have serious concerns about the requirement in the legislation for school districts to negotiate the selection of alternative assessments through collective bargaining. This represents a step backward, as school districts presently have the authority to determine assessments used in teacher evaluations.

School boards would rather see unions disappear than work with them in a collaborative manner.

The leader of Long Island Opt Out sees the proposed law as a “small step,” and is agnostic.

Jeanette Brunelle Deutermann

Admin · May 2 at 6:31pm

I want to be clear on what went down/is going down with the new APPR legislation. First and foremost, this legislation does absolutely nothing for children. Not that all legislation has to be centered around children, but I just want to ensure that if you hear ANY INDIVIDUAL or ORGANIZATION proclaim that it is, they are lying. All this does is take away the REQUIREMENT for districts to use 3-8 scores in evaluations, the way it used to be before the moratorium. After this law is passed, districts will continue to be forced to use test scores as 50% of their evaluation, but now in addition to local computer assessments, the science test, or regents tests being a choice, now 3-8 assessments are back on the list. A minuscule positive detail – they don’t HAVE to use 3-8 tests (as they would have had to use as the moratorium comes to an end). Those involved in creating this bill are celebrating this as a huge win. A more appropriate response would be “very sad that this is all our elected officials could muster.” Some have said “but it’s a step.” I guess that all depends on what shoes you’re wearing.

 The legislature will plod along, adjournment around the end of the third week in June and won’t return, except for an unusual special session, until January, 2019; the governor has until the end of the year to sign bills that pass both houses.

Will the Republican leader bring the bill to the floor for a vote? Will the governor sign the bill?

Flanagan has a conundrum,

  •  Should he try and extract a quid prop quo from the Democrats in the Assembly – signing the bill in exchange for, let’s say, raising the charter school cap in New York City, or, approving pre-K classes in the Success Academy charter schools? Actions probably resulting in dollars from the charter school political action committees.
  • Should Flanagan, sub rosa, try and get NYSUT, the teacher union, not to vigorously oppose fellow Republicans in the November general election? Unlikely
  • Should he simply “say no,” it’s a bad bill, without changes palatable to the commissioner and the school board association? In other words use the commissioner and the school board association as cover.

Or, oppose the bill and try and trash the teacher union as being self-serving.

And, will the governor sign the bill without an endorsement from NYSUT?  The small number of union locals that endorsed Zephyr Teachout four years ago might decide to endorse Cynthia Nixon this time, clearly engendering the animosity of the governor.

The end of the legislative session is called “the big ugly” for a reason.

Oddly, this is not an issue in New York City. The City and the Union agreed upon a system of using SLOs and MOSLs and a sophisticated set of alghorisms that satisfies their needs. Under the last year of Bloomberg 2.7% of teachers received unsatisfactory ratings, under the current matrix, less than 1% of teachers received ineffective ratings. The question of the number of observations will be part of the upcoming collective bargaining negotiations.

Maybe a sentence that should be posted above the Albany legislative chambers:

“No mans life, liberty or property are safe while the Legislature is in session.”

Gideon J. Tucker, NYS Surrogate, 1866

Was I a Highly Effective Teacher? How Do You Assess the Impact of a Teacher? A Test Score? A Supervisory Observation? Over a School Year, or Over a Career?

Principal: “In my school all teachers will be highly effective.”
Charlotte Danielson, shaking her head, “No one lives in highly effective, we visit it occasionally.”

Every couple of months the alumni bulletin at the high school at which I taught for three decades arrives in the mail. An alumnus who was a journalist is writing a history of the school, a decade in each edition, primarily using the school newspaper as a resource. The school opened in the mid 1920’s and the history is up to the 70’s. The journalist contacted me, we chatted for a while, and I reminisced about the halcyon days of yore.

I opened the spring, 2013 edition and read,

…Larry Weinstein, Gary Calka, ’71, and Steve Lustig, ’71 conducted a poll for a paper assigned in Peter Goodman’s Urban Affairs class. They surveyed 250 Madison juniors on a variety of topics, including pot smoking, sex education and contraception distribution, the need for school security guards, and satisfaction with Madison education. “[I taught units on polling techniques, stratified random samples, focus groups etc.] Larry remembers how he and his classmates were called into the principal’s office, where “we feared for our lives.” Instead Dr. Forsheit was “fascinated and wanted to know our methodology. He was very encouraging and excited that we had done this on our own.”

Students remembered, in detail, what I had taught more than forty years ago!

Did I achieve my Student Learning Objectives?

Did my lesson comply with the Danielson Frameworks?

Was I a Highly Effective teacher?

Teaching is far more of an art than a science – we strive to be creative – to “connect” with our students, to find the activity that motivates students to delve into the subject matter – and we’re never quite sure of how successful we are.

We do know the total emphasis on exam scores is a poor substitute for creative, challenging lessons.

If students re-took a regents exam a month after taking the original exam would they achieve a similar grade? Probably a much lower grade. What we call project-based learning or portfolio learning, whether in English or Social Studies or a Science class results in learning/skills acquisition that builds upon the past and provide steps to future learning, far more meaningful than a score on one 3-hour exam.

Beginning next year every teacher will receive a numerical score based on student test scores, the mystical Student Learning Objectives and supervisory observations based on the Danielson frameworks.

The “science” that creates a teacher score is highly unstable; the scores vary significantly from year to year.

Economic Policy Institute, “Problems with the use student test scores to evaluate teachers,” by Richard Rothstein and other, August 27, 2010, tells us,

For a variety of reasons, analyses of VAM results have led researchers to doubt whether the methodology can accurately identify more and less effective teachers. VAM estimates have proven to be unstable across statistical models, years, and classes that teachers teach. One study found that across five large urban districts, among teachers who were ranked in the top 20% of effectiveness in the first year, fewer than a third were in that top group the next year, and another third moved all the way down to the bottom 40% …. Thus, a teacher who appears to be very ineffective in one year might have a dramatically different result the following year. The same dramatic fluctuations were found for teachers ranked at the bottom in the first year of analysis. This runs counter to most people’s notions that the true quality of a teacher is likely to change very little over time and raises questions about whether what is measured is largely a “teacher effect” or the effect of a wide variety of other factors.

If Value-Added Modeling, (VAM), using a complex metric to compare teachers and assign them a numerical grade (in New York State the numerical score converts to Highly Effective, Effective, Developing and Ineffective) is unstable, prone to errors, and widely suspect, how should we assess teacher effectiveness?

At a meeting a teacher exclaimed, “Why don’t they leave me alone, I know what I’m doing,” many heads nodded, sotto voce, I mumbled, “but do the kids know what you’re doing?”

The process of teaching is inextricably tied to learning, you cannot blame the kids, yes, external factors impact, it is our job, school leaders and teachers, to figure it out, how do we take kids from where we find them and move them forward?

I listened to a presentation by a high school senior: a research project about childhood prostitution in India and Bangladesh; the student had been in the country for three years, she and her family had fled Tibet and lived in Nepal and India before immigrating to the land of hope and opportunity.

A quiz, a well-crafted test, daily writing assignments, period long in-class writing assignments, a research paper, all filling a portfolio, a defense of the research project before peers, teachers and critical friends are more powerful and impactful than one 3-hour regents exam.

Teachers might agree and remind us that they teach five periods a day with thirty kids in a class – 150 kids a day. They are absolutely correct!! The school leader should redesign the school – flexible, block scheduling would allow teachers to teach longer classes with fewer students, learning communities, cohorts of teachers and kids within a school personalizes student-teacher interactions, all the responsibility of a school leader, to create a learning environment to maximize the abilities and the effectiveness of the teacher.

In some schools kids are highly motivated and in others the motivation is low, we only measure the cognitive skills, those skills we can numerically measure, we fail to measure, or teach the non-cognitive skills which may be more important in the world of work.

See “Data-driven Student Success Solution: The Role of Non-cognitive Abilities in Predicting and Promoting Student Success,” here

See “The Role of Non-cognitive Skill in Academic Success,” here

Unfortunately the teacher evaluation arbitration decision crafted by the Commissioner is rigid, it does not allow for a “design your own,” the plan requires, depending on the model chosen by the teacher, four or six observations a year.

For a number of years in my school teachers voluntarily divided into triads, two teachers observed the third teacher, within a few days the teachers observed each other, the supervisor facilitated a discussion of the lessons and the minutes served as an observation report. It was an eye-opener for the teachers who had never watched a colleague teach a lesson. Did it improve practice? I have no idea; it did create a conversation about teaching.

“Walk-throughs,” brief observations of lessons, are encouraged, although some teachers derisively call them “drive-byes.” Brief observations are useful – Is the class engaged? Does the teacher overly dominate the lesson? Do the kids spend too much time on worksheets? Does the teacher move around the room checking on student work? If the brief observation is followed up with a brief discussion it is a useful tool.

Next year the increased number of observations will morph into a compliance system – the principal probably entering the time and date of the observation into a database, the department issuing some sort of a warning, In red flashing lights on the principal portal: “Warning, Warning, it is November 1 and you are 31 observations behind the required annual 240 teacher observations.” On the other hand I heard an arrogant principal applaud, “Post-observation conferences are a waste of time – they can read.”

I hear constant criticism of newer principals, let not idolize the principals of the past. For decades schools struggled to fill classroom positions, it was commonplace for schools to suffer from vacancies for months. Teacher pay was low and in a booming economy jobs were readily available. Over the last decade teaching has become a desirable profession – the Teaching Fellows Program is overwhelmed with applicants, CUNY and SUNY are graduating many teaching candidates – there are scores of candidates for each job – and thousands of certified candidate seeking jobs.

In the mid- nineties 17% of teacher were uncertified, when I asked a principal his definition of a satisfactory teacher he snapped back, “They arrive on time, every day and blood doesn’t run out from under the door.”

Maybe you were observed one or twice a year, maybe not observed at all. High school graduation rates in scores of schools were appalling, no one seemed to care. We idolized principals of yore because if we “came to school every day, on time” principals left us alone. It may have been a comfortable era for teachers, I remember at Taft High School one year there were no graduates with regents diplomas – that’s right – none – in a school with a 40% graduation rate – all local diplomas with the low skill Regent Competency Test (RCT) exam – a ninth grade skills level exam. No one hassled the staff, unfortunately, or the kids.

Today the world has changed, yes; the teacher evaluation plan (APPR) on the surface looks deeply flawed. Teachers cannot reject APPR and reject school leader teacher observations.

Teachers and school leaders must fight together to create schools in which teacher growth is as important as student growth, a climate of lifelong learning. The macro policies of school governance and school-system structure dominate the debates and the headlines; the micro policies, how do we create learning communities for students and teachers within schools will determine our future.

The answer is not, “leave us alone, we know what we’re doing,” the answer should be, “help us to work together and improve – teachers and school leaders.”

A few decades down the road when that Tibetan young lady is leading a worldwide organization to extinguish childhood prostitution we’d like her to reminisce about that teacher who encouraged and motivated her.