APPR (Teacher Evaluation) Complexity: If Principals, Teachers and Parents Don’t Understand It, It Will Fail.

On Wednesday night the UFT and the City submitted their versions of a teacher evaluation plan to the state and over the next two weeks the areas of disagreement will be narrowed and a plan will be agreed upon or imposed by the commissioner.

Otto Von Bismarck was the architect of the unification of the German nation. He was an apostle of “realpolitik,” (politics based on practical and material factors rather than on theoretical or ethical objectives), both reviled and admired as a master of international political wheeling and dealing.

Two provinces on the Denmark-German border, Schleswig and Holstein, were part of an extremely complex dispute.

Years later Bismarck was asked to explain the details of the dispute, he replied, “Only three people have fully understood the issues, one died, one is mad, and I’ve forgotten.”

I fear the New York State APPR teacher and principal evaluation plans (see all approved plans here) are so complex, that the implementation has so many moving parts; it is highly unlikely that mere mortals fully understand the plans.

* 20% of the plan is made up of NYS student test data – a Value-Added Model (VAM), measuring student growth adjusted for a number of variables (Disability, English Language Learners, Title 1 eligibility, etc.), well, not exactly – only 21% of teachers teach students who are tested – 79% of teachers are not impacted by state tests. Teachers in grades pre-K – 2, all elementary school specialty teachers, all middle school teachers other than English and Math and all high school teachers are not impacted by the state tests.

Additionally middle and high school teachers teach five classes a day across multiple grades – how do you “measure” teachers who are teaching multiple grades? Check out the “50% rule” on the EngageNY website. Think you’re principal will be able to figure this out?

* locally negotiated 20% depends on Student Learning Objectives (SLOs) – sort of a pre-test/post-test measurement of students that is highly subjective and in no way passes any test of “validity and reliability”

Once again, take a look at Student Learning Objective 101 on the EngageNY website. 70,000 teachers in 1600 schools creating SLOs to measure the progress of 1.1 million students, probably can knock it off in a prep period!! Or, hire a few hundred SLO Creation and Implementation Developers (SLOCID) to scurry from school to school…

* The 60% supervisory/peer assessment through classroom observations are the major section of the APPR. I doubt many (any?) plans include peers in the classroom observation section. In my random sampling of plans many call for “one announced and one unannounced observation per year” utilizing a scoring rubric approved by the bargaining agent, in New York City, the Danielson Frameworks.

The Danielson Frameworks 2013 are a 114-page document describing in minute detail classroom behaviors divided in Domains, Components and Elements. Last year I sat through two days of “training” on an earlier version of the Frameworks – we watched videos, discussed, took a “mini quiz,” and assessed lessons. I sat with six principals – we watched a video of lesson – “scored” the lesson, and compared our scoring – with wide differences among us. At the end of the session Teachscape made a pitch – for many thousands of dollars principals could buy software to train themselves and staffs – I was told more than half of the schools in NYC purchased all or part of the Teachscape package.

Danielson, strangely, is not a fan of standardized testing ,

I do understand the policy people desperately wanting to assure their public that they are paying attention to, not only what teachers do, which is teacher practice but to also the effect that their teaching has on student learning. I think that is a reasonable request.

The real question is what counts as evidence and how can you attribute that evidence to individuals? That is the challenge and I do not believe anyone has figured that out yet. There is a great deal of psycho-metric evidence that indicates that any standardized tests are a poor measure of student learning for this purpose, but in many cases that’s all that’s available.

I am not thrilled about saying that a teacher’s performance is a level 2 and not a level 3. However, we are in a world of high-stakes evaluation, and so it may be necessary to do so.

Can a teacher receive a high grade in one category and a low grade in another category, and, if so, what does that say about the system?

When New York City distributed scores to teachers, percentiles scores, they also gave the teacher a wide range – a standard error of measurement,

The standard error of measurement (SEM) is an estimate of error to use in interpreting an individual’s test score. A test score is an estimate of a person’s “true” test performance. Using a reliability coefficient and the test’s standard deviation, we can calculate this value:

SEM = s  1 – r), Where:
S = the standard deviation for the test
r = the reliability coefficient for the test

I’m sure your supervisor will have no problem working out the formula to assure that you fully understand your grade and the ramifications.

Do not fear: the folks at the State Education Department have provided a number of detailed reports and webinars – click here for “Resources about State Growth Measures.”

After you complete reading the resources an online test will be administered (only kidding!!)

Teachers complain that supervisors “walk-throughs” are “drive-bys,” implying that supervisors are using a few minutes to derogate the lesson. I demur: as you walk into a classroom sometimes the class is jumping with high levels of student engagement, other classrooms are boring, kids filling in worksheets or a teacher rambling, most classrooms fall somewhere in the middle.

After a presentation by Charlotte Danielson I asked, “Justice Potter Stewart in a Supreme Court decision wrote, ‘ …pornography was hard to define, but that I know it when I see it.'” Isn’t it the same with a good lesson?

Charlotte was not pleased.

An irony: Charlotte Danielson’s other book, “Talk About Teaching: Leading Professional Conversations,” 2009, discusses the conversations supervisors have with teachers after a classroom visit that are not intended to be evaluative.

Take six minutes and watch an excellent video (http://www.youtube.com/watch?v=NqeMKQhTcLQ).

Maybe we should just dump the entire APPR plan and coach teachers and principals how to talk with each other about teaching.