Are New York State Common Core Grades 3-8 Tests Valid, Reliable and Stable Reflectors of Student Achievement?

Are New York State grades 3-8 tests valid, reliable and stable reflectors of student achievement? In other words, are these “good” tests? And, are they useful to parents and teachers, or, just poorly designed accountability tools to measure teacher, principal and school performance?

On the eve of the release of the 2014 ELA/Math test results parents and teachers are wary.

Last year, the first year of the common core tests, the state tried, unsuccessfully, to mollify parents. The commissioner tells us, yes, your kids did terribly, swallow the medicine, in the long run your kids will be better prepared for college and life

The 2013 State presser:

“These proficiency scores do not reflect a drop in performance, but rather a raising of standards to reflect college and career readiness in the 21st century,” King said. “I understand these scores are sobering for parents, teachers, and principals. It’s frustrating to see our children struggle. But we can’t allow ourselves to be paralyzed by frustration; we must be energized by this opportunity. The results we’ve announced today are not a critique of past efforts; they’re a new starting point on a roadmap to future success.

• 31.1% of grade 3-8 students across the State met or exceeded the ELA proficiency standard; 31% met or exceeded the math proficiency standard
• The ELA proficiency results for race/ethnicity groups across grades 3-8 reveal the persistence of the achievement gap: only 16.1% of African-American students and 17.7% of Hispanic students met or exceeded the proficiency standard
• 3.2% of English Language Learners (ELLs) in grades 3-8 met or exceeded the ELA proficiency standard; 9.8% of ELLs met or exceeded the math proficiency standard
• 5% of students with disabilities met or exceeded the ELA proficiency standard; 7% of students with disabilities met or exceeded the math proficiency standard

As the parent anger grew the commissioner embarked on a listening tour – at the first stop in Poughkeepsie parents responded angrily. See U-Tube of meeting: (https://www.youtube.com/watch?v=P_Eiz406VAs)

The rumors are the scores across the state this year are “flat.” Over 30,000 parents in the state opted-out of the state tests and the parent revolt around the state morphed from a brush fire to a conflagration. At the end of the legislative session a law stayed the impact of student test scores on students and teachers for two years. The Republicans have just filed petitions, with over 60,000 signatures to add a “Stop the Common Core” line to the November gubernatorial ballot.

Annual state tests only began in 2003 with the passage of a reauthorization of the 1965 Elementary and Secondary Education Act (ESEA), renamed, No Child Left Behind (NCLB). The law required the testing of all students in grades 3 through 8 (previously only students in grade 4 and 8 were tested), the setting of a progress goal, called Adequate Yearly Progress (See detailed description here), the public reporting of scores by sub-group with the goal of having all student achieve on grade level by 2014.

Each year the state tests were administered in April, all the tests questions released a few months later, the student scores released in August and a Technical Report released in December, a detailed analysis of the test.

Beginning with the 2013 common core tests the state changed the practice – only 25% of the questions were released and an abbreviated Technical Report was not issued until July 2014, fifteen months after the exam was administered; for the 2014 round of tests the state has just released 50% of the test questions.

Fred Smith, a former testing guru in the NYC Department of Education raises the cogent questions (Read entire article here),

… the State Education Department took a half-step by releasing 50 percent of the English and math questions from the April 2014 exams. It was a half-step not just because it falls halfway short of full disclosure, but also because SED fails to provide data at its disposal that would enable objective evaluation of the questions, each of which is a brick in the wall of the testing program.

Until 2011, SED made complete copies of the annual statewide exams available on its web site shortly after they were given, along with the answer keys. People could ponder the test content and thought processes required of students to answer the questions. Every year, no later than December, a technical report was issued giving specific information about each item.

These data included the percentage of students who answered the item correctly (the p-value) and the percentage who chose each incorrect option (distractor) or omitted the item. In addition, a vital statistic was reported showing how well the item functioned (the discrimination index). This allowed reviewers to learn about how every item was working, to identify which ones might be weak and to reach an overall judgment about test quality.

The ability to see all operational items in a timely manner and match them one-to-one against analytic data afforded a reasonable way to study the tests. Since Pearson became the test publisher in 2011, however, SED has hoarded the kind of item-level data needed to render an informed opinion about the tests.

The state has never given a satisfactory answer to the crucial question: why they closed the door to full transparency? At one point claiming the cost of releasing all the questions is too high, or, due to the nature of the contract with Pearson there are confidentiality issues, are not acceptable. The standards setting, the process by which each question is assigned a difficulty level (Level 1-4) is also cloaked in secrecy – the members of the standards setting team must sign confidentiality agreements, principals and teachers are banned from discussing individual questions; secrecy only leads to speculation: does Pearson or the state have something to hide?

In the pre-2013 era the tests reflected curricula content – to what extent did students “learn” the particular content in ELA and math that was required for each grade, the common core exams were significantly different, rather than testing content the exams test the standards – to be more accurate the application of the standards. Schools are now expected to teach content through the lens of specific standards set by distant anonymous committees under the direction of the National Governors Association, and, the actual curriculum modules have been released piecemeal by the state and vary in quality. The state maintains curriculum decisions should be made at the local level.

If the purpose of the tests are to measure student progress, sampling techniques, the procedures used by NAEP. PISA and TIMMS, would be sufficient. If the purpose is to inform parents and teachers of student weakness the tests do not achieve that purpose. The sole purpose of the exams appears to be to hold teachers, principals and schools accountable. The state maintains the 2013 tests set a baseline and each year the results measure the growth, or lack thereof, from the baseline.

Whether an improvement in scores is due to test sophistication or improved instruction is impossible to tell. It will be years before we can determine whether the common core standards have any impact on student learning.

Unfortunately the mishandling of the rollout of the tests, the refusal to delay the implementation, the ragged release of curriculum modules, and especially an arrogance toward parent concerns have thrust the common core into the political arena.

Peter DeWitt, a former New York State principal, writes in Education Week,

We should stop talking about achievement and start having real dialogue about growth. Achievement means that leaders like King only care about the adults in the room (teacher evaluation), and his lack of experience in the classroom doesn’t help him understand the importance of getting students to become assessment capable (Hattie. 2012).

Focusing on growth will help students become assessment-capable, college and career ready, 21stcentury learners. Until King discusses growth and not achievement, and can provide schools with feedback that is “Just in time and just for them in a timely manner that matters most (Hattie)” he will just be sending out another memo that infuriates school leaders and gets deleted in much quicker fashion than it took to write.

Unless we radically change the testing process in New York State the winds of politics will prevail and the common core will become another failed innovation.

Advertisements

2 responses to “Are New York State Common Core Grades 3-8 Tests Valid, Reliable and Stable Reflectors of Student Achievement?

  1. Eric Nadelstern

    Does that mean DeWitt is suggesting a value added approach to assessment? And you’re endorsing that?

    It’s about time!

    Like

  2. If the purpose of state tests are to inform principals and teachers about the progress of students to guide instructional practice the tests are not useful.

    If the purpose of the tests are to compare schools the tests are not useful.

    If the purpose of the tests are to set a state baseline and measure growth the tests are useful, but, who cares? If scores increase by “x” percent what does it show? We have no idea. We can speculate – better PD, better hiring practices – less snow – few solar flares – more immigrants from Finland.

    How many times have you walked through a high-achieving school with a selected student body or located in a high income zip code and saw mediocre instruction, and, the converse, walking through a school with average data in a high poverty, low income student body and see excellent instruction?

    Common Core standards are aspirational goals – students will progress at varying rates, and, monitoring both student progress and teacher practice is the essence of what we do … if creating a collaborative community of reflective educators is growth, I support it!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s