Tag Archives: Daniel Koretz

Feds Release ESSA Alternate Assessment Pilot Regulations: Will New York State Apply? Will Parents, Teachers and School Districts Choose to Participate?

The feds have posted the regulations for the ESSA Alternative Assessment pilot. The competitive pilot allows seven state pilots and encourages states to apply as consortia. The regulations (Read full text here) sets an April 2nd filing date and the regulations sets forth specific requirements. The pilot is three years, possibility for a fourth year, with the goal the moving the pilot to the statewide assessment tool.

I know there is enthusiasm among many parents in the state, especially among the opt-out parents, moving from an examination-based accountability system to a project-based system, at first glance, is attractive.

I have heard: “Instead of a test at the end of the year students can submit a portfolio and a project.”

Unfortunately the application is far more specific.

Generate results, including annual summative determinations …. that are valid, reliable, and comparable for all students and for each subgroup of students;

  Provide for the participation of all students, including children with disabilities and English learners;  

 As a significant portion of the innovative assessment system in each required grade and subject in which both an innovative and statewide assessment are administered, items or performance tasks from the statewide assessment system that, at a minimum, have been previously pilot tested or field tested for use in the statewide assessment system.

 Align with the challenging State academic content standards … including the depth and breadth of such standards, for the grade in which a student is enrolled;

 The regulations are 45-pages long and includes the specificity noted in the sections above.

What do the terms “valid, reliable and comparable for all students” mean?

If you move to a system in which teachers grade/evaluate or assess student work: how do you assess inter-rater reliability? How do you assure the teachers/raters in Buffalo, Rochester, New York City, Scarsdale and Great Neck grade/assess projects/portfolios at the same level?

Vermont moved to a portfolio system in the early nineties and asked the Rand Corporation to assess the program, Daniel Koretz, now a professor at Harvard conducted the study.

“For a variety of reasons, such as the variability of tasks used, it may be unrealistic to expect a portfolio program to reach as high a level of reliability as a standardized performance-assessment program” … the report states. “However, the reliabilities obtained in Vermont in 1992 are sufficiently low to limit severely the uses to which the results can be put.”

 On the positive side, the study also found no evidence that teachers assigned higher or lower scores to their own students than did other raters.

 In the ensuing years technology has improved the rater reliability issue; in many schools in New York City regents essays are scanned and teachers grade anonymous papers, the LEA or the SED can review and monitor reliability, although with 700 school districts in the state, a complex process.

A number of states currently have alternative assessments waivers under No Child Left Behind, New Hampshire is in its fourth year and each year the state has added two school districts to a performance task system.

What are performance tasks?

A complicated question: SCALE, Stanford Center for Assessment, Learning and Equity, has developed a data bank of tasks,

SCALE provides task and resource materials to schools and districts that have committed to adopting performance-based assessment as part of a multiple measures system for evaluating student learning.

 Check out the SCALE performance assessment resource bank here.

What does the New Hampshire alternative assessment look like?

The principles,

  • common performance tasks that have high technical quality,
  • locally designed performance tasks with guidelines for ensuring high technical quality,
  • regional scoring sessions and local district peer review audits to ensure sound accountability systems and high inter-rater reliability,
  • a web-based bank of local and common performance tasks, and
  • a regional support network for districts and schools.

The New Hampshire pilot has changed the face of teaching and learning, teaching in a performance task system is very different from teaching in a current classroom.

I suggest accessing the New Hampshire site here.

The state works with a consultancy, 2Revolutions, that has played a major role in the training of staffs, much more than training, working with teachers and schools to change cultures, to change the face of teaching and learning in a process that totally engages all the stakeholders.

Are the parents, teachers and school leaders willing to jump off the diving board, to walk into a new world, to move away from rigid testing accountability to performance tasks, to move to a student-centered, highly individualized classroom?

The fed proposal requires consultation with all stakeholders, in a limited period of time.

How will participants be selected? Do you consult with stakeholders, submit the application, and choose actual participants after the application has been approved? Or, work with high opt-out districts in the application creation process? Do you choose a subset of schools within districts, for example, the PROSE schools in New York City? Or, do you expand the Internationals Network for new immigrant arrivals? And. all these decisions within a ten week window.

Another core issue: funding. The fed regs do not come with any additional dollars; the governor/legislature will have to add funds to the budget in a restrictive funding year, or, State Ed will have to find funding from external grants.

Daniel Koretz, the current Harvard scholar who wrote the 1992 Rand Report criticizing the Vermont Portfolio Project has a new book, The Testing Charade, Pretending to Make Schools Better (2017); although he is not anti-testing he does skewer the current use of testing – Read review here.

Can you sever testing from accountability and simply use testing a tool to guide instruction?

A weighty nuanced discussion that would normally take many months is squeezed into a narrow time frame; the folks in Albany have an extremely difficult task.

Should New York State End Regents Exams? Can Authentic Assessments Replace the Regents? Or, Will We Diminish the Value of a Diploma?

If you meet anyone who went to high school in New York State I’m sure they’ll remember Regents tests; they’ve been around since the 1870’s.  The Regents were intended for college-bound students; most students left high school and moved onto jobs that allowed them to live a middle class life; jobs, good jobs, were plentiful, commonly union jobs with fair pay and benefits.

In the high achieving school in which I taught only a quarter of students bothered to earn a Regents diploma, three-quarters of the kids earned a local diploma, the requirement, the 9th grade level Regents Competency Test, the RCT, and the accompanying diploma referred to as the RCT diploma. Today we would call the system multiple pathways.

By the mid-nineties the world of work had changed, a college degree was essential for a job. After a few years of discussion the Board of Regents moved to a single Regents diploma system, the RCT diploma was phased out. The plan, originally scheduled to take five years took a decade.

John King was appointed state commissioner,  the state won a  $700 million Race to the Top grant, and, adopted the Common Core State Standards.

Failure rates on the Common Core Algebra 1 Regents increased and the state decided to “scale’ the scores; currently students can receive a passing grade with fewer than half correct answers The state plan was to increase the number of correct answers to achieve a passing grade over time; it hasn’t been happening.

Unless student grades on the Algebra 1 exam increase graduation rates may be impacted, See “Rough Calculations: Will the Common Algebra 1 Regents Exam Threaten NYC’s Graduation Rates? (2015).

If you haven’t seen Regents exams recently look at the Global Studies here and the English here.

Click and try the Regents  ….  How’d you do?

The June, 2016 New York State rate graduation rate was 80%, the glass half full, the graduation rates keep edging up, the glass half empty, one in five kids fails to graduate in four years; six percent have dropped out and twelve percent are still registered in school. Although more kids are graduating more kids are not prepared for college and must take remedial courses in college.

The Board of Regents have been creating additional pathways to graduation,  4 + 1, CDOS, the “safety net” for students with disabilities, the re-scoring option, all part of multiples pathways to graduation options .

The members of the board and the commissioner are beginning to ask whether the emphasis on passing examinations is the best measurement of college and career readiness.

At the October Regents Meeting the members began to explore a move away from Regents exams. The commissioner set forth “potential goals,”

  • Prepare students for 21st century post secondary options, for example, Baccalaureate :programs in STEM, Humanities and Arts, Technical degree programs, Career training certificate programs, Adult education programs leading to certifications, Military service, Employment
  • Offer more flexibility in completing credit requirements, relevant pathway choice and student interest
  • Expand external certification assessment options
  • Allow students to demonstrate proficiency in multiple ways.

And the commissioner when on to list questions: called “Key Considerations”

  • How do we ensure that all students including students with disabilities and English language learners are able to access rigorous coursework?
  • Should students have the opportunity to demonstrate proficiency in a specific area of graduation through a district designed Capstone project?

 The commissioner could appoint a “blue ribbon” commission, experts, who could review the literature, ask for public input and submit recommendations, or, appoint a regents work group who would work with state education staff to draft a plan.

New York State is one of only seven states that requires exit exams, on the other hand critics defend regents exams; every school should meet the same standards, the same exams. The NY Post, the Manhattan Institute and others on the conservative side might accuse the commissioner and the chancellor of eroding the quality of a diploma.

On the other hand the opt-out parents would applaud, one in five students in the state opts-out of state tests and on Long Island more than half of families opt-out. Opting out of regents exams is not an option.

Daniel Koretz, a leading expert on testing has soured on the emphasis on test-based accountability.

High-stakes tests. Lots of them. And that has become a major problem. Daniel Koretz, one of the nation’s foremost experts on educational testing, argues in The Testing Charade that the whole idea of test-based accountability has failed—it has increasingly become an end in itself, harming students and corrupting the very ideals of teaching. 

Are alternative methods of measuring accountability, such as a portfolio of student work, a viable alternative?

The state of Vermont tried to move to a portfolio system which it abandoned; rater reliability was poor.

 A report analyzing Vermont’s pioneering assessment system has found severe problems with it and raised serious questions about alternative forms of assessment.

The Vermont system, which is being closely watched by educators around the country, is the first statewide assessment program to measure student achievement in part on the basis of portfolios.

 But the report by the RAND Corporation … found that the “rater reliability” in scoring the portfolios–the extent to which scorers agreed about the quality of a student’s work–was very low …

 … the report’s author, said the low levels of reliability indicate that the scores are essentially meaningless, since a different set of raters could come up with a completely different set of scores.

“If you’re not rating reliably, you’re not rating,” he said. “You can’t measure anything unless you measure it reliably.”

 Can the state move backwards, to a dual testing, dual diploma system aimed at improving graduation rates for students with disabilities and English language learners?

The state ESSA plan does not include this option.

The commissioner did endorse district-based Capstone projects.

Capstone projects are an excellent example of authentic assessment; at the college level a project might require an entire term to prepare.

The following comes from a partial description of the requirements of a college Capstone project

Capstone Expectations:

The capstone marks the culmination of the student’s studies. Accordingly, the topic selected should require application of a broad range of the skills and knowledge … The final paper must reflect thorough research, analysis, critical thinking and clear writing.

Capstone Content:

  • The topic students choose must be one they develop and work on independently.
  • The paper must showcase a deep understanding of an area….
  • The finished capstone must be a minimum of xx pages and include: an abstract; a background statement; a literature review; objectives; an analysis of existing research; an original analysis of the … challenges; opportunities, threats and possible solutions, critical and thoughtful conclusions; along with a bibliography, charts and any necessary illustrations.
  • The paper may contain primary research, ….Alternatively and more commonly, students may write their paper based on an analysis of secondary research. This approach may include a secondary data analysis or other specified metrics plan.
  • All secondary research must be attributed throughout the paper and in the bibliography.

This is a significant project: the commissioner suggests a “district-designed Capstone project,” how can we assure rater reliability in 770 school districts?

The commissioner and the regents are beginning a long journey with no clear outcome. Students pass courses and fail regents exams: should the failure prevent a student from graduating?  Should one three-hour exam determine graduation? On the other hand bar exams determine who becomes a lawyer; civil service exams determine who becomes a police officer or fire fighter.

I look forward to a deep discussion with experts and public participation and, I would recommend that the state hold hearings around the state.

Are we too wedded to Regents tests?

Are we jumping on a reform wave which may diminish a diploma?

Can/should we change the nature of instruction from the current modality to an authentic, project-based educational modality?

What do you think?