Smarter Assessments, Better Student Outcomes

After two years of hard work, sweat, and many intense discussions, most districts in Maine have put together impressive Local Assessment Systems to meet the requirements of Ch. 127 (Maine’s current law governing graduation and assessment requirements). Teachers, administrators, and the Maine Department of Education (MDOE) personnel have worked tirelessly to create unique, student-centered assessments that show students’ progress toward the Maine Learning Results. These professionals are to be congratulated for their work.

There is an expanding set of online tasks for schools to use, and never before in the history of education in Maine have teachers been so intimately involved in the assessment process. In every school around the state there are conversations about assessment. While not all educators agree on the amount or type of assessments we are using in Maine, most have had the chance to be involved in this transformation to a standards-based system for accountability and certification.

The question that faces schools now is how to ensure that all students can have the maximum opportunity to “show what they know.” One challenge is determining how to address the significant hurdle students with disabilities face in meeting the requirements for graduation now that the bar has been raised even higher by State statute. In addition to conversations about assessing students with general needs, we must consider the implications of our assessment format on students with disabilities.

Assessments must specify what they are measuring and educators need to be aware that all tests measure more than they say they do. The critical issue to address is whether the various hidden measures negatively impact one segment of the student population. For example, math and science tests often measure reading and writing; reading assessments may measure chart reading; and Social Studies tests may test all of these areas. In short, it is difficult to test only the intended construct and nothing else. For many students, this is usually not an issue, but for students with diverse learning styles or those with cognitive disabilities, reading can be the barrier that prevents them from being successful. Typical students, with average to above-average literacy skills, are able to demonstrate what they know about science because they do not struggle with the reading and writing components of test taking. As a result, it is clearer that what is truly being measured is the students’ knowledge of science. Consider figure 1.1 (below) in relation to this issue.

Figure 1.1

The pink and yellow colored areas, on the left and the right, show the unique skills measured for science and for reading on a test for the hypothetical, “typically-performing” student. The area of overlap in the middle (with the question mark) represents the area these two characteristics share in this assessment. In this figure, the area of overlap is relatively small, accounting for about of one-third of the total content area. We know that reading ability influences content area performance, so this comes as no surprise. Now consider figure 1.2 and the student with a disability in reading.

Figure 1.2

In this figure, the two characteristics (reading on the left and science on the right) overlap almost completely. This represents the performance of a hypothetical student with significant reading problems. Notice how much reading ability impacts science performance in this scenario. In this case, the area of overlap accounts for 90 percent of the total area. On this assessment we cannot make valid inferences about the student’s science performance because reading affects it so significantly. Consider the following assessment, pulled directly from the Maine Math and Science Alliance Local Assessment Development (LAD) task bank, (available by clicking on Baseball Fever) and is representative of many high-quality assessments used by local assessment systems around the state.

Figure 2

Maine Math and Science Alliance (2004, May). Baseball Fever. Retrieved March 1, 2005 from http://www.state.me.us/education/lsalt/LAD/Tasks/Math5-8/BaseballFever.pdf

This assessment does an excellent job of measuring Numbers and Number Sense (Content Standard A) and Computation (Content Standard K) in a format that is appealing to many youth. There are several parts to this task, including asking students to graph their response and write a letter to the commissioner of baseball.

For a typical student with average to above-average literacy skills, this test accomplishes exactly what it sets out to do: measure Number Sense and Computation. For the struggling reader, this test presents a considerable reading demand in order to get to the math. Reading is not part of the standards being assessed, nor is it part of the scoring rubric for this assessment. While this task requires the student to complete 17 different, multi-step calculations, it also requires the student to read more than 550 words. The overall reading level for this assessment is about the 5.5 grade level. Students with decoding issues, receptive language difficulties, processing speed delays, or comprehension problems likely find this task very difficult to complete without additional assistance or time. In addition, the writing demands also impact students who struggle with written language.

Currently, accommodations are provided to many students with disabilities for tasks like this one. Unfortunately, researchers are finding that providing accommodations on an “as needed” basis does not improve student scores (Helwig, Rozek-Tedesco & Tindal, 2002; Helwig & Tindal, 2003). In nearly half of all cases, teachers did not accurately match students to accommodations that improved their performance. What these authors suggest instead is an assessment approach that attempts to provide as many different ways as possible for students to interact with the test. A better-designed assessment, one that builds in these options from the start, allows all students to benefit.

Effective learning is optimized when information is presented graphically, orally, and experientially (Woolfolk, 2002). In this assessment, the information is presented graphically but the items are not presented orally or experientially. Additional research from Meloy, Deville and Frisbie (2002) suggests the read-aloud accommodation impacts students’ performance in unpredictable ways—and should be used with great caution on reading tests—but may serve to assist students on math and science assessments if it is presented to all students as a part of standard administration.

Relatively small changes to the administration procedure for and test design of this particular task would provide more students with the information they need to perform better. Minimal test adjustments would further support the goal of truly understanding what students have learned by more accurately measuring their knowledge.

Improvements to this task might include:

First, reading the instructions and questions aloud to the entire class to improve the comprehension of struggling readers.
Second, asking several students to paraphrase the instructions briefly to engage those students who learn best by doing. (The pause to paraphrase also gives students with slower processing time a chance to reflect on the task for an extra minute or two.)
Finally, offering more intensive support to students with identified needs only after they have had the chance to hear and experience the directions through pre-teaching.

The lesson to be learned from the research is the importance of building in multiple methods of presentation right from the start, and giving them to each and every student. Helwig’s and Tindal’s findings suggest teachers are not always accurate in determining the exact support that will best benefit students. Providing flexibility and choice for all students gives everyone a better chance to show what they know!

References

Helwig, R., Rozek-Tedesco, M., Tindal, G. (2002). An oral versus a standard administration of a large-scale mathematics test. Journal of Special Education, 36(1), 39-47.

Helwig, R., & Tindal, G. (2003). An experimental analysis of accommodation decisions on large-scale mathematics tests. Exceptional Children, 69(2), 211-225.

Maine Math and Science Alliance (2004, May). Baseball Fever. Retrieved March 1, 2005 from http://www.state.me.us/education/lsalt/LAD/Tasks/Math5-8/BaseballFever.pdf

Meloy, L. L., Deville, C., & Frisbie, D. A. (2002). The effect of a read aloud accommodation on test scores of students with and without a learning disability in reading. Remedial and Special Education, 23(4), 248-255.

Woolfolk, A. E. (2002). Readings in educational psychology (8th ed.). Boston, MA: Allyn & Bacon.

Smarter Assessments, Better Student Outcomes

Local assessments that measure what they say they measure allow all students to show what they know!

References