National Study of the Effectiveness of Reading Comprehension Interventions

Study Purpose
Methods
Schedule

Study Purpose

In today’s era of accountability, the need to help struggling readers has become increasingly important. The most recent National Assessment of Educational Progress revealed that 37 percent of the nation’s fourth-graders have difficulty reading. Other estimates suggest that as many as 30 percent of elementary, middle, and high school students have reading problems that severely curtail their educational progress and ultimately hurt their educational attainment. The National Reading Panel, in its most comprehensive review to date of what is known about teaching young children to read, states that our knowledge base on how to help children become better readers is inconsistent. We know a lot about improving certain aspects of reading instruction and little about others.

Mathematica’s evaluation, sponsored by the U.S. Department of Education’s Institute for Education Sciences, addresses critical questions for understanding the effectiveness of reading comprehension interventions, especially for children from low-income households. There are increasing cognitive demands on student knowledge in upper elementary grades where students become primarily engaged in reading to learn rather than learning to read. Children from disadvantaged backgrounds lack general vocabulary as well as vocabulary related to academic concepts that enable them to comprehend what they are reading and gain content knowledge.

The interventions will be selected by a panel of reading experts after review for content, strategies, materials, and empirical evidence on effectiveness. An experimental design will yield rigorous findings on intervention impacts and on the use and applicability of comprehension interventions with fifth-grade students who read commonly used social studies and science textbooks and other types of materials.

The evaluation will address three major questions:

Can reading comprehension approaches improve student reading achievement in either social studies or science?
What are the most effective reading comprehension approaches for improving student reading achievement in either social studies or science?
Under what conditions and practices do reading comprehension approaches improve student reading achievement in either social studies or science?

Methods

The expert panel will recommend reading comprehension interventions on the basis of several criteria, including theoretical and empirical support for the intervention approach to improving reading comprehension in social studies or science; evidence of the efficacy or effectiveness of the specific intervention; appropriateness for the target population, which includes children from low-income families; quality of teacher training and support materials; capacity of developers to implement the intervention in schools throughout the country; and qualifications of staff.

We will identify districts and recruit 100 schools from geographically diverse areas of the country, with a focus on those serving a high percentage of low-income children. Within districts, schools will be randomly assigned to one of the four interventions or to a control group. Schools in the control group will provide students with the same instruction and curriculum they would usually provide in the absence of the evaluation. Reading intervention developers will train fifth-grade teachers in the intervention schools (about three teachers per school, or about 60 teachers per developer) to implement the interventions assigned to their school.

To evaluate implementation, we will observe classroom instruction three times during the year for each teacher, whether assigned to an intervention or a control group. We will also administer a standardized reading comprehension test at the beginning and end of the school year to assess students’ reading comprehension in social studies or science (about 20 students per classroom; approximately 6,000 students in total). Other assessments may be used as well.

Schedule

The expert panel will be established to advise on the selection and development of the interventions. In early 2005, an RFP will be issued, and the panel will recommend interventions for study. During the course of the year, developers will create or revise teacher training and classroom activity materials for interventions, and the evaluation team will develop the instruments for classroom observation, a teacher survey, and school records abstraction. Districts and schools for the pilot implementation will also be selected. The remainder of the study is planned as follows:

Pilot (2005-2006)

Developers implement reading comprehension approaches in 12 schools (3 schools per developer).
Developers train 36 teachers (12 schools x 3 classrooms) at the beginning of the year to use the approaches and provide additional support as needed.
Intervention team monitors and supports developers.
Evaluation team establishes a technical work group for the evaluation to advise on the impact evaluation of reading comprehension approaches.
Evaluation team prepares training materials and conducts training of classroom observers.
Evaluation team pilots classroom observation forms and teacher surveys.
Evaluation team identifies and recruits districts and schools for full implementation.
Developers revise intervention materials on the basis of the pilot experience.

Full Implementation (2006-2007)

Developers implement reading comprehension approaches in about 20 schools each.
Developers each train approximately 50 third-grade teachers at the beginning of the year to use the approaches and provide additional support as needed.
Evaluation team conducts surveys of 300 teachers at the beginning of the year and collects students’ school records at the end of the year.
Evaluation team observes 300 classroom teachers three times each during the year to assess fidelity of implementation and quality of instruction.
Evaluation team administers reading comprehension assessments to 6,000 fifth-grade students at the beginning and end of the year.

Analysis and Dissemination (2007-2008)

Evaluation team assesses the feasibility of administering the reading comprehension tests to participating students at the end of their second year in the study.
Evaluation team analyzes the findings from the impact and implementation evaluations and disseminates the results.