|
|
Tweet |
|
The Local Site Research Initiative: Learning Important Lessons and Setting Future Direction
By: Linda Friedrich, Paul LeMahieu
Publication:
The Voice, Vol. 10, No. 3
Date: 2005
Summary: Through the Local Site Research Initiative, NWP is building rigorous, context-specific evaluation studies focusing on writing project sites' work. Participating sites design and conduct a research study framed by locally important questions.
Through the Local Site Research Initiative (LSRI), the National Writing Project is building a portfolio of rigorous, context-specific evaluation studies focusing on writing project sites' work. In designing this initiative, the NWP took on an evaluation challenge familiar to many national education reform organizations: How do we design a research undertaking that both illuminates the NWP's common values, principles of instruction, and elements of the model, and remains sensitive to the variety of programs that sites design and deliver to meet the needs and the reform priorities of their local communities?
Typical national evaluations employ singular designs that risk reducing a study's sensitivity to local nuances and effects. With LSRI, the NWP took a different approach. Each participating site, with a $20,000 grant and technical support from NWP research staff, designed and conducted a study framed by locally important questions. At the same time, all studies addressed a common concern: the influence of a key component of the site's work on teachers' practices and students' writing. The studies employed various kinds of data to examine student learning, including performance on state and local assessments, student responses to writing prompts and revision tasks, and analyses of student portfolios.
This article tells the story of five pioneering sites that took this challenge and have helped shape this uniquely NWP approach to evaluation while learning from their own work.
Studies and Results
During 2003–2004, five writing project sites—Mississippi Writing/Thinking Institute, Pennsylvania Writing and Literature Project, South Coast Writing Project (CA), National Writing Project at Kent State University (OH), and Western Massachusetts Writing Project—conducted studies of local programs.
All five sites reported generally strong and favorable results for teachers. These included changes in teachers' approaches to teaching writing, the development of classroom communities of writers, and the formation of communities of teachers who support each other's growth and development. While student outcome data ranged from moderately to strongly favorable, methodological and administrative challenges were encountered during the research. The second half of this article addresses these issues and highlights lessons learned while conducting the studies.
Studies of Inservice
Mississippi Writing/Thinking Institute (MWTI): Using the Results for Strategic Benefit. The MWTI conducted two studies of multiyear school/MWTI partnerships. One study focused on fourth grade, the other on seventh. The fourth grade study investigated MWTI's impact in a district with a several-year history of involvement with the institute. Student performance on the state writing test in this district was compared with student performance in two smaller, but demographically similar, districts. Writing assessment results in all three intervention years were significantly higher for the MWTI group. The results also indicate that participation in MWTI professional development over time increases the likelihood that teachers will implement MWTI-advocated practices. Further, the greater the extent to which teachers implement these practices, the more positive the effect on student performance.
The seventh grade study, conducted over two years, compared student performance in four schools engaged in MWTI partnerships for a minimum of two years with student performance in a similar set of schools. Among the teachers whose practices were examined, almost 100 percent reported that they had implemented key practices. During the first intervention year, scores on the state writing assessment were significantly higher for students in the MWTI schools than for students in the comparison schools. This difference, however, did not hold up in the second year of the intervention. MWTI schools experienced a 60 percent teacher turnover in the second year; and professional development was less intensive as schools juggled time allocations for inservice and teaching. Further, the professional development structure changed from workshops and classroom demonstrations to coaching and mentoring, and, in two MWTI schools, professional development focused on content-area teaching rather than on writing and language arts.
The results of both studies suggest that MWTI programs—and likely those at other NWP sites—should be treated as long-term enterprises in which the likelihood and the magnitude of the desired "payoff" increases as teachers' knowledge of and experience with the MWTI-advocated practices increases.
Reflection from Sherry Swain, MWTI
The MWTI state network took advantage of the LSRI research by including teacher-consultants on our research team and updating sites on the progress of the studies. Here are some specific ways we strategically used LSRI to advance our work:
- Key to Policy Doors: Initially we enlisted the assistance of the Mississippi Department of Education (MDE), which provided disaggregated student data so that our results could be reported by NCLB categories. We have kept the deputy superintendent informed and have shared our research in several statewide forums. A presentation to our state board of education resulted in an invitation for MWTI to help MDE in the development of a new language arts framework and the design of teaching strategies that support the state's writing standards.
- Reflection Tool: The seventh grade study underscores the NWP principles that change happens over time and that teachers need consistent support as they implement new strategies. This evidence is useful for our own planning processes and for sharing with school administrators.
- Marketing Tool: We have developed a fact sheet summarizing our research that serves as a handout for conference presentations and provides talking points for administrators and teachers advocating for MWTI professional development.
Research is now enmeshed in the culture of our state network. Summer institutes, continuity meetings, and professional development sessions all now include teacher-consultants talking comfortably about research.
*****
Pennsylvania Writing and Literature Project (PAWLP): The Importance of Sustained Engagement and Effort. PAWLP, which has long offered professional development partnership programs to school districts in southeastern Pennsylvania, was invited to present a program of professional development for all elementary school teachers in a rural district and used this opportunity to examine the impact of its work. End-of-year scores on a direct assessment of writing for students in kindergarten through second grade whose teachers had participated in the professional development program were higher and had increased at a greater rate than those of students whose teachers had not participated. In addition, interviews and observations documented a high degree of implementation of recommended practices in program teachers' classrooms. In third through fifth grades, the same pattern of stronger increase in performance for students of the PAWLP program participants was also observed but the figures were not statistically significant at the given sample size.
Reflection from Mary Buckelew and Andrea Fishman, PAWLP
On reflection, the lack of significant improvement in the performance of older students does not mean that such an effect could not be found under different circumstances, but it does raise questions about why this pattern of results might have occurred. The statistically significant improvement in test scores and change in teacher practice in kindergarten through second grade seem directly attributable to the high level of engagement and receptiveness of the K–2 teachers to the professional development activities. The teachers in third through fifth grades did not embrace the professional development, were reluctant to give up time for writing workshop, and eventually refused to participate in certain aspects of the professional development model. For these teachers, significant improvements were not observed. In a real sense, the NWP model is affirmed in that beneficial impact was observed where it was embraced but not where it was denied.
*****
South Coast Writing Project (SCWriP): Evaluation as Reflective Practice. This study examined the effects of participation in a professional development program designed to help middle school teachers work effectively with students in low-performing public schools in which a high proportion of students come from families whose home language was not English. The program employed structures such as teacher inquiry and classroom modeling to support teachers in deepening their pedagogical content knowledge (e.g., the composing process and criteria for evaluating writing) and expanding their repertoires of instructional practices (e.g., responding to students' writing). Participation in the SCWriP program had substantial positive effects on how teachers conceive of themselves as teachers and writers and on their teaching practice. These teachers employed a greater range of writing assignments and more consistently used writing in the interest of learning. Participants also thought of themselves as more professional and as part of a collaborative professional community.
An analysis of student performance, however, did not yield statistically significant differences between students in the program schools and comparison schools. One possible explanation for the student results is that student performance measures may have been insufficiently sensitive to the kinds of academic skills and habits of mind that were emphasized in the program, such as developing an ability to use writing for constructing knowledge and acquiring a deeper understanding of reading materials.
Reflection from Sheridan Blau, Rosemary Cabe, and Anne Whitney, SCWriP
Participating in this research has affected the way the program is planned and implemented and has influenced the pedagogical thinking of the research team. As the research project forced us to articulate detailed hypotheses about the classroom changes we expected to see, we discovered implicit criteria we were employing in evaluating teachers' performance. While our work had been guided by broad and clearly stated objectives, we came to recognize that these often presupposed more subtle expectations (like the art of pacing in lessons) that we had never before articulated. Further, we may have overestimated the power of modeling and underestimated the need to make explicit the thinking that grounds the strategies and practices we share in our demonstration lessons and coaching process. Thus we have made subtle but significant revisions in our written program goals. More importantly, we have enhanced our own understanding of the choices we make as classroom teachers and of the thinking behind those choices that we need to make clear in our professional development work.
*****
Studies of Summer Institutes and Continuity Programs
National Writing Project at Kent State University (NWP@KSU): Research Findings Beyond Evaluation Results. This study analyzed the impact of the invitational summer institute on teachers' practice and on the performance of their students in fifth through twelfth grades on three writing tasks: two responses to on-demand prompts, and one related revision task involving higher-level writing skills. The results revealed that NWP@KSU teacher-consultants (TCs), in comparison with other similarly experienced and professionally motivated teachers in their schools, focused more on higher-order writing concerns such as audience than on mechanical aspects of writing. Further, the tone of TCs' responses to student work was consistent with the idea that both students and teachers participate as members of a community of writers.
Students of NWP@KSU teacher-consultants obtained better scores on all three tasks than did the other students. However, the superior performance of the teacher-consultants' students cannot be reliably attributed to the TCs' participation in the summer institute. Because these students scored higher on the first assessment, intended as a baseline, their better performance may be the result of higher proficiency in writing prior to receiving instruction. Alternatively, for the younger students it may stem from the fact that their beginning assessments were conducted after several months of instruction. Thus, the higher scores of TCs' students on the initial assessment may be a result of differences in instruction that occurred before this measure was taken. For older students in semester-long high school classes, however, the initial measure was collected at the beginning of the course, making it a better basis for comparison. In these classes, the first performance of the teacher-consultants' students was similar to that of students whose teachers had not participated in NWP@KSU, and the TCs' students obtained better scores on the final assessment. In circumstances where a better baseline measure was available, teachers' participation in the summer institute appeared to have a positive effect on student performance.
Reflection from Nancy McCracken and David Bruce, NWP@KSU
The LSRI prompted our leadership team to think more deliberately and creatively about ways to measure the impact of our site work. For us, the effort to measure the impact of our work by quantitative measures of student performance was both an ethical and an intellectual challenge. Our assumption has always been that the work of students and teachers cannot be, and should not be, measured by large-scale scoring of student work produced under testing conditions. However, in light of the current No Child Left Behind Act's definition of "research-based" staff development, we designed what we consider a compromise measure. The data we gathered as part of the LSRI project include a revision task that lets us study the way that students of NWP@KSU teachers think about what makes good writing. We are currently at work on analysis of these data and will publish our findings. We are continuing to use this measure along with other context-sensitive measures at our local site. Along with extended time to talk with other site directors about research, the LSRI provided an impetus and the support for us to begin a line of research, focusing on the relation of student work to teacher practice, that goes well beyond the typical questions of evaluation research.
*****
Western Massachusetts Writing Project (WMWP): The Importance of Time and Collaboration. This study, conducted in two elementary schools in western Massachusetts, was motivated by the question of whether having a substantial number of TCs working together in a school makes a difference in teachers' practices and students' writing performance. In one school, seven faculty members were WMWP teacher-consultants; in the other school, the only writing project connection was the presence of a single teacher who had worked closely with a former colleague who was a WMWP teacher-consultant. The research found that writing project teachers both used those practices presented in the WMWP programs and articulated the theory that guided these practices more frequently than non-writing-project teachers. This group of TCs exercised influence on writing instruction within their school. They communicated with each other frequently, honing their own teaching, and shared ideas with teachers who had not participated in WMWP activities.
The study also examined student writing performance, using a locally developed pre-post writing sample. While the study found no statistically significant differences in student performance between NWP-taught students and non-NWP-taught students, over a two-year period, students taught by NWP TCs did improve more than the comparison group. A study of a larger group of students for a longer time might clarify the meaning of this trend.
Reflection from Susan Biggs and Charles Moran, WMWP
This study suggests that the presence of a large number of TCs at a school does not alone impact student performance; nor does it guarantee the transfer of best practices to non-WMWP teachers at the same school if time and space for thoughtful reflection and in-depth conversation around practice in the teaching of writing are not available. As one WMWP TC said, changes in the teaching of writing could occur if "[teachers] get time to come together and talk about the teaching of writing and look at research, not simply sharing practice amongst each other."
We have learned that if transfer of teaching practice is to occur in a school between WMWP teachers and other teachers, this transfer needs to be planned for and staged. This is something beyond our past understanding of"continuity programs," which are typically site-based and designed to keep TCs in touch with the NWP site after the summer institute. What we saw was a need for school-based programs to support a group of TCs in their school. In fall 2004, therefore, WMWP sponsored a teacher-initiated and TC-led semester-long inservice course in the target school, helping the TCs and other teachers to build into their schedules time to reflect on research and practice in the teaching of writing. WMWP is identifying other groups of TCs in other schools to work with in 2005–06.
Further, we have made adjustments in our summer institute 2005 schedule so that we can be more explicit about the ways in which transfer of practice might take place. In addition, we will share models of programs that TCs can create in their schools to foster reflection, research, and the sharing of practice. These models will include not only school-based courses such as the one described above, but also reading-and-discussion groups, writing groups, and collaborative classroom projects (such as student publications).
*****
Looking Back: Lessons Learned
We now take a step back from the specific LSRI studies to highlight some lessons learned and their implications for our work in the currently active second cohort of LSRI studies.
Timing. In the first year, our schedule allowed only limited time to carry out preparatory work such as finalizing research instrumentation, securing approval from university institutional review boards and school district research offices, and selecting program and comparison teachers. In two instances, this meant that pretest measures were not collected until November and December; therefore any student growth in writing that may have taken place during the first three months of the school year was not measured. In the second LSRI cohort, earlier notification assisted sites in completing prework and administering baseline writing prompts at the beginning of the year.
Forming Appropriate Comparison Groups. Identifying, recruiting, and maintaining appropriate "pristine" comparison groups proved difficult because a site may have saturated its service area or writing project practice may have spread within a school. In the second LSRI cohort, NWP research staff and sites have collaborated to prioritize the characteristics of potential comparison groups and met with district administrators to secure support for participation in the studies. In addition, sites increased overall sample sizes and over-sampled comparison teachers to help address problems with attrition.
Appropriateness and Sensitivity of Available Measures of Student Writing. First, developing, pilot testing, and refining writing prompts and rubrics is inordinately difficult for each site to do independently, given the size of the challenge and the modest resources available. Second, while many readily available rubrics use a relatively small number of scale points to increase an assessment's reliability, this also means that it is more difficult to detect fine distinctions and, therefore, more challenging to measure growth. To address these issues, the NWP has developed an archive of writing prompts and rubrics that make accessible field-tested assessment schemes that can provide more discrimination and measure growth more accurately. The LSRI conducted a national scoring conference at which all the student writing gathered was scored independently by certified scorers using a well-established rubric. This method ensures the independence, technical quality, and credibility of the student writing performance data.
Looking Ahead
Continued interest in evaluations of the NWP will mean increased future investment in research. Sites' generally favorable response to the LSRI suggests that it provided a significant opportunity to research the impact of their work on student outcomes. With these successes in mind, the NWP will continue with a transformed LSRI in the upcoming year. The initiative will invite sites to join with like-minded colleagues to learn together as they develop research studies that explore local impact and serve local needs. The exceptional efforts of these first LSRI sites will benefit the NWP for years to come.
Readers may view the National Writing Project Local Site Research Initiative Report: Cohort 1 on NWP's website.
