Extending Accountability: From compliance to learning

The mandates of the US National Reporting System (NRS) accountability framework presented adult education at the state and local levels with a quantitative, outcomes-based, mandatory system. In the early years of the NRS, researchers explored the challenges of implementing largescale accountability systems and documented its early impact. With the continued implementation of the NRS over the past four to five years, this paper takes a fresh look at the impact of those accountability requirements with particular reference to implementation by the Connecticut State Department of Education (CSDE). It presents some positive developments with respect to curriculum, instruction, and the use of data, while highlighting some of the shortcomings within the NRS approaches. It discusses the organisational implications for local programs, presents strategies for state agencies, and offers suggestions for improving the NRS.


Introduction
In 1998, the federal law that governed US adult education activities was included within the Workforce Investment Act (WIA): an Act primarily concerning job training and workforce development.In response to the requirements of WIA, the federal government established the National Reporting System (NRS) as a unifying accountability framework for states to report learner outcomes (Division of Adult Education and Literacy (DAEL) 2005).Qualitative, process-oriented, and voluntary approaches to program standards were thus replaced by a quantitative, outcomes-based, mandatory system (Condelli 2007).
As the WIA and NRS dawned on adult education, researchers explored the challenges of implementing large-scale accountability systems (Merrifield 1998) and documented the early impact of these mandates on local programs and state agencies (Belzer 2003).With their continued implementation over the past five years, this paper takes a fresh look at the impact of those accountability requirements on local programs in one state, Connecticut.The following questions guide this inquiry: -How have the responses of local adult education programs to the WIA accountability requirements changed since the early years?
-How have the services offered by local programs changed because of these requirements?What lessons have they learned about their services?-How have the policies and practices of the state agency influenced the local responses and services?-What future considerations do these developments offer for the accountability mandates of the NRS?

Background
Two critical performance outcome measures are mandated by WIA.
Educational Gain: The NRS established educational functioning levels (EFL) as a framework for measuring improvements in literacy and numeracy abilities (DAEL 2005).Educational gain reflects the progress achieved by learners on these EFLs as determined through standardised assessments that are administered pre-and post instruction; Follow-up Outcomes: These are the successes achieved by learners after they leave adult education and include entry into postsecondary education/training, job attainment, job retention, and achievement of a high school diploma.
The NRS expects state agencies to have assessment policies that mandate pre-and post-testing using standardised assessments and to establish electronic data systems in order to collect individual learner data with regard to demographics, entry characteristics, attendance, and test results.The NRS also requires states to attest to the quality of their data and related processes through a comprehensive checklist (DAEL 2005).
The Connecticut State Department of Education (CSDE) had implemented a standards-based framework and an individual learner data system within adult education for several years prior to the NRS .The standards-based framework, developed by the Comprehensive Adult Student Assessment System (CASAS), connects curriculum, assessment, and instruction (CASAS 2004).
To administer standardised CASAS assessments and ensure their incorporation into instruction, Connecticut providers are expected to maintain Program Facilitators with the necessary training and certification.These Facilitators are teacher leaders who play a critical role in maintaining the integrity and quality of the assessment process (Alamprese 1993).They also assist teachers with curriculum issues and help integrate test results into instruction.Each provider must also have staff who are trained annually by the CSDE to accurately collect and enter data into the Connecticut Adult Reporting System (CARS).

Research approach
Systematic data about the impact of the accountability requirements on local programs were gathered through phone interviews with ten Program Facilitators across a range of urban and regional programs Each interview lasted about 45 minutes.With one exception, these Facilitators are members of a CSDE advisory group and had been involved with assessment/accountability issues in a local program prior to WIA implementation.
Thematic analysis (Boyatzis 1998) was applied to these data and then supplemented with relevant comments from the Facilitators, NRS data from Connecticut, and insights from the author's experiences with accountability in adult education.
The results are presented through four themes -curriculum, assessment and instruction; from 'data for accountability' to 'data for program improvement'; organisational alignment through instructional leadership; and mixed results with follow-up outcomes.Each theme is explored through changes in practitioner responses and the modifications to program services that have occurred since the early years of WIA.Also discussed is the impact of the NRS on the complete and accurate presentation of the successes achieved by learners in adult education.The mediating role of the state agency in implementing these accountability requirements is also examined.
Theme 1: Curriculum, assessment and instruction: A return to an integrated past Because of Connecticut's history with CASAS implementation, the NRS requirement for standardised pre-post assessment was not a new concept or expectation.However, with the onset of the NRS, several Facilitators felt that the stakes had been raised.They began to take the accountability requirements much more seriously.
This new attitude has brought about a renewed interest in utilising assessment results for instruction.In the past, the impact of CASAS implementation was borne mostly by the central office staff and not diffused throughout the program (Rogers 1995).Teachers may have reflected a 'give the test and file it' mentality with regard to standardised testing and delivered instruction that ignored the results of this assessment.With each year since WIA, however, this attitude has changed.One Facilitator observed: We used to just give tests, but I think as the accountability and data requirements have evolved, we have really started to be much more focused and clearer with our teachers about connecting the testing and the instruction, realising the part it plays in terms of their role as teachers and instructors and encouraging them to recognise that piece, and take it more seriously.
Typically, assessments used for accountability purposes do not yield results in a timely manner to inform instruction (Linn 2001).NRS approved assessments, however, are administered locally and the results are readily available.Therefore, teachers are receiving extensive professional development in using pre-test results at the individual and class level to identify learner needs and develop lesson plans.
These efforts to utilise test results for instructional planning are focusing attention on the broader curriculum standards that undergird the standardised assessments: the competencies and content standards (CASAS 2006).They are prompting a renaissance of the integrated competency-based education (CBE) model (Figure 1) that was more prevalent in the initial years of CASAS implementation during the late 1980s.Though standards-based education was being implemented in Connecticut prior to the introduction of the NRS, the increased stakes of the NRS requirements is prompting programs to reassess their commitment and attentiveness to the diligent implementation of those standards.For example, this renaissance is casting employability/workforce skills in a new light.Programs that did not incorporate workforce skills extensively in the early years of WIA are revisiting the potential for a workforce-infused,

Assessment
Standardized Tests and Individual/Class Profile contextualised curriculum (Jurmo 2004).Workforce skills are also being integrated within some secondary programs where the curriculum usually has a strong academic orientation (Vernon Public Schools 2006, Ferraiolo 2007).This change has not been without resistance from teachers, 'more because of its newness than because of the employability focus.'Nonetheless, all Facilitators spoke of these developments in positive terms.One Facilitator commented that aligning lessons to curriculum standards and test results has required teachers to offer instruction that is sometimes outside their comfort zone but ultimately beneficial to learners.In the past, the content of this instruction may have been up to whatever the teacher felt was important, or determined completely between the tutor and the learner.Now, the minimum expectation is that the curriculum standards and the results from standardised assessments will be incorporated into instruction and lesson planning.Such instruction is seen as being more goaldriven, focused, and targeted, that also meets the broader needs of all learners.
While the NRS has definitely renewed interest in the comprehensive and integrated CASAS model, it has not been without its shortcomings.For many learners, attainment of a secondary school diploma through passage of the General Educational Development (GED) test is their primary purpose for attending adult education.According to NRS policy, however, passage of the GED tests cannot serve as an indicator to complete an EFL, except at the adult secondary high level, primarily because the GED tests have not been evaluated against the NRS criteria.This presents practitioners with a dilemma: should they help learners to pass the GED test or achieve progress on the NRS levels?
To analyse the extent of overlap between these two purposes, the CSDE studied the relationship between learner abilities on CASAS and GED tests (CSDE 2008).Though these tests differ with regard to their purpose, content, and administration, the results from this study demonstrate that as learner abilities on CASAS assessments increase, so does their general likelihood of passing the GED tests.Facilitators are learning from the standardised testing [results] that many students who come to prepare for the GED exam are in fact very deficient in basic skills.The study encourages local practitioners to use 'results from CASAS assessments to place learners into GED programs, present learners with progress benchmarks toward GED readiness, and recommend learners for the administration of the Official GED Practice Test and the 'real' GED exam'.
This study also illustrates that learners functioning below the NRS adult secondary high level, especially in math, can also pass the GED test at reasonably high rates.In light of such evidence (CASAS 2003, CSDE 2008), it is odd to think that the national adult education reporting system does not acknowledge passing the GED tests as a success at all levels.Assisting adults to complete secondary school is one of the three purposes of adult education in the WIA legislation.However, because level completion cannot be determined unless a learner is post-tested (DAEL 2005), programs are forced to utilise their limited resources to procure a post-test score from learners who are close to passage of the GED test or may have even accomplished that goal.These learners have limited incentive to demonstrate their progress on the NRS levels.If diploma attainment were reportable as an achievement, Connecticut's performance on the NRS at the ABE High Intermediate and Adult Secondary Low levels for 2007-08 would have been greater by 62% and 91% respectively!Because gain cannot be reported within the NRS system without a post-test score, some providers may feel pressured to post-test learners before they have attended the suggested minimum number of hours.This happens especially when learners start towards the end of the year, have few instructional hours to attend, but are expected to complete the EFL in that year.Premature post-testing can be a waste of resources because greater gains generally occur after learners have attended 100 hours (CSDE, 2007).
The focus on educational gain also does not permit programs to demonstrate the life-changing achievements made by some learners.
The NRS accepts reporting on 'secondary outcome measures' (DAEL 2005) such as learner achievements in family literacy, community, or citizenship.However, a majority of states do not report on those measures probably because: a) they are optional; b) they are not used to determine state incentive awards; c) there are no clear standards for collecting such achievements; and d) they usually have limited reliability (Condelli and Kutner 1997).
Theme 2: From 'data for accountability' to 'data for program improvement' By establishing a uniform reporting framework and supporting it with high quality training, the NRS helps states to establish data systems, collect high quality data, and use NRS data for program improvement.
In Connecticut the mechanics of data collection were firmly in place when WIA was introduced.The central office staff at local programs used data to complete grant reports; however, the use of data for program management and improvement was not fully realised.Facilitators observed that 'in the past, we were more concerned with getting the paperwork in' and that 'nothing made sense to teachers, nothing clicked.' A key turning point was the institution by the CSDE in 2004 of a report card called the Program Profile report.Though Facilitators could previously access multiple reports that provided comparable data, the Profile consolidates the key process and outcome information into one succinct report.Facilitators use the 'common language' of the Profile report to converse with teachers about the impact of their class on the programs's performance.
A culture of testing assumptions against the data has permeated the adult education community.For example, one Facilitator's inquiry into transient learner attendance patterns revealed 'the need for three terms.'Another program expanded its 'ESL program from three to six levels to accommodate the wide range of English language proficiency that we discovered.'A volunteer program discovered that 'the more students come, the more intensity and duration, the better progress they are making'.
The CSDE also models the use of data for decision-making by conducting and presenting analyses that make explicit its rationale for new policies and initiatives.For example, one study regarding learner persistence in adult secondary education (Gopalakrishnan 2008) introduced a longitudinal approach to learner retention and informed the CSDE's policy decision to expand access to secondary completion programs for adult learners.The profile report was also revised to indicate the number of learners who returned to adult education from a prior fiscal year.One Facilitator spoke of using the longitudinal data to work on 'retention plans to re-engage students in the next year'.
However, though NRS training sessions encourage and support the use of accountability data for program improvement, some NRS policies and practices limit the usability of NRS data for that very purpose.For example, the NRS 'level completion' approach is to some degree predicated on where a learner enters in the level.Because each EFL is wide (spanning between two to three grade levels), a learner who enters in the upper range of a level is that much closer to the finish line than someone who enters in the lower range of that same level.
A report commissioned by the US Department of Education levelled the same criticism about this arbitrary aspect of the NRS 'level completion' approach (Rose and Wright 2004) and stated that: For analytic purposes, the best way to measure learning gains is to compute the change in scale scores when the student takes an appropriate matched test from the same company.By contrast, change in NRS level is a much cruder measure.…These boundaries [of the NRS levels] may not be that important for summary measures of gains by program, but they severely undermine the power of statistical models to identify the factors most strongly associated with learning gains.( 22) To counteract the limitations of measuring gain by levels, the CSDE reports on the percent of learners who achieve a four-point CASAS scale score gain from pre-to post-test.Table 2 expands on the data from Table 1 to contrast the NRS methodology with Connecticut's alternative approach.
Compared to the NRS method, the Connecticut method for educational gain captures, more fully, the learning gains that are achieved by learners in the lower range of the level.Unlike the NRS method, the Connecticut approach also reflects minimal difference between the performance of learners who are placed in either the upper or the lower range of that level ( Another drawback of the NRS approach is that the NRS results do not separate learner performance into the discrete basic skills of reading, writing, math, listening and speaking.This conflated view diminishes the usefulness of NRS results for evaluating the impact of instruction on specific skill acquisition.The CSDE addressed this limitation by presenting in the profile report, disaggregated results by skill area that are computed using the Connecticut method. Under WIA, states can qualify to receive financial incentives if their performance on the core outcomes exceeds their negotiated targets.Because 11 of the 15 targets relate to educational gain, this critical policy relies heavily on the results from standardised testing based on the NRS methodology and does not consider the impact of other milestone achievements (e.g.high school diploma) or measurement error (Linn 2001).
Contrary to this 'test-centric' approach to accountability, the CSDE's framework for accountability and program improvement evaluates each local program on a range of measures such as program retention and completion.Success on the NRS is only one of the indicators.Utilising a broad approach to accountability that considers both process (i.e.recruitment, retention) as well as outcome (i.e.test-score gains, credits earned, diplomas attained) measures sends a message to all programs that the CSDE is concerned not only with test-score gains, but also with accessibility, learner persistence, credit attainment, and diploma achievement.
The CSDE expects local programs to learn from these results when establishing goals and objectives (Reeves 2004).Though there were early misgivings about a report card, practitioners 'are [now] saying that the Profile is a working document that helps you to know what you are doing and what you are not doing; not to point fingers but to give you insight'.
CASAS assessments, CARS data collection, and CARS reporting are no longer activities performed only to satisfy funders.They are tools that are integral to program management.Prior to WIA, assessment and data collection for accountability purposes were viewed as being separate from assessment and data collection for program improvement and classroom instruction.Today, there is overlap between these two components as standardised assessment results support instruction, and data collection for accountability informs program improvement.

Theme 3: Organisational alignment through instructional leadership
The increased alignments of instruction to curriculum standards and test results, combined with the availability of data at the program and classroom levels, are changing the relationship between the leadership and instructors in adult education programs.Elmore's (2000) description of the impact of standards-based education on the 'loose-coupling' paradigm of public education systems helps to understand this changing relationship.According to Elmore, the loose-coupling model assumes the following: -Decisions about curriculum objectives, lesson content, instructional approaches, and evaluation are best made in individual classrooms.-This classroom 'expertise' cannot be replicated or subjected to a reliable external evaluation.-The role of school leadership is to manage the processes that surround instruction but not instruction itself.
With each passing year of WIA and NRS implementation, administrators and Facilitators are viewing their role as more than a 'buffer' for teachers from outside influences (Elmore 2000).They are proactively implementing systematic instructional and organisational interventions to improve the quality of teaching and learning.The following examples exemplify this hands-on approach.
One program administrator articulated his goal that all learners must demonstrate improvements on the state-approved standardised writing assessment.He knew that students did not write consistently in all classes.Therefore, to infuse writing instruction, he expected teachers in all areas to have students write in class at least once a week.Teachers were trained to score student writing 'informally', but using the standard rubric.They also learned instructional strategies to teach writing.
When the above expectations and staff development activities were introduced in 2004-05, 43% of learners in this program completed an NRS level in terms of their writing skills.When the instructional enhancements were implemented in the following year, 2005-06, 75% of learners completed an NRS level in writing.In the two subsequent years of 2006-07 and 2007-08, 77% and 79% of learners achieved the same outcome.
Some programs are expecting teachers to submit lesson plans weekly to the Facilitator/Director.Such oversight is not to dictate the content of daily instruction but to see evidence that testing data is informing instruction.One Facilitator observed that, 'if teachers receive no formal monitoring or supervision, it almost makes them feel like no one cares about what they do'.Recruiting and retaining high quality instructional staff who possess a solid background in education and teaching has become an increasing priority.
In summary, leaders are beginning to see accountability not as a gimmick but as a stimulus for action.They are not content with hoping for results; they know what needs to be done and are willing to take action to make that happen.These changes have not been without growing pains.Facilitators talked about the importance of 'buy-in' at all levels including students.They also emphasised the importance of the Director's public validation of their role as Facilitators in order to give them the authority to work with teachers and influence classroom practice (Gopalakrishnan 2006).
Theme 4: Mixed results with follow-up outcomes Because learners come to adult education with a variety of goals, the NRS follow-up outcomes are required only for those who have the corresponding goal(s).These outcomes have spurred interesting conversations about goal-setting and the intake process.The CSDE has tried to promote the use of goals as motivators for learner persistence (Comings, Parrella, and Soricone 1999).One Facilitator commented that goal-setting was something new and not paid much attention.NRS forced the conversation around goal-setting.ESL students were saying they needed to learn English but we are now asking why.
However, methodological and policy issues with the follow-up outcome measures appear to limit their ability to inform practice.Goal-based outcomes are double-edged swords.On the one hand, they ensure that learners are not expected to achieve outcomes that are not appropriate for them.On the other hand, because goal-setting is a voluntary process, it can lead to an under-reporting of goals and outcomes even if there is no intent to inflate outcomes artificially (Belzer 2003).One Facilitator observed that she does not 'pressure students to commit to something unless they are ready tothey may not be driven or not ready to share yet'.
The notion of exit also muddles the follow-up outcome measures.NRS policy specifies that follow-up is required only for those learners who have 'exited' i.e. they have not received instruction for 90 days, and are not scheduled to receive further instruction (DAEL 2005).Though the policy is clear, if exit status is established by program-reports, then it may be unreliable and under-reporting can occur.In 2004-05, when the CSDE began utilising learner attendance from CARS to establish the exit date instead of relying on program-reports, it saw a dramatic increase in the number of learners being reported as exited.
In light of these issues, the CSDE has utilised other state reports instead of NRS reports for a complete picture of the employment outcomes achieved by learners (Connecticut Employment and Training Commission 2006).It is also considering strategies to ascertain the total number of graduates who enter postsecondary education, not just the subset of those with the NRS goal.One unintended consequence of the CSDE conducting data matches for the employment outcomes is that local programs rely more on the state for follow-up.Not conducting the follow-up locally may deprive programs of the opportunity to learn directly from their students about happens to them after adult education.

State-wide results
State-wide CARS data from the past seven years of learners in ABE, GED and ESL programs areanalysed to determine the extent of overall improvements in performance.Because of changes in state practices and NRS policies over the past eight years, the NRS reports are not comparable across years.Therefore, CARS data from past years were re-analysed using the current NRS methods.Moreover, to compensate for the limitations with the NRS methods discussed earlier, the CARS data are also analysed based on a method that is used in Connecticut for reporting outcomes to the State legislature (Table 3).A learner achieves a measurable educational outcome by attaining either a four-point scale score gain from pre-to post-test or a high school diploma.
In light of the changes in practitioner responses and program services that were explored through the thematic analysis, it is not surprising that overall performance is greater in the years after the early WIA years of 2001-02 and 2002-03.Additionally, the performance using the Connecticut method has continued to inch upward.

Conclusion
The response of local programs to the NRS accountability requirements has changed.In the early years of WIA, programs may have been satisfied with meeting the minimum requirements.Today, accountability is not implemented solely to satisfy reporting or funding requirements; it is a tool for instructional and organisational learning (Reeves 2004) and increasingly representative of program quality (Condelli 2007).
A state's ability to respond to accountability requirements can pave the way for the local response and impact (Belzer 2003).Connecticut's experience confirms that its history with implementing the CASAS system, collecting individual student data, and building critical local capacity enabled the CSDE and the local programs to implement the minimum requirements of WIA seamlessly as well as progress to higher levels of learning and action.
The services offered by local programs have also changed substantially.Curriculum standards are being diffused among all instructional areas; even volunteer programs are expecting tutors to utilise these standards.Data from accountability frameworks are increasingly informing new program offerings (e.g.transition-to-postsecondary classes), intake requirements, attendance policies, class schedules, class groupings, and program priorities.
Sincere efforts to implement accountability requirements at all levels in the program have changed the very nature of these programs.Hands-on instructional leadership has led to increased organisational alignment while the utilisation of data for decision-making, coupled with a willingness to change and grow, has made these organisations more adaptable , credible, and viable over the longer term.Staff roles to support the implementation of standards-based education and facilitate the collection and use of student data have become further institutionalised.Some of these changes may have been attainable without the NRS.For example, the CASAS approach to standards-based education was being implemented in several states prior to the NRS, which also likely fostered some level of program alignment.Reliable management information systems may also have been in use in some states, as it was in Connecticut, prior to the NRS.Despite these early efforts, Connecticut cannot overlook the critical impetus provided by the NRS to spur these changes and encourage their continued refinement.States that had minimal systems with regard to assessment policies and practices, management information systems, data collection procedures and data-use supports have utilised the NRS to institute those systemic components.
For accountability to also serve as a learning tool, its goals, measures and methods must be consistent with the reality of the system it is evaluating.The NRS 'educational gain' measure has proved to be a fair reflection of the outcomes achieved by learners and programs but it does have some limitations.To improve this measure, the NRS should consider the following: -Expect states to maintain longitudinal databases and evaluate learner gain across fiscal years; -Evaluate learner progress based on scale score gain to counteract the arbitrary elements in the 'level completion' approach (Rose and Wright 2004); -Include the attainment of a secondary credential as an indicator of success in the educational gain paradigm; and -Display the post-test rate alongside the NRS results in the NRS outcomes table to improve the use of data for program improvement.An accountability framework should also be meaningful to the learner.At a minimum, learners should receive feedback on their test scores using the NRS level descriptors (DAEL 2005).This feedback can be more valuable to learners if their NRS performance can be correlated to their potential for achieving future success such as passing the GED, attaining employment, achieving higher earnings, and meeting the requirements for entry into postsecondary education.Such correlation data exist for other performance scales such as the National Assessment of Adult Literacy (National Centre for Education Statistics 2007).New research is needed to establish the relationship between performance on NRS assessments and the outcomes that learners achieve beyond adult education.
The NRS follow-up measures fail to provide a complete picture and present limited value for program improvement and policy development.
The goal-based approach to these outcomes needs to be revisited.For example, in today's environment where postsecondary education is considered a minimum requirement for achieving self-sufficiency, the NRS should consider a national follow up through an agency like the National Student Clearinghouse on all learners who graduate from adult education, not just those with the corresponding goal.
State administrators can also adopt policy leadership roles to minimise unintended consequences from test-based accountability frameworks such as the NRS.
Accountability does not have to be punitive.Time spent on accountability activities such as testing, data collection, and reporting can directly affect program quality and help to improve instruction (Reeves 2004).Non-financial consequences such as on-site monitoring or even the public dissemination of local program data can raise the stakes sufficiently and bring attention to curriculum standards, instructional approaches, and the data.

Table 2 :
Comparison of approaches to educational gain for learners at the ABE High Intermediate Level who attended between 60 and 80 hours